Skip to content
System Design Lab
Microservice Patterns

Retry

Transparently repeat a failed call when the failure is likely temporary — with backoff, jitter and a strict limit, so retries heal glitches instead of amplifying outages.

Beginner 7 min read· Updated 2026-11-18

What you will learn

  • Tell transient failures (worth retrying) from permanent ones (never worth it).
  • Explain exponential backoff and why jitter matters when many clients retry at once.
  • Recognise a retry storm and know the guard rails that prevent it.
  • Configure a Resilience4j retry and make the retried operation idempotent.

Overview

When a web page fails to load, you press refresh and it usually works the second time. Distributed systems are full of these momentary hiccups: a pod restarting, a dropped TCP connection, a database failover. The Retry pattern does the refresh for you — automatically, a limited number of times, and politely.

A retry policy wraps a remote call. If the call fails with an error that is likely transient, the policy waits a little and tries again, up to a maximum number of attempts. The caller sees either a success (possibly a bit slower) or the final error.

The problem: networks fail briefly and often

In a system with dozens of services, short failures are normal, not exceptional. Without retries, every hiccup becomes a user-visible error. But retrying the wrong way is dangerous: if the failure is caused by overload, every retry adds more load to the component that is already struggling.

FailureRetry?Why
Connection reset, 503, gateway timeoutYesUsually transient: the next attempt may land on a healthy instance.
429 Too Many RequestsYes, slowlyHonour the Retry-After header; retrying immediately makes it worse.
400 Bad Request, 404, validation errorNoThe same request will fail the same way forever.
Card declined, insufficient fundsNoA business answer, not a technical failure.
Timeout on a non-idempotent writeOnly with an idempotency keyThe first attempt may have succeeded — a blind retry can charge twice.

The solution: bounded retries with backoff and jitter

  • Limit attempts — typically 2–3 in total. More attempts rarely help and always cost.
  • Exponential backoff — wait 100 ms, then 200 ms, then 400 ms… so a recovering dependency gets breathing room.
  • Jitter — randomise each wait. Without it, thousands of clients that failed together retry together, in synchronised waves.
  • Retry only what is safe — idempotent operations, or writes protected by an idempotency key.
Order ServiceRetry policyPayment

Press play or step through the interaction.

One logical call, three physical attempts.

Run it: from glitch to storm

Scenarios A and B show retries at their best: transient errors disappear. Scenarios C and D show the dark side: when Payment is overloaded, retries cannot create capacity — they only multiply the work. Compare the Retries tile and Payment's utilization between C and D.

Loading simulation…

Retry storms and how to prevent them Advanced

Advanced Retries multiply across layers. If the browser, the API gateway, the BFF and the service client each retry 3 times, one user click can become 3⁴ = 81 calls to the bottom service — exactly when it is least able to handle them. Defences, from most to least important:

Advanced
  • Retry at one layer only, ideally the one closest to the failing dependency.
  • Retry budgets — allow retries to add at most ~10–20% extra traffic per client; beyond that, fail fast.
  • Combine with a [Circuit Breaker](topic:circuit-breaker) placed inside the retry: once the circuit opens, remaining retries fail instantly instead of hitting the dependency.
  • Respect server hints — Retry-After on 429/503 responses.
  • Set a [Timeout](topic:timeout) per attempt and an overall deadline, so the total time stays below what the caller is willing to wait.

Implementation with Spring Boot and Resilience4j

application.yml
resilience4j:
  retry:
    instances:
      payment:
        max-attempts: 3                     # including the first call
        wait-duration: 100ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2
        exponential-max-wait-duration: 2s
        enable-randomized-wait: true        # jitter
        randomized-wait-factor: 0.5
        retry-exceptions:
          - java.io.IOException
          - org.springframework.web.client.HttpServerErrorException
          - org.springframework.web.client.ResourceAccessException
        ignore-exceptions:
          - org.springframework.web.client.HttpClientErrorException
PaymentClient.java
@Service
public class PaymentClient {

    private final RestClient restClient;

    public PaymentClient(RestClient.Builder builder) {
        this.restClient = builder.baseUrl("http://payment-service").build();
    }

    @Retry(name = "payment")
    public PaymentResult charge(ChargeRequest request) {
        return restClient.post()
                .uri("/charges")
                // Same key on every attempt: the server charges at most once.
                .header("Idempotency-Key", request.orderId())
                .body(request)
                .retrieve()
                .body(PaymentResult.class);
    }
}

Trade-offs

BenefitsCosts
AvailabilityTransient faults become invisible to users.Permanent faults take longer to surface.
LatencyA slow success beats a fast error.Each retry adds the backoff plus another full call to the tail latency.
LoadMinimal when failures are rare.Multiplies load during overload — the moment you can least afford it.
CorrectnessSafe for reads and idempotent writes.Dangerous for non-idempotent writes without an idempotency key.

Production considerations

  • Log and count retries separately from first attempts — a rising retry rate is an early warning that a dependency is degrading.
  • Keep total retry time within the caller's own timeout; retrying after the user has given up wastes everyone's resources.
  • Do not retry inside a database transaction that holds locks.

Further reading

Related topics