Microservice Patterns
Retry
Transparently repeat a failed call when the failure is likely temporary — with backoff, jitter and a strict limit, so retries heal glitches instead of amplifying outages.
What you will learn
- Tell transient failures (worth retrying) from permanent ones (never worth it).
- Explain exponential backoff and why jitter matters when many clients retry at once.
- Recognise a retry storm and know the guard rails that prevent it.
- Configure a Resilience4j retry and make the retried operation idempotent.
Overview
When a web page fails to load, you press refresh and it usually works the second time. Distributed systems are full of these momentary hiccups: a pod restarting, a dropped TCP connection, a database failover. The Retry pattern does the refresh for you — automatically, a limited number of times, and politely.
A retry policy wraps a remote call. If the call fails with an error that is likely transient, the policy waits a little and tries again, up to a maximum number of attempts. The caller sees either a success (possibly a bit slower) or the final error.
The problem: networks fail briefly and often
In a system with dozens of services, short failures are normal, not exceptional. Without retries, every hiccup becomes a user-visible error. But retrying the wrong way is dangerous: if the failure is caused by overload, every retry adds more load to the component that is already struggling.
| Failure | Retry? | Why |
|---|---|---|
| Connection reset, 503, gateway timeout | Yes | Usually transient: the next attempt may land on a healthy instance. |
| 429 Too Many Requests | Yes, slowly | Honour the Retry-After header; retrying immediately makes it worse. |
| 400 Bad Request, 404, validation error | No | The same request will fail the same way forever. |
| Card declined, insufficient funds | No | A business answer, not a technical failure. |
| Timeout on a non-idempotent write | Only with an idempotency key | The first attempt may have succeeded — a blind retry can charge twice. |
The solution: bounded retries with backoff and jitter
- Limit attempts — typically 2–3 in total. More attempts rarely help and always cost.
- Exponential backoff — wait 100 ms, then 200 ms, then 400 ms… so a recovering dependency gets breathing room.
- Jitter — randomise each wait. Without it, thousands of clients that failed together retry together, in synchronised waves.
- Retry only what is safe — idempotent operations, or writes protected by an idempotency key.
Press play or step through the interaction.
Run it: from glitch to storm
Scenarios A and B show retries at their best: transient errors disappear. Scenarios C and D show the dark side: when Payment is overloaded, retries cannot create capacity — they only multiply the work. Compare the Retries tile and Payment's utilization between C and D.
Retry storms and how to prevent them Advanced
Advanced Retries multiply across layers. If the browser, the API gateway, the BFF and the service client each retry 3 times, one user click can become 3⁴ = 81 calls to the bottom service — exactly when it is least able to handle them. Defences, from most to least important:
- Retry at one layer only, ideally the one closest to the failing dependency.
- Retry budgets — allow retries to add at most ~10–20% extra traffic per client; beyond that, fail fast.
- Combine with a [Circuit Breaker](topic:circuit-breaker) placed inside the retry: once the circuit opens, remaining retries fail instantly instead of hitting the dependency.
- Respect server hints —
Retry-Afteron 429/503 responses. - Set a [Timeout](topic:timeout) per attempt and an overall deadline, so the total time stays below what the caller is willing to wait.
Implementation with Spring Boot and Resilience4j
resilience4j:
retry:
instances:
payment:
max-attempts: 3 # including the first call
wait-duration: 100ms
enable-exponential-backoff: true
exponential-backoff-multiplier: 2
exponential-max-wait-duration: 2s
enable-randomized-wait: true # jitter
randomized-wait-factor: 0.5
retry-exceptions:
- java.io.IOException
- org.springframework.web.client.HttpServerErrorException
- org.springframework.web.client.ResourceAccessException
ignore-exceptions:
- org.springframework.web.client.HttpClientErrorException@Service
public class PaymentClient {
private final RestClient restClient;
public PaymentClient(RestClient.Builder builder) {
this.restClient = builder.baseUrl("http://payment-service").build();
}
@Retry(name = "payment")
public PaymentResult charge(ChargeRequest request) {
return restClient.post()
.uri("/charges")
// Same key on every attempt: the server charges at most once.
.header("Idempotency-Key", request.orderId())
.body(request)
.retrieve()
.body(PaymentResult.class);
}
}Trade-offs
| Benefits | Costs | |
|---|---|---|
| Availability | Transient faults become invisible to users. | Permanent faults take longer to surface. |
| Latency | A slow success beats a fast error. | Each retry adds the backoff plus another full call to the tail latency. |
| Load | Minimal when failures are rare. | Multiplies load during overload — the moment you can least afford it. |
| Correctness | Safe for reads and idempotent writes. | Dangerous for non-idempotent writes without an idempotency key. |
Production considerations
- Log and count retries separately from first attempts — a rising retry rate is an early warning that a dependency is degrading.
- Keep total retry time within the caller's own timeout; retrying after the user has given up wastes everyone's resources.
- Do not retry inside a database transaction that holds locks.