Real Problems
Ten Seconds of Slowness, Endless Outage: Timeouts and Retry Storms
A database was slow for ten seconds. Retries at three layers multiplied its load 27× and the outage outlived the cause. How retry amplification works, and how one retry layer, budgets and jitter stop it.
Advanced 4 min read· Updated 2026-11-18
What you will learn
- Calculate how retries at several layers multiply load.
- See why timed-out work keeps loading the dependency.
- Fix it with one retry layer, retry budgets and jittered backoff.
- Propagate deadlines so nobody works on requests that are already lost.
Before you start
Symptoms
- A slow query plan made the pricing database slow for about ten seconds.
- Traffic to the database jumped far above user traffic; connection pool and queue were full.
- The query plan was fixed, but errors continued — the outage lasted until traffic was shed manually.
- Every service's logs were full of timeouts and retries.
System context
Root cause
- Each layer retried up to 3 times. Retries multiply: 3 × 3 × 3 = 27 database calls for one user request in the worst case.
- Timeouts abandoned calls, but the database kept executing them — abandoned work still consumed connections.
- As the database slowed, more calls timed out, which created more retries, which slowed it further.
- When the cause disappeared, the queue of retried work was still far above capacity: the system stayed in the overloaded state (a metastable failure).
Interactive reproduction
The database is 6× slower between t = 10 s and 20 s. In A, compare Retries with requests sent and watch the throughput chart after t = 20 s. B retries only at the gateway; C keeps retries everywhere but caps them with a 10% budget and full jitter.
Loading simulation…
Solution alternatives
| Technique | Effect |
|---|---|
| Retry at one layer only | Amplification bounded by one policy (×3, not ×27). Inner layers fail fast. |
| Retry budget (e.g. ≤ 10% of requests) | Retries help with blips but switch off during overload. |
| Exponential backoff with full jitter | Spreads retries out instead of synchronizing them. |
| Deadline propagation | Each hop knows the remaining time and skips work that can no longer succeed. |
| Circuit breaker | Stops calling a failing dependency entirely for a while. |
| Load shedding at the dependency | Rejects early (cheaply) instead of queueing work that will time out. |
Recommended fix
resilience4j:
retry:
instances:
gateway-orders: # the only layer that retries
max-attempts: 3
wait-duration: 100ms
enable-exponential-backoff: true
exponential-backoff-multiplier: 2
enable-randomized-wait: true
randomized-wait-factor: 0.5
orders-pricing:
max-attempts: 1 # fail fast; let the edge decide
pricing-db:
max-attempts: 1Instant deadline = Instant.parse(request.getHeader("X-Deadline"));
Duration remaining = Duration.between(Instant.now(), deadline);
if (remaining.compareTo(Duration.ofMillis(20)) < 0) throw new DeadlineExceededException(); // don't start doomed work
pricingClient.get()
.uri("/prices/{id}", id)
.header("X-Deadline", deadline.toString())
.retrieve()
.bodyToMono(Price.class)
.timeout(remaining)
.block();Verification strategy
- Chaos test: slow the deepest dependency 5× for 30 s under peak load; the system must recover within seconds after it ends.
- Measure calls to the dependency per user request during the test (amplification); alert if it exceeds the budget.
- Review every client library's default retry settings — many retry silently.
Monitoring and prevention
- Dashboard the retry rate per dependency next to its request rate.
- Track work started after its deadline (wasted work) at the dependency.
- Document one retry policy per call path in the service template.
Lessons learned
- Retries compound across layers; decide where retrying happens, once.
- Overload needs less traffic, not more: budgets and circuit breakers make retries back off.
- A timeout does not cancel the work — deadlines and load shedding do.
Further reading
Related topics
Microservice Patterns
Retry
Transparently repeat a failed call when the failure is likely temporary — with backoff, jitter and a strict limit, so retries heal glitches instead of amplifying outages.
Microservice Patterns
Timeout
Put an upper bound on how long you wait for any remote call, so one hanging dependency cannot quietly hold every thread, connection and user request hostage.
Microservice Patterns
Circuit Breaker
Stop calling a dependency that is failing, fail fast instead, and probe carefully until it recovers — so one broken service cannot drag down everything that calls it.
Real Problems
Cascading Failures: One Slow Service Takes Down Everything
A pricing service three hops from the user became slow. Within seconds every service above it ran out of threads and the whole site returned errors. Reproduce the cascade and contain it with timeouts, a circuit breaker and a bulkhead.
Microservice Patterns
Backpressure
When a consumer cannot keep up, push the problem back to the producer — bounded queues and 429 Retry-After — instead of buffering in memory until the process dies and takes accepted work with it.