Skip to content
System Design Lab
Real Problems

Ten Seconds of Slowness, Endless Outage: Timeouts and Retry Storms

A database was slow for ten seconds. Retries at three layers multiplied its load 27× and the outage outlived the cause. How retry amplification works, and how one retry layer, budgets and jitter stop it.

Advanced 4 min read· Updated 2026-11-18

What you will learn

  • Calculate how retries at several layers multiply load.
  • See why timed-out work keeps loading the dependency.
  • Fix it with one retry layer, retry budgets and jittered backoff.
  • Propagate deadlines so nobody works on requests that are already lost.

Before you start

Symptoms

  • A slow query plan made the pricing database slow for about ten seconds.
  • Traffic to the database jumped far above user traffic; connection pool and queue were full.
  • The query plan was fixed, but errors continued — the outage lasted until traffic was shed manually.
  • Every service's logs were full of timeouts and retries.

System context

Mobile appMobile appGateway retry×3, timeout 2 sGateway retryOrdersOrdersOrders retry×3, timeout 1 sOrders retryPricingPricingPricing retry×3, timeout 400 msPricing retryPricing DBPricing DB
Gateway → Orders → Pricing → database; each hop has a timeout and a retry policy of 3 attempts.

Root cause

  1. Each layer retried up to 3 times. Retries multiply: 3 × 3 × 3 = 27 database calls for one user request in the worst case.
  2. Timeouts abandoned calls, but the database kept executing them — abandoned work still consumed connections.
  3. As the database slowed, more calls timed out, which created more retries, which slowed it further.
  4. When the cause disappeared, the queue of retried work was still far above capacity: the system stayed in the overloaded state (a metastable failure).

Interactive reproduction

The database is 6× slower between t = 10 s and 20 s. In A, compare Retries with requests sent and watch the throughput chart after t = 20 s. B retries only at the gateway; C keeps retries everywhere but caps them with a 10% budget and full jitter.

Loading simulation…

Solution alternatives

TechniqueEffect
Retry at one layer onlyAmplification bounded by one policy (×3, not ×27). Inner layers fail fast.
Retry budget (e.g. ≤ 10% of requests)Retries help with blips but switch off during overload.
Exponential backoff with full jitterSpreads retries out instead of synchronizing them.
Deadline propagationEach hop knows the remaining time and skips work that can no longer succeed.
Circuit breakerStops calling a failing dependency entirely for a while.
Load shedding at the dependencyRejects early (cheaply) instead of queueing work that will time out.

Recommended fix

Resilience4j: retries only at the edge, with jitter
resilience4j:
  retry:
    instances:
      gateway-orders:              # the only layer that retries
        max-attempts: 3
        wait-duration: 100ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2
        enable-randomized-wait: true
        randomized-wait-factor: 0.5
      orders-pricing:
        max-attempts: 1            # fail fast; let the edge decide
      pricing-db:
        max-attempts: 1
Propagate the deadline instead of a fixed timeout per hop
Instant deadline = Instant.parse(request.getHeader("X-Deadline"));
Duration remaining = Duration.between(Instant.now(), deadline);
if (remaining.compareTo(Duration.ofMillis(20)) < 0) throw new DeadlineExceededException();   // don't start doomed work

pricingClient.get()
    .uri("/prices/{id}", id)
    .header("X-Deadline", deadline.toString())
    .retrieve()
    .bodyToMono(Price.class)
    .timeout(remaining)
    .block();

Verification strategy

  • Chaos test: slow the deepest dependency 5× for 30 s under peak load; the system must recover within seconds after it ends.
  • Measure calls to the dependency per user request during the test (amplification); alert if it exceeds the budget.
  • Review every client library's default retry settings — many retry silently.

Monitoring and prevention

  • Dashboard the retry rate per dependency next to its request rate.
  • Track work started after its deadline (wasted work) at the dependency.
  • Document one retry policy per call path in the service template.

Lessons learned

  • Retries compound across layers; decide where retrying happens, once.
  • Overload needs less traffic, not more: budgets and circuit breakers make retries back off.
  • A timeout does not cancel the work — deadlines and load shedding do.

Further reading

Related topics