Skip to content
System Design Lab
Real Problems

Cascading Failures: One Slow Service Takes Down Everything

A pricing service three hops from the user became slow. Within seconds every service above it ran out of threads and the whole site returned errors. Reproduce the cascade and contain it with timeouts, a circuit breaker and a bulkhead.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Trace how latency in a deep dependency propagates upward as resource exhaustion.
  • Read utilization charts to find the origin of a cascade.
  • Combine timeouts, circuit breakers, bulkheads and fallbacks to limit the blast radius.
  • Plan load-shedding and verification for the fix.

Before you start

Symptoms

  • Error rate jumps from 0% to ~40% for the whole site within a minute.
  • Every service on the request path shows high latency and "connection pool exhausted" or "rejected execution" errors.
  • CPU on most services is low — they are not busy computing, they are waiting.
  • The on-call engineer restarts the web tier; it recovers for 30 seconds and falls over again.

System context

UsersUsersWeb BFF20 workersWeb BFFOrders10 workersOrdersInventory10 workersInventoryPricingthe weak linkPricing
Web BFF → Orders → Inventory → Pricing. Each service uses a fixed worker pool and calls the next one synchronously.

Root cause

Pricing became 30× slower (a bad query plan after a statistics refresh). Inventory's workers each waited ~900 ms instead of ~30 ms for Pricing, so Inventory's pool filled up. Orders' workers then waited on Inventory, filling Orders' pool, and so on up to the Web BFF. Little's law explains it: concurrency = throughput × latency. Multiply latency by 30 at constant traffic and every pool needs 30× more workers than it has.

Interactive reproduction

In scenario A, watch the utilization per service chart: Inventory saturates first, then Orders, then the Web BFF — the cascade moves toward the user. Scenario B adds protections at both hops and keeps every service healthy.

Loading simulation…

Solution alternatives

DefenceWhat it boundsAlone it…
Timeout on Pricing callsHow long one worker can wait.…still wastes 250 ms per call and keeps hammering Pricing.
Circuit BreakerWhether calls are attempted at all.…needs a timeout to notice slowness, and failures until it opens.
Bulkhead (concurrency limit) on Inventory callsHow many Orders workers can be tied up by Inventory.…protects Orders but not Inventory itself.
Fallback (cached / default price)What the user gets during the failure.…requires a product decision about acceptable degraded answers.
More threads / instancesNothing — it moves the ceiling.…delays the cascade and adds load to Pricing.

Recommended fix: defence in depth

BeforeUnprotected chain
UsersUsersWeb BFF20 workersWeb BFFOrders10 workersOrdersInventory10 workersInventoryPricingthe weak linkPricing
Any slow dependency can exhaust every pool above it.
AfterProtected chain
UsersUsersWeb BFF20 workersWeb BFFOrders10 workersOrdersBulkheadmax 6 concurrentBulkheadInventory10 workersInventoryTimeout 250 msTimeout 250 msCircuit Breakerfallback: cached …Circuit BreakerPricingthe weak linkPricing
A 250 ms timeout and a circuit breaker with a cached-price fallback in front of Pricing; a bulkhead caps Orders → Inventory concurrency.
inventory-service application.yml
resilience4j:
  timelimiter:
    instances:
      pricing:
        timeout-duration: 250ms
  circuitbreaker:
    instances:
      pricing:
        sliding-window-size: 20
        minimum-number-of-calls: 10
        failure-rate-threshold: 50
        slow-call-duration-threshold: 200ms
        slow-call-rate-threshold: 60
        wait-duration-in-open-state: 4s
  bulkhead:
    instances:
      inventory:              # used by orders-service for its calls to Inventory
        max-concurrent-calls: 6
        max-wait-duration: 0

Advanced Beyond per-call protections, services should shed load when saturated: reject early (HTTP 503 with Retry-After) when queue time exceeds the caller's deadline, prioritise critical traffic (checkout over recommendations) and make sure clients do not retry aggressively into the overload (see Retry).

Verification strategy

  • Load test with a fault injector that adds latency to Pricing (e.g. Toxiproxy) and assert the error rate of unrelated endpoints stays flat.
  • Run a game day: degrade one deep dependency in staging and confirm breakers open, fallbacks serve and dashboards show the origin.
  • Check that every outbound client in every service has a timeout — automate it in code review or with a lint rule.

Monitoring and prevention

  • Dashboard per service: worker pool utilization, queue time, and latency per dependency — the dependency whose latency rose first is the origin.
  • Alert on circuit breakers that stay open and on bulkhead rejections.
  • Use distributed tracing to see where time is spent across hops.

Lessons learned

  • Slow is worse than down: a crashed dependency fails fast, a slow one holds resources.
  • Every remote call needs a timeout, and important dependencies need a breaker and a fallback.
  • Isolate resources per dependency so one failure has a bounded blast radius.

Further reading

Related topics