Real Problems
Cascading Failures: One Slow Service Takes Down Everything
A pricing service three hops from the user became slow. Within seconds every service above it ran out of threads and the whole site returned errors. Reproduce the cascade and contain it with timeouts, a circuit breaker and a bulkhead.
What you will learn
- Trace how latency in a deep dependency propagates upward as resource exhaustion.
- Read utilization charts to find the origin of a cascade.
- Combine timeouts, circuit breakers, bulkheads and fallbacks to limit the blast radius.
- Plan load-shedding and verification for the fix.
Before you start
Symptoms
- Error rate jumps from 0% to ~40% for the whole site within a minute.
- Every service on the request path shows high latency and "connection pool exhausted" or "rejected execution" errors.
- CPU on most services is low — they are not busy computing, they are waiting.
- The on-call engineer restarts the web tier; it recovers for 30 seconds and falls over again.
System context
Root cause
Pricing became 30× slower (a bad query plan after a statistics refresh). Inventory's workers each waited ~900 ms instead of ~30 ms for Pricing, so Inventory's pool filled up. Orders' workers then waited on Inventory, filling Orders' pool, and so on up to the Web BFF. Little's law explains it: concurrency = throughput × latency. Multiply latency by 30 at constant traffic and every pool needs 30× more workers than it has.
Interactive reproduction
In scenario A, watch the utilization per service chart: Inventory saturates first, then Orders, then the Web BFF — the cascade moves toward the user. Scenario B adds protections at both hops and keeps every service healthy.
Solution alternatives
| Defence | What it bounds | Alone it… |
|---|---|---|
| Timeout on Pricing calls | How long one worker can wait. | …still wastes 250 ms per call and keeps hammering Pricing. |
| Circuit Breaker | Whether calls are attempted at all. | …needs a timeout to notice slowness, and failures until it opens. |
| Bulkhead (concurrency limit) on Inventory calls | How many Orders workers can be tied up by Inventory. | …protects Orders but not Inventory itself. |
| Fallback (cached / default price) | What the user gets during the failure. | …requires a product decision about acceptable degraded answers. |
| More threads / instances | Nothing — it moves the ceiling. | …delays the cascade and adds load to Pricing. |
Recommended fix: defence in depth
resilience4j:
timelimiter:
instances:
pricing:
timeout-duration: 250ms
circuitbreaker:
instances:
pricing:
sliding-window-size: 20
minimum-number-of-calls: 10
failure-rate-threshold: 50
slow-call-duration-threshold: 200ms
slow-call-rate-threshold: 60
wait-duration-in-open-state: 4s
bulkhead:
instances:
inventory: # used by orders-service for its calls to Inventory
max-concurrent-calls: 6
max-wait-duration: 0Advanced Beyond per-call protections, services should shed load when saturated: reject early (HTTP 503 with Retry-After) when queue time exceeds the caller's deadline, prioritise critical traffic (checkout over recommendations) and make sure clients do not retry aggressively into the overload (see Retry).
Verification strategy
- Load test with a fault injector that adds latency to Pricing (e.g. Toxiproxy) and assert the error rate of unrelated endpoints stays flat.
- Run a game day: degrade one deep dependency in staging and confirm breakers open, fallbacks serve and dashboards show the origin.
- Check that every outbound client in every service has a timeout — automate it in code review or with a lint rule.
Monitoring and prevention
- Dashboard per service: worker pool utilization, queue time, and latency per dependency — the dependency whose latency rose first is the origin.
- Alert on circuit breakers that stay open and on bulkhead rejections.
- Use distributed tracing to see where time is spent across hops.
Lessons learned
- Slow is worse than down: a crashed dependency fails fast, a slow one holds resources.
- Every remote call needs a timeout, and important dependencies need a breaker and a fallback.
- Isolate resources per dependency so one failure has a bounded blast radius.