Skip to content
System Design Lab
Real Problems

The Cache Expired and Took the Database With It: Cache Stampede

Popular product pages expired together, hundreds of identical queries hit the database, and it never recovered on its own. Reproduce it, then fix it with jitter, single flight and stale-while-revalidate.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Explain how synchronized expiry turns a cache miss into an overload.
  • See why the overload can become self-sustaining (metastable).
  • Fix it with TTL jitter, request coalescing and stale-while-revalidate.
  • Know which signals reveal a stampede early.

Before you start

Symptoms

  • Minutes after a deploy (cold cache) or at regular intervals, database CPU jumps to 100% and the connection pool is exhausted.
  • Slow-query logs show the same query hundreds of times within one second.
  • Cache hit ratio drops sharply, then does not come back even though traffic is normal.
  • Restarting the application or shedding traffic is the only thing that helps.

System context

on missShoppersShoppersProduct APIProduct APIRedisfixed TTL 8 sRedisProducts DB200 ms aggregationProducts DB
Product pages are cached in Redis with a fixed TTL; a miss runs a 200 ms aggregation query.

Root cause

  1. After a cold start, most keys were cached within the same second — so with a fixed TTL they also expire together.
  2. Every request for an expired key misses. With no coordination, each one runs the same expensive query: dozens of duplicates per key.
  3. The database runs 6 queries at a time; the rest queue, then get rejected or time out.
  4. Failed queries cache nothing, so the next requests miss as well. The overload sustains itself even after the trigger is gone — a metastable failure.

Interactive reproduction

In A the keys expire together around t = 11 s and the database never catches up: watch Database queries and the failed line. B adds ±30% TTL jitter and single flight; C serves stale values while refreshing in the background. Press Flush cache to simulate a deploy at any moment.

Loading simulation…

Solution alternatives

TechniqueWhat it preventsNotes
TTL jitterKeys expiring at the same instantCheap: ttl × random(0.8, 1.2).
Single flight / lock per keyDuplicate queries for one keyOne request rebuilds, the others wait or get stale data. Use a lock with timeout in Redis across instances.
Stale-while-revalidateUsers waiting on a rebuildServe the old value, refresh in the background; needs a hard expiry too.
Probabilistic early refreshThe expiry moment itselfRefresh slightly before expiry with a probability that rises near the TTL (XFetch).
Pre-warmingCold start after deployLoad hot keys before taking traffic.
Load shedding at the databaseSelf-sustaining overloadFail fast instead of queueing; a circuit breaker lets the cache refill.

Recommended fix

Jittered TTL, single flight and stale-while-revalidate with Caffeine
LoadingCache<Long, ProductPage> pages = Caffeine.newBuilder()
    .maximumSize(50_000)
    // Soft expiry: after 8 s the entry is refreshed in the background while the old value is served.
    .refreshAfterWrite(Duration.ofSeconds(8))
    // Hard expiry, randomised per entry so keys cached together do not expire together.
    .expireAfter(Expiry.creating((Long id, ProductPage p) -> Duration.ofSeconds(ThreadLocalRandom.current().nextLong(10, 16))))
    .build(id -> productQueries.renderPage(id));   // loads are coalesced: one per key at a time

Advanced With a shared Redis cache across many instances, single flight needs a distributed lock (SET key:lock NX PX 2000): the winner rebuilds, the others serve the stale copy stored next to it (keep the value with a longer TTL than its logical expiry) or retry briefly.

Verification strategy

  • Load test at peak traffic, flush the cache, and assert the database query rate stays below its capacity.
  • Count database queries per key per second during the test — more than one means coalescing is not working.
  • Test a deploy with a cold cache in staging before every major release.

Monitoring and prevention

  • Alert on hit-ratio drops and on database query rate per normalized query.
  • Track "duplicate in-flight loads" per key in the cache layer.
  • Keep database pools and queues bounded so overload fails fast instead of building a backlog.

Lessons learned

  • A cache protects the database only while it is warm; design for the moments it is not.
  • Identical work should happen once: coalesce it.
  • Systems with a cache can have two stable states — plan how to get back from the bad one.

Further reading

Related topics