10 topics
Real Problems
Production incidents reproduced in a sandbox: symptoms, root cause, the fix and how to prove it worked.
Consistency incidents
Customers Charged Twice: Duplicate Kafka Processing
A slow downstream made a Kafka consumer miss its poll deadline. The group rebalanced, the same payment messages were processed again — and nothing crashed. How to reproduce, fix and prevent it.
ProblemsOrders Without Invoices: Lost Events After Commit
Orders were saved but Billing never heard about some of them. The culprit: committing to the database and publishing to the broker as two separate steps. Reproduce it, then fix it with an outbox.
ProblemsOne Crash Freezes Checkout: Two-Phase Commit Gone Wrong
A checkout instance crashed between prepare and commit. Its locks stayed held, and the healthy instance started failing too. Why 2PC blocks under failure, and how a saga keeps the damage local.
Problems400 Phones Sold, 100 in Stock: Race Conditions in a Flash Sale
A flash sale sold three times the available stock without a single error. The code checked stock, then saved stock − 1 — and hundreds of buyers did it at once. Reproduce it, then fix it properly.
ProblemsWithdrawal Before Deposit: Message Ordering Problems
Account balances went negative even though every event was valid. Events for one account were applied out of order — first because of a missing key, then because of a thread pool. How to keep order per key without giving up throughput.
ProblemsResilience incidents
Cascading Failures: One Slow Service Takes Down Everything
A pricing service three hops from the user became slow. Within seconds every service above it ran out of threads and the whole site returned errors. Reproduce the cascade and contain it with timeouts, a circuit breaker and a bulkhead.
ProblemsTen Seconds of Slowness, Endless Outage: Timeouts and Retry Storms
A database was slow for ten seconds. Retries at three layers multiplied its load 27× and the outage outlived the cause. How retry amplification works, and how one retry layer, budgets and jitter stop it.
ProblemsPerformance incidents
Database Bottleneck Under a Traffic Spike
A campaign multiplied traffic and the Orders API started timing out while its own CPUs sat idle. The database connection pool was the limit. Reproduce it and compare three fixes: an index, read replicas and a cache.
ProblemsKafka Consumer Lag That Never Recovers
Consumer lag grew all day, and doubling the consumers changed nothing. Learn how partitions cap parallelism, how hot keys create one slow partition, and how to size a consumer group.
ProblemsThe Cache Expired and Took the Database With It: Cache Stampede
Popular product pages expired together, hundreds of identical queries hit the database, and it never recovered on its own. Reproduce it, then fix it with jitter, single flight and stale-while-revalidate.
Problems