Skip to content
System Design Lab
Real Problems

One Crash Freezes Checkout: Two-Phase Commit Gone Wrong

A checkout instance crashed between prepare and commit. Its locks stayed held, and the healthy instance started failing too. Why 2PC blocks under failure, and how a saga keeps the damage local.

Advanced 5 min read· Updated 2026-11-18

What you will learn

  • Explain the prepare and commit phases of two-phase commit.
  • Reproduce the blocking problem: in-doubt transactions holding locks.
  • Compare 2PC with a saga of local transactions under the same crash.
  • Choose monitoring that exposes in-doubt transactions and lock waits.

Before you start

Symptoms

  • One checkout pod is restarted by Kubernetes. Seconds later, checkouts on the *other* pods start failing too.
  • Inventory and Payment logs fill with lock wait timeout exceeded.
  • pg_prepared_xacts shows transactions that are neither committed nor rolled back.
  • The errors stop only once the crashed pod is back and has resolved its transactions.

System context

Shoppers AShoppers AShoppers BShoppers BCheckout #12PC coordinatorCheckout #1Checkout #22PC coordinatorCheckout #2Inventorylocks rows on pre…InventoryPaymentlocks rows on pre…Payment
Two checkout instances act as 2PC coordinators over the same Inventory and Payment databases.

In phase 1 the coordinator asks every participant to prepare: each locks the rows, writes the change durably and votes yes or no. In phase 2, if all voted yes, the coordinator tells everyone to commit; otherwise to abort. A participant that voted yes has given up the right to decide on its own.

Root cause

  1. Instance 1 crashed after its participants prepared but before it sent commit.
  2. The participants were left in doubt: they cannot commit (maybe someone voted no) and cannot abort (maybe the coordinator already told others to commit). So they keep the rows locked.
  3. The most popular products were among those locked rows. Instance 2's transactions queued for the same locks, waited past the lock timeout and voted no.
  4. Only when instance 1 recovered and read its log did it resolve the in-doubt transactions (presumed abort) and release the locks.

Interactive reproduction

A is the healthy baseline. In B, instance 1 crashes at t = 8 s mid-transaction and stays down for 12 s: watch Rows locked in Inventory freeze and Instance 2: failed climb even though instance 2 never crashed. C runs the same crash with a saga.

Loading simulation…

Solution alternatives

OptionEffectCost
Shorter lock timeoutsHealthy instances fail fasterDoes not release the in-doubt locks; more aborted work.
Highly available coordinator (replicated log)Recovery in seconds instead of minutesOperational complexity; still blocks during failover.
Manual heuristic commit/rollbackUnblocks immediatelyRisks inconsistency; needs an on-call human.
Saga of local transactionsNo cross-service locks; failures stay localIntermediate states are visible; compensations must be designed.

Recommended fix

Across services, replace 2PC with a Saga: each step is a local transaction that commits immediately, and a failure triggers compensating steps (release stock, refund). No locks are held between steps, so a crashed orchestrator affects only its own in-flight sagas — which it resumes from its log when it comes back.

Reserve-then-confirm instead of lock-and-wait
-- Step 1 (Inventory, local transaction): reserve with an expiry instead of holding a lock.
UPDATE stock SET reserved = reserved + :qty
 WHERE sku = :sku AND available - reserved >= :qty;
INSERT INTO reservations(id, sku, qty, expires_at) VALUES (:sagaId, :sku, :qty, now() + interval '15 minutes');

-- Step 2 (Payment): charge the card. On failure the orchestrator runs the compensation:
UPDATE stock SET reserved = reserved - :qty WHERE sku = :sku;
DELETE FROM reservations WHERE id = :sagaId;

-- A sweeper releases reservations whose saga never finished — the lock-free version of "in doubt".

Advanced 2PC is still fine *inside* one database cluster or between two resources you fully control with a highly available coordinator (e.g. Kafka transactions, Spanner). The problem is stretching it across independently deployed services, where any participant or coordinator can disappear at any time.

Verification strategy

  • Kill one orchestrator instance under load; success rate on the other instances must not drop.
  • Inject failures at each saga step and assert that compensations leave stock and payments balanced.
  • Check that expired reservations are released within the sweeper interval.

Monitoring and prevention

  • Alert on any prepared transaction older than a few seconds (pg_prepared_xacts, XA recover lists).
  • Track lock wait time and lock timeouts per table.
  • For sagas: count stuck sagas and compensations per hour.

Lessons learned

  • A protocol that holds locks across the network turns one crash into a shared outage.
  • Across services, prefer local commits plus compensation over global locks.
  • Design for the intermediate states (reserved, pending) instead of pretending they do not exist.

Further reading

Related topics