Real Problems
One Crash Freezes Checkout: Two-Phase Commit Gone Wrong
A checkout instance crashed between prepare and commit. Its locks stayed held, and the healthy instance started failing too. Why 2PC blocks under failure, and how a saga keeps the damage local.
What you will learn
- Explain the prepare and commit phases of two-phase commit.
- Reproduce the blocking problem: in-doubt transactions holding locks.
- Compare 2PC with a saga of local transactions under the same crash.
- Choose monitoring that exposes in-doubt transactions and lock waits.
Before you start
Symptoms
- One checkout pod is restarted by Kubernetes. Seconds later, checkouts on the *other* pods start failing too.
- Inventory and Payment logs fill with
lock wait timeout exceeded. pg_prepared_xactsshows transactions that are neither committed nor rolled back.- The errors stop only once the crashed pod is back and has resolved its transactions.
System context
In phase 1 the coordinator asks every participant to prepare: each locks the rows, writes the change durably and votes yes or no. In phase 2, if all voted yes, the coordinator tells everyone to commit; otherwise to abort. A participant that voted yes has given up the right to decide on its own.
Root cause
- Instance 1 crashed after its participants prepared but before it sent commit.
- The participants were left in doubt: they cannot commit (maybe someone voted no) and cannot abort (maybe the coordinator already told others to commit). So they keep the rows locked.
- The most popular products were among those locked rows. Instance 2's transactions queued for the same locks, waited past the lock timeout and voted no.
- Only when instance 1 recovered and read its log did it resolve the in-doubt transactions (presumed abort) and release the locks.
Interactive reproduction
A is the healthy baseline. In B, instance 1 crashes at t = 8 s mid-transaction and stays down for 12 s: watch Rows locked in Inventory freeze and Instance 2: failed climb even though instance 2 never crashed. C runs the same crash with a saga.
Solution alternatives
| Option | Effect | Cost |
|---|---|---|
| Shorter lock timeouts | Healthy instances fail faster | Does not release the in-doubt locks; more aborted work. |
| Highly available coordinator (replicated log) | Recovery in seconds instead of minutes | Operational complexity; still blocks during failover. |
| Manual heuristic commit/rollback | Unblocks immediately | Risks inconsistency; needs an on-call human. |
| Saga of local transactions | No cross-service locks; failures stay local | Intermediate states are visible; compensations must be designed. |
Recommended fix
Across services, replace 2PC with a Saga: each step is a local transaction that commits immediately, and a failure triggers compensating steps (release stock, refund). No locks are held between steps, so a crashed orchestrator affects only its own in-flight sagas — which it resumes from its log when it comes back.
-- Step 1 (Inventory, local transaction): reserve with an expiry instead of holding a lock.
UPDATE stock SET reserved = reserved + :qty
WHERE sku = :sku AND available - reserved >= :qty;
INSERT INTO reservations(id, sku, qty, expires_at) VALUES (:sagaId, :sku, :qty, now() + interval '15 minutes');
-- Step 2 (Payment): charge the card. On failure the orchestrator runs the compensation:
UPDATE stock SET reserved = reserved - :qty WHERE sku = :sku;
DELETE FROM reservations WHERE id = :sagaId;
-- A sweeper releases reservations whose saga never finished — the lock-free version of "in doubt".Advanced 2PC is still fine *inside* one database cluster or between two resources you fully control with a highly available coordinator (e.g. Kafka transactions, Spanner). The problem is stretching it across independently deployed services, where any participant or coordinator can disappear at any time.
Verification strategy
- Kill one orchestrator instance under load; success rate on the other instances must not drop.
- Inject failures at each saga step and assert that compensations leave stock and payments balanced.
- Check that expired reservations are released within the sweeper interval.
Monitoring and prevention
- Alert on any prepared transaction older than a few seconds (
pg_prepared_xacts, XA recover lists). - Track lock wait time and lock timeouts per table.
- For sagas: count stuck sagas and compensations per hour.
Lessons learned
- A protocol that holds locks across the network turns one crash into a shared outage.
- Across services, prefer local commits plus compensation over global locks.
- Design for the intermediate states (reserved, pending) instead of pretending they do not exist.