Case Studies
Design a Payment System
Charge cards reliably when networks time out and providers are slow: idempotency keys, a double-entry ledger, state machines, webhooks and reconciliation — and why a naive retry bills customers twice.
Advanced 5 min read· Updated 2026-11-18
What you will learn
- Explain why timeouts plus retries create double charges.
- Design idempotent payment APIs with idempotency keys.
- Model payments as a state machine backed by a double-entry ledger.
- Close the gaps with webhooks, an outbox and daily reconciliation.
Before you start
Requirements
- Functional: checkout creates a payment, the card is authorised and captured through a payment service provider (PSP), refunds are possible, merchants are paid out.
- Non-functional: never charge twice, never lose a payment; every state change is auditable; correctness beats latency.
- Constraints: the PSP is external, can be slow, and may complete a charge even when your request timed out.
Back-of-the-envelope estimates
| Quantity | Assumption | Result |
|---|---|---|
| Payments | 10 M per day | ~120/s average, peaks of ~1,000/s on sale days |
| Ledger entries | ~4 entries per payment (debit/credit, fees) | ~40 M rows per day, kept for years |
| Latency budget | Checkout waits for authorisation | PSP p50 ~200 ms, p99 several seconds |
Throughput is modest; the hard part is correctness under partial failure. A timeout tells you nothing about whether the bank charged the card.
Evolve the design
Stage 1 of 2
Retry on timeout
- Current architecture
- Checkout calls the PSP with a 1 s timeout and retries twice.
- Limitation
- A timed-out request may still succeed at the bank; the retry is a brand-new charge.
- Bottleneck
- Not capacity — correctness. Every slow authorisation can become a double charge.
- Improvement
- Make the request idempotent: the same payment must map to the same charge, however often it is sent.
- Performance impact
- Fast, but ~15% of slow attempts produce a second charge (127 in this run).
- Complexity & operational cost
- Refunds, chargebacks, support tickets and lost trust.
- Trade-off
- Retries improve success rate only if repeating is safe — here it is not.
Loading simulation…
Payment state machine and ledger
CREATE TABLE payments (
id uuid PRIMARY KEY,
idempotency_key text UNIQUE NOT NULL, -- the client's key: a retry finds the same row
amount_minor bigint NOT NULL, currency char(3) NOT NULL,
status text NOT NULL, -- CREATED → AUTHORISED → CAPTURED | FAILED | REFUNDED
psp_reference text,
updated_at timestamptz NOT NULL
);
-- Money never appears or disappears: each movement is two rows that sum to zero.
INSERT INTO ledger_entries (payment_id, account, amount_minor) VALUES
(:id, 'customer_receivable', -:amount),
(:id, 'merchant_payable', :amount);- Transitions are guarded:
UPDATE payments SET status = 'CAPTURED' WHERE id = ? AND status = 'AUTHORISED'— out-of-order webhooks cannot move a payment backwards. - Webhooks from the PSP confirm final outcomes; process them idempotently and verify signatures.
- Outbox for side effects (receipts, order fulfilment): write the event in the same transaction as the status change (transactional outbox).
- Reconciliation: every day, compare your ledger with the PSP's settlement file; unexplained differences page a human.
Trade-offs
| Benefits | Costs | |
|---|---|---|
| Synchronous authorisation | The shopper knows the result at checkout. | Checkout latency depends on the PSP; timeouts leave unknown states to resolve. |
| Ledger | Every cent is traceable and balances by construction. | More writes and a strict data model to maintain. |
| Asynchronous confirmation | Correct even when calls time out or clients disappear. | Final state arrives later; the UI must show 'processing'. |
Interview checklist
- Say early that correctness beats latency, and that timeouts are ambiguous.
- Idempotency keys end to end: client → your API → PSP.
- State machine + double-entry ledger + webhooks + reconciliation.
- Mention security (PCI scope, tokenised cards) and how refunds and payouts fit.
Further reading
Related topics
Microservice Patterns
Idempotent Consumer
Make processing a message twice have the same effect as processing it once, because every practical message broker will, sooner or later, deliver the same message more than once.
Microservice Patterns
Retry
Transparently repeat a failed call when the failure is likely temporary — with backoff, jitter and a strict limit, so retries heal glitches instead of amplifying outages.
Microservice Patterns
Transactional Outbox
Save a business change and the event announcing it in the same database transaction, then publish the event from that table — so you never lose an event or publish one for a change that rolled back.
Microservice Patterns
Saga
Keep data consistent across services without a distributed transaction: run a sequence of local transactions and, when one fails, undo the completed ones with compensating actions.
Real Problems
400 Phones Sold, 100 in Stock: Race Conditions in a Flash Sale
A flash sale sold three times the available stock without a single error. The code checked stock, then saved stock − 1 — and hundreds of buyers did it at once. Reproduce it, then fix it properly.
Microservice Patterns
Asynchronous Request-Reply
For work that takes seconds or minutes, answer at once with 202 Accepted and a status URL, do the work in the background, and let the client poll or be notified — instead of holding a connection open until something times out.