Skip to content
System Design Lab
Case Studies

Design a Payment System

Charge cards reliably when networks time out and providers are slow: idempotency keys, a double-entry ledger, state machines, webhooks and reconciliation — and why a naive retry bills customers twice.

Advanced 5 min read· Updated 2026-11-18

What you will learn

  • Explain why timeouts plus retries create double charges.
  • Design idempotent payment APIs with idempotency keys.
  • Model payments as a state machine backed by a double-entry ledger.
  • Close the gaps with webhooks, an outbox and daily reconciliation.

Before you start

Requirements

  • Functional: checkout creates a payment, the card is authorised and captured through a payment service provider (PSP), refunds are possible, merchants are paid out.
  • Non-functional: never charge twice, never lose a payment; every state change is auditable; correctness beats latency.
  • Constraints: the PSP is external, can be slow, and may complete a charge even when your request timed out.

Back-of-the-envelope estimates

QuantityAssumptionResult
Payments10 M per day~120/s average, peaks of ~1,000/s on sale days
Ledger entries~4 entries per payment (debit/credit, fees)~40 M rows per day, kept for years
Latency budgetCheckout waits for authorisationPSP p50 ~200 ms, p99 several seconds

Throughput is modest; the hard part is correctness under partial failure. A timeout tells you nothing about whether the bank charged the card.

Evolve the design

Stage 1 of 2

Retry on timeout

Current architecture
Checkout calls the PSP with a 1 s timeout and retries twice.
Limitation
A timed-out request may still succeed at the bank; the retry is a brand-new charge.
Bottleneck
Not capacity — correctness. Every slow authorisation can become a double charge.
Improvement
Make the request idempotent: the same payment must map to the same charge, however often it is sent.
Performance impact
Fast, but ~15% of slow attempts produce a second charge (127 in this run).
Complexity & operational cost
Refunds, chargebacks, support tickets and lost trust.
Trade-off
Retries improve success rate only if repeating is safe — here it is not.
Loading simulation…

Payment state machine and ledger

Idempotent creation and a double-entry ledger
CREATE TABLE payments (
    id               uuid PRIMARY KEY,
    idempotency_key  text UNIQUE NOT NULL,               -- the client's key: a retry finds the same row
    amount_minor     bigint NOT NULL, currency char(3) NOT NULL,
    status           text NOT NULL,                       -- CREATED → AUTHORISED → CAPTURED | FAILED | REFUNDED
    psp_reference    text,
    updated_at       timestamptz NOT NULL
);

-- Money never appears or disappears: each movement is two rows that sum to zero.
INSERT INTO ledger_entries (payment_id, account, amount_minor) VALUES
  (:id, 'customer_receivable', -:amount),
  (:id, 'merchant_payable',     :amount);
  • Transitions are guarded: UPDATE payments SET status = 'CAPTURED' WHERE id = ? AND status = 'AUTHORISED' — out-of-order webhooks cannot move a payment backwards.
  • Webhooks from the PSP confirm final outcomes; process them idempotently and verify signatures.
  • Outbox for side effects (receipts, order fulfilment): write the event in the same transaction as the status change (transactional outbox).
  • Reconciliation: every day, compare your ledger with the PSP's settlement file; unexplained differences page a human.

Trade-offs

BenefitsCosts
Synchronous authorisationThe shopper knows the result at checkout.Checkout latency depends on the PSP; timeouts leave unknown states to resolve.
LedgerEvery cent is traceable and balances by construction.More writes and a strict data model to maintain.
Asynchronous confirmationCorrect even when calls time out or clients disappear.Final state arrives later; the UI must show 'processing'.

Interview checklist

  • Say early that correctness beats latency, and that timeouts are ambiguous.
  • Idempotency keys end to end: client → your API → PSP.
  • State machine + double-entry ledger + webhooks + reconciliation.
  • Mention security (PCI scope, tokenised cards) and how refunds and payouts fit.

Further reading

Related topics