Skip to content
System Design Lab
Real Problems

Withdrawal Before Deposit: Message Ordering Problems

Account balances went negative even though every event was valid. Events for one account were applied out of order — first because of a missing key, then because of a thread pool. How to keep order per key without giving up throughput.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Identify the places where message order is lost: producer, partitioning, consumer.
  • Reproduce out-of-order processing with round-robin keys and thread pools.
  • Keep per-key order while still processing in parallel.
  • Defend against reordering with versions and sequence checks.

Before you start

Symptoms

  • Some account balances drop below zero; replaying the same events offline gives the correct balance.
  • Logs show MoneyWithdrawn processed a few milliseconds before the MoneyDeposited that funds it.
  • It happens more under load and for the busiest accounts.
  • No consumer errors and no lag spikes.

System context

Accounts APIAccounts APIaccount-eventsround-robin parti…account-eventsledger-1one at a timeledger-1ledger-2one at a timeledger-2ledger-3one at a timeledger-3ledger-4one at a timeledger-4
The Accounts API publishes account events; four ledger consumers apply them to balances.

Root cause

  1. No key: events were published without a key, so the partitioner spread one account's events across partitions. Different consumers then raced each other.
  2. The fix that wasn't: after adding the account id as key, a developer made each consumer hand records to an 8-thread pool to cope with load. Kafka delivered in order; the pool completed them in any order.
  3. Kafka only guarantees order within a partition, to a consumer that processes sequentially. Any parallelism after the poll must preserve per-key order itself.

Interactive reproduction

Compare Applied out of order and Consumer lag across the four scenarios. C is correct but cannot keep up; D keeps order per account and still processes in parallel.

Loading simulation…

Solution alternatives

OptionOrderThroughputNotes
Key = account id, single-threaded consumerPer accountLimited by partitionsSimple; add partitions and consumers to scale.
Key-ordered parallel processingPer accountHighOne in-flight event per key, many keys at once (Confluent Parallel Consumer, or a map of single-thread executors).
More partitionsPer accountHigherPlan ahead: remapping keys breaks order during the change.
Version check on applyDetects / rejectsHighStore the last applied sequence per account; reject or park older ones.

Recommended fix

Parallel across keys, sequential within a key
// One single-thread executor per "stripe"; all events of an account go to the same stripe.
private final ExecutorService[] stripes = IntStream.range(0, 8)
    .mapToObj(i -> Executors.newSingleThreadExecutor()).toArray(ExecutorService[]::new);

@KafkaListener(topics = "account-events", groupId = "ledger")
void onBatch(List<ConsumerRecord<String, AccountEvent>> records, Acknowledgment ack) {
    List<Future<?>> work = records.stream()
        .map(r -> stripes[Math.floorMod(r.key().hashCode(), stripes.length)].submit(() -> ledger.apply(r.value())))
        .toList();
    work.forEach(this::await);     // commit only after the whole batch has been applied
    ack.acknowledge();
}
Defence in depth: refuse stale events
UPDATE accounts
   SET balance = balance + :delta, last_seq = :seq
 WHERE id = :accountId AND last_seq = :seq - 1;   -- 0 rows: out of order or duplicate → park and retry

Verification strategy

  • Produce sequence numbers per key in tests and assert they are applied in increasing order.
  • Load-test with a skewed key distribution — hot keys expose ordering bugs first.
  • Run the same events through the consumer twice and compare final state with an offline replay.

Monitoring and prevention

  • Count events rejected by the sequence check; any non-zero value is a bug signal.
  • Lint producers: every send on an ordered topic must set a key.
  • Review any executor, parallelStream or @Async in a consumer path.

Lessons learned

  • Ordering is a chain: producer, key, partition and consumer all have to preserve it.
  • You rarely need global order — you need order per entity. Key on the entity and parallelise across entities.
  • Make consumers verify order (sequence numbers) so a future change cannot break it silently.

Further reading

Related topics