Real Problems
Withdrawal Before Deposit: Message Ordering Problems
Account balances went negative even though every event was valid. Events for one account were applied out of order — first because of a missing key, then because of a thread pool. How to keep order per key without giving up throughput.
Intermediate 4 min read· Updated 2026-11-18
What you will learn
- Identify the places where message order is lost: producer, partitioning, consumer.
- Reproduce out-of-order processing with round-robin keys and thread pools.
- Keep per-key order while still processing in parallel.
- Defend against reordering with versions and sequence checks.
Before you start
Symptoms
- Some account balances drop below zero; replaying the same events offline gives the correct balance.
- Logs show
MoneyWithdrawnprocessed a few milliseconds before theMoneyDepositedthat funds it. - It happens more under load and for the busiest accounts.
- No consumer errors and no lag spikes.
System context
Root cause
- No key: events were published without a key, so the partitioner spread one account's events across partitions. Different consumers then raced each other.
- The fix that wasn't: after adding the account id as key, a developer made each consumer hand records to an 8-thread pool to cope with load. Kafka delivered in order; the pool completed them in any order.
- Kafka only guarantees order within a partition, to a consumer that processes sequentially. Any parallelism after the poll must preserve per-key order itself.
Interactive reproduction
Compare Applied out of order and Consumer lag across the four scenarios. C is correct but cannot keep up; D keeps order per account and still processes in parallel.
Loading simulation…
Solution alternatives
| Option | Order | Throughput | Notes |
|---|---|---|---|
| Key = account id, single-threaded consumer | Per account | Limited by partitions | Simple; add partitions and consumers to scale. |
| Key-ordered parallel processing | Per account | High | One in-flight event per key, many keys at once (Confluent Parallel Consumer, or a map of single-thread executors). |
| More partitions | Per account | Higher | Plan ahead: remapping keys breaks order during the change. |
| Version check on apply | Detects / rejects | High | Store the last applied sequence per account; reject or park older ones. |
Recommended fix
// One single-thread executor per "stripe"; all events of an account go to the same stripe.
private final ExecutorService[] stripes = IntStream.range(0, 8)
.mapToObj(i -> Executors.newSingleThreadExecutor()).toArray(ExecutorService[]::new);
@KafkaListener(topics = "account-events", groupId = "ledger")
void onBatch(List<ConsumerRecord<String, AccountEvent>> records, Acknowledgment ack) {
List<Future<?>> work = records.stream()
.map(r -> stripes[Math.floorMod(r.key().hashCode(), stripes.length)].submit(() -> ledger.apply(r.value())))
.toList();
work.forEach(this::await); // commit only after the whole batch has been applied
ack.acknowledge();
}UPDATE accounts
SET balance = balance + :delta, last_seq = :seq
WHERE id = :accountId AND last_seq = :seq - 1; -- 0 rows: out of order or duplicate → park and retryVerification strategy
- Produce sequence numbers per key in tests and assert they are applied in increasing order.
- Load-test with a skewed key distribution — hot keys expose ordering bugs first.
- Run the same events through the consumer twice and compare final state with an offline replay.
Monitoring and prevention
- Count events rejected by the sequence check; any non-zero value is a bug signal.
- Lint producers: every send on an ordered topic must set a key.
- Review any executor,
parallelStreamor@Asyncin a consumer path.
Lessons learned
- Ordering is a chain: producer, key, partition and consumer all have to preserve it.
- You rarely need global order — you need order per entity. Key on the entity and parallelise across entities.
- Make consumers verify order (sequence numbers) so a future change cannot break it silently.
Further reading
Related topics
Design to Scale
Kafka Partitioning
Partitions are Kafka's unit of parallelism and ordering. Choose the message key and partition count well and consumers scale linearly; choose badly and one hot partition holds everything back.
Design to Scale
Consumer Groups
Consumer groups let many services read the same Kafka topic independently and let each service scale by adding members — with rebalances whenever membership changes.
Microservice Patterns
Idempotent Consumer
Make processing a message twice have the same effect as processing it once, because every practical message broker will, sooner or later, deliver the same message more than once.
Microservice Patterns
Event Sourcing
Store every change as an immutable event and derive current state by replaying them — gaining a full audit trail and time travel, at the cost of replay time that snapshots must keep in check.