Real Problems
Customers Charged Twice: Duplicate Kafka Processing
A slow downstream made a Kafka consumer miss its poll deadline. The group rebalanced, the same payment messages were processed again — and nothing crashed. How to reproduce, fix and prevent it.
What you will learn
- Connect duplicate side effects to consumer rebalances and max.poll.interval.ms.
- Reproduce the incident deterministically and see why no error was logged.
- Compare the fixes and pick idempotency as the real cure.
- Know which metrics would have caught the incident early.
Before you start
Symptoms
- Support tickets: a handful of customers see two identical card charges for one order.
- No errors in the payment service logs. No crashes, no restarts, no deployments.
- Consumer logs show occasional
CommitFailedExceptionand "Revoking previously assigned partitions". - Consumer lag spikes for a few seconds at the same times.
System context
Each consumer polls a batch, calls the card gateway for each message, then commits offsets. Processing normally takes ~250 ms per message. max.poll.interval.ms is configured low (2 s in the simulation) to detect stuck consumers quickly.
Root cause
- The card gateway became slow for one consumer's requests (2.6 s instead of 0.25 s).
- The consumer did not call
poll()again withinmax.poll.interval.ms, so the broker considered it dead and rebalanced: its partition moved to another consumer and rewound to the last committed offset. - The new owner re-processed the uncommitted messages and charged the cards.
- Meanwhile the original consumer finished its slow call — charging the card a second time — and its offset commit was rejected (
CommitFailedException). - It rejoined the group, triggering another rebalance, and the cycle repeated while the gateway stayed slow.
Interactive reproduction
Scenario A slows payments-1 at t = 8 s. Watch the Rebalances and Double charges tiles and the event log (show every request in advanced mode). Scenario B runs the identical timeline with an idempotent consumer.
Solution alternatives
| Option | Effect | Verdict |
|---|---|---|
Raise max.poll.interval.ms | Fewer rebalances caused by slow processing. | Helps, but hides genuinely stuck consumers longer; does not remove duplicates from crashes. |
Lower max.poll.records | Smaller batches finish within the deadline. | Good hygiene; same limitation. |
| Commit before processing (at-most-once) | No duplicates. | Loses payments on a crash. Unacceptable. |
| Idempotency key to the card gateway + processed-message table | Duplicates become harmless no matter why they happen. | Recommended. |
Recommended fix
Make the consumer idempotent end to end: record the message id in a processed-messages table inside the local transaction, and send the same id as the Idempotency-Key to the card gateway so the gateway deduplicates even if your transaction is the part that fails. Then tune the poll settings so rebalances stay rare.
spring:
kafka:
consumer:
group-id: payments
enable-auto-commit: false
max-poll-records: 20 # batch × worst-case latency must fit the interval
properties:
max.poll.interval.ms: 120000
partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
listener:
ack-mode: record@KafkaListener(topics = "payments", groupId = "payments")
@Transactional
public void on(PaymentRequested event) {
if (!processed.markIfNew("payments", event.eventId())) { // INSERT ... ON CONFLICT DO NOTHING
log.info("duplicate {} skipped", event.eventId());
return;
}
gateway.charge(event.orderId(), event.amount(), event.eventId()); // eventId as Idempotency-Key
payments.recordCharged(event.orderId());
}Verification strategy
- Integration test: deliver the same message twice (and concurrently to two listeners) and assert exactly one charge.
- Chaos test: slow the gateway stub beyond
max.poll.interval.msin a test environment and assert zero duplicate charges while rebalances happen. - Reconcile: a daily job compares charges with orders and reports any order with more than one charge.
Monitoring and prevention
- Alert on consumer group rebalance rate and on
CommitFailedException— both should be near zero in steady state. - Track "duplicates skipped" as a metric: non-zero is fine (the fix is working), a spike means rebalances.
- Record per-message processing time and compare its p99 ×
max.poll.recordswithmax.poll.interval.ms.
Lessons learned
- At-least-once is the contract; design every consumer to be idempotent from day one (see Idempotent Consumer).
- Slow dependencies cause rebalances; rebalances cause redelivery. Put timeouts on calls made from consumers.
- Silence in the logs is not proof of correctness — reconcile business outcomes.