Skip to content
System Design Lab
Real Problems

Customers Charged Twice: Duplicate Kafka Processing

A slow downstream made a Kafka consumer miss its poll deadline. The group rebalanced, the same payment messages were processed again — and nothing crashed. How to reproduce, fix and prevent it.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Connect duplicate side effects to consumer rebalances and max.poll.interval.ms.
  • Reproduce the incident deterministically and see why no error was logged.
  • Compare the fixes and pick idempotency as the real cure.
  • Know which metrics would have caught the incident early.

Before you start

Symptoms

  • Support tickets: a handful of customers see two identical card charges for one order.
  • No errors in the payment service logs. No crashes, no restarts, no deployments.
  • Consumer logs show occasional CommitFailedException and "Revoking previously assigned partitions".
  • Consumer lag spikes for a few seconds at the same times.

System context

CheckoutPaymentRequestedCheckoutpayments topic3 partitionspayments topicpayments-1calls card gatewaypayments-1payments-2calls card gatewaypayments-2payments-3calls card gatewaypayments-3
Checkout publishes PaymentRequested to a 3-partition topic; three consumers in one group call an external card gateway.

Each consumer polls a batch, calls the card gateway for each message, then commits offsets. Processing normally takes ~250 ms per message. max.poll.interval.ms is configured low (2 s in the simulation) to detect stuck consumers quickly.

Root cause

  1. The card gateway became slow for one consumer's requests (2.6 s instead of 0.25 s).
  2. The consumer did not call poll() again within max.poll.interval.ms, so the broker considered it dead and rebalanced: its partition moved to another consumer and rewound to the last committed offset.
  3. The new owner re-processed the uncommitted messages and charged the cards.
  4. Meanwhile the original consumer finished its slow call — charging the card a second time — and its offset commit was rejected (CommitFailedException).
  5. It rejoined the group, triggering another rebalance, and the cycle repeated while the gateway stayed slow.

Interactive reproduction

Scenario A slows payments-1 at t = 8 s. Watch the Rebalances and Double charges tiles and the event log (show every request in advanced mode). Scenario B runs the identical timeline with an idempotent consumer.

Loading simulation…

Solution alternatives

OptionEffectVerdict
Raise max.poll.interval.msFewer rebalances caused by slow processing.Helps, but hides genuinely stuck consumers longer; does not remove duplicates from crashes.
Lower max.poll.recordsSmaller batches finish within the deadline.Good hygiene; same limitation.
Commit before processing (at-most-once)No duplicates.Loses payments on a crash. Unacceptable.
Idempotency key to the card gateway + processed-message tableDuplicates become harmless no matter why they happen.Recommended.

Recommended fix

Make the consumer idempotent end to end: record the message id in a processed-messages table inside the local transaction, and send the same id as the Idempotency-Key to the card gateway so the gateway deduplicates even if your transaction is the part that fails. Then tune the poll settings so rebalances stay rare.

application.yml
spring:
  kafka:
    consumer:
      group-id: payments
      enable-auto-commit: false
      max-poll-records: 20              # batch × worst-case latency must fit the interval
      properties:
        max.poll.interval.ms: 120000
        partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
    listener:
      ack-mode: record
PaymentListener.java
@KafkaListener(topics = "payments", groupId = "payments")
@Transactional
public void on(PaymentRequested event) {
    if (!processed.markIfNew("payments", event.eventId())) {   // INSERT ... ON CONFLICT DO NOTHING
        log.info("duplicate {} skipped", event.eventId());
        return;
    }
    gateway.charge(event.orderId(), event.amount(), event.eventId());  // eventId as Idempotency-Key
    payments.recordCharged(event.orderId());
}

Verification strategy

  • Integration test: deliver the same message twice (and concurrently to two listeners) and assert exactly one charge.
  • Chaos test: slow the gateway stub beyond max.poll.interval.ms in a test environment and assert zero duplicate charges while rebalances happen.
  • Reconcile: a daily job compares charges with orders and reports any order with more than one charge.

Monitoring and prevention

  • Alert on consumer group rebalance rate and on CommitFailedException — both should be near zero in steady state.
  • Track "duplicates skipped" as a metric: non-zero is fine (the fix is working), a spike means rebalances.
  • Record per-message processing time and compare its p99 × max.poll.records with max.poll.interval.ms.

Lessons learned

  • At-least-once is the contract; design every consumer to be idempotent from day one (see Idempotent Consumer).
  • Slow dependencies cause rebalances; rebalances cause redelivery. Put timeouts on calls made from consumers.
  • Silence in the logs is not proof of correctness — reconcile business outcomes.

Further reading

Related topics