Skip to content
System Design Lab
Microservice Patterns

Idempotent Consumer

Make processing a message twice have the same effect as processing it once, because every practical message broker will, sooner or later, deliver the same message more than once.

Intermediate 6 min read· Updated 2026-11-18

What you will learn

  • Explain why at-least-once delivery is the realistic default and where duplicates come from.
  • Implement deduplication with a processed-message table in the same transaction as the side effect.
  • Choose between natural idempotency and explicit deduplication.
  • Understand the limits of broker-level "exactly once".

Before you start

Overview

Pressing a lift button five times does not call five lifts — the second press changes nothing. Setting a thermostat to 21 °C twice leaves it at 21 °C. Operations like these are idempotent. An idempotent consumer makes message handling behave the same way: the second delivery of "charge order #42" does not charge again.

Brokers offer at-least-once delivery: a message is redelivered unless the consumer confirms it was processed. Any crash, timeout or rebalance between doing the work and confirming it means the work is done again. The consumer must therefore detect repeats — usually by remembering message ids it has already handled.

Where duplicates come from

  • A consumer crashes after the side effect but before committing its offset.
  • Processing takes longer than the broker's deadline; the partition is reassigned and redelivered (see Duplicate Message Processing).
  • A producer retries a send whose acknowledgement was lost, or an outbox relay publishes a row twice.
  • Someone replays a topic from an old offset to recover from a bug.
KafkaBilling consumerBilling DB

Press play or step through the interaction.

The processed-message record and the charge commit together, so a redelivery is detected.

Run it

Both scenarios crash consumers at the same moments and redeliver the same messages. Only the red Duplicate charges tile differs. Use Crash billing-1 now and the network-duplication fault to create more redeliveries.

Loading simulation…

Making a handler idempotent

TechniqueExampleWhen it fits
Natural idempotencyUPDATE orders SET status = 'SHIPPED' WHERE id = ?The operation sets a state instead of incrementing or appending.
Processed-message tableInsert the message id with a unique constraint in the same transaction as the work.Any side effect in your own database. The default choice.
Idempotency key downstreamPass the message id as Idempotency-Key to the payment API.The side effect happens in another system that supports keys.
Conditional write / versionUPDATE … WHERE version = ? (optimistic locking).Updates where the message carries the expected previous state.

Implementation with Spring Kafka

processed_messages
CREATE TABLE processed_messages (
    consumer_group VARCHAR(100) NOT NULL,
    message_id     VARCHAR(100) NOT NULL,
    processed_at   TIMESTAMPTZ  NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_group, message_id)
);
BillingListener.java
@Component
class BillingListener {

    private final JdbcClient jdbc;
    private final BillingService billing;

    BillingListener(JdbcClient jdbc, BillingService billing) {
        this.jdbc = jdbc;
        this.billing = billing;
    }

    @KafkaListener(topics = "order-events", groupId = "billing")
    @Transactional
    public void on(OrderPlaced event) {
        int inserted = jdbc.sql("""
                INSERT INTO processed_messages (consumer_group, message_id)
                VALUES ('billing', :id) ON CONFLICT DO NOTHING
                """)
                .param("id", event.eventId())
                .update();
        if (inserted == 0) {
            return;                     // already processed: skip, offset is committed after return
        }
        billing.charge(event);          // same database transaction as the insert above
    }
}

What about Kafka's exactly-once? Advanced

Advanced Kafka transactions give exactly-once semantics for read-process-write within Kafka: consume, produce results and commit offsets atomically. They do not cover side effects outside Kafka — a database write, an email, a card charge. For those, an idempotent consumer is still required. Treat "exactly once" as "effectively once": at-least-once delivery plus idempotent processing.

Trade-offs

BenefitsCosts
CorrectnessRedeliveries become harmless.Only if the dedup record and the work share a transaction.
PerformanceOne indexed insert per message.Extra write per message; the table must be pruned.
DesignLets you use simple at-least-once infrastructure everywhere.Every message needs a stable, unique id assigned by the producer.

Production considerations

  • Assign the id at the source (e.g. the outbox row id), never on redelivery — a new id per attempt defeats the purpose.
  • Keep processed ids at least as long as messages can be redelivered (topic retention plus replay window), then prune.
  • Count skipped duplicates as a metric: a sudden rise points to rebalances or producer retries.

Further reading

Related topics