Microservice Patterns
Idempotent Consumer
Make processing a message twice have the same effect as processing it once, because every practical message broker will, sooner or later, deliver the same message more than once.
What you will learn
- Explain why at-least-once delivery is the realistic default and where duplicates come from.
- Implement deduplication with a processed-message table in the same transaction as the side effect.
- Choose between natural idempotency and explicit deduplication.
- Understand the limits of broker-level "exactly once".
Before you start
Overview
Pressing a lift button five times does not call five lifts — the second press changes nothing. Setting a thermostat to 21 °C twice leaves it at 21 °C. Operations like these are idempotent. An idempotent consumer makes message handling behave the same way: the second delivery of "charge order #42" does not charge again.
Brokers offer at-least-once delivery: a message is redelivered unless the consumer confirms it was processed. Any crash, timeout or rebalance between doing the work and confirming it means the work is done again. The consumer must therefore detect repeats — usually by remembering message ids it has already handled.
Where duplicates come from
- A consumer crashes after the side effect but before committing its offset.
- Processing takes longer than the broker's deadline; the partition is reassigned and redelivered (see Duplicate Message Processing).
- A producer retries a send whose acknowledgement was lost, or an outbox relay publishes a row twice.
- Someone replays a topic from an old offset to recover from a bug.
Press play or step through the interaction.
Run it
Both scenarios crash consumers at the same moments and redeliver the same messages. Only the red Duplicate charges tile differs. Use Crash billing-1 now and the network-duplication fault to create more redeliveries.
Making a handler idempotent
| Technique | Example | When it fits |
|---|---|---|
| Natural idempotency | UPDATE orders SET status = 'SHIPPED' WHERE id = ? | The operation sets a state instead of incrementing or appending. |
| Processed-message table | Insert the message id with a unique constraint in the same transaction as the work. | Any side effect in your own database. The default choice. |
| Idempotency key downstream | Pass the message id as Idempotency-Key to the payment API. | The side effect happens in another system that supports keys. |
| Conditional write / version | UPDATE … WHERE version = ? (optimistic locking). | Updates where the message carries the expected previous state. |
Implementation with Spring Kafka
CREATE TABLE processed_messages (
consumer_group VARCHAR(100) NOT NULL,
message_id VARCHAR(100) NOT NULL,
processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (consumer_group, message_id)
);@Component
class BillingListener {
private final JdbcClient jdbc;
private final BillingService billing;
BillingListener(JdbcClient jdbc, BillingService billing) {
this.jdbc = jdbc;
this.billing = billing;
}
@KafkaListener(topics = "order-events", groupId = "billing")
@Transactional
public void on(OrderPlaced event) {
int inserted = jdbc.sql("""
INSERT INTO processed_messages (consumer_group, message_id)
VALUES ('billing', :id) ON CONFLICT DO NOTHING
""")
.param("id", event.eventId())
.update();
if (inserted == 0) {
return; // already processed: skip, offset is committed after return
}
billing.charge(event); // same database transaction as the insert above
}
}What about Kafka's exactly-once? Advanced
Advanced Kafka transactions give exactly-once semantics for read-process-write within Kafka: consume, produce results and commit offsets atomically. They do not cover side effects outside Kafka — a database write, an email, a card charge. For those, an idempotent consumer is still required. Treat "exactly once" as "effectively once": at-least-once delivery plus idempotent processing.
Trade-offs
| Benefits | Costs | |
|---|---|---|
| Correctness | Redeliveries become harmless. | Only if the dedup record and the work share a transaction. |
| Performance | One indexed insert per message. | Extra write per message; the table must be pruned. |
| Design | Lets you use simple at-least-once infrastructure everywhere. | Every message needs a stable, unique id assigned by the producer. |
Production considerations
- Assign the id at the source (e.g. the outbox row id), never on redelivery — a new id per attempt defeats the purpose.
- Keep processed ids at least as long as messages can be redelivered (topic retention plus replay window), then prune.
- Count skipped duplicates as a metric: a sudden rise points to rebalances or producer retries.