Microservice Patterns
Dead Letter Queue
Park messages that keep failing in a separate queue after a few attempts, so one poison message cannot block or burn a consumer forever — then fix the cause and redrive them.
What you will learn
- Recognise a poison message and how it blocks ordered processing.
- Configure a retry limit and a dead-letter destination.
- Operate a DLQ: alerting, inspection and redrive.
- Set up dead-letter publishing with Spring Kafka.
Before you start
Overview
A post office cannot deliver a letter with an impossible address. Instead of trying forever while the rest of the mail waits, it moves the letter to a "dead letter office" where someone looks at it later. A dead letter queue does the same for messages.
A poison message is one that fails every time it is processed: malformed data, a missing reference, a bug for one specific customer. Retrying it does not help. A dead letter queue (DLQ) receives such messages after a configured number of attempts so the consumer can move on.
The problem: one message stops everything
With an ordered queue (or a Kafka partition), a message that fails is retried before anything behind it. One bad invoice for customer k7 is enough to stop invoicing for every customer. Even with unordered queues, poison messages burn worker time and fill logs with the same stack trace.
Run it
In A, watch Processed freeze the moment the first k7 message arrives while Waiting in queue climbs. In B the stream continues and k7 messages collect in the DLQ. In C the bug fix is deployed and the DLQ redriven — use the Redrive DLQ button yourself at any time.
Operating a DLQ
- Alert on any message in the DLQ (or on its growth rate). A DLQ nobody watches is just a slower way to lose data.
- Keep context: original topic/queue, partition and offset, exception message and attempt count travel with the message (as headers).
- Fix the cause: deploy the bug fix, correct the reference data, or decide the message is invalid.
- Redrive the messages to the main queue — or discard them deliberately with an audit trail.
- Make consumers idempotent: redriven messages may have partially succeeded before (see Idempotent Consumer).
Implementation with Spring Kafka
@Configuration
class KafkaErrorHandlingConfig {
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
// Failed records go to <topic>.DLT, same partition, with exception details in headers.
var recoverer = new DeadLetterPublishingRecoverer(template);
var handler = new DefaultErrorHandler(recoverer, new ExponentialBackOff(500, 2.0) {{
setMaxElapsedTime(5_000); // a few retries, then dead-letter
}});
// Never retry errors that cannot succeed.
handler.addNotRetryableExceptions(DeserializationException.class, InvalidInvoiceException.class);
return handler;
}
}Advanced Blocking retries in the listener (above) preserve order but stall the partition while they wait. For long backoffs use non-blocking retries (@RetryableTopic): failed records move to retry topics with delays and finally to a DLT, so the main partition keeps flowing — at the cost of ordering for the retried record.
Trade-offs
| Benefits | Costs | |
|---|---|---|
| Availability | One bad message cannot stop the stream. | Dead-lettered messages are delayed until someone acts. |
| Ordering | Healthy keys keep their order. | A dead-lettered message is now out of order relative to later messages for the same entity. |
| Operations | Failures are visible and replayable. | Needs alerting, tooling and an owner. |
Production considerations
- Set DLQ retention longer than your worst-case time to fix and redrive.
- If later messages for the same entity depend on a dead-lettered one, park the whole entity (or pause it) instead of skipping a single message.
- Track DLQ size, age of the oldest message and redrive counts on a dashboard.