Skip to content
System Design Lab
Real Problems

Kafka Consumer Lag That Never Recovers

Consumer lag grew all day, and doubling the consumers changed nothing. Learn how partitions cap parallelism, how hot keys create one slow partition, and how to size a consumer group.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Compute whether a consumer group can keep up from arrival rate and processing time.
  • Explain why consumers beyond the partition count sit idle.
  • Diagnose a hot partition from per-partition lag.
  • Choose between more partitions, faster processing and better keys.

Before you start

Symptoms

  • Analytics dashboards are hours behind; kafka-consumer-groups --describe shows LAG growing on every check.
  • The team scales the consumer deployment from 3 to 6 pods. Lag keeps growing at almost the same speed.
  • Three of the new pods log "no partitions assigned" and do nothing.
  • After a fix, one partition still lags far behind the others.

System context

ClickstreamproducerClickstreamevents topic3 partitionsevents topicconsumer-1consumer groupconsumer-1consumer-2consumer groupconsumer-2
A clickstream producer writes ~36 events/s to a topic; a consumer group processes each event in ~100 ms.

Root cause

  1. Not enough capacity: each consumer processes ~10 events/s sequentially. Two consumers handle 20/s while 36/s arrive, so lag grows by ~16 events every second.
  2. Partitions cap parallelism: within a group, each partition is consumed by exactly one consumer. With 3 partitions, at most 3 consumers do work; the 4th, 5th and 6th get nothing. Capacity stays at 30/s < 36/s.
  3. Key skew: after repartitioning, events are keyed by customer id and a few customers produce most of the events. All events for one key land in one partition, so that partition's consumer is overloaded while others idle.

Interactive reproduction

Walk through the scenarios in order. In B, look at the idle consumer nodes in the diagram. In D, switch to advanced mode and compare the per-partition lag tiles. Change partitions, consumers and processing time yourself to test the sizing rule.

Loading simulation…

Solution alternatives

OptionWhen it helpsWatch out for
Add consumersOnly while consumers < partitions.Idle consumers beyond the partition count; each scale event triggers a rebalance.
Add partitionsWhen all partitions are busy.Changes key → partition mapping for new messages (ordering per key across the change); cannot be reduced later.
Process fasterAlways: batch writes, async I/O, remove per-message remote calls.Keep per-key ordering if downstream needs it.
Parallelise inside a consumerOne partition is the bottleneck but order only matters per key.Use a key-ordered worker pool (e.g. Confluent Parallel Consumer); offsets get more complex.
Fix the keyHot partitions caused by skewed keys.Salting a hot key breaks per-key ordering for that key.

Recommended fix

  1. Make processing cheaper first (batching to the database cut processing time the most in this case).
  2. Size partitions for the target peak with headroom (here: 6–12), then run as many consumers as partitions.
  3. Monitor per-partition lag; for hot keys, aggregate or split them (e.g. customerId#bucket) when ordering allows.
Diagnosing lag
kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group clickstream-analytics

# GROUP                 TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG    CONSUMER-ID
# clickstream-analytics events  0          1840211         1840398         187    consumer-1-...
# clickstream-analytics events  1          1839877         1912030         72153  consumer-2-...   <- hot partition
# clickstream-analytics events  2          1840102         1840260         158    consumer-3-...
Advanced
Batch listener (Spring Kafka)
@KafkaListener(topics = "events", groupId = "clickstream-analytics", concurrency = "6", batch = "true")
public void onBatch(List<ClickEvent> events) {
    // One multi-row insert instead of one round trip per event.
    repository.insertAll(events);
}

// application.yml
// spring.kafka.consumer.max-poll-records: 500
// spring.kafka.listener.type: batch

Verification strategy

  • Load test the consumer alone: measure events/s per consumer, then compare with the peak arrival rate per partition.
  • After the change, lag should return to near zero after bursts, and the time to drain a burst should be predictable.
  • Confirm every consumer pod has partitions assigned.

Monitoring and prevention

  • Alert on lag trend (growing for N minutes) and on time lag (age of the oldest unconsumed event), not just message count.
  • Graph lag per partition to spot hot keys.
  • Autoscale consumers on lag (KEDA's Kafka scaler) with a maximum equal to the partition count.

Lessons learned

  • Partitions are the unit of parallelism in Kafka. Plan them for future peak, not today's load.
  • Total capacity is not enough — capacity per partition matters when keys are skewed.
  • Scaling consumers is the last step, after making processing efficient.

Further reading

Related topics