Consumer lag grew all day, and doubling the consumers changed nothing. Learn how partitions cap parallelism, how hot keys create one slow partition, and how to size a consumer group.
Intermediate 4 min read· Updated 2026-11-18
What you will learn
Compute whether a consumer group can keep up from arrival rate and processing time.
Explain why consumers beyond the partition count sit idle.
Diagnose a hot partition from per-partition lag.
Choose between more partitions, faster processing and better keys.
Analytics dashboards are hours behind; kafka-consumer-groups --describe shows LAG growing on every check.
The team scales the consumer deployment from 3 to 6 pods. Lag keeps growing at almost the same speed.
Three of the new pods log "no partitions assigned" and do nothing.
After a fix, one partition still lags far behind the others.
System context
A clickstream producer writes ~36 events/s to a topic; a consumer group processes each event in ~100 ms.
Root cause
Not enough capacity: each consumer processes ~10 events/s sequentially. Two consumers handle 20/s while 36/s arrive, so lag grows by ~16 events every second.
Partitions cap parallelism: within a group, each partition is consumed by exactly one consumer. With 3 partitions, at most 3 consumers do work; the 4th, 5th and 6th get nothing. Capacity stays at 30/s < 36/s.
Key skew: after repartitioning, events are keyed by customer id and a few customers produce most of the events. All events for one key land in one partition, so that partition's consumer is overloaded while others idle.
Interactive reproduction
Walk through the scenarios in order. In B, look at the idle consumer nodes in the diagram. In D, switch to advanced mode and compare the per-partition lag tiles. Change partitions, consumers and processing time yourself to test the sizing rule.
Loading simulation…
Solution alternatives
Option
When it helps
Watch out for
Add consumers
Only while consumers < partitions.
Idle consumers beyond the partition count; each scale event triggers a rebalance.
Add partitions
When all partitions are busy.
Changes key → partition mapping for new messages (ordering per key across the change); cannot be reduced later.