Design to Scale
Consumer Groups
Consumer groups let many services read the same Kafka topic independently and let each service scale by adding members — with rebalances whenever membership changes.
What you will learn
- Explain how groups keep their own offsets and read independently.
- Scale a group by adding members, up to the partition count.
- Understand rebalances: when they happen and what they cost.
- Reduce rebalance pain with cooperative rebalancing and static membership.
Before you start
Overview
A newspaper is read by the sports desk and by the archive team. Each keeps its own bookmark, reads at its own speed and never waits for the other. Inside each team, people split the sections between them — more people, faster reading.
In Kafka a consumer group is a set of consumers sharing a group.id. Each group stores its own committed offsets, so two groups (billing and analytics) read the same topic independently — that is publish/subscribe. Inside a group, partitions are divided among the members — that is competing consumers. When members join, leave or die, the group rebalances: partitions are reassigned.
Run it
A has only billing. In B a slow analytics group reads the same topic: compare Billing lag and Analytics lag, then drag the analytics consumers up. In C analytics scales from 2 to 6 at t = 15 s — the group pauses for the rebalance, then drains. Crash analytics-1 to see a member removed after the session timeout.
Rebalances
| Trigger | Example | Mitigation |
|---|---|---|
| Member joins | Scale out, rolling deploy | Cooperative rebalancing: only moved partitions pause. |
| Member leaves cleanly | Scale in, shutdown | Close the consumer properly so the group reacts immediately. |
| Missed heartbeats | Crash, long GC pause, network | Tune session.timeout.ms; static membership survives quick restarts. |
| Too slow between polls | Processing a batch takes longer than max.poll.interval.ms | Smaller batches, faster processing — see duplicate processing. |
Configuration
spring:
kafka:
consumer:
enable-auto-commit: false
properties:
partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
session.timeout.ms: 45000
max.poll.interval.ms: 300000
group.instance.id: ${HOSTNAME} # static membership: a quick restart is not a rebalance
# Two listeners on the same topic, different groups → both receive every event.
# @KafkaListener(topics = "order-events", groupId = "billing", concurrency = "2")
# @KafkaListener(topics = "order-events", groupId = "analytics", concurrency = "6")Trade-offs
| Benefits | Costs | |
|---|---|---|
| Independence | New consumers of the data need no change to the producer or other groups. | Every group re-reads the data: more broker egress. |
| Scaling | Add members to share partitions and drain lag. | Capped by partition count; each membership change rebalances. |
| Failure handling | A dead member's partitions move automatically. | Detection waits for the session timeout; uncommitted work is processed again. |
Production considerations
- Name groups after the purpose (
billing,search-indexer), never reuse a group id for a different service. - Alert on lag per group and on rebalance rate — frequent rebalances usually mean slow processing or unstable pods.
- Use cooperative-sticky assignment and static membership for groups with many members or frequent deploys.
- Make consumers idempotent: every rebalance can redeliver uncommitted messages.