Skip to content
System Design Lab
Design to Scale

Consumer Groups

Consumer groups let many services read the same Kafka topic independently and let each service scale by adding members — with rebalances whenever membership changes.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Explain how groups keep their own offsets and read independently.
  • Scale a group by adding members, up to the partition count.
  • Understand rebalances: when they happen and what they cost.
  • Reduce rebalance pain with cooperative rebalancing and static membership.

Before you start

Overview

A newspaper is read by the sports desk and by the archive team. Each keeps its own bookmark, reads at its own speed and never waits for the other. Inside each team, people split the sections between them — more people, faster reading.

In Kafka a consumer group is a set of consumers sharing a group.id. Each group stores its own committed offsets, so two groups (billing and analytics) read the same topic independently — that is publish/subscribe. Inside a group, partitions are divided among the members — that is competing consumers. When members join, leave or die, the group rebalances: partitions are reassigned.

Run it

A has only billing. In B a slow analytics group reads the same topic: compare Billing lag and Analytics lag, then drag the analytics consumers up. In C analytics scales from 2 to 6 at t = 15 s — the group pauses for the rebalance, then drains. Crash analytics-1 to see a member removed after the session timeout.

Loading simulation…

Rebalances

TriggerExampleMitigation
Member joinsScale out, rolling deployCooperative rebalancing: only moved partitions pause.
Member leaves cleanlyScale in, shutdownClose the consumer properly so the group reacts immediately.
Missed heartbeatsCrash, long GC pause, networkTune session.timeout.ms; static membership survives quick restarts.
Too slow between pollsProcessing a batch takes longer than max.poll.interval.msSmaller batches, faster processing — see duplicate processing.

Configuration

Spring Kafka: two independent groups, cooperative rebalancing
spring:
  kafka:
    consumer:
      enable-auto-commit: false
      properties:
        partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
        session.timeout.ms: 45000
        max.poll.interval.ms: 300000
        group.instance.id: ${HOSTNAME}      # static membership: a quick restart is not a rebalance

# Two listeners on the same topic, different groups → both receive every event.
# @KafkaListener(topics = "order-events", groupId = "billing",   concurrency = "2")
# @KafkaListener(topics = "order-events", groupId = "analytics", concurrency = "6")

Trade-offs

BenefitsCosts
IndependenceNew consumers of the data need no change to the producer or other groups.Every group re-reads the data: more broker egress.
ScalingAdd members to share partitions and drain lag.Capped by partition count; each membership change rebalances.
Failure handlingA dead member's partitions move automatically.Detection waits for the session timeout; uncommitted work is processed again.

Production considerations

  • Name groups after the purpose (billing, search-indexer), never reuse a group id for a different service.
  • Alert on lag per group and on rebalance rate — frequent rebalances usually mean slow processing or unstable pods.
  • Use cooperative-sticky assignment and static membership for groups with many members or frequent deploys.
  • Make consumers idempotent: every rebalance can redeliver uncommitted messages.

Further reading

Related topics