Skip to content
System Design Lab
Design to Scale

Kafka Partitioning

Partitions are Kafka's unit of parallelism and ordering. Choose the message key and partition count well and consumers scale linearly; choose badly and one hot partition holds everything back.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Explain how the message key decides the partition and the ordering guarantee.
  • Recognise a hot partition caused by a skewed key.
  • Understand why consumers beyond the partition count sit idle.
  • Pick a partition count with room to grow.

Before you start

Overview

A supermarket with six checkout lanes assigns customers to lanes by the first letter of their surname. Each lane serves its customers in order. It works — until a coach full of people named Nguyen arrives, and one lane has a queue out of the door while the others are empty.

A Kafka topic is split into partitions, each an ordered, append-only log. The producer picks a partition from the message key — hash(key) % partitions — so all messages with the same key land in the same partition, in order. Within a consumer group each partition is read by exactly one consumer. That gives two hard rules: ordering exists only within a partition, and parallelism is capped by the partition count.

Run it

Watch Lag per partition. In A the key is the order id and load is even. In B the key is the merchant id and one big merchant overloads a single partition. In C traffic grows and the team adds consumers: 9 consumers for 6 partitions.

Loading simulation…

Choosing the key

KeyOrderingDistributionUse when
Entity id (order, account)Per entityEven if there are many entitiesEvents about one thing must stay in order — the usual choice.
Tenant / merchant idPer tenantSkewed if tenants differ in sizeYou need cross-entity order per tenant — watch for whales.
No key (null)NoneEven (sticky / round-robin)Order does not matter at all (logs, metrics).
Composite (tenant:entity)Per entityEvenA big tenant would otherwise create a hot partition.

In practice

Key by entity, not by tenant
// Hot: every order of a large merchant lands in one partition.
kafka.send("orders", order.merchantId(), event);

// Even: per-order ordering is all the consumers need.
kafka.send("orders", order.id(), event);
Find the hot partition
kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group fulfilment
# TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG    CONSUMER-ID
# orders  4          18822           18830           8      worker-5
# orders  5          9120            31544           22424  worker-6   ← one partition, all the lag

Trade-offs

BenefitsCosts
More partitionsMore consumers in parallel; more room to grow.More files, memory and open connections on brokers; slower rebalances and leader elections.
Coarse keyStronger ordering (everything for a tenant).Skewed load — one partition can become the bottleneck.
Fine key / no keyEven load across partitions.Weaker or no ordering guarantees.

Production considerations

  • Monitor lag per partition, not just per topic — a single hot partition hides in the total.
  • Size partitions for peak throughput ÷ per-consumer throughput, then add headroom.
  • For a whale tenant, split its key (tenant:entity) or give it a dedicated topic.
  • Scaling consumers past the partition count does nothing; consumer groups explains why.

Further reading

Related topics