Design to Scale
Kafka Partitioning
Partitions are Kafka's unit of parallelism and ordering. Choose the message key and partition count well and consumers scale linearly; choose badly and one hot partition holds everything back.
What you will learn
- Explain how the message key decides the partition and the ordering guarantee.
- Recognise a hot partition caused by a skewed key.
- Understand why consumers beyond the partition count sit idle.
- Pick a partition count with room to grow.
Before you start
Overview
A supermarket with six checkout lanes assigns customers to lanes by the first letter of their surname. Each lane serves its customers in order. It works — until a coach full of people named Nguyen arrives, and one lane has a queue out of the door while the others are empty.
A Kafka topic is split into partitions, each an ordered, append-only log. The producer picks a partition from the message key — hash(key) % partitions — so all messages with the same key land in the same partition, in order. Within a consumer group each partition is read by exactly one consumer. That gives two hard rules: ordering exists only within a partition, and parallelism is capped by the partition count.
Run it
Watch Lag per partition. In A the key is the order id and load is even. In B the key is the merchant id and one big merchant overloads a single partition. In C traffic grows and the team adds consumers: 9 consumers for 6 partitions.
Choosing the key
| Key | Ordering | Distribution | Use when |
|---|---|---|---|
| Entity id (order, account) | Per entity | Even if there are many entities | Events about one thing must stay in order — the usual choice. |
| Tenant / merchant id | Per tenant | Skewed if tenants differ in size | You need cross-entity order per tenant — watch for whales. |
| No key (null) | None | Even (sticky / round-robin) | Order does not matter at all (logs, metrics). |
Composite (tenant:entity) | Per entity | Even | A big tenant would otherwise create a hot partition. |
In practice
// Hot: every order of a large merchant lands in one partition.
kafka.send("orders", order.merchantId(), event);
// Even: per-order ordering is all the consumers need.
kafka.send("orders", order.id(), event);kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group fulfilment
# TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
# orders 4 18822 18830 8 worker-5
# orders 5 9120 31544 22424 worker-6 ← one partition, all the lagTrade-offs
| Benefits | Costs | |
|---|---|---|
| More partitions | More consumers in parallel; more room to grow. | More files, memory and open connections on brokers; slower rebalances and leader elections. |
| Coarse key | Stronger ordering (everything for a tenant). | Skewed load — one partition can become the bottleneck. |
| Fine key / no key | Even load across partitions. | Weaker or no ordering guarantees. |
Production considerations
- Monitor lag per partition, not just per topic — a single hot partition hides in the total.
- Size partitions for peak throughput ÷ per-consumer throughput, then add headroom.
- For a whale tenant, split its key (
tenant:entity) or give it a dedicated topic. - Scaling consumers past the partition count does nothing; consumer groups explains why.