Case Studies
Design a Chat Service
Deliver messages in real time to millions of connected users: why polling is slow and wasteful, how WebSockets push instantly, and what breaks — and how pub/sub fixes it — once connections span many servers.
Intermediate 5 min read· Updated 2026-11-18
What you will learn
- Compare polling, long polling and persistent connections.
- Route messages between users connected to different servers.
- Store messages for history, ordering and offline delivery.
- Handle presence, receipts and reconnects.
Before you start
Requirements
- Functional: one-to-one and group conversations, message history, delivery and read receipts, online presence, push notifications for offline users.
- Non-functional: messages delivered in well under a second; no message lost; messages in a conversation shown in the same order to everyone.
- Scale: tens of millions of concurrent connections — far more than one server can hold.
Back-of-the-envelope estimates
| Quantity | Assumption | Result |
|---|---|---|
| Concurrent users | 50 M online at peak | ~50 M open connections |
| Connections per server | ~100 k–500 k WebSockets per node | 100–500 gateway servers |
| Messages | 50 M users × 40 messages/day | ~25,000 messages/s, more with group fan-out |
| Storage | ~200 B per message | ~400 GB/day before replication |
Evolve the design
Stage 1 of 4
HTTP polling
- Current architecture
- Each client asks "anything new?" every 2 s.
- Limitation
- Average delay is half the poll interval; most polls in a quiet conversation return nothing.
- Bottleneck
- Wasted requests: load grows with users × poll frequency, not with messages.
- Improvement
- Keep a connection open and let the server push.
- Performance impact
- ~1.1 s average delay; about a third of polls empty even in an active chat.
- Complexity & operational cost
- Simple HTTP; heavy, mostly useless load at scale.
- Trade-off
- Easy to build, poor latency and efficiency.
Loading simulation…
Storing and ordering messages
- Write first, then fan out: persist the message (partitioned by conversation id) before pushing, so history and offline users are covered.
- Order per conversation: assign a per-conversation sequence number on write; clients sort by it and detect gaps (message ordering).
- Offline users: they are not connected anywhere — enqueue a push notification and let them fetch history on reconnect (
messages after seq N). - Receipts and presence are just more small messages, but rate-limit presence updates — they dwarf chat traffic.
function connect(conversationId: string, lastSeq: number) {
const ws = new WebSocket(`wss://chat.example.com/ws?c=${conversationId}&after=${lastSeq}`);
ws.onmessage = (e) => {
const msg = JSON.parse(e.data) as { seq: number; text: string };
if (msg.seq <= lastSeq) return; // duplicate after reconnect
lastSeq = msg.seq;
render(msg);
};
ws.onclose = () => setTimeout(() => connect(conversationId, lastSeq), backoffWithJitter());
}Trade-offs
| Benefits | Costs | |
|---|---|---|
| Push | Instant delivery, no wasted requests. | Stateful servers; deploys and failures drop many connections at once. |
| Pub/sub routing | Any server can reach any user. | All traffic crosses the broker; large groups amplify load. |
| Per-conversation ordering | Everyone sees the same order. | A conversation's writes go through one partition — very busy groups need special handling. |
Interview checklist
- Choose WebSockets (or similar) and justify against polling.
- Explain how a message reaches a user on another server.
- Persist before fan-out; order per conversation with sequence numbers.
- Cover offline delivery, receipts, presence and reconnects.
Further reading
Related topics
Microservice Patterns
Publish/Subscribe
Publish an event once and let any number of independent subscribers receive their own copy — new consumers can be added without touching the producer.
Design to Scale
Stateless Application Design
Keep no user state in the memory of an application instance, so any instance can serve any request — the precondition for load balancing, autoscaling and painless deploys.
Real Problems
Withdrawal Before Deposit: Message Ordering Problems
Account balances went negative even though every event was valid. Events for one account were applied out of order — first because of a missing key, then because of a thread pool. How to keep order per key without giving up throughput.
Design to Scale
Kafka Partitioning
Partitions are Kafka's unit of parallelism and ordering. Choose the message key and partition count well and consumers scale linearly; choose badly and one hot partition holds everything back.
Design to Scale
Load Balancing
Spread requests across instances so none is overloaded — and see how the choice of algorithm and health checks decides your tail latency when instances are not identical.