Skip to content
System Design Lab
Case Studies

Design a Chat Service

Deliver messages in real time to millions of connected users: why polling is slow and wasteful, how WebSockets push instantly, and what breaks — and how pub/sub fixes it — once connections span many servers.

Intermediate 5 min read· Updated 2026-11-18

What you will learn

  • Compare polling, long polling and persistent connections.
  • Route messages between users connected to different servers.
  • Store messages for history, ordering and offline delivery.
  • Handle presence, receipts and reconnects.

Before you start

Requirements

  • Functional: one-to-one and group conversations, message history, delivery and read receipts, online presence, push notifications for offline users.
  • Non-functional: messages delivered in well under a second; no message lost; messages in a conversation shown in the same order to everyone.
  • Scale: tens of millions of concurrent connections — far more than one server can hold.

Back-of-the-envelope estimates

QuantityAssumptionResult
Concurrent users50 M online at peak~50 M open connections
Connections per server~100 k–500 k WebSockets per node100–500 gateway servers
Messages50 M users × 40 messages/day~25,000 messages/s, more with group fan-out
Storage~200 B per message~400 GB/day before replication

Evolve the design

Stage 1 of 4

HTTP polling

Current architecture
Each client asks "anything new?" every 2 s.
Limitation
Average delay is half the poll interval; most polls in a quiet conversation return nothing.
Bottleneck
Wasted requests: load grows with users × poll frequency, not with messages.
Improvement
Keep a connection open and let the server push.
Performance impact
~1.1 s average delay; about a third of polls empty even in an active chat.
Complexity & operational cost
Simple HTTP; heavy, mostly useless load at scale.
Trade-off
Easy to build, poor latency and efficiency.
Loading simulation…

Storing and ordering messages

  • Write first, then fan out: persist the message (partitioned by conversation id) before pushing, so history and offline users are covered.
  • Order per conversation: assign a per-conversation sequence number on write; clients sort by it and detect gaps (message ordering).
  • Offline users: they are not connected anywhere — enqueue a push notification and let them fetch history on reconnect (messages after seq N).
  • Receipts and presence are just more small messages, but rate-limit presence updates — they dwarf chat traffic.
Client: reconnect and catch up from the last sequence number
function connect(conversationId: string, lastSeq: number) {
  const ws = new WebSocket(`wss://chat.example.com/ws?c=${conversationId}&after=${lastSeq}`);
  ws.onmessage = (e) => {
    const msg = JSON.parse(e.data) as { seq: number; text: string };
    if (msg.seq <= lastSeq) return;            // duplicate after reconnect
    lastSeq = msg.seq;
    render(msg);
  };
  ws.onclose = () => setTimeout(() => connect(conversationId, lastSeq), backoffWithJitter());
}

Trade-offs

BenefitsCosts
PushInstant delivery, no wasted requests.Stateful servers; deploys and failures drop many connections at once.
Pub/sub routingAny server can reach any user.All traffic crosses the broker; large groups amplify load.
Per-conversation orderingEveryone sees the same order.A conversation's writes go through one partition — very busy groups need special handling.

Interview checklist

  • Choose WebSockets (or similar) and justify against polling.
  • Explain how a message reaches a user on another server.
  • Persist before fan-out; order per conversation with sequence numbers.
  • Cover offline delivery, receipts, presence and reconnects.

Further reading

Related topics