Case Studies
Design a Notification System
Send push, SMS and email at scale without letting a marketing blast delay a login code. From a synchronous call to a queue with workers and priority lanes, plus retries, preferences and deduplication.
Intermediate 5 min read· Updated 2026-11-18
What you will learn
- Separate accepting a notification from delivering it.
- Protect urgent notifications from bulk traffic with priority lanes.
- Handle provider failures with retries, dead letters and idempotency.
- Respect user preferences, quiet hours and rate limits per channel.
Before you start
Requirements
- Functional: services request a notification for a user (login code, order shipped, marketing); the system renders a template, picks channels (push, SMS, email) from the user's preferences, and delivers.
- Non-functional: urgent messages (login codes, security alerts) arrive within seconds; no notification is lost; users are never spammed with duplicates; bulk campaigns must not hurt urgent traffic.
- Constraints: third-party providers (APNs/FCM, an SMS gateway, an email service) are slow, rate-limited and sometimes down.
Back-of-the-envelope estimates
| Quantity | Assumption | Result |
|---|---|---|
| Transactional | 10 M users, ~5 per user per day | ~600/s average, bursts ×5 |
| Campaigns | A blast to 10 M users | 10 M messages that should drain within ~1 hour (~2,800/s) |
| Provider latency | 100 ms (push) to 1 s (SMS) | In-flight sends = rate × latency (thousands at once) |
| Storage | ~1 KB per notification record, 90 days | ~5 TB, mostly cold |
High-level design
- Notification API validates the request, checks preferences and quiet hours, stores a notification record and enqueues one message per channel — then returns
202 Accepted. - Queues per priority and channel buffer work (queue-based load leveling).
- Sender workers (competing consumers) render and call providers, with per-provider rate limits and retries.
- Delivery status (sent, delivered, failed, opened) flows back via provider callbacks into the record.
Evolve the design
At t = 10 s marketing launches a 1,600-message campaign while users keep requesting login codes. Watch what each version does to the codes.
Stage 1 of 3
Send inside the API request
- Current architecture
- The API calls the provider and answers when the message is sent.
- Limitation
- API threads are held for the provider's full latency; throughput = threads ÷ latency.
- Bottleneck
- The API thread pool: the campaign fills it, login codes slow past a second and some are rejected.
- Improvement
- Accept quickly and deliver in the background.
- Performance impact
- Most campaign messages fail; login-code p95 > 1 s with failures.
- Complexity & operational cost
- No extra infrastructure.
- Trade-off
- Simple, but every slow provider becomes your outage.
Loading simulation…
Deep dives
- Exactly-once feel: queues deliver at least once, so give each notification an id and make workers idempotent (skip ids already sent); pass the id to providers that support idempotency keys.
- Failures: retry transient provider errors with backoff, move permanent failures to a dead-letter queue, and fail over to a second provider for critical channels.
- Provider limits: apply rate limiting per provider account so workers do not get throttled or blocked.
- User experience: respect opt-outs and quiet hours, collapse duplicates ("5 new messages" instead of 5 pushes), and expire messages that are useless when late (a login code after 5 minutes).
Trade-offs
| Benefits | Costs | |
|---|---|---|
| Asynchrony | API stays fast and providers can fail without losing requests. | Callers learn delivery status later, via callbacks or polling. |
| Priorities | Urgent messages arrive in seconds during any campaign. | More queues and capacity planning per lane. |
| At-least-once | Nothing is silently dropped. | Deduplication is required to avoid double sends. |
Interview checklist
- Split accept from deliver, and say why (slow, flaky providers).
- Prioritise transactional over bulk; mention per-provider rate limits.
- Cover retries, DLQ, idempotency and provider failover.
- Mention preferences, quiet hours, templates and delivery tracking.
Further reading
Related topics
Microservice Patterns
Queue-Based Load Leveling
Put a queue between a bursty producer and a system that cannot scale, so the system works at a steady pace it can sustain while the queue absorbs the peaks.
Microservice Patterns
Priority Queue
Let urgent work overtake routine work in a queue — with priority lanes or separate queues and consumers — so a batch flood never makes interactive users wait, without starving the batch forever.
Microservice Patterns
Competing Consumers
Let several identical workers pull from the same queue so throughput scales with the number of workers — and understand what you give up: ordering, and exactly-once processing.
Microservice Patterns
Dead Letter Queue
Park messages that keep failing in a separate queue after a few attempts, so one poison message cannot block or burn a consumer forever — then fix the cause and redrive them.
Microservice Patterns
Idempotent Consumer
Make processing a message twice have the same effect as processing it once, because every practical message broker will, sooner or later, deliver the same message more than once.
Microservice Patterns
Rate Limiting
Cap how many requests a client may make per unit of time to protect services from bursts and abuse — and pick the algorithm whose behaviour at the edges you can live with.