Back to Notes

Design a Notification System

1. Problem & Real-World Context

Design a system that sends push, email, and SMS notifications triggered by backend events (order placed, message received, price drop) to potentially millions of users. The class-design version of channel modeling is covered in [[Notification System (OOD)]] — this page is the distributed-systems version: durability, fan-out at scale, retry semantics, and the specific trade-off between "must arrive" (a 2FA code) and "best effort, can be dropped" (a marketing blast).

2. Requirements

Functional: support push (mobile), email, SMS channels; trigger notifications from backend events; respect user preferences (opt-out per channel, quiet hours, frequency caps).

Non-functional: high availability — a notification must not be silently lost (at-least-once delivery, idempotent consumers); low latency for push/critical notifications, best-effort/batchable for email/marketing; must not overwhelm a user with duplicate or excessive notifications.

3. Capacity Estimation

  • At, say, 500M users each triggering an average of 5 notification-worthy events/day → ~2.9M events/sec at peak is an overstatement for illustration — more realistically, average load is ~29K events/sec (2.5B events/day ÷ 86,400s), with peaks 3–5x during high-activity windows (e.g., a flash sale). The exact number matters less than the exercise: this is squarely a message-queue-scale problem, not something a request-response API can absorb directly.
  • Fan-out multiplier: one event can trigger multiple channel sends (a message received → push AND an email digest if the user's been offline). Budget for a 2-3x multiplier from event count to actual send count.
  • Template rendering + personalization adds CPU cost per notification (name substitution, localization) — this is why the notification workers are typically a separately scaled tier from the event ingestion tier, not the same process.

4. API Design

Internal, not user-facing:

POST /internal/v1/notify
{
  "event_type": "order_placed",
  "user_id": "u_123",
  "template_id": "order_confirmation",
  "params": {"order_id": "o_456", "amount": 49.99},
  "priority": "high"          # high | normal | low — governs queue routing, see §7
}

A dedicated notification ID, generated at ingestion (not left to the client), is what makes downstream retries idempotent (§8) — every notification, however triggered, gets exactly one ID that follows it through the whole pipeline.

5. High-Level Architecture

flowchart LR
    ES[Event Source /<br/>Backend Services] --> MQ[Message Queue<br/>Kafka]
    MQ --> NW[Notification Workers]
    NW --> PREF[(User Preferences DB)]
    NW --> TMPL[Template Service<br/>localization/personalization]
    NW -->|allowed + rendered| PUSH[Push: FCM/APNs]
    NW -->|allowed + rendered| EMAIL[Email: SendGrid]
    NW -->|allowed + rendered| SMS[SMS: Twilio]
    NW -->|failed N times| DLQ[Dead Letter Queue]

Walking an event through: a backend service publishes an event to Kafka → notification workers (a consumer group, scaled independently of event producers) pick it up → look up the user's channel preferences and quiet-hours setting → render the message via the template service (localization + personalization) → dispatch to the appropriate third-party channel provider(s) → on failure, retry with backoff, eventually routing to a dead letter queue after N attempts.

6. Key Components

  • Event producers — any backend service publishing "something happened" events; decoupled from notification logic entirely (see [[Message Queues & Kafka]]'s async-decoupling rationale).
  • Message queue (Kafka) — durability (an event isn't lost if workers are temporarily down) and fan-out (multiple consumer groups can each independently process the same event stream — e.g., a notification consumer group and an analytics consumer group both reading the same topic).
  • Notification workers — the actual business logic: resolve preferences, render templates, dispatch, retry.
  • User Preferences DB — per-user opt-outs, channel preference order, quiet-hours window; a simple key-value or document store keyed by user_id (see [[SQL vs NoSQL]]) since the access pattern is a point lookup, not relational querying.
  • Template Service — separates what triggered a notification from how it's worded, enabling localization and personalization without touching event-producing code.
  • Channel providers (third-party) — FCM/APNs for push, SendGrid for email, Twilio for SMS — the system explicitly does not reimplement delivery infrastructure, only orchestration around it.

7. Priority Tiers & Quiet Hours

Not all notifications are equal — a 2FA code and a "someone liked your photo" push have very different urgency and failure tolerance.

from enum import Enum

class Priority(Enum):
    HIGH = "high"      # 2FA, security alerts, payment confirmations — near-real-time, own queue/partition
    NORMAL = "normal"   # social notifications, order updates
    LOW = "low"         # marketing, digests — batchable, can be delayed or dropped under load

def should_send_now(user_prefs: dict, priority: Priority) -> bool:
    if priority == Priority.HIGH:
        return True                                # quiet hours never suppress high-priority
    if user_prefs.get("quiet_hours_enabled") and is_within_quiet_hours(user_prefs):
        return False                                # queue for after quiet hours instead
    return True

Routing by priority, not just processing order: high-priority notifications should go through a separate Kafka topic/partition (or a higher-priority consumer pool) so that a marketing-blast backlog can never delay a security alert — mixing all priorities into one queue means a burst of low-priority sends can starve high-priority ones behind it, which is the single most common design flaw in a naive version of this system.

8. Idempotency & Retry

At-least-once delivery from the queue means a notification worker can process the same event more than once (consumer crash after processing but before committing its offset — see [[Message Queues & Kafka]]). Sending a duplicate push notification is annoying; sending a duplicate SMS costs real money and looks broken.

def process_notification_event(event: dict) -> None:
    notification_id = event["notification_id"]

    if redis.sismember("sent_notifications", notification_id):
        return   # already dispatched — safe no-op on redelivery

    prefs = get_user_preferences(event["user_id"])
    if not is_channel_enabled(prefs, event) or not should_send_now(prefs, event["priority"]):
        return

    rendered = template_service.render(event["template_id"], event["params"])
    success = dispatch_to_channel(event["user_id"], rendered, prefs.preferred_channel)

    if success:
        redis.sadd("sent_notifications", notification_id)   # mark AFTER confirmed dispatch, not before
    else:
        raise RetryableError()   # triggers the retry/backoff path below

Why mark as sent only after a confirmed dispatch, not before: marking it sent before attempting delivery would mean a crash between the mark and the actual send permanently loses the notification (it'll never be retried, since it now looks "already sent") — the ordering here is deliberate, trading a small window where a true duplicate could theoretically occur (dispatch succeeds, but the process crashes before the sadd completes) for the much worse failure mode of silent loss. This is the same at-least-once-plus-idempotent-consumer trade-off as [[Message Queues & Kafka]]'s general delivery-guarantees discussion.

Retry with exponential backoff + dead letter queue:

import time

def dispatch_with_retry(user_id: str, message: dict, max_attempts: int = 5) -> bool:
    for attempt in range(max_attempts):
        try:
            return dispatch_to_channel(user_id, message)
        except TransientError:
            time.sleep(min(2 ** attempt, 30))   # exponential backoff, capped
    send_to_dead_letter_queue(user_id, message)  # exhausted retries — ops can inspect/replay
    return False

A poison-pill notification (malformed template, permanently invalid phone number) that fails every retry must not block the queue behind it — routing to a DLQ after max_attempts isolates the failure (see [[Message Queues & Kafka]]'s DLQ pattern) rather than retrying forever or crashing the worker.

9. Rate Limiting / Frequency Capping

A user who takes 50 actions in a minute (liking 50 photos) shouldn't trigger 50 separate push notifications to their followers. Two complementary mechanisms:

  • Per-user frequency cap — a [[Rate Limiter (OOD)]]-style token bucket per (user_id, channel), capping total notifications/hour regardless of how many distinct events fired.
  • Batching/digesting — for low-priority notifications specifically, accumulate events over a window (e.g., 15 minutes) and send one combined digest ("5 people liked your post") instead of 5 separate pushes — trading immediacy for a much better user experience at high event volume, and directly reducing the send-count multiplier from §3.

10. Deep Dives / Follow-ups

  • Multi-channel fallback: if push fails to deliver (device unreachable) within a timeout, fall back to email for the same logical notification — requires tracking delivery confirmation per channel attempt, not just per notification.
  • Delivery tracking / read receipts: a separate lightweight event stream (delivered, opened) feeding back into analytics — decoupled from the send path itself via the same Observer-style decoupling as the [[Notification System (OOD)]] page's design.
  • A/B testing notification copy: the Template Service resolving template_id to variant text based on an experiment assignment — orthogonal to the delivery pipeline, changes nothing about queuing/retry/idempotency.

11. Staff-Level Lens

The naive version of this system treats every notification identically — one queue, one priority, retry-forever-or-crash. That design works in a demo and fails specifically the first time a marketing campaign and a security-alert burst land in the same minute: the marketing blast (huge volume, low individual urgency) backs up the shared queue, and the security alerts queue behind it — a real, embarrassing production incident pattern (a delayed fraud alert because the queue was full of "50% off" emails). Naming the priority-isolation requirement unprompted, before being asked "what if there's a huge marketing send," is the single highest-value thing to say in this interview — directly analogous to naming the celebrity fan-out problem unprompted in the News Feed interview.

12. Interview Drill Questions

  1. Why not just use one Kafka topic for all notification priorities? — A backlog of low-priority sends (a large marketing campaign) would delay processing of high-priority ones (security alerts, 2FA codes) sitting behind them in the same queue — separate topics/partitions per priority tier let a dedicated high-priority consumer pool process urgent notifications independently of backlog on the low-priority tier.
  2. How do you avoid double-charging a user's phone bill if an SMS notification is processed twice due to at-least-once delivery? — Idempotency via a notification_id checked against a "sent" set (Redis SISMEMBER) before dispatch, marked only after confirmed delivery — the same idempotent-consumer pattern as [[Message Queues & Kafka]]'s general delivery-guarantees discussion, applied specifically to a channel where duplicates have real monetary cost.
  3. A user opts out of push notifications entirely. Where in the pipeline is that enforced, and why there specifically? — In the notification worker, immediately after fetching user preferences and before rendering/dispatching — enforcing it there (rather than, say, at the event-producer level) means producers never need to know about per-user preferences at all, keeping the event-source side of the system completely decoupled from notification policy.
  4. How would you implement "quiet hours" without simply dropping notifications that arrive during them? — Queue them (don't discard) and redeliver at the end of the quiet-hours window for any notification that isn't high-priority — high-priority notifications bypass quiet hours entirely, as shown in should_send_now.
  5. Design digesting: how do you combine multiple low-priority events into one notification? — Accumulate events per (user_id, notification_category) in a time-windowed buffer (Redis list or a scheduled aggregation job); at window expiry, render one combined template summarizing the count/content instead of dispatching one notification per individual event — directly reduces both send volume (§3's fan-out multiplier) and user-facing notification fatigue.

Cross-links

[[Notification System (OOD)]] · [[Message Queues & Kafka]] · [[Rate Limiter (OOD)]] · [[Caching & Redis]] · [[SQL vs NoSQL]] · [[News Feed - Twitter Feed]]