Design WhatsApp

Scale: 2B users, 100B messages/day, 65B messages/day (pre-2022).

---

Requirements Clarification

Functional:

• 1-1 messaging
• Group messaging (up to 1024 members)
• Message delivery status (sent ✓, delivered ✓✓, read ✓✓ blue)
• Online presence (last seen)
• Media sharing (images, videos, documents)

Non-functional:

• Messages must not be lost (at-least-once delivery)
• Ordering guaranteed within conversation
• End-to-end encryption
• Low latency (< 100ms for delivery when both online)

---

Core Challenge: Message Delivery

When Alice sends to Bob:

1. Bob online → deliver immediately via WebSocket
2. Bob offline → store message, deliver when Bob reconnects

---

High-Level Architecture

[Alice's Phone]
    │  WebSocket (persistent connection)
    ▼
[Connection Service / Chat Server]
    │
    ├── Bob online on same server → direct delivery
    │
    ├── Bob online on different server:
    │   → Pub/Sub (Redis) or service mesh → route to Bob's server
    │
    └── Bob offline:
        → Message Queue → Message Store (Cassandra)
        → Push notification (FCM/APNs) to wake Bob's device
        → When Bob reconnects → fetch undelivered messages

---

Connection Management

Each chat server maintains WebSocket connections.
User-to-server mapping stored in Redis:
  user:{user_id}:server → server_id

On connect: register in Redis, subscribe to user's channel
On disconnect: update last_seen, unregister

For cross-server delivery:
  Publish to Redis channel: msg:{user_id}
  Bob's server subscribes → receives → delivers via WebSocket

---

Message Flow (detailed)

1. Alice sends message
   → WebSocket to Chat Server A
   → Server A assigns message_id (Snowflake)
   → Sends ACK to Alice (message_id) ← client marks as "sent ✓"

2. Server A checks Bob's server
   → Bob on Server B: publish to Redis channel msg:{bob_id}
   → Server B delivers to Bob via WebSocket
   → Bob's client sends delivery receipt to Server B
   → Server B publishes receipt to Alice's channel
   → Server A delivers receipt to Alice ← client marks as "delivered ✓✓"

3. Bob opens conversation
   → Bob's client sends read receipt
   → Alice's client marks as "read ✓✓ blue"

---

Message Storage

Cassandra schema (write-heavy, time-series):
  messages table:
    partition key: conversation_id
    clustering key: message_id (snowflake — time-sortable)
    columns: sender_id, content, type, status, created_at

Why Cassandra:
  - High write throughput (100B messages/day)
  - Time-series access pattern (fetch last N messages)
  - Horizontal scale

Retention:
  WhatsApp: messages stored on device, NOT permanently on server
  Server stores only until delivered (then deletes)
  → Reduces storage, privacy-friendly

---

Group Messaging

Group has up to 1024 members.
Two approaches:

Fan-out on write (WhatsApp's approach for small groups):
  Message → Fan-out service → message stored per recipient
  Each member has own message queue

For very large groups (1000+ members):
  Store one copy, reference per member
  Trade-off: storage efficiency vs read complexity

---

Presence (Online Status / Last Seen)

Heartbeat approach:
  Client sends heartbeat every 5s while active
  Server updates Redis: user:{user_id}:last_seen = timestamp

On query:
  if last_seen > now - 10s → "online"
  else → "last seen {time}"

Privacy setting: user can hide last_seen

---

Media Sharing

Large files NOT sent through chat server (would overwhelm it).

Flow:
1. Client uploads media to S3 directly (presigned URL)
2. Get back CDN URL for the file
3. Send message with media_url (not the file itself)
4. Recipient downloads from CDN

End-to-end encrypted: client encrypts file before upload,
includes decryption key in message (encrypted for recipient).

---

Key Design Decisions

---

Message Ordering

Within a conversation: Snowflake IDs guarantee ordering
  (time-based, generated by server, not client)

Clock skew between servers:
  Use Hybrid Logical Clocks (HLC) or central Snowflake service
  → Monotonically increasing even across servers

---

Failure Modes

• Chat server crash → client reconnects to another server, fetches undelivered from Cassandra
• Message not ACKed → client retries with same message_id (idempotent)
• Redis pub/sub fails → fallback: recipient polls for messages on reconnect

---

Related

• [[Message Queues & Kafka]] — message queuing concepts
• [[Caching & Redis]] — presence tracking, cross-server routing
• [[System Design/Problem Designs/Notification System]] — push notifications