Message Queues & Kafka

Message queues decouple producers from consumers, enabling async processing and absorbing traffic spikes. Kafka is the dominant distributed event-streaming platform — it shows up in notification, feed, ride-sharing, and most distributed system designs.

Why Message Queues?

Problems they solve:

• Async decoupling — producer doesn't wait for the consumer to process.
• Load leveling — absorbs traffic spikes; producer writes fast, consumer processes at its own pace.
• Reliability — message persists in the queue even if a consumer crashes.
• Fan-out — one message reaches multiple independent consumers.
• Ordering — process events in sequence (per-key, see below).

Without a queue: payment API directly calls the email service; email service down → payment fails. With a queue: payment API → queue → email service; payment succeeds regardless of email service health.

Queue Types

Point-to-Point (SQS standard): one producer, one consumer group; message consumed and deleted; at-least-once delivery (can deliver twice on failure).

Publish-Subscribe (Kafka, SNS): one producer, multiple independent consumer groups, each reading all messages independently; messages retained for the configured retention window even after consumption.

Producer → [Topic] → Consumer Group A (reads all messages)
                   → Consumer Group B (reads all messages)
                   → Consumer Group C (reads all messages)

Kafka Architecture

Producers → Topics → Partitions → Consumer Groups

Partitioning for scale:

Topic "orders" with 3 partitions, consumer group with 3 consumers → each gets 1 partition
Consumer group with 6 consumers → 3 sit idle (partitions < consumers = wasted capacity)

Rule: max throughput = number of partitions. To scale consumer throughput, add partitions.

Message ordering: guaranteed within a partition, not across partitions. Solution: key-based partitioning — the same key always lands on the same partition.

# Same user_id always goes to the same partition → that user's events are ordered
producer.send("orders", key=str(user_id).encode(), value=order_json)

Delivery Guarantees

In practice: at-least-once + idempotent consumer is the standard pragmatic combination.

def process_order(order_id, data):
    if redis.sismember("processed_orders", order_id):
        return  # already processed — duplicate delivery, no-op
    # ... process ...
    redis.sadd("processed_orders", order_id)

Kafka vs SQS vs RabbitMQ

Common Patterns

Fan-out (notification systems):

User posts → "events" topic →
    [email-service group]   → sends email
    [push-service group]    → sends push notification
    [feed-service group]    → updates follower feeds
    [analytics group]       → logs event

One event touches 4 services, all decoupled — see the fanout section of the Notification System problem page.

Dead Letter Queue (DLQ): after N failed processing attempts, move a message to a DLQ instead of blocking the queue behind it (prevents one poison-pill message from stalling everything downstream); ops can inspect/replay DLQ messages.

Event Sourcing: store every state change as an event; current state = replay all events from the log. Full audit trail, and state can be rebuilt at any point in time — Kafka's immutable, replayable log is a natural fit.

CQRS (Command Query Responsibility Segregation): write path produces events → Kafka → read models updated asynchronously; queries hit the read model (optimized for reads), never the write DB directly.

Back-Pressure

When consumers are slower than producers, the queue fills up. Options, roughly in order of preference: scale consumers horizontally; add more partitions (increases consumer parallelism, requires re-keying); rate-limit producers; drop messages (acceptable for metrics/analytics, never for orders/payments).

Redis Pub/Sub vs Kafka

Use Redis Pub/Sub for ephemeral signals where losing a message occasionally is fine. Use Kafka for anything that needs reliability or replay.

Interview Drills

1. "Why Kafka over a simple DB queue?" — Kafka persists messages in an ordered, replayable log; DB-based queues require polling, don't scale to high throughput, and offer no clean replay story. Kafka separates storage from compute — consumers can rewind to any offset.
2. "How do you ensure message order?" — Partition by the entity key (user_id, order_id); all events for that entity land in the same partition → same consumer → ordered.
3. "How would you handle a slow consumer?" — Scale the consumer group up to the partition count; if still slow, add partitions (requires re-keying/rebalancing). Also consider batching or async processing within each consumer.
4. "Difference between a queue and a topic?" — Queue: message consumed and removed, one consumer group. Topic: message retained, multiple consumer groups each read independently.

Cross-links

[[Caching & Redis]] · [[Database Replication & Sharding]] · [[CAP Theorem & Consistency Models]] · [[API Gateway]]