AWS SQS & SNS

SQS = managed message queue (point-to-point). SNS = managed pub/sub (fan-out). Used together for reliable async decoupling.

---

SQS — Simple Queue Service

Queue Types

Key Concepts

Visibility Timeout (default 30s):
After receive, message hidden from other consumers.
If not deleted within timeout → reappears (assumes processing failed).
Set to: max expected processing time × 1.5

Message Retention: 1 min–14 days (default 4 days)
Message Size: up to 256KB (use S3 + SQS for larger)
Delay Queue: delay delivery by 0–15 minutes
Long Polling: wait up to 20s for message (reduces empty receives, saves cost)

Dead Letter Queue (DLQ)

Main Queue → [maxReceiveCount failures] → DLQ

Configure maxReceiveCount (e.g., 5). After 5 failed processing attempts, message moves to DLQ. Ops team inspects / replays DLQ.

Lambda Integration

SQS Queue → Lambda trigger → Lambda fn processes batch

• Lambda polls SQS automatically (no manual polling needed)
• batch_size: 1–10,000 messages per invocation
• On failure: entire batch re-processed (use idempotency)
• DLQ for unrecoverable messages

---

SNS — Simple Notification Service

Push-based pub/sub. Topic-based fan-out. Supports multiple subscriber types.

Subscribers

SNS → SQS Fan-out Pattern

SNS Topic
├── SQS Queue A (email service)
├── SQS Queue B (push notification service)
├── SQS Queue C (analytics pipeline)
└── Lambda (real-time processing)

Why queue between SNS and Lambda? SQS buffers messages — Lambda can fall behind without losing messages. SNS → Lambda directly = messages lost if Lambda throttled.

---

SQS + SNS Together

# Example: e-commerce order placed
# 1. SNS topic: "order-created"
# 2. Subscribers: inventory-queue, shipping-queue, analytics-queue, email-queue
# 3. Each queue feeds its own Lambda/service

import boto3

sns = boto3.client('sns')
sqs = boto3.client('sqs')

# Publish to topic
sns.publish(
    TopicArn='arn:aws:sns:us-east-1:123456789:order-created',
    Message=json.dumps({'order_id': '42', 'total': 99.99}),
    MessageAttributes={
        'event_type': {
            'DataType': 'String',
            'StringValue': 'order.created'
        }
    }
)

# Receive from queue (for manual consumer)
response = sqs.receive_message(
    QueueUrl='https://sqs.us-east-1.amazonaws.com/123/inventory-queue',
    MaxNumberOfMessages=10,
    WaitTimeSeconds=20,         # long polling
    VisibilityTimeout=60
)

for msg in response.get('Messages', []):
    process(json.loads(msg['Body']))
    sqs.delete_message(
        QueueUrl=queue_url,
        ReceiptHandle=msg['ReceiptHandle']  # delete after successful processing
    )

---

Message Filtering (SNS)

SNS subscription filter policies — subscribers only receive relevant messages:

// Subscription filter for "order.created" events only
{
    "event_type": ["order.created"],
    "total": [{"numeric": [">=", 50]}]
}

Reduces downstream processing — queues only get messages they care about.

---

SQS vs Kafka

MSK (Managed Streaming for Kafka): AWS-managed Kafka. Use when you need Kafka but want managed infra.

---

Common Patterns

Async Decoupling

API Gateway → SQS → Lambda → DB

API response in < 100ms. Lambda processes in background.

Fan-out

S3 event → SNS → SQS (resize) + SQS (metadata) + Lambda (scan)

S3 upload triggers multiple downstream processes.

FIFO for Payment Processing

Payment API → FIFO Queue (deduplication enabled) → Payment Processor Lambda

Exactly-once, ordered processing of payment events.

Retry with Exponential Backoff

Queue → Lambda (fails) → Visibility timeout expires → Retry
→ After maxReceiveCount → DLQ

---

Exam Notes (MLA-C01)

• SNS + SQS fan-out is standard AWS event-driven architecture
• Lambda can poll SQS (event source mapping) — no need to write polling code
• FIFO queues for ordered, exactly-once processing
• DLQ for fault tolerance and debugging
• SNS filter policies reduce unnecessary downstream invocations

---

Related

• [[AWS/Lambda]] — Lambda triggered by SQS
• [[Message Queues & Kafka]] — concepts + Kafka comparison
• [[System Design/Problem Designs/Notification System]] — fan-out pattern
• [[AWS/AWS Topics]] — full AWS index