Problem 01 — Design a ChatGPT-style Assistant

streaming · context management · GPU serving · safety

Requirements

Functional: multi-turn text chat, streamed responses, history across devices, regenerate/edit, thumbs feedback. Optional: file upload, web search, memory. Non-functional: 10M DAU, TTFT p95 < 1 s, TPOT < 50 ms, 99.9% availability, no conversation loss, moderation on input and output.

Estimates

10M DAU × 10 messages = 100M messages/day → ~1,200 QPS avg, ~3,500 peak
Per message: ~1,500 input tokens (history + system), ~400 output tokens
Output: 100M × 400 = 40B tokens/day → ~460k output tokens/s average
Assume one 8×H100 replica of a 70B model sustains ~2,500 output tok/s
→ ~185 replicas average, ~550 at peak → ~4,400 GPUs at peak for a single model size
History storage: 100M msgs × 2 KB ≈ 200 GB/day of text

That GPU bill is why routing and caching dominate the deep dive.

Architecture

flowchart LR
  C[Client] -->|SSE| G[API Gateway]
  G --> CS[Chat Service]
  CS --> H[(Conversation Store)]
  CS --> CB[Context Builder]
  CB --> M[(Memory / Retrieval)]
  CB --> IF[Input Moderation]
  IF --> R[Model Router]
  R --> S1[Small model pool]
  R --> S2[Large model pool]
  S1 --> OF[Streaming Output Filter]
  S2 --> OF
  OF --> CS
  CS --> L[[Event Log]]
  L --> E[Eval and Training Data]

Walkthrough

1. Gateway: auth, per-user limits counted in tokens (not requests), holds the SSE connection.
2. Chat service: writes user message, loads recent history, calls context builder.
3. Context builder: system prompt + memory + history. Over budget → keep last N turns verbatim, replace older turns with a rolling summary.
4. Input moderation: fast classifier, run in parallel with routing to save latency.
5. Router: picks model by plan tier + cheap difficulty classifier.
6. Serving: vLLM / TensorRT-LLM with continuous batching, paged KV cache, prefix caching for the shared system prompt.
7. Output filter: checks the stream in small windows, can cut mid-stream.
8. Persist final assistant message; log event for analytics/eval.

Deep dives

Streaming transport

On disconnect: keep generating server-side, persist; client fetches by message ID on reconnect.

Conversation storage

Wide-column (Cassandra/DynamoDB): partition user_id, cluster conversation_id, message_ts. Append-only writes, "latest N" reads. Edits create branches → store parent_message_id (conversation is a tree).

GPU capacity and autoscaling

70B model load takes minutes → reactive autoscaling too slow. Warm buffer; scale on queue depth + KV-cache utilization, not CPU. Shed load gracefully: route free tier to the small model before rejecting anyone.

Cost levers (by impact)

1. Route most traffic to a smaller/distilled model.
2. Prefix caching for system prompts and shared docs.
3. Quantize to fp8/int8 after eval-set check.
4. Summarize history instead of resending.
5. Cap max output tokens per tier.

Evaluation

Offline: versioned golden set by category (coding, reasoning, refusals), pairwise LLM-as-judge vs production model, safety red-team set. Online: thumbs ratio, regenerate rate, conversation length, TTFT/TPOT percentiles, cost per conversation. Ship changes behind A/B flags.

Likely follow-ups

• Add long-term memory without leaking across users?
• GPU node dies mid-stream: what does the user see?
• Support 1M-token documents?
• Stop a cached harmful answer from being served again?

Related: [[GenAI Capacity Math]] · [[Design LLM Gateway]] · [[index|GenAI index]]