Back to Notes

GenAI SD Framework

GenAI System Design — Interview Framework

Same skeleton as a classic SD round ([[SD Interview Framework & Pitfalls]]), with three GenAI additions: model choice, context construction, evaluation.

The 7 steps (45-min budget)

  1. Clarify product and scope (5 min). Who uses it, input/output modalities, single- or multi-turn, does it take actions, what's out of scope. Ask hosted API vs self-hosted open weights; this changes half the design.
  2. Non-functional requirements (3 min). Latency split into TTFT (time to first token) and TPOT (time per output token). Throughput, availability, cost ceiling per request, safety/compliance (PII, data residency), knowledge freshness.
  3. Estimate (4 min). QPS, input/output tokens per request, tokens/s, GPU count or API $/day, embedding and log storage. See [[GenAI Capacity Math]].
  4. High-level architecture (10 min). Request path end to end: client → gateway → orchestration service → context builder (retrieval, history, tools) → safety filters → model serving → stream back. Then the offline path: ingestion, logging, feedback, evaluation.
  5. Model and context strategy (8 min). Which model size and why. Prompting vs RAG vs fine-tuning. How the context window gets filled and what gets dropped on overflow.
  6. Deep dives (10 min). The two hardest parts, usually serving efficiency (batching, KV cache, routing) and quality (retrieval, hallucination). Failure modes and how you degrade.
  7. Evaluation, safety, iteration (5 min). Offline golden set, LLM-as-judge calibrated on human labels, online metrics + A/B, guardrails, feedback loop into the next prompt/model version.

Senior signal

Juniors draw boxes. Seniors name the trade-off each box makes, say what they'd measure, and separate v1 from v2.

  • v1 (ships in weeks): hosted API + RAG + prompt engineering + basic guardrails.
  • v2 (wins at scale): self-hosted, distilled/quantized models, routing, caching, fine-tuned components.

Latency targets to quote

ProductTarget
ChatTTFT < 1 s, TPOT 20–50 ms (faster than reading)
Autocomplete< 100–300 ms end to end
Agents, image/video genseconds–minutes → async + progress updates

Checklist of "GenAI-specific" boxes to remember

  • Context builder (system prompt, history truncation/summary, retrieval, memory)
  • Input + output moderation (streaming output filter)
  • Model router / cascade
  • Inference servers (vLLM / TensorRT-LLM, continuous batching, paged KV cache, prefix cache)
  • Token-based rate limiting
  • Cache (exact, semantic, prefix)
  • Eval pipeline (golden set, LLM-as-judge, A/B) + feedback logging

Related: [[GenAI Vocabulary]] · [[index|GenAI index]]