GenAI Question Bank

Short questions that come up between or inside design rounds. Answer aloud before reading the answer.

1. RAG or fine-tuning: how do you choose? RAG for knowledge that changes, must be cited, or is permission-scoped. Fine-tuning for behaviour: format, tone, domain style, or a narrow task where a small tuned model replaces a big general one. Combine: fine-tune the format, retrieve the facts.

2. Why is LLM decode slow, and what speeds it up? Each token reads all weights from GPU memory → bandwidth-bound, not compute. Batching (share each read), quantization (fewer bytes), speculative decoding (verify several tokens per pass), smaller models.

3. What limits concurrent users per GPU? Usually KV-cache memory; each active sequence holds cache proportional to its length. PagedAttention, GQA models, cache quantization, shorter contexts.

4. How do you reduce hallucinations in production? Ground in retrieved sources, instruct refusal when context is missing, require + verify citations, lower temperature for factual tasks, schema-constrained output, measure faithfulness on an eval set. (Training-side fix: [[Learning/AI & ML/Karpathy - Deep Dive into LLMs|Karpathy notes]].)

5. How would you evaluate an LLM feature before launch? Golden set from real/realistic inputs → rubric → LLM-as-judge + human review of a sample to calibrate → per-category results, not just average → online A/B with product + guardrail metrics (safety flags, latency, cost).

6. What is prompt injection; how do you defend? Instructions hidden in untrusted input (user text, web pages, docs, tool output) overriding system instructions. Defence in depth: least-privilege tools, authorization in code not the model, human confirmation for risky actions, separate instructions from data, output filtering. No prompt alone is a full defence.

7. HNSW vs IVF-PQ? HNSW: graph, high recall, low latency, full vectors in memory, slower build. IVF-PQ: clusters + compression, far less memory, some recall loss, good for 100M–billions. Common: IVF-PQ candidates → exact-distance rerank.

8. How do you choose chunk size? Tuned parameter. Start 300–800 tokens on structural boundaries with overlap; measure recall@k for a few sizes. Small = precise but loses context; large = context but diluted relevance + more tokens.

9. Hosted API or self-hosted open model? Hosted: fastest to ship, strongest models, no GPU ops, pay per token. Self-hosted: data control, predictable cost at high steady volume, deep fine-tuning, custom serving. Path: start hosted, move high-volume narrow tasks to self-hosted distilled models.

10. TTFT vs TPOT — what affects each? TTFT = queueing + prefill: prompt length, model size, prefix-cache hits. TPOT = decode: model size, batch size, memory bandwidth, quantization. Long prompts hurt TTFT; big batches raise throughput but can hurt TPOT.

11. How do you make agents reliable? Workflows where path is known. Narrow tools, clear schemas. Cap steps/time/cost. Validate tool args. State outside the prompt. Full traces. Sandbox testing with simulated users + scenario evals.

12. How do you roll out a new model version safely? Golden set vs production → shadow traffic (run both, show old) → canary with auto-rollback on guardrail metrics. Pin model versions; version prompts with models.

13. What is LoRA and why useful in serving? Small low-rank matrices added to frozen weights. Adapters are MBs not GBs → one base model in GPU memory serves hundreds of customer adapters, swapped per request.

14. How would you cut LLM cost 5×? Measure where tokens go first. Then: route to smaller models, cache (exact/semantic/prefix), shorten prompts + history, cap output length, batch offline jobs at lower priority / batch-API pricing, distill a small model for the highest-volume task.

Related: [[GenAI Vocabulary]] · [[index|GenAI index]]