GenAI Vocabulary for Interviews
Know a one-line definition + one trade-off for each. Deeper theory: [[Learning/AI & ML/LLM Fundamentals]], [[Learning/AI & ML/Karpathy - Deep Dive into LLMs]].
Model fundamentals
| Term | Definition / trade-off |
|---|
| Token | Unit a model reads/writes (subword, BPE). ~0.75 English words per token. Pricing, context limits, latency all counted in tokens. |
| Transformer / attention | Each token attends to earlier tokens. Attention cost grows with the square of sequence length → long context is expensive. |
| Context window | Max input + output tokens per call. Bigger helps, but facts in the middle of very long prompts get missed ("lost in the middle"). |
| Pretraining → SFT → RLHF/DPO | Next-token prediction on web text → supervised tuning on instruction examples → preference tuning. DPO trains on preference pairs directly, no separate reward model. |
| Embedding | Fixed-length vector of meaning; similar texts are close under cosine similarity. Core of semantic search and RAG. |
| Temperature, top-p | Temperature 0 picks most likely token (near-deterministic). Top-p samples from smallest set with cumulative probability p. |
| Hallucination | Fluent output not supported by facts/context. Mitigate: RAG grounding, citations, constrained output, verification. |
| Mixture of Experts (MoE) | Only a few expert sub-networks run per token → compute per token ≪ total params. Memory still holds all experts. |
| Multimodal | Image/audio/video encoded into the same token space as text. |
Inference and serving
| Term | Definition / trade-off |
|---|
| Prefill vs decode | Prefill processes the whole prompt in parallel, compute-bound → sets TTFT. Decode generates one token at a time, memory-bandwidth-bound → sets TPOT. |
| KV cache | Stored attention keys/values for past tokens so decode doesn't recompute. Often the main GPU memory consumer at high concurrency. |
| PagedAttention | vLLM: KV cache in fixed-size blocks like OS pages → no fragmentation, many more concurrent sequences. |
| Continuous batching | New requests join the running batch every decode step. Big throughput gain over static batching. |
| Prefix / prompt caching | Reuse KV cache for a shared prefix (system prompt, long doc). Cuts TTFT and cost. |
| Quantization | Weights in int8/fp8/4-bit instead of fp16 → ½–¼ memory, faster decode, small quality cost (measure it). |
| Speculative decoding | Small draft model proposes tokens; big model verifies several in one pass. Same output distribution, lower latency. |
| Tensor / pipeline parallelism | Split one model across GPUs: tensor = split each layer (needs NVLink); pipeline = split layers into stages. |
| Distillation | Train small student to imitate big teacher. Path to cheap, fast narrow models. |
| Model routing / cascade | Easy requests → small model, hard → big; or try small first and escalate on low confidence. |
| Structured output | Constrain decoding to a JSON schema/grammar so downstream code always parses. |
Retrieval (RAG)
| Term | Definition / trade-off |
|---|
| RAG | Retrieve relevant docs at query time and put them in the prompt. Fresh, private, citable knowledge without retraining. |
| Chunking | Split docs into retrievable pieces. Too small loses context; too large dilutes relevance and wastes tokens. Structure-aware > fixed size. |
| Vector DB / ANN | Approximate nearest neighbour. HNSW: graph, high recall, memory-heavy. IVF-PQ: clustered + compressed, cheaper, lower recall. DiskANN: SSD-resident, billion scale. |
| Vector quantization | Store embeddings in fewer bits: int8 (4×), binary (32×), PQ (16–64×). Oversample + rescore to recover recall. Deep dive: [[Vector Quantization]]. |
| Hybrid search | BM25 keywords + dense vectors. Keywords catch exact IDs, error codes, acronyms. Merge with Reciprocal Rank Fusion (RRF). |
| Reranker | Cross-encoder scores (query, passage) jointly. More accurate, slow → only on top 50–100. |
| Query rewriting | LLM turns vague/follow-up question into standalone query, or multiple queries (multi-query, HyDE). |
| GraphRAG | Entity-relationship graph from docs for cross-document / global questions. |
Adapting models
| Term | Definition / trade-off |
|---|
| Prompt engineering | System prompt, few-shot, chain-of-thought. Cheapest lever, always first. |
| Fine-tuning | Good for style, format, narrow skills. Bad for frequently changing facts (use RAG). |
| LoRA / PEFT | Train small low-rank adapters, not all weights. Cheap; one base model can serve many adapters (multi-LoRA). |
| Synthetic data | Strong model generates training/eval data. Filter + human spot-check. |
Agents and tools
| Term | Definition / trade-off |
|---|
| Tool / function calling | Model emits structured call (name + JSON args); your code executes, result goes back into context. |
| Agent loop | think → call tool → observe, until done. Needs step limits, timeouts, cost caps. |
| Workflow vs agent | Workflow = code-defined path with LLM steps; agent = model picks the path. Prefer workflows when path is known. |
| MCP | Model Context Protocol: open standard for exposing tools/data to LLM apps. See [[Learning/AI & ML/MCP]]. |
| System One model (Jev) | TypeSafe AI's Jev (Sept 2026): non-autoregressive model that returns only typed decisions (choice/score/yes-no) with calibrated probabilities in 70–500 ms. Use for routing, classification, tool-call risk checks; can't generate text. See [[Learning/AI & ML/LLM Frontier Learning Path]]. |
| Memory | Short-term = conversation in context; long-term = external store retrieved when relevant. |
Safety and evaluation
| Term | Definition / trade-off |
|---|
| Guardrails | Input/output checks: moderation, PII detection/redaction, topic restrictions, schema validation. |
| Prompt injection | Malicious instructions in user input or retrieved/tool content. Treat as untrusted data; least-privilege tools; confirm risky actions. |
| LLM-as-judge | Strong model grades against a rubric. Biases: position, length, self-preference → calibrate on human labels. |
| Golden set | Curated, versioned inputs + expected outputs/rubrics; run on every prompt/model change like regression tests. |
| RAG metrics | Retrieval: recall@k, MRR, nDCG. Generation: faithfulness, answer relevance, citation accuracy. |
| Online metrics | Thumbs, regenerate rate, acceptance rate, task completion, escalation rate, cost + latency per request. |
Related: [[GenAI SD Framework]] · [[GenAI Capacity Math]] · [[index|GenAI index]]