Karpathy — Deep Dive into LLMs like ChatGPT

Source: Andrej Karpathy, YouTube, ~3h31m (Feb 2025) · Notes built from the full transcript. Goal of video: mental models for what ChatGPT is — how it's built (3 training stages) and its "psychology" (why it hallucinates, can't count, needs tokens to think). Lecture whiteboard: [[Karpathy - LLM Intro.excalidraw]] (the drawing used in the video; details from it are merged below and marked 🖼). Interview use: this is the vocabulary layer for the GenAI SD round → apply it in [[Interview Prep/Content/System Design/GenAI/index|GenAI System Design]]. Theory companion: [[LLM Fundamentals]].

---

TL;DR — the whole video in 10 lines

1. ChatGPT = 3 stages: pretraining (knowledge) → SFT (assistant persona) → RL (thinking/reasoning).
2. Pretraining = predict next token on ~15T tokens of filtered internet → base model = "internet document simulator", a lossy zip of the internet.
3. Text → tokens via BPE (GPT-4: 100,277-token vocab). Model sees token IDs, never characters.
4. SFT = same algorithm, swap dataset to human-written conversations. ChatGPT's answer ≈ simulation of an expert human labeler following labeling instructions.
5. Hallucination = model imitates the confident style of training answers. Fixes: train "I don't know" where model provably doesn't know + tools (web search).
6. Parameters = vague recollection; context window = working memory. Paste the source → better answers.
7. Fixed compute per token → models need tokens to think. Spread reasoning out; use code tools for math/counting.
8. RL = practice problems: sample many solutions, reinforce the ones that reach the right answer. Chain-of-thought emerges (DeepSeek-R1 "aha moments").
9. RLHF = RL in unverifiable domains via a reward model that simulates human rankings. Gameable → run only a few hundred steps. "RLHF is not real RL."
10. Swiss-cheese capabilities: brilliant overall, random holes (9.11 vs 9.9, strawberry Rs). Use as a tool, verify output.

---

The pipeline

flowchart LR
  A[Internet<br/>Common Crawl] --> B[Filter: URL, text extract,<br/>language, dedup, PII]
  B --> C[Tokenize<br/>BPE ~100k vocab]
  C --> D[Pretraining<br/>next-token prediction]
  D --> E[Base model<br/>internet simulator]
  E --> F[SFT on<br/>conversations]
  F --> G[Assistant<br/>e.g. GPT-4o]
  G --> H[RL on verifiable<br/>problems]
  H --> I[Thinking model<br/>e.g. o1, R1]
  G --> J[RLHF<br/>reward model]
  J --> G

---

1. Pretraining — [0:00–1:00:00]

Data [0:00]

• Source: Common Crawl (2.7B pages indexed as of 2024) → filtered datasets like HuggingFace FineWeb.
• 🖼 The FineWeb pipeline (8 steps):

flowchart LR
  A[URL filtering<br/>blocklists] --> B[Text extraction<br/>HTML to text]
  B --> C[Language filtering<br/>keep >65% English]
  C --> D[Gopher filtering<br/>quality heuristics]
  D --> E[MinHash dedup<br/>near-duplicates]
  E --> F[C4 filters]
  F --> G[Custom filters]
  G --> H[PII removal]

• URL blocklists drop malware, spam, marketing, adult sites. Language threshold is a design choice that sets how multilingual the model will be.
• MinHash = locality-sensitive hashing to find near-duplicate pages cheaply (same idea as dedup in a web crawler, see [[Web Crawler]]).
• Source write-up: FineWeb blog post.
• Result: FineWeb ≈ 44 TB ≈ 15 trillion tokens. Not huge — fits on a few hard drives.

Tokenization [0:06]

• NN needs a 1-D sequence from a finite symbol set. Trade-off: vocab size vs sequence length (sequence length is precious compute).
• bits (2 symbols) → bytes (256) → Byte Pair Encoding: repeatedly merge the most frequent adjacent pair into a new symbol.
• 🖼 Compression example: ~5,000 characters ≈ 40,000 bits (vocab 2) ≈ 5,000 bytes (vocab 256) ≈ 1,300 GPT-4 tokens (vocab 100,277). ~4 chars per token.
• GPT-4 (cl100k_base) vocab = 100,277 tokens. "hello world" = 2 tokens. Case- and space-sensitive.
• Tool: tiktokenizer website to see tokens.

Training [0:15]

• Take random windows (up to e.g. 8k tokens) → NN outputs a probability over all 100,277 tokens for the next one → nudge weights so the true next token gets higher probability. Done in parallel over huge batches.
• Loss = single number, lower is better; the thing researchers watch.
• 🖼 Parameters are like knobs on a DJ mixer: training = finding the knob setting whose predictions match the data. The network is just a giant mathematical expression mixing inputs x with weights w (nested multiply/add/exp, e.g. 1/(1+exp(-(w0·x1 + w1·x2 + w2)))…).
• 🖼 See it live: bbycroft.net/llm, a 3D walkthrough of a tiny nano-gpt with 85,584 params (embedding → attention → MLP blocks ×3 → logits).
• Transformer internals: token embeddings → attention + MLP blocks → logits/softmax. Fixed, stateless mathematical function of (input tokens, parameters). Not like biological neurons — no memory.

Inference [0:24]

• Feed prefix → get distribution → sample (biased coin flip) → append → repeat.
• Stochastic: output is a remix of training data, statistically similar but not verbatim.
• ChatGPT at use time = inference only; weights frozen since training.

Worked example: GPT-2 (2019) [0:30]

• Cost drop: better data + much faster GPUs + better software.
• Training run view: each step processes ~1M tokens, ~7s per step, 32k steps ≈ 33B tokens. Sample text goes from random → locally coherent → fluent.
• 🖼 Actual log line: step 540/32000 | loss 4.927 | norm 0.46 | lr 4.63e-04 | 6908 ms | 59.7% bf16 MFU | 152,376 tok/s
  • norm = gradient norm (spikes signal instability) · lr = learning rate (warms up, then decays) · MFU = Model FLOPs Utilization, % of the GPU's peak FLOPs actually used (~60% is very good) · tok/s = training throughput.
• Paper: "Language Models are Unsupervised Multitask Learners" (OpenAI 2019). Reproduction: llm.c discussion #677.

Compute [0:39]

• Rented 8×H100 node (~$3/GPU/hr on Lambda). GPUs fit because matrix multiply is massively parallel.
• GPU → node (8) → data center (e.g. 100k GPUs). This demand = Nvidia's $3.4T valuation. All of it is "predict the next token, faster."

Base models [0:42]

• A model release = code (forward pass, ~200 lines of Python) + parameters (the valuable part — a list of billions of numbers).
• 🖼 "Psychology" of a base model: token-level internet document simulator · stochastic (different output each run) · "dreams" internet documents · can regurgitate training docs verbatim · params = lossy zip of the internet (lots of world knowledge) · usable via clever prompts (few-shot, fake-conversation) · "but we can do better" → post-training.
• Try a base model: Llama 3.1 405B base on Hyperbolic.
• Base model ≠ assistant. Ask "what is 2+2" → it rambles; it's autocomplete of internet documents.
• 405B params ≈ lossy compression of the internet. Frequent facts remembered better than rare ones.
• Regurgitation: recites Wikipedia verbatim (high-quality sources seen ~10× during training). Usually undesirable.
• Beyond knowledge cutoff → hallucinates plausible parallel universes.
• In-context learning / few-shot prompts: give 10 English→Korean pairs, it continues the pattern. Can even fake an assistant by formatting the prompt as a human/assistant transcript.

---

2. Post-training: SFT — [1:00:00–1:20:00]

• Same training algorithm, dataset swapped to conversations. Much cheaper (~3 hours vs ~3 months).
• Conversation tokenization [1:06]: special tokens like <|im_start|>user<|im_sep|>…<|im_end|> turn a structured chat into a 1-D token sequence (like a TCP/IP packet format). At inference the server appends <|im_start|>assistant<|im_sep|> and samples.
• 🖼 Tiktokenizer (gpt-4o) view of a 2-turn chat = 50 tokens:

• <|im_start|>user<|im_sep|>What is 2+2?<|im_end|>
  <|im_start|>assistant<|im_sep|>2+2 = 4<|im_end|>
  ...
  token IDs: <|im_start|>=200264  <|im_sep|>=200266  <|im_end|>=200265
  

  IDs above 200k show GPT-4o uses a bigger tokenizer (o200k_base, ~200k vocab) than GPT-4's 100k. The special tokens are new tokens added in post-training; never seen in pretraining.
• Data source [1:09]: InstructGPT (OpenAI 2022) — contractors (Upwork/Scale AI) write prompts + ideal responses following labeling instructions (helpful, truthful, harmless; real docs are hundreds of pages). Open reproduction: OpenAssistant.
• 🖼 Exact rule from the InstructGPT instructions: "For most tasks, being truthful and harmless is more important than being helpful." This is the root of model refusals.
• Modern [1:15]: mostly synthetic (LLM-generated, human-edited) — e.g. UltraChat, millions of conversations, "SFT mixtures."
• 🖼 UltraChat topic map: conversations span thousands of clusters (TV shows, healthcare, Cold War, customer service, poetry, diet...). Diversity of the SFT set is what makes the persona generalize to prompts it never saw.
• Key mental model: a ChatGPT answer ≈ instant simulation of a skilled human labeler following company instructions. If the prompt is in the SFT set, you get roughly that labeler's answer; otherwise an emergent blend with pretraining knowledge.

---

3. LLM psychology — [1:20:00–2:07:00]

Hallucinations [1:20]

• Cause: training answers to "Who is X?" are always confident → model imitates the confident style even for unknown names (Falcon-7B invents 3 different "Orson Kovats" bios).
• Fix 1 — know what you don't know (Meta Llama 3 "factuality"): take docs → LLM generates Q&A → ask model several times → LLM-judge compares → if model is wrong, add training example with answer "I'm sorry, I don't know." Wires the internal "uncertainty neuron" to the words.
• Fix 2 — tools [1:31]: model emits <search_start>query<search_end> → inference program pauses, runs search, pastes results into context → model answers with citations. Taught by a few thousand example conversations.

Parameters vs context [1:38]

Knowledge in parameters = vague recollection (something read a month ago). Knowledge in the context window = working memory.

• Practical: "Summarize chapter 1 of Pride and Prejudice" works OK; pasting the chapter works much better. → This is the whole argument for RAG.

Knowledge of self [1:41]

• "What model are you?" is nonsensical — no persistent self; boots up per conversation. Unprogrammed models say "ChatGPT by OpenAI" because that's the most common answer on the internet.
• Identity is bolted on via hardcoded SFT conversations (OLMo has 240 of them) or a hidden system message.

Models need tokens to think [1:45]

• Each token gets a fixed, small amount of compute (~100 layers). Can't do big leaps in one token.
• Bad label: "The answer is $3. Because…" (forces the answer into one token; the rest is post-hoc justification). Good label: intermediate steps first, answer last.
• Single-token answer with bigger numbers → wrong. With steps → right. Better still: "use code" — Python interpreter beats mental arithmetic.
• Counting fails (dots → a few tokens). "Use code" works because copy-pasting is easy for the model and Python does the counting.

Tokenization quirks [2:00]

• Models don't see characters → spelling tasks fail ("every 3rd char of ubiquitous", "Rs in strawberry" = can't see letters + can't count).
• 9.11 > 9.9 errors: interpretability found Bible-verse neurons firing (9:11 comes after 9:9 in verses).
• → Swiss cheese model: solves Olympiad math, fails trivial tasks at random. Treat as a stochastic tool.

---

4. Reinforcement learning — [2:07:00–3:07:00]

Why RL [2:09]

• Textbook analogy: exposition = pretraining, worked examples = SFT, practice problems with only the final answer = RL.
• Humans can't write the optimal token sequence for the model: what's easy/hard for us ≠ for the LLM. The model must discover sequences that reliably work for it.

How [2:18]

• For one prompt, sample many (thousands of) solutions → check against the known answer → train on the ones that are correct (e.g. best/shortest). Repeat across tens of thousands of prompts.
• SFT = initialization near good solutions; RL = where it gets dialed in.
• 🖼 Concrete loop from the drawing: 15 sampled solutions → only 4 reach "$3" → take the top one (correct and short) → train on it → repeat many, many times.
• Pretraining + SFT are standard; RL is new and finicky in its details → DeepSeek-R1 paper (Jan 2025, "Incentivizing Reasoning Capability in LLMs via Reinforcement Learning") made it public [2:27].

Emergent reasoning [2:30]

• During RL, accuracy climbs and response length grows: model learns to say "Wait, let me re-evaluate…", backtrack, check from other angles = chain of thought emerges; nobody hardcoded it.
• 🖼 The famous example is from DeepSeek-R1-Zero (RL applied directly to the base model, no SFT first), Table 3 of the paper: mid-way through solving √(a − √(a+x)) = x it writes "Wait, wait. Wait. That's an aha moment I can flag here. Let's reevaluate this step-by-step…" inside <think> tags. The final R1 adds a small SFT "cold start" for readability.
• Thinking models: DeepSeek-R1 (open weights, MIT; via together.ai), OpenAI o1/o3-mini (show only CoT summaries — distillation risk), Gemini Flash Thinking. GPT-4o ≈ mostly SFT (+RLHF).
• Karpathy's usage: ~80–90% plain models for knowledge Qs; thinking models for hard math/code.

AlphaGo parallel [2:42]

• Supervised (imitating experts) plateaus below top humans; RL surpasses them. Move 37 — a 1-in-10,000 human move that was brilliant.
• LLM RL could find novel strategies, analogies, maybe its own thinking language. Bottleneck: large, diverse practice-problem environments for every domain.

RLHF — unverifiable domains [2:48]

• Jokes, poems, summaries can't be auto-checked; 1B human ratings infeasible.
• Trick: indirection — humans rank ~5k outputs (ranking easier than scoring) → train a reward model (scores prompt+response 0–1) to match rankings → run RL against the reward model.
• 🖼 Cost math:

• 🖼 Training signal example: human ordering of 5 jokes 2 1 3 5 4 vs reward model scores 0.1 0.8 0.3 0.4 0.5 → loss pushes the RM so its scores rank the jokes the same way the human did.
• Origin paper: "Fine-Tuning Language Models from Human Preferences" (Ziegler, Stiennon, … Amodei, Christiano, Irving; OpenAI 2019). Several authors later co-founded Anthropic.
• Upside: RL in any domain; exploits the discriminator–generator gap (easier to judge than to write).
• Downside: reward model is a lossy, gameable simulation. Run too long → adversarial outputs ("the the the the" scores 1.0). So stop after a few hundred steps.
• "RLHF is not RL" in the magical sense — it's a small fine-tune. Verifiable RL (math, code, Go) can run indefinitely.

---

5. Future & where to find models — [3:07:00–end]

Coming: native multimodality (audio spectrogram slices and image patches as tokens — same recipe), long-running agents with human supervision ("human-to-agent ratio"), computer use (OpenAI Operator), test-time training (models currently can't learn after deployment beyond in-context; context window is finite).

Stay updated: LM Arena leaderboard (human pairwise votes; somewhat gamed), AI News newsletter (swyx), X/Twitter.

Where to run: proprietary → provider sites; open weights → together.ai; base models → Hyperbolic (Llama 3.1 405B base); local → LM Studio (distilled, quantized models).

Final mental model: your query → tokens in conversation format → model autocompletes. GPT-4o-style answer = simulation of a labeler. Thinking-model answer = something new, emergent from RL. Use as a tool, check the work, own the result.

---

Interview angle — what to say

Numbers to remember: 15T tokens / 44 TB (FineWeb) · 100,277 vocab (GPT-4) · GPT-2 1.6B/1,024 ctx/100B tokens · Llama 3 405B/15T · H100 ~$3/hr · SFT ~hours vs pretrain ~months.

---

Related

• [[Karpathy - LLM Intro.excalidraw]] — lecture whiteboard
• [[Learning/AI & ML/LLM Frontier Learning Path|LLM Frontier Learning Path]] - what to watch/read next (agents, reasoning, multimodal, world models)
• [[LLM Fundamentals]] — transformer stack, LoRA, quantization, RAG theory
• [[Learning/LLM & AI Stack]] — synthesis / positioning
• [[Interview Prep/Content/System Design/GenAI/index]] — apply this in GenAI system design
• [[Interview Prep/Content/System Design/GenAI/GenAI Vocabulary]] — interview glossary