LLM Frontier Learning Path
LLM Frontier Learning Path
What to watch/read after [[Karpathy - Deep Dive into LLMs]], to understand where LLMs are now: agents (goal → tasks → loop), agent graphs, reasoning models, multimodal models, Jev / System One models. Ordered; links verified 2026-10-05.
Legend: 🎯 interview-critical (senior SDE / GenAI SD round) · 📚 depth / curiosity · ⏱ rough time
⭐ AskTGE-aligned shortlist (read this first)
Filtered from everything below + the [[Interview Prep/Content/System Design/GenAI/index|GenAI SD]] resource list. Kept only what maps to your AskTGE work (AWS Bedrock multi-agent assistant, Knowledge Base tools, prompt caching, asyncio.gather parallel tool calls, sliding-window summarization, SSE delta streaming, DynamoDB chat history, RAG hallucination fixes) and shows up in senior interviews.
Certifications
⚠️ Your [[Interview Prep/Main Plan]] has certs on the kill list until after an offer (MLA-C01 notes are parked in
Archive/Deferred/AWS-ML-Cert/). Free course certificates (#2, #3) cost no money and come with courses already in this list. The paid AWS exam (#1) is the one to decide on deliberately.
| # | Certification | AskTGE match | Cost / effort |
|---|---|---|---|
| 1 | AWS Certified Generative AI Developer – Professional (AIP-C01) | Almost 1:1 with AskTGE: Bedrock FM integration + Knowledge Bases (31%), Agents/implementation (26%), Guardrails/security (20%), cost + latency optimization like prompt caching (12%), testing/troubleshooting (11%). Turns your project into a credential recruiters filter on | Paid AWS Professional exam, 75 Qs; ~4–6 weeks prep alongside the job. Reuse the archived MLA-C01 Bedrock notes |
| 2 | Andrew Ng — Agentic AI (certificate also via Coursera) | Reflection, tool use, planning, multi-agent, evals + error analysis: the vocabulary for describing AskTGE's agents in interviews | ~10 h |
| 3 | Hugging Face — AI Agents Course (free certificate) | Hands-on multi-agent + tool calling in LangGraph/smolagents; gives you public code to show, since AskTGE code is private | ~2–3 weekends |
Resources (in order)
| # | Resource | AskTGE link → interview use | ⏱ |
|---|---|---|---|
| 1 | Anthropic — Building Effective AI Agents | Name AskTGE's pattern (routing / orchestrator-workers) and say why not a free-running agent | 30 min |
| 2 | Anthropic — Effective context engineering | Industry framing for your sliding-window summarization + prompt caching work (compaction, just-in-time retrieval) | 30 min |
| 3 | Anthropic — Multi-agent research system | Parallel sub-agents ≈ your parallel KB tool calls; token-cost trade-off (15×), checkpointing, tracing: senior follow-up material | 30 min |
| 4 | Chip Huyen — Building A Generative AI Platform | Where AskTGE's pieces sit: context construction, guardrails, router, cache, observability | 1 h |
| 5 | Eugene Yan — Patterns for LLM Systems | Evals, caching, guardrails, defensive UX: answers "how did you know the hallucination fixes worked?" | 1.5 h |
| 6 | Lilian Weng — LLM Powered Autonomous Agents | Planning / memory / tool-use vocabulary; AskTGE's chat history in DynamoDB = long-term memory | 45 min |
| 7 | Microsoft GraphRAG | Next step beyond AskTGE's KB retrieval for cross-document questions; good "what would you do next" answer | 1 h |
| 8 | LangChain Academy — Intro to LangGraph | Agents as state graphs with checkpoints + human-in-the-loop; common interview stack, LangChain already on your resume | modules 1–3, ~3 h |
| 9 | Stanford CME295 Lecture 7 (Agentic LLMs) + Lecture 8 (Evaluation) | Structured review of agents + LLM-as-judge evals; evaluation is the weakest part of most AskTGE-style stories | ~4 h |
| 10 | Raschka — Understanding Reasoning LLMs | "When would you use a thinking model in AskTGE?" (latency/cost vs accuracy per agent) | 40 min |
| 11 | [[#Phase 6 — Jev and System One models (TypeSafe AI)|Jev / System One models]] (below) | Fast typed classifier for AskTGE-style agent routing + tool-call risk checks without a full LLM call; fresh (Sept 2026) talking point | 1 h |
Skipped for this goal: 3Blue1Brown, Let's build GPT, DeepSeek-R1 paper, RLHF book, ReAct paper, Dwarkesh episode, multimodal posts, CS336, Berkeley MOOC. Good for depth, but they don't map to AskTGE or a senior SDE loop.
Vault practice that matches AskTGE: [[Interview Prep/Content/System Design/GenAI/Design Support Agent with Tools]], [[Interview Prep/Content/System Design/GenAI/Design LLM Gateway]], [[Interview Prep/Content/System Design/GenAI/Design Enterprise RAG Q&A]], and the war stories in [[Interview Prep/Behavioral & Past Projects/index]].
Full path
Phase 1 — Close the foundation gaps (≈ 4 h)
| # | Resource | Why | Tag |
|---|---|---|---|
| 1 | Karpathy — [1hr Talk] Intro to Large Language Models | Short version of what you watched + LLM as an operating system idea, scaling laws, jailbreaks/prompt injection | 🎯 ⏱ 1 h |
| 2 | 3Blue1Brown — Transformers, the tech behind LLMs (Ch 5) + Attention in transformers (Ch 6) | Best visual intuition for embeddings + attention; Karpathy skipped the internals | 🎯 ⏱ 1 h |
| 3 | Karpathy — How I use LLMs | Practical sequel: thinking models, tools, deep research, file uploads, voice/image I/O, custom GPTs | 🎯 ⏱ 2 h |
| 4 | Karpathy — Let's build GPT: from scratch, in code | Write a transformer yourself in PyTorch. Optional unless you want builder depth | 📚 ⏱ 2 h |
Phase 2 — Reasoning models (≈ 3 h)
How o1/o3, DeepSeek-R1, Claude/Gemini "thinking" work: RL with verifiable rewards, test-time compute, CoT faithfulness.
| # | Resource | Why | Tag |
|---|---|---|---|
| 5 | Sebastian Raschka — Understanding Reasoning LLMs | 4 ways to build reasoning models (inference-time scaling, pure RL, SFT+RL, distillation), DeepSeek-R1 pipeline explained clearly | 🎯 ⏱ 40 min |
| 6 | Lilian Weng — Why We Think | Test-time compute theory: parallel sampling vs sequential revision, thinking tokens, CoT faithfulness, reward hacking | 🎯 ⏱ 45 min |
| 7 | DeepSeek-R1 paper | Primary source for the "aha moment"; read sections 1–2 + Table 3 | 📚 ⏱ 1 h |
| 8 | Nathan Lambert — RLHF Book | Free book: reward models, PPO/DPO/GRPO, RL with verifiable rewards. Reference, read chapters as needed | 📚 |
Phase 3 — Agents: goal → tasks → loop (≈ 6 h + course)
| # | Resource | Why | Tag |
|---|---|---|---|
| 9 | Lilian Weng — LLM Powered Autonomous Agents | The classic map: planning (task decomposition, reflection), memory (short/long-term), tool use. ReAct, Reflexion | 🎯 ⏱ 45 min |
| 10 | Anthropic — Building Effective AI Agents | Workflow patterns (prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer) vs autonomous agents. Already in your [[Interview Prep/Content/System Design/GenAI/index|GenAI SD]] list | 🎯 ⏱ 30 min |
| 11 | Karpathy — Software Is Changing (Again) (YC AI Startup School 2025) | Software 1.0/2.0/3.0, LLMs as utilities/OS, partial autonomy + "autonomy slider", keeping AI on a leash | 🎯 ⏱ 40 min |
| 12 | Anthropic — Effective context engineering for AI agents | Successor to prompt engineering: curating the token budget across long agent loops (compaction, note-taking, sub-agents, just-in-time retrieval) | 🎯 ⏱ 30 min |
| 13 | Anthropic — How we built our multi-agent research system | Real production case: orchestrator + parallel sub-agents, +90% vs single agent, 15× tokens, evals, checkpointing. Great SD interview material | 🎯 ⏱ 30 min |
| 14 | Andrew Ng — Agentic AI (DeepLearning.AI) | Structured course: reflection, tool use, planning, multi-agent, plus evals/error analysis. ~10 h, 31 videos | 🎯 ⏱ 10 h |
| 15 | Hugging Face — AI Agents Course | Free, hands-on: build agents with smolagents, LangGraph, LlamaIndex; certificate | 📚 hands-on |
| 16 | Karpathy on Dwarkesh — "AGI is still a decade away" | Critical counterweight: why agents aren't reliable yet ("march of nines"), why RL is weak, model collapse | 📚 ⏱ 2.5 h |
| 17 | ReAct paper (Yao et al. 2022) | Origin of the Reason + Act loop most agents use | 📚 ⏱ 30 min |
Phase 4 — Graphs (agent graphs + GraphRAG) (≈ 7 h)
Two meanings of "graphs" in LLM land; know both.
| # | Resource | Why | Tag |
|---|---|---|---|
| 18 | LangChain Academy — Introduction to LangGraph (free) | Agents as state machines/graphs: nodes, edges, cycles (the loop), state, checkpointing, human-in-the-loop, memory. 6 h, 55 lessons | 🎯 ⏱ 6 h |
| 19 | DeepLearning.AI — AI Agents in LangGraph | Shorter alternative to #18; builds ReAct agent from scratch, then in LangGraph | 📚 ⏱ 1.5 h |
| 20 | Microsoft GraphRAG + paper "From Local to Global" | Knowledge-graph RAG: entity graph + community summaries for global questions plain vector RAG can't answer | 🎯 ⏱ 1 h |
Vault link: [[Langchain]] (framework notes), [[MCP]] (tool protocol).
Phase 5 — Multimodal models (≈ 2 h)
| # | Resource | Why | Tag |
|---|---|---|---|
| 21 | Sebastian Raschka — Understanding Multimodal LLMs | The 2 architectures: unified embedding decoder (image patches → tokens) vs cross-attention; Llama 3.2, Qwen2-VL, Pixtral, NVLM | 🎯 ⏱ 45 min |
| 22 | Chip Huyen — Multimodality and Large Multimodal Models | CLIP, Flamingo, contrastive learning: the foundations behind vision-language models | 📚 ⏱ 1 h |
Phase 6 — Jev and System One models (TypeSafe AI)
What: Jev by TypeSafe AI (launched 2026-09-15; CEO Diogo Almeida, ex-OpenAI RLHF researcher). It's the first "System One model", named after Kahneman's fast/slow thinking. A frontier LLM is the slow, deliberate System 2. Jev is a fast, intuitive decision-maker for software.
How it differs from an LLM:
| LLM (System 2) | Jev (System 1) | |
|---|---|---|
| Output | Free text, one token at a time (autoregressive) | Typed decisions only: choice / score / yes-no, with calibrated probabilities. No text at all |
| Decoding | Sequential | Parallel sampler: all outputs in one pass (non-autoregressive) |
| Post-training | RLHF (human preference) / RLVR (verifiable answers) | RLCD: Reinforcement Learning for Calibrated Decisions, so confidence matches accuracy |
| Type errors / out-of-schema hallucination | Possible even in JSON mode | Impossible by construction: options defined in the schema before the call |
| Latency | seconds to minutes | 70–500 ms |
| Price | $ per M input + output tokens | ~$0.042 per M input tokens, output free |
| Can't do | — | Generate text/code, explain itself, open-ended answers |
API shape (state + typed questions, all answered in parallel):
requests.post("https://api.typesafe.ai/v1/systemone", json={
"model": "jev-latest",
"state": "Customer emailed twice about a failed payment...",
"questions": {
"category": {"type": "choice", "options": ["billing", "technical", "sales"]},
"urgency": {"type": "score", "min": 0, "max": 100},
},
})
# → per question: answer + probability distribution + confidence
Launch eval (TypeSafe's own, 4 workflows; treat as vendor numbers): Jev 67.8% accuracy at $0.0004/case, 0.4 s · GPT-5.6 Terra 67.9%, $0.03, 10 s · Claude Opus 5 73.1%, $0.18, 38 s. Similar accuracy to mid frontier models on bounded decisions at ~100× lower cost and latency; the best LLM is still more accurate.
Where it fits in agent systems (LangChain integration):
- Router: classify each request to the right model/agent (cheap model vs strong model)
- Tool-call guardrail: score each proposed tool call for risk and block it before execution
- Classify / extract / score / branch inside workflows; map-reduce over big datasets; LLM-as-judge replacement for verification; real-time loops (games, robots, trading)
Interview angle: "Not every step in an agent loop needs a text generator. Route and guard with a fast calibrated classifier, and spend LLM tokens only on generation and reasoning." This is the same idea as model routing in [[Interview Prep/Content/System Design/GenAI/Design LLM Gateway]].
| # | Resource | Why | Tag |
|---|---|---|---|
| 23 | TypeSafe — Introducing System One Models & Jev | Primary source: thesis, RLCD, benchmarks, limitations | 🎯 ⏱ 30 min |
| 24 | Latent Space — "Jev: System One models for Prod, not God" (Diogo Almeida) / YouTube: Why We Made Jev | Founder explains why typed, calibrated decisions matter for production | 📚 ⏱ 1 h |
| 25 | LangChain — Building a harness with Jev | Jev + LLM in one agent: routing, AutoModeMiddleware tool-call blocking | 🎯 ⏱ 20 min |
| 26 | TypeSafe docs · TypeLLM (open source, Jev-inspired type-safe generation for regular LLMs) | Try the API; open-source way to get schema-guaranteed outputs from existing models | 📚 |
JEPA / world models (Meta V-JEPA 2, LeCun) is a different topic, parked for later: V-JEPA 2 blog.
Phase 7 — One full course that ties it all together (pick one)
| Resource | Covers | Tag |
|---|---|---|
| Stanford CME295 — Transformers & LLMs (Autumn 2025) | 9 lectures × ~1h50: transformer → training → tuning → reasoning (L6) → agentic LLMs (L7) → evaluation (L8) → trends (L9). Best single course for your goal. At minimum watch L6–L8 | 🎯 |
| Stanford CS336 — Language Modeling from Scratch | Builder depth: tokenizer, training, GPUs/kernels, scaling laws, alignment. Heavy | 📚 |
| Berkeley Advanced LLM Agents MOOC | Research-level agent lectures: reasoning, tool use, multi-agent, safety | 📚 |
Stay current (weekly, 30 min)
- AI News (swyx): daily digest; read top summaries only
- Latent Space podcast: AI engineering interviews
- Interconnects (Nathan Lambert): post-training/RL, open models
- Simon Willison's blog: hands-on take on every new model/tool
- Ahead of AI (Raschka): monthly research roundups
- LM Arena: leaderboard (treat as a first pass; somewhat gamed)
Suggested schedule (fits beside the Oct 2026 crash course)
~1 h/day, interview-critical 🎯 items only first:
| Week | Do |
|---|---|
| 1 | Phase 1 (#1–3) + Phase 2 (#5–6) |
| 2 | Phase 3 (#9–13) + start #14 Agentic AI course |
| 3 | Finish #14 (2× speed, skip labs) + #18 LangGraph modules 1–3 + #20 GraphRAG + #21 multimodal |
| Later | Phase 6 extras, CME295 full, 📚 items, hands-on HF course |
After each phase, add the key ideas to [[Interview Prep/Content/System Design/GenAI/GenAI Vocabulary]] or a new note here.
Related
- [[Karpathy - Deep Dive into LLMs]] · [[LLM Fundamentals]] · [[Learning/LLM & AI Stack]]
- [[Interview Prep/Content/System Design/GenAI/index]]