Back to Notes

Design a Code Assistant

Problem 04 — Design a Code Assistant (Copilot-style)

fill-in-the-middle · repo context · request cancellation · security

Requirements

Two features: inline completions while typing (p95 < 300 ms) and a chat panel that explains/edits code across the repo. Enterprise: code not used for training, secrets never leave the machine.

Architecture

flowchart LR
  IDE[IDE Extension] --> CX[Context Collector]
  CX --> SF[Secret Scrubber]
  SF --> GW[Gateway]
  GW -->|inline| FIM[Small FIM Model]
  GW -->|chat| BIG[Large Model]
  BIG --> TL[Tools: search, read file, run tests]
  CX --> IX[(Local Repo Index)]
  FIM --> PF[Post-filter]
  PF --> IDE
  BIG --> IDE

Deep dives

Prompt for inline completion

Fill-in-the-middle (FIM): send code before cursor (prefix) + after cursor (suffix); model fills the gap. Add snippets from recently opened tabs and symbols referenced near the cursor (language server or local embedding index). Rank snippets, pack into a fixed budget (~2–8k tokens).

Latency tactics

  • Debounce (~75 ms) and cancel stale requests on the next keystroke; most requests get cancelled.
  • Small fast model inline; big model only for chat.
  • Speculative decoding + prefix caching (consecutive requests share almost the whole prefix).
  • Stop at natural boundary: end of line/block/function.

Repo-scale chat

Give the chat model tools, not the whole repo in the prompt: semantic code search, grep, read file, list symbols, run tests. Agent loop with step budget. Show edits as diffs the user approves.

Security and compliance

Client-side secret scanning (API keys, private keys) before sending. Filter suggestions matching public code verbatim beyond a length threshold (licences). Zero-retention mode for enterprise: no prompt logging, no training on customer code.

Evaluation

Offline: pass@k on unit-test benchmarks from real repos, exact match on held-out edits. Online: acceptance rate, % of accepted code still present after 30 s and 5 min (accepted-then-deleted = false positive), latency, cancellation rate.

Likely follow-ups

  • Evaluate the chat agent's multi-file edits?
  • Monorepo with 50M lines?
  • Per-enterprise fine-tuned model serving (multi-LoRA)?

Related: [[Design Support Agent with Tools]] · [[index|GenAI index]]