Back to Notes

Design Smart Compose

Problem 03 — Design Smart Compose for an Email Client

latency budget · small models · personalization · privacy

Classic cousin: [[Typeahead - Autocomplete]].

Requirements

Suggest the rest of the current phrase inline while typing; accept with Tab. End-to-end < 100 ms, grammatical, never offensive, never leak another user's data. 1B+ users.

Key insight

The latency budget rules out large models. This is a small-model, high-QPS problem: distilled transformer ~100M–1B params, served at the edge or on-device, triggered sparingly.

Estimates

Raw keystrokes are far too many to send. Trigger only after a word boundary
plus ~100 ms pause: maybe 20 triggers per email × 1B emails/day = 20B/day
→ ~230k QPS average. Output is short (≤ 8 tokens), input ≤ 256 tokens.
A 300M-param model on one GPU can serve thousands of short requests/s,
so ~100–300 GPUs globally, or zero server cost if run on-device.

Architecture

flowchart LR
  E[Editor] --> T[Trigger Logic]
  T --> CA[Prefix Cache]
  CA -->|miss| SV[Edge Inference]
  SV --> RK[Confidence and Safety Filter]
  RK --> E
  E --> LG[[Accept / Reject Log]]
  LG --> TR[Offline Training]
  TR --> SV

Deep dives

  • Context features: email prefix, subject, email being replied to, recipient, time of day, locale.
  • Decoding: small beam search (2–4) beats sampling for short completions. Stop at end of phrase.
  • Show-or-not threshold: display only above a tuned confidence. A wrong suggestion costs more attention than none.
  • Caching: common prefixes ("Thanks for", "Let me know if") hit cache. Cancel in-flight requests when user keeps typing.
  • Personalization: small per-user adapter or feature layer. Federated learning keeps raw email on device.
  • Safety: blocklists + classifier; never guess gendered pronouns; never train on content that could leak secrets.

Evaluation

Offline: exact/prefix-match accuracy on held-out emails, latency on target hardware. Online: acceptance rate, characters saved per email, display rate, complaints. Shown-but-rarely-accepted = negative signal even if accurate.

Likely follow-ups

  • On-device vs server trade-offs?
  • Ship a model update to a billion clients?
  • Support 30 languages?

Related: [[index|GenAI index]]