Problem 07 — Design an LLM Gateway for a Large Company

multi-provider · token quotas · semantic cache · observability

Classic cousins: [[API Gateway]], [[Rate Limiter]].

Requirements

200 internal teams call several LLM providers + self-hosted models. One platform: unified API, per-team budgets, failover, caching, PII redaction, cost attribution. < 20 ms overhead, supports streaming.

Architecture

flowchart LR
  A[Team Apps] --> GW[Gateway]
  GW --> AU[Auth and Quotas]
  GW --> PII[PII Redaction]
  GW --> CC[(Exact and Semantic Cache)]
  GW --> RT[Router]
  RT --> P1[Provider A]
  RT --> P2[Provider B]
  RT --> SH[Self-hosted Models]
  GW --> OB[[Usage and Trace Log]]
  OB --> BI[Cost Dashboards and Evals]

Deep dives

Quotas counted in tokens

Requests vary 1,000× in cost → rate-limit on tokens. Token bucket per team in Redis: reserve input_tokens + max_tokens before the call, refund unused when the response finishes.

Routing and failover

Route by declared model or policy (cheapest model meeting a quality tier). On 429/5xx: backoff + retry on another region/provider with an equivalent model. Circuit breaker per provider. Caveat: prompts tuned for one model behave differently on another → fallback list must be tested.

Caching

Exact cache	Semantic cache	Provider prompt caching
Hash of model + full prompt + params. Safe, low hit rate. Temperature 0 only.	Embed prompt, return cached answer above similarity threshold. Higher hit rate, can return wrong answer for subtly different question. Opt-in per use case.	Keep stable content (system prompt, docs) at the start so provider prefix caching applies.

Streaming through a proxy

Forward chunks without buffering; count tokens on the fly; output filters on a sliding window. Stateless gateway → horizontal scale.

Observability

Per request: team, model, input/output tokens, cost, latency (TTFT, total), cache hit, error type. Redact/hash prompts by default. Feed sampled traffic into eval jobs so teams compare models on their own data.

Likely follow-ups

Migrate 200 teams to a new model version safely?
Stop one team's runaway job exhausting a shared provider quota?
Guardrails: gateway or application?

Related: [[Design ChatGPT-style Assistant]] · [[index|GenAI index]]