GenAI Back-of-Envelope Math

Four formulas cover most estimation questions. Order-of-magnitude only; say assumptions aloud. Classic estimation: [[Back-of-Envelope Estimation]].

1. Weight memory

memory ≈ parameters × bytes per parameter
70B @ fp16 (2 B)   = 140 GB  → needs 2× H100 80GB minimum, more for KV cache
70B @ int4 (0.5 B) =  35 GB  → fits on 1 GPU with room for cache
8B  @ fp16         =  16 GB  → one mid-range GPU

2. KV cache per token

kv_bytes/token = 2 (K and V) × layers × kv_heads × head_dim × bytes
Llama-3-70B: 2 × 80 × 8 × 128 × 2 B ≈ 0.33 MB per token
One 8k-token conversation ≈ 2.6 GB of cache
→ 100 concurrent 8k sessions ≈ 260 GB: cache, not weights, limits concurrency

3. Decode speed (memory-bound upper limit)

tokens/s per sequence ≤ memory bandwidth ÷ bytes read per step
H100 ≈ 3.35 TB/s; 70B fp16 = 140 GB read per token
→ ≈ 24 tokens/s at batch 1 on one GPU's worth of bandwidth
Batching reads weights once for many sequences: throughput scales with batch
size until compute or KV memory runs out.

4. Prefill compute

FLOPs ≈ 2 × parameters × input tokens
70B × 2k-token prompt ≈ 2.8 × 10^14 FLOPs
H100 ≈ 1 × 10^15 dense fp16 FLOP/s at good utilization (~0.3 s ideal)

Traffic template

DAU × requests/user/day ÷ 86,400        = average QPS   (peak ≈ 2–3×)
QPS × output tokens per request          = output tokens/s to serve
output tokens/s ÷ tokens/s per replica   = replicas needed
tokens/day × price per 1M tokens         = API cost per day

Embedding storage

vectors × dimensions × bytes
100M chunks × 1024 dims × 4 B (fp32) ≈ 410 GB
same with int8                       ≈ 100 GB
HNSW graph adds roughly 10–30% on top

Interview takeaways

Prefill is compute-bound → long prompts hurt TTFT. Decode is bandwidth-bound → model size and batch size drive TPOT.
At high concurrency the KV cache, not the weights, caps users per GPU. Fixes: PagedAttention, GQA models, KV quantization, shorter context.
Reactive GPU autoscaling is slow (model load takes minutes). Keep a warm buffer and scale on queue depth / KV utilization.

Related: [[GenAI Vocabulary]] · [[Design ChatGPT-style Assistant]] · [[index|GenAI index]]

GenAI Capacity Math