Latency vs Throughput
The Distinction
| Low Latency | High Throughput | |
|---|---|---|
| Goal | Fast single request | Maximum requests/sec, individual delay less critical |
| Technique | Caching, in-memory state, fewer network hops | Batching, queuing, parallelism |
| Example | Video calls, gaming, autocomplete | Log ingestion pipelines, batch ETL |
Sometimes both matter for the same product but on different paths — Netflix needs low latency for playback start and high throughput for background encoding. Naming this split ("these are different sub-problems with different SLOs") is a senior-level move; treating "the system" as one undifferentiated latency target is not.
Cache Hit-Rate Math
The formula that recurs in nearly every HLD problem involving a cache:
Effective Latency = (HitRate × CacheLatency) + ((1 - HitRate) × OriginLatency)
Worked example — cache = 1ms, DB = 10ms:
| Hit Rate | Effective Latency | Net benefit? |
|---|---|---|
| 90% | 0.9×1 + 0.1×10 = 1.9ms | Yes — massive win |
| 50% | 0.5×1 + 0.5×10 = 5.5ms | Marginal |
| 10% | 0.1×1 + 0.9×10 = 9.1ms + lookup overhead | Net negative |
Long-tail problem: if most content is accessed once (e.g., 90% of shortened URLs clicked only once), cache hit rate collapses toward ~10% — adding Redis then adds latency to 90% of traffic for no benefit, plus stale-data risk.
Decision rule: if expected hit rate is below ~50%, don't add a cache yet. Start without one, instrument real access patterns, add the cache only once numbers prove it helps. The confident interview answer is "I'd skip the distributed cache here and benchmark first," not "add Redis" by reflex.
Interview Drill
Q: Give an example where high throughput and low latency are in direct tension, and how you'd resolve it. A: Batching writes to a message queue improves throughput but adds the batch-wait delay to each individual write's latency. Resolve by separating the two SLOs — a synchronous low-latency path for user-facing writes, and an async high-throughput batched path for background/analytics writes to the same underlying event.
Cross-links
[[Core Vocabulary & Quality Pillars]] · [[Caching & Redis]] · [[Rate Limiter]]