Problem 02 — Design an Enterprise RAG Question-Answering System

ingestion · hybrid retrieval · permissions · citations

Companion: [[RAG & LLM System]] (older, shorter HLD page).

Requirements

Functional: employees ask questions over Confluence, Google Drive, Slack, Jira. Answers cite sources. Users only see content they already have access to. New/edited docs searchable within 15 min. Non-functional: 50k employees, 10M documents, p95 answer < 5 s, zero permission leaks, says "I don't know" rather than invent.

Estimates

10M docs × ~20 chunks = 200M chunks
200M × 768 dims × 4 B ≈ 615 GB fp32  → int8 ≈ 155 GB, sharded across nodes
                                     → IVF-PQ (m=96) ≈ 19 GB + rescore
Queries: 50k users × 5/day = 250k/day → ~3 QPS avg, ~15 peak (small!)
Ingestion: ~1% docs change daily = 100k docs → 2M chunks re-embedded/day

Query load is small. Hard parts: ingestion correctness, permissions, answer quality.

Architecture

flowchart TB
  subgraph Ingestion
    SRC[Connectors] --> Q[[Change Queue]]
    Q --> P[Parse and OCR]
    P --> CH[Chunker]
    CH --> EM[Embedding Model]
    EM --> VDB[(Vector Index)]
    CH --> KW[(BM25 Index)]
    SRC --> ACL[(ACL Store)]
  end
  subgraph Query
    U[User] --> QR[Query Rewriter]
    QR --> HY[Hybrid Retrieve]
    HY --> VDB
    HY --> KW
    HY --> F[ACL Filter]
    F --> RR[Cross-encoder Reranker]
    RR --> LLM[LLM with Citations]
    LLM --> V[Citation Check]
    V --> U
  end
  ACL --> F

Deep dives

Chunking

Split on structure (headings, paragraphs, table boundaries), target 300–800 tokens, 10–15% overlap. Prepend doc title + section path before embedding ("Step 3: restart the pod" keeps meaning). Metadata: doc_id, chunk_index, updated_at, source, ACL group IDs.

Why hybrid retrieval

Dense handles paraphrase ("how do I get reimbursed" ↔ "expense policy"). BM25 handles exact tokens: ERR_4012, ticket IDs, codenames. Top 50 from each → RRF merge → cross-encoder rerank to top 5–8.

Permissions — the part that gets people fired

Pre-filter (recommended)	Post-filter
User's group IDs as metadata filter inside ANN search. No leak possible; needs a vector DB with efficient filtered search.	Retrieve then drop forbidden chunks. Simple, but may return too few results; leak risk if a step is skipped.

Resolve group membership at query time from the IdP (cache minutes). Sync ACL changes through the same change queue. Never cache final answers across users.

Freshness and deletes

Webhooks where available, else polling. Hash chunk content; re-embed only changed chunks. Deletes must propagate fast to every index + invalidate answer cache.

Grounding and hallucination

Prompt: answer only from passages, cite passage IDs, say "I don't know" if insufficient. Post-check: every cited ID exists in context; optionally a small model verifies each claim is supported.

Changing the embedding model

Means re-embedding all 200M chunks. Build new index in parallel (blue-green), compare retrieval metrics on golden set, then switch reads.

Evaluation

300–500 real questions with labelled relevant docs. Measure retrieval (recall@10, MRR) separately from generation (faithfulness, answer relevance, citation precision via LLM-as-judge calibrated on human labels) to know which stage failed. Online: thumbs, citation click-through, "no answer" rate.

Likely follow-ups

A question needs facts from 40 documents: what breaks?
Tables and diagrams inside PDFs?
Malicious wiki page injecting instructions?
When fine-tune instead?

Related: [[Vector Quantization]] · [[GenAI Vocabulary]] · [[index|GenAI index]]

Design an Enterprise RAG Q&A System