RAG at 10M Documents: Architecture Deep Dive
RAG at scale is not a retrieval problem. It's an infrastructure problem that retrieval lives inside of — and the entire AI industry has been teaching it backwards.
Every tutorial I've seen starts with LangChain, a PDF, and FAISS.from_documents(). That's fine for a demo. It is catastrophically wrong as a mental model for production. When r/programming lit up this week with a thread on RAG at 10 million documents, the comments were full of engineers who had already hit the wall — blown query budgets, index rebuild nightmares, retrieval quality that fell off a cliff past 100K documents. I've been in those trenches. This is what actually works.
The Toy Example Mental Model Will Destroy Your Production System
Here's the uncomfortable truth about the RAG tutorials dominating the AI integration space right now: they optimize for time-to-first-demo, not time-to-production-stability. You embed 500 documents, run a cosine similarity search, pipe the top-k chunks into GPT-4, and it looks like magic. Then you load 10 million documents and everything falls apart simultaneously.
What breaks first? Usually the index. A flat FAISS index at 10M vectors with 1536-dimensional embeddings (OpenAI's text-embedding-3-large output) occupies roughly 60GB of RAM just for the raw float32 vectors — before you add IVF clustering overhead, metadata, or the document store backing it. You cannot fit that on a single machine and expect sub-100ms p99 query latency. You need to stop thinking about your vector index as a data structure and start thinking about it as a distributed system with its own sharding topology.
The second thing that breaks is chunking strategy. At small scale, naive fixed-size chunking (512 tokens, 10% overlap) is fine. At 10M documents you're looking at potentially 50-150M chunks depending on average document length. Your chunking decisions now directly impact:
- Index shard count and rebalancing cost
- Embedding API spend (at $0.00002/1K tokens for
text-embedding-3-small, 150M chunks at 400 tokens average = ~$1,200 just to build the index once) - Retrieval precision, because bad chunks mean bad context windows
This is an architectural problem, not a prompt engineering problem.
LLM Integration Starts at the Index Design, Not the Prompt
Before you write a single line of LLM integration code, you need to make three irreversible architectural decisions. Get these wrong and you're rebuilding from scratch.
Decision 1: Hierarchical Chunking Over Flat Chunking
At scale, flat chunking is a retrieval quality killer. The approach that actually holds up is hierarchical chunking — sometimes called "parent-child" chunking — where you index small chunks (128-256 tokens) for retrieval precision but return larger parent chunks (512-1024 tokens) to the LLM for context richness.
LlamaIndex's HierarchicalNodeParser implements this directly:
from llama_index.core.node_parser import HierarchicalNodeParser, get_leaf_nodes
node_parser = HierarchicalNodeParser.from_defaults(
chunk_sizes=[2048, 512, 128]
)
nodes = node_parser.get_nodes_from_documents(documents)
leaf_nodes = get_leaf_nodes(nodes)
You embed and index the leaf nodes (128-token chunks). At query time, after retrieving leaf nodes, you walk up the parent relationship to fetch the 512 or 2048-token parent for the actual LLM context. This gives you semantic precision on retrieval without starving the model of context. At 10M source documents this is not optional — it's the difference between a system that retrieves the right sentence from the right paragraph versus one that retrieves vaguely related noise.
Decision 2: Shard Your Vector Index by Document Taxonomy, Not by Document Count
The naive sharding approach is round-robin by document ID. Every engineer I've seen do this regrets it. When a query comes in, you have to fan out to every shard, collect results, re-rank, and merge. At 20 shards that's 20 parallel network calls on every single query. Your p99 latency is now dominated by your slowest shard.
The correct approach: shard by domain or taxonomy. If you're indexing a corpus of legal documents, financial filings, and technical manuals — those are three separate indexes. Your query router (more on this below) determines which index or combination of indexes to hit before the vector search ever runs. You go from 20 parallel fan-outs to 1-3 targeted lookups.
Qdrant's collection-based architecture is built for this pattern. Each collection maintains its own HNSW index with independent configuration:
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, HnswConfigDiff
client = QdrantClient(url="http://localhost:6333")
client.create_collection(
collection_name="legal_documents",
vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
hnsw_config=HnswConfigDiff(
m=16,
ef_construct=200,
full_scan_threshold=10000,
),
)
The m parameter controls the number of bi-directional links in the HNSW graph. At 10M vectors, bumping m from the default 16 to 32 increases memory usage by ~30% but cuts recall degradation significantly. That tradeoff is worth it for high-value retrieval domains. For archival or low-traffic collections, keep it at 16.
Hybrid Retrieval Is Non-Negotiable for Machine Learning Workloads at This Scale
Pure vector search has a dirty secret: it fails on exact matches. If a user queries for a specific contract number, a regulation citation, or a product SKU, cosine similarity on embeddings will return semantically adjacent documents instead of the exact document containing that string. At small scale this is annoying. At 10M documents serving real users, it erodes trust in the entire system fast.
The production answer is hybrid retrieval: BM25 sparse retrieval running in parallel with dense vector search, results merged via Reciprocal Rank Fusion (RRF).
Elasticsearch's knn + BM25 hybrid search handles this natively as of 8.x:
POST /legal-documents/_search
{
"query": {
"bool": {
"should": [
{
"match": {
"content": {
"query": "force majeure clause termination",
"boost": 0.3
}
}
}
]
}
},
"knn": {
"field": "content_vector",
"query_vector": [0.1, 0.2, ...],
"k": 20,
"num_candidates": 100,
"boost": 0.7
},
"size": 10
}
The boost ratio between BM25 and kNN is something you tune empirically per domain. Legal and compliance corpora typically want higher BM25 weight (exact citation matching matters). Conversational knowledge bases lean toward higher kNN weight. Don't guess — instrument your retrieval pipeline and measure MRR (Mean Reciprocal Rank) against a golden evaluation set before you tune.
Weaviate's hybrid search uses the same RRF principle with a simpler API surface if you're not already on the Elastic stack.
Your Query Router Is the Actual Intelligence in the System
This is the piece that almost nobody talks about in AI integration tutorials, and it's arguably the most important component in a RAG at scale system. The query router runs before retrieval. It classifies the incoming query and makes decisions about:
- Which collection(s) to search — domain routing
- Which retrieval strategy to use — pure vector, hybrid, or metadata-filtered
- Whether to retrieve at all — some queries (arithmetic, date calculations, factual lookups about the system itself) should bypass retrieval entirely and go straight to the LLM
A naive implementation uses keyword matching. A production implementation uses a small, fast classification model — something like a fine-tuned text-classification model on your query taxonomy, or a structured output call to a cheap LLM tier (gpt-4o-mini, claude-haiku) that's faster and cheaper than your primary model.
from openai import OpenAI
from pydantic import BaseModel
from typing import Literal
client = OpenAI()
class QueryRoute(BaseModel):
collection: Literal["legal", "financial", "technical", "general"]
strategy: Literal["vector", "hybrid", "metadata_filter", "no_retrieval"]
confidence: float
def route_query(query: str) -> QueryRoute:
response = client.beta.chat.completions.parse(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": (
"Classify this query for document retrieval routing. "
"Return the most appropriate collection and retrieval strategy."
)
},
{"role": "user", "content": query}
],
response_format=QueryRoute,
)
return response.choices[0].message.parsed
OpenAI's structured outputs with Pydantic make this reliable — no JSON parsing fragility. The router call costs a fraction of a cent and saves you from fan-out queries across irrelevant shards on every request.
The Reranking Layer You're Probably Skipping
Top-k retrieval from a vector index gives you candidates ranked by embedding similarity. That ranking is good but not great — it doesn't account for query-document interaction the way a cross-encoder does. At small scale, skipping reranking is fine. At 10M documents where retrieval recall is lower and you're pulling k=50 candidates to ensure coverage, sending raw vector-ranked results to your LLM is leaving significant quality on the table.
Cross-encoder reranking with a model like cross-encoder/ms-marco-MiniLM-L-6-v2 (via sentence-transformers) runs inference on query-document pairs and produces a relevance score that's substantially more accurate than cosine similarity alone:
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
pairs = [[query, doc] for doc in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
return [doc for doc, _ in ranked[:top_n]]
This runs on CPU in under 100ms for 50 candidates. You pull k=50 from the vector index, rerank to top-5, send those 5 to the LLM. Your context window is cleaner, your answers are better, and you haven't increased LLM token spend.
Cohere's Rerank API is the managed alternative if you don't want to host the model yourself — worth it at scale to eliminate the operational overhead.
Why I'm Not Backing Down: Scale Changes Everything About AI Integration
The YC CEO shipping 37K lines of AI-generated code per day story burning up Hacker News this week is a perfect encapsulation of the problem. Volume of output is not the same as architectural soundness. You can generate a thousand RAG tutorials with an LLM. None of them will tell you about HNSW m parameter tuning, cross-encoder reranking latency budgets, or why routing by taxonomy beats routing by document count.
RAG at scale is not an AI problem. It's a distributed systems problem with an AI interface bolted on top. The embedding model, the LLM, the prompt template — those are the last 20% of the work. The first 80% is index architecture, sharding strategy, hybrid retrieval, query routing, and reranking. Every engineer who has shipped a RAG system past 1M documents knows this. The tutorials are still catching up.
Build the infrastructure first. The magic follows.