Why Your RAG Falls Apart at Scale

The real failure rate of RAG in production is 26.4% — not the 2.3% that passes surface-level evals. Dense-only systems produce "fluent, confident-sounding answers" from wrong retrieval sets. The errors don't announce themselves.

Air Canada found this out in court. Their chatbot told a passenger he could book a full-fare flight and retroactively apply for a bereavement discount — a policy that didn't exist. The BC Civil Resolution Tribunal ruled Air Canada liable. The tribunal explicitly noted that users shouldn't have to cross-check AI claims against other parts of the same website. That's the new standard.

RAG reduces hallucinations by 71% on average. But domain-specific rates remain 17–33% in legal tools and 10–20% in medical tools even with retrieval enabled. The gap between "I have RAG" and "my RAG is reliable" is where most production systems live.

The chunking problem

Starting point: 512 tokens, 10–20% overlap. This is the most widely recommended baseline for a reason — it's where precision and context preservation cross. But the relationship isn't linear.

Reducing chunk size from 1,024 to 64 tokens improved fact-based recall@1 by 10–15 percentage points on entity-heavy datasets. Beyond ~2,500 tokens of total context per prompt, generation quality starts to degrade — not improve. More context isn't always better.

Hierarchical chunking — small chunks for retrieval, parent chunks for generation — improved retrieval quality by 18–25% over flat methods in HiCBench results. NVIDIA testing showed it improved accuracy from 61% to 89% on presentation decks. The mechanism: small chunks give you precision, the parent gives the LLM enough context to answer coherently.

The misconception worth killing: semantic chunking doesn't reliably beat fixed-size. A NAACL 2025 paper found fixed 200-word chunks matched or beat semantic chunking across retrieval and answer generation benchmarks. The computational overhead of sentence boundary detection and embedding similarity comparisons often isn't justified unless your corpus has highly variable topic density.

Chunking strategy comparison:
  Fixed-size (512t, 10% overlap)  → precision: high, complexity: low   ← start here
  Hierarchical (parent/child)     → precision: high, context: good      ← upgrade if needed
  Semantic                        → compute: high, gains: mixed
  LLM-driven (propositions)      → quality: best, latency: 10-100x

Why vector search alone lies

Dense retrieval silently fails on: error codes (0x80070005, ENOMEM), product SKUs (RTX-4090 vs RTX-4070 — near-identical in embedding space, completely different products), function names (torch.nn.functional.cross_entropy), invoice IDs, rare named entities.

These aren't edge cases. They're the queries your users actually type when something breaks.

Hybrid search numbers that matter:

• Tuned BM25+dense hybrid beats BM25 alone by +7.5% NDCG (Elasticsearch Wands dataset, 2025)
• Two-stage pipeline (hybrid + neural reranking): Recall@5 0.816 vs 0.695 for hybrid RRF alone, 0.644 for BM25, 0.587 for pure dense
• E5+BM25: MAP from 0.523 → 0.797

Perplexity serves 200M queries/day on Vespa.ai and treats BM25 as "load-bearing infrastructure, not optional." That's not a philosophical position — it's operational experience at scale.

Alpha tuning by domain:

• Technical documentation: α ≈ 0.3 (lean sparse)
• Conversational/policy docs: α ≈ 0.7 (lean semantic)
• Mixed: α ≈ 0.6
• 2026 frontier: per-query dynamic alpha — detect at runtime whether the query is keyword-heavy or semantic

Reranking: add it when retrieval is the bottleneck

Cross-encoders: 10–25% accuracy improvement over pure retrieval; +5 to +15 NDCG@10 across MTEB/BEIR. But they add 50–500ms latency. MiniLM cross-encoder on CPU with N=50 candidates: 100–250ms. FlashRank: 15–30ms. Cohere Rerank API: 150–400ms plus network.

Real measured overhead: 9.2x latency multiplier (0.22s → 2.02s). That's the trade you're making.

Cost: Cohere Rerank API charges $1/1,000 queries. BGE Reranker is free (self-hosted). LLM inference remains the dominant cost in most RAG bots — reranking is cheap by comparison.

Don't add it speculatively. Measure retrieval precision on a domain-specific eval set first. Add reranking when you can prove retrieval precision is the bottleneck, not generation quality or chunking.

Failure modes nobody monitors

Embedding drift kills RAG systems slowly. Three mechanisms:

1. Model drift: switching embedding models without reindexing. Vectors from the old and new model live in incompatible latent spaces. Retrieval quality drops immediately; it's not always obvious why.
2. Corpus drift: adding new document types creates unexpected vector clusters, shifting centroid positions. Your HR documents start retrieving alongside your engineering docs.
3. Query drift: user vocabulary evolves faster than embeddings. Terms like "LLM fine-tuning" or "vector database" were nonsense two years ago — your embeddings may not handle the current version of your users' language.

Outdated embeddings cause performance declines of up to 20%. Detection: monitor Mean Reciprocal Rank (MRR) over time with a canary query set. Set an alert at 5% degradation.

Stale index. Notion's initial architecture took more than a day for end-to-end ingestion. After rebuilding: minutes for small tables, ~2 hours for the largest. Freshness wasn't a nice-to-have — it was the prerequisite for AI features being reliable at all.

The lost-in-the-middle problem. LLMs exhibit a U-shaped performance curve: accuracy is highest for information at the start and end of the context window, lowest in the middle. Stanford/Berkeley research measured a 30%+ accuracy drop on multi-document QA when the answer document moved from position 1 to position 10 in a 20-document context. Fix: place highest-ranked chunks at the beginning and end of the context window.

Context stuffing. Running retrievals too frequently can push end-to-end latency to nearly 30 seconds — breaking production viability. RAGAS 2.0: faithfulness scores dropped by up to 40% when irrelevant filler overloaded context windows. Larger top_k values actively hurt answer quality while increasing cost.

Vector DB: the cost trap

The OpenSearch Serverless trap deserves its own callout: minimum 4 OCUs × $0.24/OCU-hr = $701/month floor before a single document is indexed. Swapping to Aurora PostgreSQL + pgvector for Bedrock Knowledge Bases drops this to under $50/month. 90% reduction. This is the AWS RAG default most teams should be using.

Selection guide:

• pgvector: up to ~10M vectors with HNSW, excellent cost, 94.2% recall. Start here.
• Qdrant: best filtered-search performance at every scale. Rust-native, best for self-hosted performance-critical systems. Switch from pgvector at 50M+ vectors.
• Pinecone: zero-ops managed. Consistent p99 regardless of scale. The only realistic option at 1B+ vectors without a dedicated infra team.
• Weaviate: native hybrid BM25+vector in a single query. Pick it when hybrid search matters and you don't want to run Elasticsearch separately.

Notion's two-year arc

Notion went from 0 to 10B+ vectors across 200B+ blocks. Numbers:

• Nov 2023: pod-based clusters, Spark + Kafka, workspace-ID sharding
• May 2024: serverless migration → 50% cost reduction immediately
• Jan 2025: Turbopuffer migration → additional 60% on search costs, 35% on EMR; p50 latency from 70–100ms → 50–70ms
• Net result: 10x scale, 90% cost reduction, 600x increase in daily workspace onboarding capacity

The architecture evolved faster than the model did. The lesson: how you retrieve matters more than which embedding model you use.

What to actually measure

RAGAS metrics: faithfulness (does the answer stick to what the retrieved docs say?), answer relevance, context precision (is each retrieved chunk relevant?), context recall (did you retrieve everything needed?). Not just "did it answer."

Per-query logging minimum: retrieved chunk IDs, similarity scores, reranker scores, final answer. Without this you're debugging production failures with one hand tied behind your back.

Canary query set: 50–100 representative queries with known correct answers. Run weekly. Alert on MRR degradation above 5%. This is how you detect embedding drift before users do.