RAG Is Dead. Who Killed It? - DEV Community

Posted on Jul 21 • Originally published at blog.expedora.com
Your RAG pipeline just returned the wrong answer. Again.
The query was simple. The doc had the answer. You watched the top-5 chunks come back and none of them held it. The right paragraph sat two pages away from anything your embedding model thought was "similar." So the LLM hallucinated something plausible, your user trusted it, and you spent the afternoon tuning chunk_size from 512 to 1024 and back.
This is not a tuning problem. This is the architecture.
Vector RAG was a clever hack for a 2022 problem: LLMs had tiny context windows, so we shredded documents into chunks, embedded them, and prayed cosine similarity would find the right shred. It worked well enough on FAQ pages and Notion dumps that we called it a pattern and shipped it into production at every company on earth.
Then we pointed it at a 200-page 10-K. A Kubernetes operator manual. A pharma SOP. A legal contract. And it fell apart.
Vector RAG rests on four assumptions. All four break on real documents.
Assumption 1: Similarity equals relevance. It doesn't. "revenue declined" and "revenue grew" sit next to each other in embedding space — opposite meanings, near-identical vectors. Domain jargon drifts even worse. "margin" means one thing in finance, another in typography, another in ML. Your general-purpose embedding model doesn't know which doc it's in.
Assumption 2: The right answer fits in k chunks. Why is k=5? Because someone's tutorial said so. Real answers span sections, reference earlier definitions, sit inside tables that got shredded on ingest. Top-k is a guess dressed up as a hyperparameter.
Assumption 3: Chunk boundaries preserve meaning. They don't. A 512-token window slices mid-sentence, mid-table, mid-recipe. The chunk that contains the answer loses the heading that gives it context. The LLM sees "...bake at 425°F for 40 minutes" and has no idea which dish "it" belongs to.
Assumption 4: Documents are bags of text. They aren't. A 10-K has a structure. A manual has chapters. A contract has clauses that reference other clauses. Vector RAG throws all of that away on ingest, then acts surprised when retrieval can't find its way back.
Every RAG failure I've debugged in the last two years traces back to one of these four. Bigger embedding models don't fix them. Reranking papers over them. Hybrid BM25 just postpones them. The problem isn't the retriever — it's the premise.
Humans don't read a 200-page doc by cosine similarity. They open the table of contents.
Forget databases for a second. Picture a 400-page cookbook, and you want the roast chicken recipe.
You don't read all 400 pages looking for the passage most "similar" to chicken. You flip to the contents, find Poultry, turn to that chapter, run your finger down to Roast Chicken, and read the whole recipe — ingredients, oven temp, and timing all together, in order.
Now imagine doing it the vector-RAG way: tear every page out, shuffle them into a bin, and pull the five loose pages whose words look closest to your question. You might grab the ingredients list but miss the oven temperature on the next page. You might grab a chicken stock recipe by mistake. And you'd have no idea which chapter any page came from.
That's the difference. Humans use reasoning over structure; vector RAG uses similarity over shredded text. The table of contents isn't decoration — it's a map of meaning the author already drew for you, and chunking throws it in the trash.
So what if retrieval kept the map?
It's called PageIndex.
Instead of embedding chunks, PageIndex builds a tree from the document's structure — chapters, sections, sub-sections — mirroring the table of contents. (No table of contents? An LLM generates the tree from the page text.) It keeps the cookbook intact instead of shredding it. Each node knows what it contains. No vectors. No cosine similarity. No k.
At query time, an LLM walks the tree the way a human walks a TOC. It starts at the root, reasons about which branch is most likely to hold the answer, descends, re-evaluates, and either drills deeper or backs out. When it lands on the right node, it reads the full, intact section — headers, tables, and context still attached.
Two things fall out of this for free:
1. Context stays whole. The answer arrives with the section that frames it. That orphaned "...bake at 425°F for 40 minutes" from earlier now carries its heading — Roast Chicken — instead of floating free.
2. Retrieval is explainable. You get the reasoning trace — the exact path from root down to the answer node. When it's wrong, you see where it turned wrong. Try debugging a cosine score.
PageIndex is built and open-sourced by Vectify AI (docs). And the numbers on structured docs are hard to argue with. On FinanceBench — QA over real financial filings — Vectify's Mafin 2.5, built on PageIndex, reports:
98.7% accuracy — vs 31% for ChatGPT-4o with search, and 45% for Perplexity, on the same documents.
Worth being honest: that's a vendor-reported number, not an independent replication. But the pattern holds — structured documents are exactly where similarity search hurts most, and where a tree wins biggest.
Vector RAG searches for text that looks like the answer. PageIndex reasons about where the answer lives.
Enough theory. Back to that 400-page cookbook, one real query: "What temperature do I roast the chicken at, and for how long?"
Vector RAG returns the top-5 chunks by similarity. Four are about chicken stock, chicken salad, chicken stir-fry — they all say "chicken." One catches "425°F for 40 minutes" but lost its heading on chunking, so the model can't tell if that's the roast chicken or the roast potatoes two pages over. Answer: confident, wrong.
PageIndex walks the tree — root → Poultry → Roast Chicken → the intact recipe, temperature and timing under their own heading, ingredients and steps still attached. It reads the answer in context and gets it right.
Same book. One retriever guessed at surface te