Study and demonstrator
RAG: the librarian and its catalogue
Needle in a haystack, part 3 of 7. What happens when the model only reads the pages a search engine hands it?
Series
- A needle in Jules Verne
- Big context: just read everything
- RAG: the librarian and its catalogue ← you are here
- Hybrid search: when exact words matter
- Agentic search: let the model look
- RLM: the model that writes its own search
- The verdict: which architecture, for which haystack
The opposite idea
In part 2, the model read the whole novel for every question. It worked well, but it hit two walls: six minutes for the first read, and a hard ceiling at about 236,000 tokens. Seven novels don’t fit. Thirty-three certainly don’t.
RAG (retrieval-augmented generation) takes the opposite approach. You never show the model the whole library. A search engine picks a handful of passages that look relevant to the question, and the model answers from those alone.
In library terms: a librarian with a catalogue organised by meaning. You ask your question at the desk, the librarian brings back the most similar pages, and you never walk the shelves yourself.
It’s the most common architecture behind “chat with your documents” products. The question is what it costs in accuracy, and how it fails.
How it works here
1. Cut the text into excerpts. Each tier is split into excerpts of about 512 tokens (roughly a page), cut at sentence boundaries, with a 64-token overlap so a sentence on the edge appears in both neighbours. An excerpt never crosses a chapter boundary.
2. Turn each excerpt into a vector. A small embedding model, Qwen3-Embedding-0.6B, turns each excerpt into a list of 1,024 numbers that represents its meaning. Excerpts about similar things get similar vectors.
3. Store them. The vectors go into Qdrant, a vector database, with the excerpt text, its chapter and its position in the book.
4. At question time, the question is turned into a vector too, and Qdrant returns the closest excerpts. The model, the same Qwen3.8-27B as in part 2, gets those excerpts and the question, nothing else.
Everything else is identical to the big context test: same model, same settings, same instruction (“answer only from the text, or say NOT FOUND”), same 16 edited facts, same scoring.
One detail matters later: each excerpt is shown to the model with a short label giving its chapter title, as most RAG systems do.
How many excerpts? That’s the main setting of any RAG system, usually called k. I tested three values:
| k | Text given to the model | Compared with big context on one novel |
|---|---|---|
| 8 | ~4,000 tokens | 1.6% of the book |
| 16 | ~7,500 tokens | 3.2% |
| 32 | ~15,000 tokens | 6.4% |
The haystacks
This time all four tiers are in play, because RAG doesn’t care how big the library is:
| Tier | Content | Size | Excerpts |
|---|---|---|---|
| T1 | Opening chapters of Vingt mille lieues | 26,000 tokens | 66 |
| T2 | The whole novel | 236,000 tokens | 599 |
| T3 | 7 novels | ~1.1 million tokens | 2,852 |
| T4 | 33 novels | ~5 million tokens | 12,322 |
In T3 and T4, Vingt mille lieues is surrounded by other Verne novels. When another book tells the same story (L’Île mystérieuse retells Nemo’s end), the edits are applied there too. When it’s a different thing with the same name, it’s left as Verne wrote it. Part 1 explains why.
Preparation cost: the index
Big context has no preparation. RAG has to build its catalogue first:
| Tier | Excerpts | Tokens embedded | Time |
|---|---|---|---|
| T1 (opening chapters) | 66 | 31,000 | 1.3 s |
| T2 (1 novel) | 599 | 281,000 | 30 s |
| T3 (7 novels) | 2,852 | 1.35 million | 50 s |
| T4 (33 novels) | 12,322 | 5.78 million | 3 min 39 s |
Under four minutes for 33 novels, on the same desk-side machine. That’s less than big context needs to read a single novel once. And it’s paid once: after that, every question costs a search of a few milliseconds plus a short answer.
Results
Each point is 16 questions × 6 runs (T1: 6 questions), pooled over two full runs.
| Accuracy | T1 | T2 | T3 | T4 |
|---|---|---|---|---|
| Big context | 100% | 95% | ❌ | ❌ |
| RAG, 8 excerpts | 83% | 73% | 65% | 60% |
| RAG, 16 excerpts | 100% | 71% | 65% | 65% |
| RAG, 32 excerpts | 100% | 85% | 79% | 71% |
Big context stops at the context window. RAG keeps going, but loses accuracy with every book added.
On noise. With 93 answers per point, each figure is good to about ±9 points. I ran the whole test twice, and on identical questions the same configuration scored 76% once and 67% the other time. So 16 excerpts isn’t really worse than 8 at T2; it’s within noise. The trends below hold across both runs.
1. RAG scales; accuracy doesn’t
The good news first: RAG runs on 33 novels without breaking a sweat, and answers in about 9 to 14 seconds (median) whatever the size of the library. There is no six-minute first question. Big context can’t even start at T3.
But on the one novel both can handle, big context wins clearly: 95% against 85% at best. And from there, every tier costs RAG accuracy: with 32 excerpts, 85%, then 79%, then 71%.
2. RAG doesn’t lose facts in the middle. It loses facts that are worded differently
This is the most interesting result, because it’s the exact opposite of big context.
For every question, I recorded where the excerpt containing the answer ranked in the search results, out of the top 50:
| Fact | Depth in the novel | T1 | T2 | T3 | T4 |
|---|---|---|---|---|---|
| Commander (Whitcombe) | ~0% | 1 | 1 | 1 | 1 |
| Crespo → Moreira | 25% | — | 1 | 1 | 1 |
| Manaar → Kalpitiya | 49% | — | 2 | 2 | 2 |
| Vigo → Ferrol | 65% | — | 10 | 10 | 10 |
| First sighting (Holloway) | ~0% | 2 | 5 | 6 | 14 |
| Date (1858) | ~0% | 10 | 14 | 19 | 24 |
| Reward | ~0% | 1 | 3 | 24 | 48 |
| Atlantis → Thalassie | 68% | — | not in top 50 | not in top 50 | not in top 50 |
| Pole date (25 March) | 81% | — | not in top 50 | not in top 50 | not in top 50 |
Depth doesn’t matter at all. The Manaar gulf, in the middle of the book, is the fact big context struggled with most; RAG finds it at rank 2 every time, at every size. The two hardest facts for big context, Manaar and Vigo, are answered correctly in every single RAG run.
What matters is how the question and the passage are worded. “Which gulf does Nemo take his companions to, to visit the pearl beds of Ceylon?” lands squarely on a passage about pearl beds in a gulf. Two facts, Atlantis and the date Nemo reaches the South Pole, were never retrieved, at any size, with any number of excerpts, not even in the single novel. The search engine simply doesn’t see the passage as close to the question. No setting of k fixes that. The model nearly always answered NOT FOUND; a few times, it described a sunken continent without naming it.
And as the library grows, some answers sink. The reward and the date slide down the ranking because other novels are full of rewards and dates: La Maison à vapeur alone mentions 1858 several times, in a completely different story. The right passage is still there; it’s just no longer among the closest.
3. More excerpts help the search and hurt the reading
The obvious fix for a fact at rank 24 is to give the model more excerpts. It works: the date, which never reached the prompt with 8 or 16 excerpts at T3 and T4, was answered correctly 6 out of 6 times with 32.
But more excerpts also mean more noise to read through. The French warship where the Nautilus stops (the Redoutable) ranks 4th every time, so it’s always in the prompt. With 8 excerpts the model found it 15 times out of 18 across T2–T4. With 32, only 11 out of 18: the rest were NOT FOUND, with the answer sitting right there in the text.
So k is a trade-off, not a dial you can turn up. Too few excerpts and the answer isn’t there; too many and the model misses it among the rest.
4. As the library grows, wrong answers appear
In part 2, the most reassuring result was that big context almost never invents: on the novel, 1 answer in 93 was neither right nor “not found”.
RAG on the same novel stays close to that: between 0 and 4 wrong answers out of 93, depending on the number of excerpts. On 7 and 33 novels, it changes. With 16 excerpts, 21 answers out of 186 were confidently wrong.
The clearest case is the Maelström. With the single novel and 32 excerpts, RAG answered “Corryvreckan” 6 times out of 6. With 7 or 33 novels, it answered “the Maelström” 5 times out of 6, the original Verne name the text no longer contains anywhere in Vingt mille lieues, even though the right excerpt was in the prompt.
What changed? Other novels use the word “maelström” figuratively: a river whirlpool in Le Village aérien, an air vortex in L’Île mystérieuse. Those excerpts come back for a question about a whirlpool, they agree with what the model remembers, and memory wins. The distractors don’t contain the answer; they give the model permission to trust itself.
The date shows the same thing: “1859” (another year from the edited novel), or “1866” (the original year, which still appears in other novels in unrelated stories). Both plausible, both wrong, both stated with confidence.
For a business system, this is the finding that matters most. “Not found” can be handled. A plausible wrong answer goes straight into a report.
5. A hidden helper: the chapter titles
One result puzzled me. The bay of Vigo excerpt ranks 10th, so with 8 excerpts it’s never in the prompt. Yet the model answered “Ferrol” 18 times out of 18.
The answer was in the labels. Each excerpt is shown with its chapter title, and the chapter where Nemo collects the galleons’ gold is called “La baie de Ferrol” (in my edited text). Any excerpt from that chapter carries the answer in its header. The tunnel question works the same way, through the chapter title “Sinaï-Tunnel”.
Labelling excerpts with their section titles is a real, recommended technique, and it clearly helps. But it means that for those two facts, the test measured the labels rather than the search. Worth knowing when you read the figures, and worth knowing when you design a system: your document structure is part of your retrieval.
Cost and speed
| Big context (1 novel) | RAG (any tier) | |
|---|---|---|
| Preparation | none | 30 s (1 novel) to 3 min 39 s (33 novels) |
| First question | 6 min 20 s | ~10 s |
| Following questions | ~15 s | 9–14 s (median) |
| Text read per question | 236,000 tokens | 4,000–15,000 tokens |
| Maximum corpus on this machine | ~236,000 tokens | tested to 5 million, no limit reached |
What I’d tell a client
When RAG fits
- The corpus is bigger than the context window, or keeps growing.
- Questions use the same words and ideas as the documents (product names, procedures, clauses).
- Latency matters from the very first question.
- Each question touches a small part of the corpus.
When to be careful
- Facts are stated in passing, in a different vocabulary from the question. Some will never be found, whatever you tune.
- The corpus contains many similar documents (contracts, reports, versions of the same procedure). Lookalikes push the right passage down and make wrong answers more likely.
- A confident wrong answer is costly. Measure how often your system gives one, not just how often it’s right.
- Don’t tune k by intuition. More excerpts isn’t always better. Test it on your own questions.
RAG is what makes large corpora possible on modest hardware. It’s not a drop-in replacement for reading: on a document that fits in context, big context was clearly more accurate, and failed more honestly.
Limits of this test
- One embedding model, one setting. Qwen3-Embedding-0.6B, 512-token excerpts, no reranker, meaning-based search only. Larger embedding models or different excerpt sizes could move the numbers.
- The “rank” diagnostic finds the first excerpt containing the edited answer. For an answer that appears in several places, that excerpt isn’t always the one that best answers the question.
- Chapter labels helped two facts (Vigo, tunnel), as explained above.
- Six answers per point. Good enough for the trends, not for small differences.
- Counterfactual facts only, as in part 2.
Next: hybrid search
Two facts were never found by meaning. A third (the reward) sinks out of reach in the full library. The next article adds an exact-word index alongside the meaning-based one, and a second pass that reorders the candidates.
If the right passage is invisible to a search by meaning, can a search by words find it? That’s part 4.