Study and demonstrator
A needle in Jules Verne
Needle in a haystack, part 1 of 7. Why finding one sentence in a large corpus is still hard on your own hardware.
Series
The problem every company has
Somewhere in your company’s documents there’s one sentence that answers today’s question.
A clause in contract number 412. An error code in a three-year-old incident report. A dosage in a 600-page technical manual. The tolerance on a part, buried in a supplier’s specification.
The documents are too big to read, too many to search by hand, and the answer is tiny. That’s the needle in a haystack. It’s also one of the most common requests I hear when companies talk about AI: “Can it find things in our documents?”
The honest answer is: it depends on the architecture. There are several ways to build it; they don’t fail in the same way, and the differences matter a lot more than the choice of model.
This series compares five of them, on the same data, on the same machine, with the same model.
Why not just use the cloud?
Frontier cloud models have context windows of a million tokens or more. On the classic needle-in-a-haystack benchmark, they score close to 100%. For many uses, that’s the end of the discussion.
But a lot of companies can’t, or won’t, send their documents to a cloud model. Contracts, medical files, industrial know-how, anything covered by a confidentiality clause. For them, the question becomes: what can you do on hardware you own?
That changes everything:
- Smaller models. Open-weight models that fit on one machine, usually quantised (compressed) to fit in memory.
- Smaller windows, in practice. The model may accept 262,000 tokens, but memory and speed decide what’s usable.
- Real costs. Every second of compute is yours. Six minutes to read a document isn’t an abstraction; it’s your machine, busy.
The near-perfect cloud benchmarks say little about this situation. So I measured it.
The five architectures
Picture your documents as a library, and the needle as one sentence slipped into one book. Each architecture is a different way of finding it.
| Architecture | In library terms | Expected strength | Expected weakness |
|---|---|---|---|
| Big context | The reader memorises every book at once, then answers | Sees everything, nothing to build | Limited by memory; slow to “read”; attention blurs in the middle |
| RAG (Retrieval Augmented Generation) | A librarian with a meaning-based catalogue hands over the most similar pages | Fast, scales to any size | Misses needles phrased differently from the question |
| Hybrid search | The same librarian with a second, exact-word index, then a second check of the best pages | Catches exact names and codes too | Still one shot: wrong pages in, wrong answer out |
| Agentic search | The reader walks the shelves, searches, opens books, and decides when to stop | Adapts its strategy to the question | Depends on the model’s judgement; can loop or give up |
| RLM (Recursive Language Models) | The reader writes small programs to slice the library and delegates pieces to copies of itself | Handles inputs far larger than its memory | Needs reliable code-writing; harder to secure and debug |
Two questions separate them:
- When does the work happen? At indexing time (RAG, hybrid search: you prepare the library in advance) or at question time (big context, agentic search, RLM: the model does the work when you ask).
- Who decides where to look? A pipeline designed by an engineer (RAG, hybrid) or the model itself (agentic search, RLM).
Big context sits apart: nobody decides where to look, because the model looks at everything.
The test bench
One machine
An ASUS Ascent GX10: one NVIDIA GB10 chip, 128 GB of memory shared between processor and GPU. It’s the kind of machine a small company, or a department, can buy and run in an office.
One model
The same model for all five architectures, with the same settings. Otherwise the comparison measures models, not architectures. For this series: Qwen3.8-27B, an open-weight model quantised to NVFP4, served by vLLM.
Some architectures need a second, much smaller model to build their index. For RAG and hybrid search, that’s Qwen3-Embedding-0.6B. The model that reads and answers is always the same.
One haystack: Jules Verne
The corpus is Jules Verne’s work in French. It comes in four nested sizes, from a few chapters to 33 novels:
| Tier | Content | Approx. size |
|---|---|---|
| T1 | Opening chapters of Vingt mille lieues sous les mers | 26,000 tokens |
| T2 | The whole novel | 236,000 tokens |
| T3 | 7 novels | ~1.1 million tokens |
| T4 | 33 novels | ~5 million tokens |
Each tier contains the previous one. Vingt mille lieues sits in the middle of T3 (from 33% to 54% of the text) and near the start of T4 (8% to 13%), surrounded by other novels.
Between T2 and T3, big context stops being possible on this machine: the library simply doesn’t fit any more. Where each architecture breaks is the headline result of the series.
Needles the model can’t cheat on
A famous novel has a problem: the model has probably read it. Ask who commands the frigate chasing the sea monster and it may answer “Farragut” from memory, without reading the text you gave it. That would look like a successful retrieval and prove nothing.
So instead of adding sentences, I edit facts that are already in the book, everywhere they appear. Farragut becomes Whitcombe. The Maelström becomes the Corryvreckan. 1866 becomes 1858.
Each answer then means something precise:
- the edited fact: the model read the text;
- the original fact: the model answered from memory and ignored the text;
- both: it read the text but couldn’t let go of what it knew;
- “not found”: it looked and gave up, which is better than inventing.
Before each test, every question is also asked with no text at all, to record what the model knows from memory. That baseline tells you whether a correct answer means “found it” or “overrode what it knew”. Without the text, the model has never once given an edited answer.
When the library grows, so do the traps
The other novels raised a question I hadn’t anticipated: what happens when another book mentions the same thing?
Verne reuses his own material. L’Île mystérieuse retells the end of Captain Nemo, including the frigate that hunted the Nautilus. Mistress Branican mentions the reefs where La Pérouse’s ships were lost. When another book talks about the same fact, I edit it there too, so the library tells one consistent story.
When it’s a different thing with the same name, I leave it alone. Another novel has its own ship called Scotia; others mention a 21 March, a year 1866, the bay of Vigo in passing. Those stay as Verne wrote them.
That’s realistic. In a real company archive, the same name means different things in different documents, and a good system has to tell them apart. It’s also where some of the most interesting failures happen, as part 3 shows.
What gets measured
For every answer:
- correct or not, and whether it came from the text or from memory;
- total time, and for big context, time to first word, because that’s where its cost is: reading the whole text before answering;
- tokens read and written; the written count includes the model’s reasoning;
- for architectures that search, whether the right passage reached the model at all, which separates “couldn’t find it” from “found it but misread it”;
- the number of model calls and tool calls (for agentic search and RLM).
For every architecture: the preparation cost (building an index) and the failure modes, because how a system fails matters as much as how often.
Noise
Every question is asked several times, at the same settings, and the results are pooled.
I learnt why the hard way. I ran the same RAG configuration twice, on the same index, with three repeats per question. On the 15 questions whose text hadn’t changed between the two runs, it scored 76% the first time and 67% the second. With three repeats per question, a gap of 10 points can be pure chance. The figures in this series come with their sample size, and differences smaller than the noise aren’t reported as findings.
Three rules I hold myself to
Same everything except the architecture. Same model, same settings, same prompt, same answer format. If one option gets a better prompt, the comparison is dead.
Test data is code. The needles are built by scripts, versioned, and checked before every run. A wrong edit in the text would produce a result that looks like a finding.
Report the noise. A result that changes from one batch to the next is reported with its range, not as a single number.
What’s next
Part 2 starts with the simplest architecture of all: big context. Put the whole novel in one prompt, 236,000 tokens, and ask.
Spoiler: it works better than I expected; it isn’t uniform across the book, and when it fails, it fails in the way you’d want.