Study and demonstrator

The verdict: which architecture, for which haystack

Christophe Cosnefroy · 20 October 2026

Needle in a haystack, part 7 of 7. Six ways to find one sentence in 33 novels, on one machine, compared.

Series

1. A needle in Jules Verne
2. Big context: just read everything
3. RAG: the librarian and its catalogue
4. Hybrid search: when exact words matter
5. Agentic search: let the model look
6. RLM: the model that writes its own search
7. The verdict: which architecture, for which haystack ← you are here

What I set out to answer

Part 1 asked a simple question: if your documents can’t leave the building, what can you do on hardware you own?

Over two weeks, one desk-side machine (an ASUS Ascent GX10) ran one model (Qwen3.8-27B) through the same test: 16 facts I secretly edited in Jules Verne’s novels, asked 5 or 6 times each, on a library growing from a few chapters to 33 novels and about 5 million tokens. Five architectures, six systems (the agent in two versions), same model, same instruction, same scoring.

This part puts them side by side, and says which one I’d use for which haystack.

The scoreboard

Accuracy	T1 · chapters	T2 · 1 novel	T3 · 7 novels	T4 · 33 novels
Big context	100%	95%	❌	❌
RAG, 32 excerpts	100%	85%	79%	71%
Hybrid, 16 excerpts	100%	94%	88%	83%
Agent, grep + read	97%	91%	90%	88%
Agent + search	100%	95%	84%	88%
RLM	100%	94%	89%	91%

On one novel, everything except plain RAG lands between 91% and 95%. On 33 novels, the systems that search for themselves hold up best.

Each figure is 80 to 114 answers (T1: 30), so about ±9 points. Read groups, not decimals:

• On one novel, big context, hybrid search, both agents and RLM are tied. Plain RAG is behind.
• On 33 novels, big context can’t run, the agents and RLM are tied at the top (88–91%), hybrid search is a few points behind (83%), and plain RAG is clearly last (71%).

Speed and cost

	Big context	RAG	Hybrid	Agent, grep + read	Agent + search	RLM
Preparation	none	index: 30 s to 3 min 39 s	RAG’s index + keywords (seconds)	none	hybrid’s index	none
Extra running	none	embedding model	embedding model + reranker	none	embedding model + reranker	sandbox container
Largest library	1 novel	33 novels	33 novels	33 novels	33 novels	33 novels
Median answer, 33 novels	—	12 s	9 s	26 s	11 s	44 s
Slowest 10%, 33 novels	—	32 s	35 s	2 min 35 s	40 s	1 min 36 s
Tokens read per question (median)	236,000 (1 novel)	15,300	7,700	10,600	5,600	11,000
Tokens written per question (median)	170 (1 novel)	210	180	480	190	840
Accuracy, 33 novels	—	71%	83%	88%	88%	91%

Tokens are on 33 novels unless noted; for the agents and RLM they add up every model call of the question, reasoning included.

Big context on one novel: 6 min 20 s for the first question (reading 236,000 tokens), then about 15 s per question while the novel stays in the model’s cache.

Accuracy costs seconds. The agent with search sits at the best compromise: hybrid-search speed, agent-level accuracy.

How each one fails

The accuracy figure says how often. For a business system, how matters as much.

	Typical failure	What it looks like
Big context	Facts in the middle of the book	“NOT FOUND”: 4 of its 5 misses on the clean novel
RAG	Facts worded differently from the question; distractors from other books	Confident wrong answers as the library grows: “the Maelström”, “1866”
Hybrid	Right passage, wrong reading	NOT FOUND with the answer in the prompt (the reward); both names (the Maelström)
Agents	Found it, then added memory	“La Thalassie (l’Atlantide de Platon)”; thinking that runs out; losing the question
RLM	Skips the search, or writes a buggy one	Answers from memory on turn 1 (6 times in 160 on large libraries); a nearby date from the right chapter

Two patterns cross every part.

The better the search, the more you see the model’s memory. Follow Atlantis, renamed Thalassie in my text. RAG and hybrid search never found the passage: 0 correct. The agents found it every time and mostly answered “Thalassie (l’Atlantide)”, adding the name they learned in training. RLM found it and answered plain “Thalassie” 9 times out of 15. Once retrieval works, the remaining errors come from the model, not the search.

Errors got better-behaved as the architectures improved. RAG’s wrong answers were plausible values: a wrong year, the original name. From hybrid search on, most errors were hedges (both names), NOT FOUNDs or non-answers. The exceptions were few: a handful of RLM answers given from memory before any search, and one date read from the right chapter but the wrong paragraph. A human can catch a hedge. A confident wrong year goes straight into a report.

Which architecture, for which haystack

One document that fits in memory (a contract, a manual, a report: up to one novel, about 236,000 tokens, on this machine) → Big context. No index, no pipeline, 95% on one novel. Pay a few minutes for the first read, then keep it in the cache for the next questions.

A large, stable corpus, many users, answers in seconds → Hybrid search (meaning + keywords + reranker). Never plain RAG: exact names, codes and dates are where meaning-only search fails, and company documents are full of them. 83% on 33 novels in 9 seconds.

A large corpus where accuracy matters more than seconds, or that changes every day → An agent with grep and read. No index to build or keep in sync: a document is searchable the moment it’s on disk. 88% on 33 novels, about 26 seconds a question. Add hybrid search as a tool if you have the index: same accuracy, 11 seconds.

Questions that need computing, not finding (count, compare, extract across thousands of places) → RLM, inside a real sandbox. On single-fact questions it behaved like an agent writing its own grep: as accurate, slower, and with code execution to secure.

And in every case: ask for the quote. Whatever the architecture, the last errors were the model blending the document with its memory. An answer with its source sentence can be checked; an answer alone can’t.

Five lessons from two weeks of runs

1. The architecture matters more than tuning it. On 33 novels, RAG’s best setting (32 excerpts) reached 71%. Hybrid search’s middle setting reached 83%, and an agent with nothing but grep, 88%. Tuning moves you along a line; changing architecture moves the line.
2. Exact words are underrated. In this test, meaning-based search never found a passage that keyword search missed. My questions use Verne’s own words, which favours keywords, but company documents are full of exact words too.
3. Run every question several times. The same configuration scored 76% and 67% on two identical runs. Several failure types (reasoning that never ends, a model that loses the question) appeared in a few percent of runs only. One run per question would have reported noise as findings.
4. Test data is code. The questions, the edits and the expected answers are built by scripts, versioned and checked before each run. A wrong edit or a too-strict expected answer looks exactly like a model failure. When a result surprises you, check your own data first.
5. Don’t assume the clever part happens. RLM’s headline feature, the model calling itself on pieces of the text, never ran in 270 questions. If your design depends on a behaviour, measure it.

What this test doesn’t tell you

• One model. Every figure is Qwen3.8-27B on one machine. Another model, especially a larger one, could reorder the agents and RLM.
• Questions in the books’ own words. That favours keyword search, grep and RLM’s regex. Questions phrased the way users actually talk would favour meaning-based search more.
• Single facts only. No question needed two facts joined, a count, or a summary. Those would change the picture, probably in favour of agents and RLM.
• Edited facts in a famous text. That was the point: it separates reading from remembering. But it also makes memory a stronger trap than in most company documents.
• No tuning per architecture. Same prompt rule, same budgets, simple loops. A production system would tune each one.

What I’d test next

• A second model on the same bench, to see which results belong to the architecture and which to Qwen.
• Questions in other words than the text, the real test for meaning-based search.
• Neighbouring excerpts in hybrid search: Atlantis was one excerpt away.
• An RLM that has to delegate, with questions that need reading everything, to test the recursion this series never saw.
• Multi-fact and aggregation questions, where I expect the ranking to change.

The short version

On a machine that fits on a desk, with documents that never leave it, a 27-billion-parameter model found edited facts in 33 novels about 9 times out of 10. The choice of architecture made the difference between 71% and 91%, and between 9 seconds and 44.

For most companies, the answer starts with hybrid search for speed, an agent with grep when accuracy matters more, and quotes for every answer, whichever you choose.