Study and demonstrator
Agentic search: let the model look
Needle in a haystack, part 5 of 7. No index, four tools, and a model that decides where to look.
Series
Where hybrid search left us
Hybrid search brought the right passage to the model for 15 of my 16 edited facts, on one novel and on 33. Two problems were left.
Atlantis (renamed Thalassie in my text) was never answered: 0 out of 60. The passage that matches the question describes the continent; the name is a few paragraphs earlier, in another excerpt. The answer was one excerpt away.
The reward was in the prompt, ranked 2nd to 4th, and the model still said NOT FOUND 16 times out of 30 on 7 and 33 novels.
A human reader would solve both by looking around: reading the page before, searching for another word. So in this part, the model gets to do that.
How it works here
No pipeline this time. The model gets the question, a few tools, and up to 25 tool calls. It decides what to search, what to read, and when it has the answer.
In library terms: no librarian and no catalogue. The reader walks the shelves with a search function and opens the books themselves.
The tools are four plain functions over the same text the other parts used:
| Tool | What it does |
|---|---|
list_works |
the novels in the library, with their number of chapters |
list_chapters(work) |
chapter titles of one novel |
grep(pattern) |
finds a word or pattern everywhere, ignoring case and accents; returns a short snippet per matching paragraph, with novel, chapter and paragraph number |
read(work, chapter, start) |
reads up to 30 consecutive paragraphs |
The model never runs code or commands; it can only call these four functions.
I ran two versions:
- Agent, grep + read: the four tools above. No index at all, no embedding model, nothing prepared in advance.
- Agent + search: the same, plus a fifth tool,
search(query), which is part 4’s hybrid search (meaning + keywords + reranker). Each passage comes with its location, so the agent canreadaround it.
The loop is about 150 lines in my test harness, not an agent framework, so nothing hidden changes between parts. Same Qwen3.8-27B model, same instruction (“answer only from the text, or say NOT FOUND”), same reasoning settings, same 16 facts, same four corpus sizes, same scoring, 5 runs per question.
One line is added to the instruction: the library may have been edited, so trust only what the tools return. That’s the honest version of a real deployment: you’d tell an agent the same about your own documents.
Budgets: 25 tool calls and 60,000 tokens of prompt per question. Running out counts as NOT FOUND. In practice, neither limit was ever reached.
Results
| Accuracy | T1 | T2 | T3 | T4 |
|---|---|---|---|---|
| Big context | 100% | 95% | ❌ | ❌ |
| RAG, 32 excerpts | 100% | 85% | 79% | 71% |
| Hybrid, 16 excerpts | 100% | 94% | 88% | 83% |
| Agent, grep + read | 97% | 91% | 90% | 88% |
| Agent + search | 100% | 95% | 84% | 88% |
Both agents end at 88% on 33 novels. The agent with only grep and read is the most stable line from one novel to 33.
Each agent figure is 80 answers (T1: 30), so about ±9 points. The differences between the agents and hybrid search are within noise at every size. The trend is what matters: the agent loses less as the library grows.
1. The agent finds the passage almost every time
For every question, I recorded whether the passage holding the answer appeared in anything the tools returned.
| Answer passage seen by the agent | T1 | T2 | T3 | T4 |
|---|---|---|---|---|
| Agent, grep + read | 97% | 96% | 96% | 94% |
| Agent + search | 100% | 100% | 99% | 99% |
And quickly. With grep and read, the median question took 2 to 3 tool calls. With search, 1: on 1, 7 and 33 novels, 61% of questions were answered after a single tool call. The longest search took 22 calls, on 33 novels, still under the budget of 25.
The pole date shows why grep works so well on this kind of question. With grep and read, the agent answered “25 mars” correctly 15 times out of 15 on 1, 7 and 33 novels, and 11 of those took a single grep: the passage (“je prends possession de cette partie du globe”) came back with the first search. That’s what keyword search did in part 4, with the model choosing the keyword itself.
2. The reward: solved by looking around
On 7 and 33 novels, hybrid search gave the model the reward passage and the model said NOT FOUND more than half the time.
Each agent answered it correctly 10 times out of 10 on 7 and 33 novels. With grep and read, it took between 4 and 13 tool calls and 25 seconds to 3 minutes. The traces show the same pattern each time: grep “prime”, get hits from several novels, grep “monstre” or “signalera”, then read the right chapter of Vingt mille lieues sous les mers, and often a last grep for “quatre mille cinq cents” to confirm before answering.
Why the agent is more confident than hybrid search with the same passage, I can’t prove from this test. One difference is visible: every grep hit comes with its novel and chapter, so the agent can tell the reward in Vingt mille lieues sous les mers from the other rewards in other novels, and it checks before answering. With hybrid search, those passages arrived together, as excerpts without much context.
3. Atlantis: the search problem became a memory problem
Atlantis was the fact no search could find. The agent found it every time: the name Thalassie appeared in a tool result in 15 runs out of 15 on 1, 7 and 33 novels, for both agents. It took a median of 7 tool calls with grep and read, 4 with search.
How it searched is telling. In 25 of the 30 runs, at some point the agent grepped for “Atlantide” or “Atlantis”, and in 14 it was the very first search. The model was looking for the name it remembered. That name isn’t in my edited novel, so grep found nothing there, and the agent moved on to words like “continent” and “disparu”, then read the chapter “Un continent disparu” until it found Thalassie. The memory pointed the search the wrong way; the tools corrected it.
And then the answer was still mostly scored wrong: 5 correct out of 15 with grep and read, 3 out of 15 with search.
The answers look like this:
La Thalassie (l’Atlantide de Platon).
La Thalassie (le continent submergé d’Atlantide).
The agent read the right passage, gave the right name, and then added the one it remembers. The passage says “la Thalassie de Platon”, and the model completes it from what it learned in training. 10 of the 15 answers with grep and read, and 12 of 15 with search, named both.
The memory didn’t come from the text. “Atlantide” never appears in my edited Vingt mille lieues sous les mers, and in only 2 of the 30 runs did any tool return it at all: a passage from another novel in the 33-novel library (“les Atlantides” of Plato, in an astronomer’s theory).
My scoring counts an answer with both names as wrong, as in every part. For a reader, “La Thalassie (l’Atlantide de Platon)” is arguably a good answer. For a system that must report only what the documents say, it isn’t: the second half came from memory, and nothing in the answer tells you which half.
That’s the most interesting result of this part. Hybrid search hid this behaviour, because it never got the model that far. The better the search, the more you see what the model does with the passage.
4. Search brings back the lookalikes
On the Maelström (renamed Corryvreckan), the two agents split:
| Maelström → Corryvreckan, correct | T1 | T2 | T3 | T4 |
|---|---|---|---|---|
| Agent, grep + read | — | 3/5 | 3/5 | 5/5 |
| Agent + search | — | 4/5 | 0/5 | 0/5 |
(— : the Maelström comes at the end of the novel, so it isn’t in the opening chapters and isn’t asked at T1.)
With search, on 7 and 33 novels, the answer was almost always “Le Corryvreckan (le Maelström)”, and twice just “le Maëlstrom”. Hybrid search returns the best passages from all novels, and several other Verne novels use “maelström” for any whirlpool. The model sees both words and hedges, as in part 4.
With grep, each hit shows which novel it comes from, so the agent can stay in Vingt mille lieues sous les mers. On 33 novels it got the Maelström right 5 times out of 5. (On one novel, it twice answered with the passage’s nickname for the whirlpool, “le Nombril de l’Océan”, instead of the name.)
So the fifth tool made the agent faster and steadier on one novel, and worse on 7 novels (84% against 90%). It’s the same trade-off as with any search: what it brings back that looks relevant, the model has to sort out.
5. What went wrong, once the passage was found
On 1, 7 and 33 novels together, each agent gave about 25 wrong answers out of 240 (10–11%).
- Named the original too (14 and 24). Atlantis (10 and 12) and the Maelström (4 and 11), as above, plus one on the pearl fishery: “golfe de Kalpitiya (golfe de Manaar)”. With search, two Maelström answers gave the original name alone.
- No answer: thinking ran out (10, grep + read only). Every time right after the first tool call, the model reasoned for the whole output allowance (4,096 tokens, the same in every part) and never answered. Each took about 2 min 40 s. It never happened with the search tool available. I don’t know why; it’s the kind of failure you only see by running each question several times.
- Lost the question (1 and 3). After a long tool result, the model answered as if the user had pasted a text: “Vous avez collé un long ensemble d’extraits de romans de Jules Verne…”, or, once, in English, “I don’t have any previous thinking to translate”. The question was still there; the model lost track of whose turn it was. The one case with grep and read was different: the model wrote its next search as plain text (“I need to search for the specific content.”) instead of calling the tool, so the loop took it as the answer.
Apart from those two “Maëlstrom” answers, no answer gave a plain wrong value: no wrong year, no wrong name alone. In part 3, RAG answered “1859” or “1866” for the date. Here, every error was a hedge, a non-answer or a confusion.
Cost and speed
| Hybrid search | Agent, grep + read | Agent + search | |
|---|---|---|---|
| Preparation | embeddings + keywords: 30 s to 3 min 39 s | none | same as hybrid |
| Extra model running | embedding model + reranker | none | embedding model + reranker |
| Tool calls per question (median; slowest 10%) | — | 2–3; 5–10 | 1; 2–4 |
| Median answer time | 8–11 s | 16–26 s | 11–12 s |
| Slowest 10% of answers | — | 42 s to 2 min 35 s | 27–40 s |
| Text in the prompt | 4,000–15,000 tokens | up to 33,000 tokens | up to 23,000 tokens |
| Tokens read per question, 33 novels (median, all calls) | 7,700 | 10,600 | 5,600 |
| Tokens written per question, 33 novels (median; slowest 10%) | 180; 780 | 480; 3,500 | 190; 720 |
| Accuracy, 1 / 33 novels | 94% / 83% | 91% / 88% | 95% / 88% |
The agent with grep and read needs nothing prepared and nothing extra running. On 33 novels, it’s the cheapest system to set up and the slowest to answer: the median is 26 seconds, and one question in ten takes more than two and a half minutes, mostly the runs where the thinking ran out.
The tokens tell the same story. A typical agent answer writes about 480 tokens, reasoning included, across all its calls: 2.5 times hybrid search. The slowest tenth writes 3,500 or more, the runs where reasoning used the whole allowance. With the search tool, the agent reads and writes about as little as hybrid search alone.
Adding search halves the time and the number of calls, and keeps the accuracy on 33 novels. It costs the whole hybrid index.
What I’d tell a client
When an agent is worth it
- When the corpus changes often, or you can’t index it. Grep needs no preparation: new documents are searchable the moment they’re on disk.
- When the question needs looking around: a fact split across two paragraphs, a name defined a page earlier, several candidates to compare. That’s where the agent fixed what hybrid search couldn’t.
- When you can wait 20 seconds to 2 minutes per question. An agent is a research assistant, not a search box.
What to watch
- Better search exposes the model. Once the agent finds the passage, what’s left is the model mixing the documents with what it remembers. Ask for quotes, or check answers against the passage, if that matters.
- Give the agent a scope. A global search tool returns lookalikes from every document. Grep with a location lets the agent stay in the right one.
- Run every question several times. Two of the failure types here (thinking that never ends, a model that loses the question) appeared in a few percent of runs only. One run per question would have missed them.
Limits of this test
- Questions in the novel’s vocabulary. As in part 4, my questions use Verne’s words, which favours grep. A question phrased differently from the text would need the agent to guess other words, or the search tool.
- One model. Qwen3.8-27B with its own tool-call format, served by vLLM. Tool calling quality varies a lot between models; another model could loop, give up early, or follow the instruction better.
- A simple loop. No planning step, no self-check, no memory between questions. Agent frameworks add these; I kept them out so the comparison stays fair.
- Five answers per point. About ±9 points. The agents and hybrid search are within noise of each other at every size.
- “Seen” means the edited answer appeared in a tool result, not that the model read it carefully.
- Counterfactual facts only, as in the earlier parts.
Next: RLM
The agent called tools one at a time, and each call’s result went back into its prompt. On 33 novels that worked, because grep returns short snippets.
Recursive Language Models go one step further: the model writes code to explore the text, and can call itself on pieces of it. That’s the last architecture of this series, and the one most likely to break on a local model. → Part 6: RLM