Study and demonstrator
Big context: just read everything
Needle in a haystack, part 2 of 7. Can a local model find one edited fact in a 235,000-token novel?
Series
The simplest idea first
You have a large document and one small fact hidden somewhere inside it. The most obvious architecture is no architecture at all: give the model the whole document and ask the question.
No index, no chunking, no embeddings, no retrieval pipeline to tune. If the model’s context window is large enough, it sees everything.
Cloud models with million-token windows have made this sound solved. The classic “needle in a haystack” benchmark, where one sentence is hidden in a long text, is close to 100% for frontier models.
But I wanted to know what happens on hardware a small company can actually own. One desk-side machine, one open-weight model, no cloud. That’s the question behind this series, and big context is the baseline every other approach will be compared to.
The setup
Hardware. An ASUS Ascent GX10: NVIDIA GB10 chip, 128 GB of memory shared between CPU and GPU. It sits on my desk.
Model. Qwen3.8-27B, quantised to NVFP4, served by vLLM 0.25.1. The settings that matter for this test:
- context window of 262,144 tokens;
- KV cache stored in fp8 (the memory that holds the “read” document, compressed);
- prefix caching on, so a document read once can be reused for the next question;
- reasoning mode on, with the same output budget (4,096 tokens) for every question.
Haystack. Jules Verne’s Vingt mille lieues sous les mers, in French. Two sizes:
| Tier | Content | Size |
|---|---|---|
| T1 | Opening chapters | 26,400 tokens |
| T2 | The whole novel | 235,700 tokens |
T2 fills 90% of the context window. It’s about as big as this setup can go. The larger tiers of the series (7 and 33 novels) don’t fit at all.
Building needles the model can’t cheat on
Here’s the problem with using a famous novel: the model has probably read it. Ask “who commands the frigate chasing the monster?” and it may answer “Farragut” from memory, without reading a word of the text you gave it.
So I didn’t add needles to the book. I changed facts that are already in it, everywhere they appear:
| Original (Verne) | Edited | Depth in the book |
|---|---|---|
| 1866 | 1858 | ~0% |
| Farragut | Whitcombe | ~0% |
| Scotia | Thessalia | ~1% |
| Crespo | Moreira | 25% |
| Vanikoro | Rotuma | 33% |
| Manaar | Kalpitiya | 49% |
| Arabian-Tunnel | Sinaï-Tunnel | 56% |
| Vigo | Ferrol | 65% |
| Atlantide | Thalassie | 68% |
| Maelström | Corryvreckan | 99% |
Sixteen edits in total: six in the opening chapters, ten spread from 25% to 99% of the book.
Each edit is a small regex rule applied to the whole text, so the book stays consistent: the Nautilus is drawn into the Corryvreckan every time it’s mentioned, not just once.
This gives each answer three possible meanings:
- context: the model gives the edited fact (Whitcombe). It read the text. Correct.
- parametric: the model gives Verne’s original (Farragut). It answered from memory and ignored the text.
- mixed: both (“Atlantis, also called Thalassie”). It saw the text but couldn’t let go of what it knew.
Anything else, including “NOT FOUND”, is a miss.
The baseline: what does the model already know?
Before giving it the book, I asked every question with no text at all, telling the model to answer from memory or say it doesn’t know.
It knows some facts well: Atlantis and the Maelström (3 out of 3), Farragut, the Scotia, Vanikoro, the Manaar gulf (2 out of 3). It doesn’t know others: the frigate’s name, the reward, the island where Nemo goes hunting.
That matters. When the model gives “Corryvreckan” with the book in front of it, the baseline proves it overrode something it knew. When it gives “Moreira”, it only proves it found something new.
Without the book, it never gives an edited answer: 0 correct out of 132 no-context answers. The edits can’t be guessed.
Repeats
Each question was asked 6 times, at temperature 0.6, with the same prompt and settings every time, on the opening chapters and on the whole novel.
One run per question isn’t enough: the same setup can miss a fact once and find it the next time.
Results
Accuracy by position in the book
T2, the whole novel (Atlantis, at 68%, is discussed below):
| Depth in the book | Needles | Correct |
|---|---|---|
| Start (0–3%) | 6 | 36 / 36 · 100% |
| First half (25–35%) | 3 | 18 / 18 · 100% |
| Middle (49–65%) | 3 | 11 / 15 · 73% |
| End (81–99%) | 3 | 18 / 18 · 100% |
The start, the first half and the end are perfect. The middle is the only band that misses.
The start and the end are perfect. The middle is where the misses are.
This is the “lost in the middle” effect researchers have described for long contexts: models attend more strongly to the beginning and the end of a long input than to what sits in between. Here it shows up on a quantised model, on a single machine, with a real book rather than a synthetic filler text.
How solid is it? Every single miss was between 49% and 68% of the book: the Manaar gulf twice, the bay of Vigo twice, and one hesitant Atlantis. Everything before and after was 100%.
Six runs per needle is a small sample, so I won’t claim a precise figure for the middle. But the pattern is clear.
On the smaller tier (T1, 26,400 tokens), every answer was correct.
When it misses, it says so
This is the result I didn’t expect, and probably the most useful one.
Almost every miss was “NOT FOUND”. The model didn’t invent an island or a ship’s name: 4 of the 5 misses were NOT FOUND; the fifth named both continents (“Thalassie, Plato’s Atlantis”).
For a business use case, that matters more than the accuracy figure. A system that says “I didn’t find it” can be caught and retried, or sent to a human. A system that confidently invents an answer can’t.
It trusts the text over its memory
The counterfactual needles were designed to catch the model answering from memory. It almost never did.
The Maelström is the clearest case. Without the book, the model answers “Maelström” every time. With the edited book, it answers “Corryvreckan” every time, even though it knows the original very well.
Atlantis is the one hesitation: 5 answers out of 6 gave Thalassie; the sixth named both continents.
The cost of reading everything
| T1 (26,400 tokens) | T2 (235,700 tokens) | |
|---|---|---|
| First question (full read) | ~21–26 s | ~374–382 s (over 6 minutes) |
| Reading speed | ~1,250 tokens/s | ~630 tokens/s |
| Next questions, time to first word | 1.7 s | 5.3 s |
| Next questions, full answer | median ~7 s | median ~15 s (10–40 s) |
Two things stand out.
Reading slows down as the document grows. At 235,000 tokens, the model reads at half the speed it reads at 26,000. Attention compares every token with every other one, so the cost grows faster than the length.
The prefix cache changes everything. The first question pays the full six minutes. Every following question reuses the already-read book: 233,600 of 235,700 tokens came from the cache (99.1%). Six minutes once, then about 15 seconds per question.
So big context is viable when you ask many questions about the same document. It’s a poor fit when every question comes with a different large document: you pay the six minutes every time.
What I’d tell a client
When big context fits
- The document fits comfortably in the window (here, up to ~235k tokens).
- You’ll ask many questions about the same document, so the first read is paid once.
- You need a first version quickly: no index, no pipeline, nothing to tune.
- A “not found” answer is acceptable and can be handled.
When it doesn’t
- The corpus is larger than the window, or keeps growing. At 236k tokens this setup was already at 90% of its limit.
- Every request brings a different large document: you pay the full read each time.
- The critical fact may sit in the middle of the document and a miss is costly. Perfect at both ends, noticeably less in the middle.
- Latency matters on the first question. Six minutes isn’t interactive.
Big context is the right baseline: simple, honest, surprisingly accurate. But it hits a hard ceiling on size, and its reliability isn’t uniform across the document.
Limits of this test
To be fair to the results, and to anyone who wants to reproduce them:
- One model, one quantisation, one book. NVFP4 weights and an fp8 KV cache. A higher-precision KV cache might reduce the middle drop; that comparison is still to run.
- Three needles in the middle band. One difficult needle weighs a lot. More needles between 40% and 70% would make the curve firmer.
- The sample is small: 6 answers per needle. Enough to see where the misses are, not enough for a precise rate.
- Counterfactual edits only. No questions yet where the answer needs two facts from different chapters, or where the question shares no words with the text.
Next: RAG
Big context reads everything, every time. The next article tests the opposite idea: cut the library into pieces, index them, and only show the model the few pieces that look relevant to the question.
It does scale far beyond 235,000 tokens: up to 33 novels on the same machine. It also misses facts that big context finds every time, and, oddly, gets right the two that big context struggles with. → Part 3: RAG