Nvidia publishes a preprint that buries the context window race: compaction beats long-context by 3.6 to 7x on cost
🔎 The race for giant context windows just received its bill
For the past three years, context window size has served as the unit of measurement for LLM progress. 128K, 1M, 2M tokens: every model announcement rolls out its figure, and agent benchmarks have aligned themselves with ever-longer conversation histories. Bigger = better — such was the implicit reasoning.
On November 3, 2026, Nvidia publishes a 31-page preprint with an unambiguous title: You do not need long context. Thirteen authors, including Peter Belcak and Pavlo Molchanov, 13 benchmarks, 9 models put under scrutiny. Their thesis: for agents, filling the window is almost always a waste — financially first, technically second.
The figure that sums it all up: on an H100 billed at $3.85/hour (Nebius, September 2026), serving a 256K context instead of 16K costs 3.6 to 7x more per generated token depending on the model. And the accuracy of the best models holds — sometimes even improves — with working contexts of 12 to 16K tokens and frequent compaction. This paper doesn't propose an optimization: it attacks the metric itself.
Key takeaways
- Cost explodes with the window: at 256K context, generating a token costs 3.6x more for GLM 5.2, 5.6x for Kimi K2.6, and 7x for Qwen 3 Coder Next than at 16K (H100 at $3.85/h, Nebius, September 2026).
- Accuracy holds up: across the 22 configurations tested (Claude Opus 4.5 and GPT 5.6 Sol, 6 agent benchmarks), at most 5 lose more than 2 points compared to the full history — some actually gain.
- Selective reading beats full ingestion: on LOONG (1,600 multi-document questions), Opus 4.5 at 16K/12K reaches an average of 79.3% versus 70.1% with full-input.
- The savings come with caveats: the spectacular gains (up to 54.6x) rest on the authors' pricing model; with caching and nominal prices, Opus 4.5's advantage on SWE-bench Verified drops from 2.2x to 1.4x.
- Smaller models can't keep up: Qwen 3 Coder Next collapses from 53.0% to 3.6% on SWE-bench Verified at the tightest compaction threshold.
Recommended tools
| Tool | Main use | Price (November 2026) | Ideal for |
|---|---|---|---|
| alphaXiv | Read the preprint and its reviews | Free | Follow the community's critique |
| mini-swe-agent | Minimalist harness used in the study | Free (open source) | Reproduce the compaction tests |
| Ollama | Open-weight models locally | Free | Prototype without an API bill |
| Hostinger | VPS for orchestrating your agents | From ~€3/month (November 2026, check hostinger.com) | Host a harness 24/7 without a GPU |
No miracle framework to report: the study was conducted with mini-swe-agent v2.4 and a home-made agent equipped with grep and sed. That's the level of complexity required — and it's minimal.
What Nvidia's preprint really says
The paper doesn't say that long context is useless. It says that agents almost never need it as working memory, and that frequent compaction coupled with selective reading does the same job for a fraction of the cost.
The team evaluated 9 models on 13 benchmarks, split into two families. Six long-horizon benchmarks — SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.1, FRAMES, BFCLv3, Tau2Bench — measure agent loops spanning dozens of steps. Seven long-input benchmarks — LongBench v2, LOONG, AA-LCR, NIAH, RULER, Counting-Stars, BABILong — measure the exploitation of very long documents.
On the model side: Claude Opus 4.5 and GPT 5.6 Sol represent the closed frontier; GLM 5.2, Kimi K2.6, Qwen 3 Coder Next, Qwen 3.6 27B, Gemma 4 31B, Step 3.7 Flash and MiniMax M2.7 cover the open-weight side. The harness fits in one line: mini-swe-agent v2.4, plus an in-house agent whose only tools are grep and sed for digging through files.
Three strategies tested
- End-of-Conversation Compaction: the history is compacted at the end of the loop, into a summary that is fed back as context.
- Single-Action Compaction: compaction happens after every action. The study shows you can go down to one action per context without the model collapsing.
- Selective reading: long documents are exposed as files that the agent browses excerpt by excerpt, instead of being injected wholesale into the prompt.
The paper ends with a research proposal, semantic state space models (SSSMs): models designed to maintain a compact semantic state rather than an exhaustive window. A program, not a product — but it says where Nvidia wants to go.
256K instead of 16K: up to 7x the cost per generated token
The longer the context, the more expensive each generated token becomes — and the gap isn't linear, it's multiplicative. That's the economic core of the paper, and the figure that announcements of giant context windows systematically leave out.
The mechanism boils down to one sentence: a token generated at step 500 of a conversation must attend to the 499 that came before. The authors show that late tokens are structurally more expensive to produce, and that on state-of-the-art open-weight models, the gap between 16K and 256K reaches 7x.
| Model | Extra cost per generated token at 256K vs 16K (H100) |
|---|---|
| GLM 5.2 | 3.6x |
| Kimi K2.6 | 5.6x |
| Qwen 3 Coder Next | 7x |
Billing assumption: H100 at $3.85/hour (Nebius, September 2026). Source: arXiv preprint 2611.05306.
Long context also comes at a price in latency. On a 128 GB DGX Spark, inter-token latency roughly doubles over the 256K range, and MiniMax M2.7 saturates as early as 60K of context. On local hardware, the giant window isn't just expensive: it becomes unusable.
This is where the race to 1M tokens reveals its true nature: an economy disguised as a technical feat. You're sold a window; you pay for every token generated inside it. We documented this in Tokens, context, costs: understanding LLM billing: billing follows the structure of attention, not your intentions. And as The long-context impossibility triangle explains, no model can maximize length, quality, and cost all at once. Nvidia provides the numerical proof of the third side of the triangle.
Compaction: accuracy holds, and sometimes climbs
On frontier models, compacting frequently costs zero to a few accuracy points — and on long-document tasks, it actually gains some. This is the hardest result to swallow for full-context advocates.
Across the 6 agent benchmarks, the authors compile 22 configurations (model × benchmark × strategy). Bottom line: at most 5 configurations lose more than 2 points compared to the full-history baseline, and some gain points. The big picture: Opus 4.5 stays within -3.7 to +5.6 points of its baseline, GPT 5.6 Sol within -3.3 to +4.4 (breakdown Learn LLM, November 11, 2026).
| Test | Baseline | With compaction / selective reading |
|---|---|---|
| LOONG — Opus 4.5, average (full-input → 16K/12K) | 70.1% | 79.3% |
| LOONG — Opus 4.5, perfect responses | 38.9% | 53.4% |
| SWE-bench Verified — Opus 4.5, one action per context | 70.0% | 70.2% |
| SWE-bench Verified — GPT 5.6 Sol, one action per context | baseline | -2.0 points |
| SWE-bench Verified — Qwen 3 Coder Next, tight threshold | 53.0% | 3.6% |
The most counter-intuitive result concerns selective reading. On LOONG — 1,600 multi-document questions — Opus 4.5 with 16K of generation context and 12K of reading reaches 79.3% average and 53.4% perfect responses, versus 70.1% and 38.9% in full-input. Giving the model less context improves its answers: less noise, more focus.
Even in degraded mode — a single action per context — Opus 4.5 matches its baseline on SWE-bench Verified (70.2% vs 70.0%), while GPT 5.6 Sol concedes 2.0 points (analysis trackfuture.ai, November 5, 2026). Compaction is not lossless compression, but it is far from the degradation one might have imagined.
What a compaction summary should contain
The study details the recipe for a useful summary, in four blocks:
- The verbatim copy of the task — never paraphrased, it's what anchors the agent.
- The current state and the plan — what's done, what remains, in what order.
- The relevant artifacts — code snippets, command outputs, file paths.
- The completion criteria — the exact stopping condition.
A summary that paraphrases the instruction or loses the exit criteria produces an agent that goes in circles. The summary's content matters more than its size.
The limitations: what the preprint doesn't prove
This is a preprint with no published code and no independent replication, and its most spectacular savings rest on a theoretical pricing model rather than on actual invoices. Might as well say so before the triumphant headlines.
First point, factual: no code, no independent replication to date (i10X News, November 4, 2026). The numbers are detailed and consistent, but they haven't yet been reproduced outside Nvidia.
Second point, economic: the advertised gains of up to 54.6x on long-input are calculated using the authors' pricing model. At nominal prices, the picture changes noticeably:
| Task | Model | Full-history cost | Cost with compaction |
|---|---|---|---|
| SWE-bench Verified | Claude Opus 4.5 | $420.00 | $295.75 |
| SWE-bench Verified | GPT 5.6 Sol | $167.68 | $188.96 |
| LOONG | Claude Opus 4.5 | $1,367.50 | $502.10 |
On SWE-bench Verified, Opus 4.5's savings are real but modest. For GPT 5.6 Sol, the cost with compaction exceeds that of the full history: on this test, compaction costs money. On LOONG with Opus, on the other hand, the near-threefold reduction holds up.
Third point, technical: caching. The full history reuses the cache at one-tenth of the price. On SWE-bench Verified, Opus 4.5's savings then drop from 2.2x to 1.4x. Still favorable, but far from the headlines.
Last point, and not the least: small open-weight models don't survive aggressive compaction. Qwen 3 Coder Next drops from 53.0% to 3.6% on SWE-bench Verified at the tightest threshold. Tight compaction is a frontier-model privilege.
My take: these limitations don't invalidate the thesis, they draw its boundaries. Compaction is a strong cost lever on very-long-document tasks, modest on short agent loops, and a trap on small models. That's already far more precise than "more context = better".
What this changes for agent builders
Stop sizing your agents to the model's context window: size them to their working memory instead, and give them tools to fetch the rest. Four concrete decisions come out of this study.
- Aim for 12 to 16K of working context. That's the range tested by the authors, the one where Opus 4.5 wins on LOONG. Beyond that, you're paying for attention you never consume.
- Expose long documents as files, not as prompt. grep and sed beat massive context injection on LOONG. An agent that knows how to dig is worth more than an agent that swallows everything.
- Compact after every action, using the four blocks described above — with the task always kept verbatim.
- Reserve tight compaction for frontier models. To make the call, our comparison of the best LLMs for AI agents details these trade-offs; for orchestration, see our selection of the best autonomous AI agents.
On the infrastructure side, nothing changes: an agent harness only needs a small VPS — Hostinger is enough for orchestration (November 2026, check pricing on their site), with inference remaining with your API provider. And to prototype your compaction summaries without an API bill, open source AI agents with local Ollama work perfectly well — keeping in mind that smaller models are more fragile there.
Last piece of advice, and the most valuable one: log the cost per generated token of your agent loops, not just the token count. That's the metric this paper turns into a must-track.
What if you really do need long context?
There are still legitimate use cases — but then treat the window as a cache to optimize, not a reservoir to fill.
Tasks requiring genuinely global attention are becoming rarer as benchmarks get finer-grained: even on tests designed for long input, selective reading wins. What's left are cases where exhaustive exploration is the product itself — full code audits, contract review — and even those can often be sequenced into successive passes.
If you have to serve long context, optimize inference first before buying tokens. Training-free techniques like KV-Fold compress the KV cache without retraining: the right lever when the window is an imposed constraint, not a choice.
The labs, for their part, won't slow down: the window remains a marketing argument and a genuine asset for consumer products. Gemini 3.1 Pro, GPT-5.5, and Claude Opus 4.7 will keep stacking up context tokens, and that's great for assisted chat. For your agents, the question is no longer "what's the largest window?" but "how much does my generated token cost?". And if Nvidia pushes its semantic state space models all the way to a pre-trained model for this use case, the answer could change in nature.
❌ Common Mistakes
Mistake 1: Confusing the context window with working memory
A 1M token window is a capacity, not a goal. Filling it by default means paying 3.6 to 7x more per generated token with no accuracy gain. Solution: a working set of 12-16K, and selective reading tools for the rest.
Mistake 2: Compacting by paraphrasing the task
A summary that rephrases the instruction loses the details that keep the agent running: exact paths, constraints, exit criteria. Solution: verbatim copy of the task, state and plan, artifacts, completion criteria — in that order of priority.
Mistake 3: Extrapolating the paper's savings to your bill
The 54.6x assumes a pricing model without caching. With a cache reused at one-tenth the price, Opus 4.5's savings on SWE-bench Verified drop to 1.4x — and GPT 5.6 Sol can end up costing more with compaction than with full history. Solution: measure on your own traces before refactoring.
Mistake 4: Applying tight compaction to a small model
Qwen 3 Coder Next drops from 53.0% to 3.6% on SWE-bench Verified at the tightest threshold. Aggressive compaction assumes a model capable of reconstructing a rich state from a short summary. Solution: more context for compact open-weight models, tight compaction reserved for frontier models.
❓ Frequently Asked Questions
Is long context dead?
No. The study shows that agents rarely need it as working memory, not that the capability is useless. Selective reading and compaction cover most agent use cases; long context remains relevant for interactive document exploration and as a product selling point.
What working context should you aim for with an agent?
12 to 16K tokens, the range tested in the preprint: Opus 4.5 at 16K/12K beats its full-input baseline on LOONG. The exact figure depends on your task, but the order of magnitude is right — and the extra cost beyond it is measurable.
Are the advertised savings, up to 54.6x, real?
They rest on the authors' pricing model, not on actual bills. At nominal prices, SWE-bench Verified drops from $420.00 to $295.75 for Opus 4.5, but rises from $167.68 to $188.96 for GPT 5.6 Sol. With caching, expect 1.4x to 2.2x on this test.
Does compaction work with open-weight models?
Not the small ones. Qwen 3 Coder Next collapses to 3.6% on SWE-bench Verified at the tightest threshold, and MiniMax M2.7 saturates around 60K of context. Frontier models and large open-weight models hold up; always test before generalizing.
Where can you read the preprint and follow the debates?
On arXiv (2611.05306, published on November 3, 2026) and on alphaXiv for reviews. No official code or independent replication to date: be cautious with the extreme figures.
✅ Conclusion
The race for giant context windows was economics disguised as a technical feat: Nvidia's preprint publishes the bill — 3.6 to 7x per generated token — and an alternative that holds up on accuracy. Before choosing your next agent model, revisit our comparison of the best LLMs for AI agents — and always ask for the cost per generated token, not the window size.