Evolving Infra: the MIT-Berkeley-Georgia Tech preprint that cuts multi-agent serving latency by 4.1x
🔎 A November 9 preprint that attacks the real bottleneck of agents
On November 9, 2026, a team bringing together MIT Han Lab, UC Berkeley (Ion Stoica), Georgia Tech (Tushar Krishna), NVIDIA, OpenAI, UIUC, and Georgia Tech published a preprint on arXiv that deserves your attention: Evolving Infra, an open source LLM serving framework specifically designed for agent workloads. The number making the rounds among industry watchers: up to 4.1x reduction in end-to-end latency and 14x on P95 time-to-first-token, against Dynamo, NVIDIA's own production framework.
Why now? Because 2026 is the year workloads shifted from single-turn chatbots to multi-agent pipelines: dozens of sub-agents, asynchronous branches, contexts that live for minutes or even hours. Yet inference infrastructure — prefill/decode disaggregation, KV pools, LRU eviction — was designed for short, independent requests. The mismatch has become brutal.
Evolving Infra's thesis boils down to one sentence: stop freezing GPU allocation and KV cache management at deployment time. Adapt them at runtime, to the actual composition of the workload. And the detail that changes everything: the system does this without restarting engines and without losing already-computed cache. As AI Weekly's analysis sums up, this isn't a lab prototype benchmarked against a strawman — it's a head-to-head against NVIDIA's production serving stack.
The essentials
- Evolving Infra (preprint arXiv 2611.12304, November 9, 2026) is an open-source agent-adaptive serving system, published by a team affiliated with Georgia Tech, UIUC, UC Berkeley, MIT, NVIDIA, and OpenAI.
- Core idea: decoupling compute allocation from KV cache management. Nodes switch between prefill and decode hot, in place, in 6.2 seconds on average — over 100x shorter than a Dynamo restart.
- Reported numbers: mean E2E latency reduced by 1.15x to 4.12x, P95 TTFT reduced by 3.25x to 13.92x, throughput of 1.10x to 2.73x that of Dynamo, and lost KV reuse reduced by 81 to 95%.
- Benchmarks: coding, math, and computer-use agents, on GLM-5.2 and Kimi K2.6, up to 256 GB300 GPUs.
- Open-source code: available at github.com/mit-han-lab/EvolvingInfra, with symmetric engines built on SGLang.
- Take with a grain of salt: no absolute latency figures in the abstract, no comparison against non-NVIDIA stacks, and one run missed its SLOs due to a poorly tuned planner switch.
Recommended Tools
| Tool | Main use | Price (November 2026) | Ideal for |
|---|---|---|---|
| Evolving Infra | Adaptive agent serving, hot prefill/decode switching | Open source (free) | Teams running multi-step agent pipelines on their own cluster |
| NVIDIA Dynamo | Reference PD disaggregated serving, baseline of the comparison | Open source (free) | Stable and predictable workloads, NVIDIA ecosystem |
| SGLang | High-performance inference engine, basis of Evolving Infra's symmetric engines | Open source (free) | Serving open source LLMs like Kimi K2.6 self-hosted |
| Mooncake | Distributed KV cache pool with RDMA transfers (approach cited by the preprint) | Open source (free) | Shared L2 memory architectures across nodes |
The problem isn't the model, it's the KV cache plumbing
Direct answer: Evolving Infra's gains don't come from a better model, but from how the infrastructure handles compute and attention memory. And this bottleneck didn't exist two years ago.
In a classic disaggregated PD serving setup — the dominant architecture in production — nodes are split into two populations: those doing prefill (processing the prompt, filling the KV cache) and those doing decode (generating tokens). The problem is that this split is frozen at deployment. Each node's KV cache is tied to that node's phase: if you want to change the prefill/decode ratio because your workload has shifted, you either have to restart the node or lose its KV.
But agentic workloads don't stay still. An agent pipeline chains together reasoning turns, tool calls, and sub-agents running in parallel. The composition of the load — prefill/decode ratio, context sizes, prefix lifetimes — changes along the way. A split calibrated in the morning through offline profiling is wrong by 2 PM. And unlike multi-stream parallel processing, which parallelizes at the model level, this problem is purely systemic: it's the queue that keeps growing.
The worst enemy is LRU eviction. As explained in CASRAI's technical analysis, LRU judges a context's value by its recency — which is catastrophic for agents. An agent paused on a code interpreter, or an asynchronous sub-agent branch, sees its prefix evicted for "inactivity" even though it will be reused 30 seconds later. Conversely, tokens invalidated by compaction stay in the cache because they were recently accessed. The result: cache thrashing, redundant re-prefills, and up to 55.5% of potential cache hits lost in alternative in-place switch mechanisms (Arrow-type) that cut off access to already-cached KV.
This is exactly the problem that our article on StreamMA addressed at the reasoning protocol level: multi-agent latency doesn't add up linearly, it compounds. Every queued turn adds to the P95, and the pipeline's P95 is the max of all P95s. Evolving Infra attacks the same enemy, but from the infrastructure side.
How it works: three designs, not one
Direct answer: Evolving Infra rests on three complementary mechanisms — a compute/KV decoupling with hot switching, an agent-aware cost model that drives an online planner, and a distributed KV manager that avoids evicting what is about to be reused.
1. Compute/KV decoupling: the switch in 6.2 seconds
In Evolving Infra's architecture, the engines are symmetric: each node can do prefill or decode, and switch phases in place, without a restart. The KV cache is not a prisoner of the node: it lives in a pool of host memory readable by all nodes, with Mooncake-style RDMA transfers. Concretely, the memory architecture is hierarchical: L1 on GPU, L2 in host memory (4x greater token capacity), a shared pool across nodes.
The key figure: on a run of 36 GB300s, a phase switch takes 6.2 seconds on average — more than 100x shorter than a Dynamo node restart. During the switch, the queued requests on the node that is switching finish elsewhere, and the KV remains readable throughout the operation. That's the fundamental difference from approaches that "shut down" in order to switch.
2. A cost model that understands agents, not just tokens
Second piece: a cost model that predicts prefill/decode load while taking sub-agents and context management into account. It's not an incoming-token counter. The model knows that an agent will fork three sub-agents, that their contexts will duplicate prefixes, that compaction will invalidate branches.
This cost model serves two purposes: the planner uses it to size the prefill/decode split — and it only switches if the predicted throughput gain persists over the planning window, which prevents short bursts from triggering unnecessary switches. The router uses it to place each request on the most suitable node, arbitrating between load balancing and cache locality — the arbitration that's impossible in architectures where KV is tied to the node.
Important point, highlighted in the repo README: all adaptation is purely online, with no offline profiling. You don't calibrate anything before deployment. The system learns the workload's composition while serving it.
3. An agent-aware KV manager: eviction by value, not by recency
Third piece: eviction is no longer a dumb LRU. The KV manager knows what an agent branch is. It evicts the KV that no agent branch needs anymore: histories replaced by compaction, contexts of finished sub-agents, duplicated copies. And it explicitly retains the KV that will be reused — even if that prefix has been "inactive" for two minutes because its agent is waiting on a tool result.
This is the paper's most interesting semantic reversal: a context's value is no longer its recency, but its predictable futureness. If you followed our coverage of ChatGPT's massive conversation storage, you can see the underlying trend: conversational memory is becoming a full-fledged infrastructure asset, with its own system layer. Evolving Infra is the serving incarnation of that trend.
The whole thing is packaged with Han Lab's classic inference optimizations: NVFP4 quantization, FP8 KV cache, speculative decoding via MTP, 4-way tensor parallelism on 4xGB300 nodes.
The Numbers Against Dynamo: What the Benchmark Really Says
Direct answer: the gains are real and measured on agent workloads, but they are ratios against Dynamo — not absolute latencies, and no comparison with other stacks.
The benchmark covers three types of agents (coding, math, computer-use) on GLM-5.2 and Kimi K2.6, on up to 256 GB300 GPUs. Here's a summary of the reported numbers:
| Metric | Evolving Infra vs NVIDIA Dynamo | Detail |
|---|---|---|
| Average E2E latency | 1.15x to 4.12x faster | Up to 4.1x depending on the workload |
| P95 TTFT | 3.25x to 13.92x faster | The most spectacular gain |
| Throughput | 1.10x to 2.73x | On every agentic workload |
| Lost KV reuse | Reduced by 81 to 95% | vs classic eviction |
| Cost of a phase switch | 6.2 s (36 GB300) | >100x shorter than a restart |
The most telling run is the mutating workload: a load that switches from math to computer-use mid-run. Result: average E2E latency of 38.7 s versus 83.1 s for Dynamo and 51.5 s for the best static split — i.e., 2.15x and 1.33x. Throughput: 2.1x that of Dynamo, and 28% better than the best static split. That last figure may be the most important one: even an omniscient operator who knew the workload in advance and calibrated the perfect static split would lose 28% to online adaptation. The problem isn't finding the right split — it's that the right split doesn't exist over the duration of an agentic run.
Why is P95 TTFT so improved? Because in an agentic pipeline, each turn waits on the TTFT of the next. Reducing P95 TTFT by 14x doesn't make one turn faster: it makes the entire chain more predictable. In multi-agent setups, tail latency dominates average latency.
Who Really Suffers from Phase-Switch Overhead
Direct answer: if you're serving short, independent chat requests, this paper almost doesn't concern you. If you're running multi-step agent pipelines, it concerns you directly.
The affected teams share three signatures:
Heterogeneous and mutating workloads. A mix of coding + computer-use + math whose proportions change depending on the time of day. The static split that's optimal at 9am is suboptimal at 2pm, and the cost of constantly re-profiling + redeploying is prohibitive.
Long-lived, long contexts. Agents that keep their history for dozens of turns, fork sub-agents, share system prompt prefixes. The more expensive your prefixes are to recompute, the more LRU eviction costs you.
Tail SLOs, not averages. If your users judge the latency of the complete pipeline — a coding agent that has to deliver its diff, a computer-use agent that has to finish its task — then every point of P95 TTFT gets paid at the end of the chain.
Conversely, a consumer chatbot service with 500-token requests and a stationary workload has almost nothing to gain: its static split is already good, and the cost of operating an adaptive system isn't justified. Be honest about your workload profile before rewriting your serving stack.
Testing it: what's realistic, what's not
Direct answer: the code is open source and clonable in five minutes, but the paper's benchmarks run on GB300s — faithful reproducibility requires a budget few teams have.
The repo is available on GitHub:
git clone https://github.com/mit-han-lab/EvolvingInfra.git
cd EvolvingInfra
pip install -e .
The symmetric engines are built on SGLang, so if you already operate SGLang, the API surface will feel familiar. The benchmark models — GLM-5.2 and Kimi K2.6 — are open-weight LLMs, and Kimi K2.6 is among the models we recommend for self-hosting in our guide to LLMs for agents.
Let's be honest about the hardware. The paper's numbers come from GB300 clusters with up to 256 GPUs. You will not reproduce a -14x P95 TTFT on two A100s. That said, three realistic use cases exist:
- Validate the thesis on your own workload. The repo lets you measure your actual prefill/decode ratio and your lost KV reuse rate. If you're losing 80% of your cache hits like in the paper's benchmarks, you have a serving architecture problem, whatever your hardware.
- Hack the KV manager. The agent-aware eviction logic is the most transferable part: even on a small cluster, value-based eviction rather than recency-based improves cache hits.
- Wait for the diffusion. Ideas from a Han Lab preprint historically end up in mainstream open-source engines. SGLang and vLLM regularly integrate this type of work. Watch the release notes rather than forking prematurely.
If your goal is to experiment with agents locally without a GPU cluster, our guide to open-source AI agents with Ollama remains the realistic entry point — Evolving Infra solves a scale problem, not a getting-started problem. And if you're looking for which agents to orchestrate once the infrastructure is set up, our comparison of the best autonomous AI agents covers the OpenClaw and AutoGPT landscape.
Impact on the inference bill: KV reuse is money
Direct answer: cutting E2E latency by 4x on an agentic workload mechanically means serving more requests per GPU — and every percentage point of KV cache reuse is prefill you don't pay for.
The economic mechanism is simple. Prefill is the most expensive part of serving: processing 100,000 context tokens costs far more FLOPs than generating 1,000. In an agentic pipeline where every sub-agent starts from a shared prefix, every missed cache hit = a full re-prefill = wasted compute that you bill to no one. The preprint reports lost KV reuse reduced by 81 to 95%: translated into units sold, that's GPU capacity recovered without buying a single additional GPU.
This context lands at the worst possible time for budgets. Gartner forecasts $4 trillion in AI spending in 2027, a growing share of it going to inference infrastructure. In that regime, serving efficiency isn't an engineering topic: it's a CFO topic. A framework that divides latency by 4 on agent workloads is potentially a whole factor off the number of GPUs you need to provision to meet a given SLO.
Two honest caveats. First, the cost of switching: 6.2 seconds of switchover is partly unproductive compute — the planner only switches if the predicted gain persists, but a poorly tuned planner gets expensive (see below). Second, operational complexity: an online adaptive system has failure modes that a static split doesn't. You save GPUs; you pay in reliability engineering.
Limitations: what the preprint doesn't say
Direct answer: the results are impressive but presented favorably, and one anecdote from the paper itself shows that the adaptive system can fail where a static system would hold up.
Three caveats, two of which come from AI Weekly's critical analysis:
No absolute latency. The abstract publishes no absolute latency figures, no per-workload breakdown, and no numbers against non-NVIDIA stacks. The 4.1x and 14x ratios are ratios against Dynamo. If your baseline is already better than Dynamo, your gain will be smaller; if it's worse, it will be larger. The absolute figures from the mutation run (38.7 s average E2E latency) also make clear that we're talking about long agent pipelines, not interactive requests.
A single, well-chosen baseline. Comparing against Dynamo, NVIDIA's production framework, is the most credible choice possible — but one comparand is just one comparand. Teams running custom serving or other stacks will have to run the test themselves.
The adaptive system can get it wrong. The paper itself acknowledges it: one run missed its SLOs at 0.75 agents/min because of a single badly-timed planner switch, with a queue of 120 to 275 seconds at P95 for 10 minutes. That's the structural price of online adaptation: the planner is a decision system that can decide badly, and its errors show up precisely at moments of critical load. A static split never gets its split wrong — it's just always slightly suboptimal. Choose your poison.
Finally, reproducibility: the benchmarks rely on GB300s and NVFP4-quantized engines, a configuration few academic labs can replicate exactly. Independent numbers will come, but not right away.
❌ Common Mistakes
Mistake 1: Reading "4.1x" as a universal gain
What goes wrong: the preprint's numbers apply to multi-step agent workloads with long-lived contexts and mutating loads. On stationary single-turn chat, the expected gain is marginal — and the operational cost of an adaptive system isn't justified. The fix: measure your actual prefill/decode ratio and your rate of lost cache hits first, before touching your serving.
Mistake 2: Confusing in-place switch with lossless switch
What goes wrong: alternative in-place switching mechanisms (Arrow-style) avoid the relaunch but cut off access to already-cached KV — up to 55.5% of potential cache hits lost, according to CASRAI. "No restart" doesn't mean "no cost." The fix: verify that the approach you're evaluating preserves access to KV during the switchover, the way Evolving Infra's shared host memory pool does.
Mistake 3: Keeping an LRU for agent context eviction
What goes wrong: LRU evicts by recency, which penalizes exactly the agent contexts — branches paused on a tool call, asynchronous sub-agents — and retains tokens invalidated by compaction. The fix: if your serving exposes the eviction policy, replace it with agent-aware eviction that reasons in terms of live branches and future reuse, even without adopting the full framework.
❓ Frequently Asked Questions
Does Evolving Infra replace Dynamo or vLLM?
Not today. It's a research preprint with an open source repo, whose engines are built on SGLang. The ideas — hot phase switching, agent-aware KV manager, online planner — have a good chance of migrating to mainstream engines in the months that follow, as is the usual pattern for Han Lab work.
Which models are supported?
The preprint's benchmarks run on GLM-5.2 and Kimi K2.6, two open weight models suited to self-hosting. Since the framework is built on SGLang, compatibility follows SGLang's for open weight models. No closed proprietary model is needed to reproduce the approach.
Do you need GB300s to benefit?
To reproduce the paper's numbers, yes — the benchmarks go up to 256 GB300s. But the thesis (agent-aware eviction, online split adaptation) is testable on more modest hardware, and software components like the KV manager are transferable regardless of hardware.
What gain to expect on P95 rather than on the average?
This is the paper's strongest gain: up to 13.92x reduction in P95 TTFT, versus 1.15x to 4.12x on average latency. Makes sense: online adaptation crushes the queue buildups caused by switches and re-prefills, which hit the high percentiles hardest — precisely what determines the perceived latency of a full agent pipeline.
Can the planner make the service less reliable?
Yes, and the paper documents it: a badly timed switch caused a run at 0.75 agents/min to miss its SLOs, with a queue of 120-275 s at P95 for 10 minutes. That's the structural trade-off of online adaptation. Safeguards exist (the planner only switches on persistent gains), but the reliability of an adaptive system depends on that of its planner.
✅ Conclusion
Evolving Infra is the first serving system that treats agent workloads as what they are — mutating workloads with persistent contexts — rather than as chat that overflows, and its numbers against Dynamo, if they hold up, will redefine what we expect from an inference stack in 2027. Clone the repo, measure how much KV reuse you're losing, and if you're losing 80% of your cache hits like in the paper, you know what you have to do.