DeepSeek open-sources DeepInfer: the inference engine behind its production, under Apache 2.0
🔎 Three days after the V4.1-Flash weights, DeepSeek releases the machine that serves them
The timing is too tight to be a coincidence. On October 17, 2026, DeepSeek releases the weights of DeepSeek-V4.1-Flash. On October 20, the lab posts DeepInfer on GitHub: the inference framework that runs this model on its own servers, in production. Apache 2.0 license, commercial use permitted with no conditions.
The move deserves a closer look. Open-sourcing a model means giving away a finished product. Open-sourcing the engine that serves this model at the scale of a global service means giving away the factory: the FlashMLA attention kernels, the DeepGEMM matrix multiplications, the DeepEP inter-GPU communication layer, the DSpark speculative decoding. Everything that, as a rule, remains carefully locked away.
The reception lives up to the gesture: first place on GitHub trending, top of Hacker News with over 2,100 points, 3,300 stars and 235 forks in a matter of days. Three questions remain. What exactly does this code contain? What are the announced figures worth? And what has DeepSeek chosen not to give?
Key takeaways
- DeepInfer is DeepSeek's production inference framework, open-sourced on October 20, 2026 under Apache 2.0 (GitHub repo).
- A single target model: DeepSeek-V4.1-Flash, whose weights were released on October 17, 2026.
- ~26,000 tokens/s per GPU in decoding (EP64, 131,072-token context, 75% acceptance rate) and prefill of 60,000 to 1.5 million tokens/s/GPU thanks to a 96% prefix cache hit rate.
- DSpark, the in-house speculative decoding, cuts rejected drafts by 30% and adds 4–7% throughput.
- DeepSeek's cluster would serve 155 trillion tokens per day, for $38 in revenue per GPU per hour (October 2026, DeepSeek figures).
- First-class dual hardware target: NVIDIA SM100 and Huawei Ascend 950.
- Acknowledged limitations: offline inference only, no KV cache management, no load balancing.
Recommended Tools: DeepInfer's Components
DeepInfer is not a monolithic repo: it is an assembly of components, most of which are already known to DeepSeek followers, now gathered in a single framework.
| Component | Primary Use | Price (October 2026) | Key Takeaway |
|---|---|---|---|
| DeepInfer | Production inference framework | Free (Apache 2.0) | Offline only, only one model supported |
| FlashMLA | Optimized attention kernel | Free (included) | The internal version is 5 to 10% faster than the published version |
| DeepGEMM | Optimized matrix multiplication | Free (included) | Targeted at both SM100 and Ascend 950 |
| DeepEP | Inter-GPU communication (expert parallelism) | Free (included) | Already published separately by DeepSeek |
| DSpark | Speculative decoding | Free (included) | Drafts rejected -30%, +4 to 7% throughput |
| SWA bounded replay | Bounded sliding window prefill | Free (included) | One of the engines for 1.5 million tokens/s/GPU prefill |
DeepSeek also publishes a second repo, deepseek-recipe, under the MIT license this time, which gathers recipes and configurations. Separating the core engine (Apache 2.0) from the usage recipes (MIT) is an unusual—and rather clever—choice: the recipes can evolve quickly without touching the core engine.
A production engine, not a research project
The answer comes down to one sentence: DeepInfer is not a lab engine adapted for production — it's the production engine published as-is.
This is what SiliconANGLE points out: the framework behind DeepSeek's production services. The code that runs serves the service's real traffic, not a demo optimized for a benchmark.
Second singularity: a single model. DeepInfer is designed to serve DeepSeek-V4.1-Flash, period. This isn't an accidental limitation, it's an architectural choice: the engine and the model are co-designed, from expert parallelism to attention kernels. This logic follows directly from the lab's V4 strategy, where the Pro/Flash duo redesigned the entire lineup (DeepSeek V4: two new models — Pro and Flash — change the game).
Third singularity: the release sequence. Weights on October 17, engine on October 20. Three days apart, so that anyone who wants to can match the weights against the code that serves them. It's technical storytelling, but it's also an invitation to audit — and few labs dare this level of exposure.
In my view, this marks the end of a cycle. For two years, labs have opened weights and closed factories. DeepSeek has just shown that you can open both.
The numbers: 26,000 tokens/s per GPU, 155 trillion tokens per day
Direct answer: DeepSeek measures roughly 26,000 tokens/s per GPU in decoding and claims its cluster serves 155 trillion tokens per day, for $38 in revenue per GPU per hour.
Let's start with decoding. The reference configuration: EP64 (the model's experts spread across 64 GPUs), a context of 131,072 tokens, and a 75% acceptance rate for speculative decoding. Prefill, for its part, was measured on 4 GPUs, decoding on 64 — two very different regimes, don't confuse them (Marktechpost). On this configuration, DeepSeek reports roughly 26,000 tokens/s/GPU.
The prefix cache, the real star of the numbers
On the prefill side, the numbers are even more spectacular: from 60,000 to 1.5 million tokens/s per GPU. How? Because 96% of DeepSeek's inputs are already in the prefix cache.
Concretely: a huge share of production traffic repeats contexts — system prompts, instructions, conversation histories. The prefill for those tokens is already done; only the delta remains to process. That 96% figure says less about the engine than about DeepSeek's traffic: it's the profile of a service with millions of returning users.
On the user experience side, dev.to's read of the repo lands at a first token around 400 ms and generation at 230 tokens/s. Real-world usage figures, not test-bench numbers.
$38 per GPU/hour: the most subversive figure
The most interesting thing isn't a speed figure, it's a money figure: $38 in revenue per GPU per hour (October 2026, DeepSeek's figures), for a cluster serving 155 trillion tokens per day. DeepSeek isn't just publishing an engine, it's publishing a reference economic framework. Every cluster operator now has a brutal point of comparison for judging their own profitability.
One caveat, though. These measurements come from DeepSeek, on its own traffic, with its own configuration. explainx.ai, for its part, devotes its analysis to the gap between the claimed and the verifiable — and notes that the internal version of FlashMLA is reportedly 5–10% faster than the published one. In other words: even the open-source code isn't exactly the production build. Take it as an upper bound, not a contractual promise.
DSpark: 30% fewer rejected drafts, 4 to 7% more throughput
Direct answer: DSpark is DeepInfer's speculative decoding layer. Its job: cut rejected drafts by 30%, for a 4 to 7% throughput gain, with a 75% acceptance rate.
A useful refresher for those who follow infra less closely. Speculative decoding has a small model guess the next tokens — the "drafts" — which the large model then verifies in parallel. Every accepted token is free throughput. Every rejected token is compute burned for nothing. The whole game is maximizing the acceptance rate, hence the significance of the announced 75%.
A 4 to 7% gain may seem modest on paper. At the scale of a service serving 155 trillion tokens per day, that's a massive amount of capacity recovered without buying a single additional GPU. That's the entire difference between optimizing a benchmark and optimizing a cost line.
Our take: DSpark is the most immediately copyable building block of the bunch. The technique is well known, but a battle-tested implementation at this scale, with publicly documented acceptance rates, was missing from the open source ecosystem.
SM100 and Ascend 950: the dual target that speaks volumes
Direct answer: DeepInfer treats the NVIDIA SM100 and the Huawei Ascend 950 as first-class targets, on equal footing. This may well be the most strategic line in the entire repo.
In the inference ecosystem, non-NVIDIA hardware is usually treated as the poor relation: a late port, less optimized, less tested. Here, the repo makes a point of noting that the Ascend 950 is a first-class target of the framework, and the DeepGEMM kernels are optimized for both platforms.
The geopolitical reading is obvious. A lab capable of serving 155 trillion tokens per day on Huawei hardware is a lab that no GPU export control can strangle. DeepInfer isn't just code: it's a technological insurance policy, published in plain sight.
It's also a signal to the industry: public, quantified proof that a leading inference infrastructure runs natively on Chinese hardware. And for operators, it's bargaining power. When your engine runs on two hardware ecosystems, vendors find themselves in competition — and, at last, so do prices.
What DeepInfer does not do: offline, no KV cache, no load balancing
Straight answer: DeepInfer only covers offline inference. No KV cache management, no load balancing, a single supported model. The trickiest part of online serving remains internal.
Three limitations, clearly documented in the repo:
- Offline only: the framework is designed for batch and throughput workloads, not for serving an interactive API as-is.
- No KV cache: the state that makes multi-turn conversations economically viable is not handled by the published code.
- No load balancing: distributing load across GPUs and nodes, a pillar of any online service, is out of scope.
How should we read these absences? Two non-exclusive hypotheses. First reading: DeepSeek is opening up the compute core — where the real technical value lies, the kernels and scheduling — and keeping the serving plumbing. Second reading: this plumbing is coming, brick by brick. DeepEP, the communication layer, had already been released separately last year.
The clarity deserves credit: the limitations are stated in black and white, not buried in a marketing README. But we should also call things by their name. DeepInfer is not a ready-to-use serving platform; it is a core of excellence to be integrated into an existing stack.
The Strategy: Opening Inference to Better Sell Intelligence
Direct answer: by publishing DeepInfer, DeepSeek transforms inference into a commodity and shifts its competitive advantage to where it is difficult to copy—the models and the services.
The reasoning is relentless. If the engine is Apache 2.0, any operator can serve models with comparable efficiency. DeepSeek therefore bets that its moat is not in secret kernels, but in the quality of its models and its cost structure. The published figures support this bet: at $38 in revenue per GPU/hour, the money is made at scale, not in the dark.
Monetization, on the other hand, plays out elsewhere. The lab has already opened its Search and Fetch APIs to agents, billed at $0.75 per thousand calls (DeepSeek opens Search and Fetch API to agents: its internal engine at $0.75 per thousand calls). The logic is consistent: the infrastructure is a cost to be optimized and shared; the service is the product.
Quartz sees this as the continuation of an open-source strategy assumed as a competitive weapon. It is difficult to contradict this diagnosis: with every publication—weights, kernels, engine—DeepSeek poses the same embarrassing question to its closed competitors. What exactly are you selling that we are not giving away for free?
Add visibility—a Hacker News front page, 3,300 stars in a few days—and you also get the best possible recruitment argument for a systems engineering team. Offensive openness is also a talent machine.
Who can actually run DeepInfer?
Straight answer: without a datacenter GPU cluster, no one. And that's a choice, not an oversight.
The hardware requirements leave no ambiguity: NVIDIA SM100 or Huawei Ascend 950. The reference decoding configuration uses 64 GPUs in expert parallelism; the prefill tests, 4. We're a long way from the workstation, and deliberately so. DeepInfer is aimed at cluster operators, clouds, and companies that make inference their business.
For these audiences, the path is clearly marked: DeepSeek-V4.1-Flash weights (October 17), framework (October 20), and a deepseek-recipe repo under MIT license documenting recipes and configurations. Enough to reproduce, on your own hardware, conditions close to those announced by DeepSeek.
For individuals, on the other hand, the entry point lies elsewhere. If the goal is to run DeepSeek V4 Flash locally, the realistic route goes through ds4, the inference engine launched by antirez that makes this model usable on a Mac (antirez launches ds4: the local inference engine that makes DeepSeek V4 Flash usable on a Mac). Otherwise, the lab's API remains the shortest path.
Our take: DeepInfer is not a gift to hobbyists, it's a gift to operators — and a message to competitors. Don't mistake the recipient.
❌ Common Mistakes
Mistake 1: comparing the 26,000 tokens/s/GPU figure to your own workload
The figure comes from a specific configuration: EP64, a 131,072-token context, a 75% acceptance rate — and above all, traffic where 96% of inputs hit the prefix cache. Your traffic doesn't look like DeepSeek's. The fix: read the deepseek-recipe configurations, then measure on your own workload before drawing any conclusions.
Mistake 2: believing DeepInfer replaces a serving platform
Offline only, no KV cache, no load balancing: the published code doesn't serve an interactive API as-is. The fix: treat DeepInfer as a compute kernel from which you extract the building blocks — FlashMLA, DeepEP, DSpark — to integrate into your existing stack.
Mistake 3: assuming the repo is the exact production build
The internal version of FlashMLA is reportedly 5–10% faster than the released version, and entire portions of the serving stack remain internal. The fix: treat the announced figures as upper bounds and wait for independent benchmarks before making decisions based on these numbers alone.
❓ Frequently Asked Questions
What is DeepInfer, in one sentence?
The inference framework used in production by DeepSeek, open-sourced on October 20, 2026, under Apache 2.0. It groups the FlashMLA, DeepGEMM, and DeepEP kernels, the DSpark speculative decoding, and a SWA bounded replay prefill, to serve a single model: DeepSeek-V4.1-Flash.
Is DeepInfer commercially usable?
Yes. The Apache 2.0 license allows commercial use, modification, and redistribution without royalty, and the deepseek-recipe repo is under the MIT license. The real barrier is not legal but hardware: you need NVIDIA SM100 or Huawei Ascend 950 GPUs to run it.
What is the difference between DeepSeek V4 Flash and V4.1-Flash?
V4.1-Flash is the variant of the Flash range served by DeepInfer, whose weights were published on October 17, 2026. It is part of the V4 strategy of the lab, which distinguishes between Pro and Flash variants with different positioning and cost profiles.
Are the announced performances verified by third parties?
No: these are DeepSeek's measurements — approximately 26,000 tokens/s/GPU in decoding (EP64, context 131,072, 75% acceptance) — relayed by MarktechPost. explainx.ai highlights the gap between the claimed and the verifiable, and the internal version of FlashMLA would be faster than the published version.
Why is DeepSeek open-sourcing its production infrastructure?
Three reasons combine: commoditizing inference to shift the advantage to the models and services, demonstrating hardware independence with the SM100/Ascend 950 duo, and capturing the technical community. The first place on Hacker News and the 3,300 stars in a few days are immediate illustrations of this.
✅ Conclusion
By opening up the engine that serves 155 trillion tokens per day, DeepSeek isn't just publishing code: it's publishing an economic blueprint, a dual hardware target, and an admission — in AI, the advantage is shifting from secrecy to execution. The next step will be independent benchmarks. To make sure you don't miss any announcements of this kind, head over to our watch of new AI tools.