Habitat, ChatGPT's storage platform: 500 petabytes, 70M requests/s, and a Python→Rust rewrite by two engineers

Habitat: behind the scenes of ChatGPT's storage — 500 petabytes, 70 million requests/s, and a Python→Rust rewrite led by two engineers

🔎 On October 23, 2026, OpenAI pulled back the curtain on the machine that powers ChatGPT

When a service exceeds one billion weekly users, every millisecond of latency becomes a strategic matter. That's exactly what OpenAI describes in the second part of its technical series on Habitat, its online storage platform, published on October 23, 2026.

The headline figure: reads went from roughly 120 ms to roughly 10 ms at the global median, a 12× improvement since the beginning of the year. Behind that number lies a deliberate architectural decision: a return to multi-tenant containers co-managed with Azure, after a detour through dedicated containers.

And there's a second story, more understated but just as revealing: two engineers rewrote the core of Habitat in Rust in the second quarter of 2026, with the help of Codex and GPT-5.5. The result — 6× more CPU-efficient, 15× more memory-efficient — says a lot about how AI is transforming the technical debt of web giants.

This article dissects OpenAI's two publications, the analyses from AI Weekly, Dataphoenix and ALFW, and then draws the lessons for anyone operating transactional storage at AI scale.

The Essentials

• Habitat is OpenAI's online storage layer: it handles more than 70 million requests per second, stores over 500 petabytes, spans nearly 40 geographic regions, and serves more than 1 billion weekly users of ChatGPT, Codex, and the API.
• The key to performance: multi-tenancy. OpenAI went back to Azure Cosmos DB co-managed containers, an approach co-engineered with the Azure team, which brought the global median read latency down from about 120 ms to about 10 ms (a 12× improvement, October 2026).
• The Python→Rust rewrite was carried out by two engineers in Q2 2026 with the help of Codex and GPT-5.5. The Rust version now handles 95% of production traffic, is 6× more CPU-efficient, and uses 15× less memory.
• QoS comes down to sorting requests: throttling the hottest keys first, priority tiers (offline traffic is throttled before interactive traffic), and a co-engineered serverless mode at 10,000 request units/s per physical partition.
• The catch: more than 90% of Habitat requests are reads, and OpenAI publishes no tail latency figures. The median means half of reads are still slower than 10 ms.

Tools and technologies at the heart of this article

What exactly is Habitat? The invisible layer between your conversations and the datacenters

Habitat is not a database. It's the service layer that sits between OpenAI's products — ChatGPT, Codex, the API — and the underlying storage systems: Azure Cosmos DB, caches, blob storage, and change-data-capture services. That's how GravityDevOps describes it in their summary.

Concretely, every time you reopen a conversation in ChatGPT, when Codex re-reads a project's context, or when a developer queries the API, it's Habitat that serves the data. At OpenAI's scale, this amounts to more than 70 million requests per second and over 500 petabytes stored, spread across nearly 40 geographic regions.

The most striking thing in the first part of the series? Habitat was not born from a grand architectural plan. It started out as a simple client-side Python library, hooked up to a single database. It was growth — 1 billion weekly users — that forced each layer to reinvent itself. This pattern is classic in large-scale software engineering: the first "naive" version often survives for a long time, until the moment it no longer can.

Part II: why OpenAI went back to shared containers — and why it changes everything

Direct answer: because dedicated containers, though simpler to isolate, turned out to be too costly and too slow to provision at the scale of a billion users. Multi-tenancy via containers co-managed with Azure is the compromise they settled on, and it's what unlocks the 12× gain on reads.

The multi-tenancy problem at scale

In a multi-tenant architecture, multiple products and multiple customers share the same physical resources. It's cost-effective, but it creates a risk: a noisy workload can degrade performance for all its neighbors. It's the classic cloud dilemma.

According to Dataphoenix, OpenAI first tried dedicated containers — maximum isolation, but massive overprovisioning and increased latency during spikes. The return to shared multi-tenant containers as the default, co-managed with the Azure Cosmos DB team, makes it possible to pool resources while keeping fine-grained control mechanisms.

The three mechanisms that make multi-tenancy viable

Three technical fixes, detailed by NewsroomAmerica, make this model tenable:

1. Hot key throttling: the most heavily accessed keys are throttled first, before contention spreads to the whole container.
2. Request tiering: every request gets a priority tier. Offline traffic — background processing, indexing — is throttled before interactive traffic. When you're waiting for a response in ChatGPT, your request goes ahead of a batch job.
3. Co-engineered serverless mode: 10,000 request units/s per physical partition, a capacity calibrated with the Azure team itself.

On this last point, the quote reported by NewsroomAmerica is telling: "Product teams would often ship their features to production and forget to pre-scale and we'd sit in SEV rooms on launch day attempting to scale up containers last minute." In other words: product teams forgot to pre-size their containers, and the infrastructure team spent launch days sitting in a war room. Co-managed serverless is an organizational answer as much as a technical one: the responsibility for capacity planning is shared with Azure.

The result: 120 ms → 10 ms

The global read p50 went from about 120 ms to about 10 ms, a 12× improvement since the beginning of the year. But be careful how you read that figure: as ALFW points out, the gain combines three levers — broad replication of small latency-sensitive datasets, the built-in cache, and the Rust rewrite — without breaking down the contribution of each. And above all, OpenAI publishes no tail latency figures. A median of 10 ms mathematically means that half of reads are slower. We'll come back to that.

Python → Rust: two engineers, one AI, and 95% of production traffic

Direct answer: the rewrite of Habitat in Rust, carried out by two engineers in Q2 2026 with the help of Codex and GPT-5.5, produced a service that is 6× more CPU-efficient and 15× more memory-efficient, which now handles 95% of production traffic.

That's the most talked-about angle of this news story, and AI Weekly provides the details. The Python version of Habitat had surpassed 20 million requests per second at peak — a feat for an interpreted language. But at that scale, every CPU cycle and every gigabyte of memory comes at a steep price, across tens of thousands of servers.

Why Rust, and why now

Rust offers memory-safety guarantees without a garbage collector. For a storage service handling 70 million requests/s, that means:

• Predictable latencies, with no unpredictable GC pauses — crucial when you're targeting p99 and not just p50.
• Far greater load density: 15× less memory per request means 15× fewer servers for the same load, or the same capacity with a huge growth margin.
• Directly reduced infrastructure costs: CPU and RAM are the two biggest line items in a database server fleet.

The role of Codex and GPT-5.5: AI-assisted migration becomes production reality

The most significant point for the industry isn't Rust itself — plenty of teams migrate to Rust. It's the method. Two engineers, not a team of twenty. And the migration tool: Codex and GPT-5.5, OpenAI's own models.

It's a textbook case of what could be called augmented migration: the AI handles the tedious, repetitive work of translating mature Python code into idiomatic Rust equivalents, while the engineers validate the semantics, design the tests, and steer the gradual rollout — 95% of production traffic, with the old Python version kept for the rest.

Worth noting: Codex appears in the agentic model rankings with GPT-5.3 Codex (score 80), while GPT-5.5 dominates the category with 98.2. OpenAI is thus using its own most advanced models to rewrite its own infrastructure — a self-improvement cycle worth highlighting.

Throttling, tiering, and change-data-capture: the anatomy of a billion-scale QoS

Direct answer: OpenAI can't simply "add more servers." Quality of service rests on an explicit hierarchy of requests and proactive throttling of the hottest workloads.

The change-data-capture deserves a closer look. According to Dataphoenix, Habitat processes millions of changes per second across thousands of containers, feeding more than 100 Kafka topics. It is this stream that enables replication, cache invalidation, and synchronization across the 40 geographic regions.

Request tiering, for its part, reflects a product philosophy: not all requests are equal. An interactive read in ChatGPT — what the user expects in under a second — takes priority over an offline job that can take minutes without any visible impact. This triage at the source, applied even before reaching Cosmos DB, is what makes it possible to sustain a 10 ms median without overprovisioning the entire fleet.

For the OpenAI team, the challenge was also organizational. Engineers Jon Lee, Hrachya Hakobyan, and Chaomin Yu describe a world where capacity planning was the weak human link in the chain. The co-managed containers offload part of that cognitive burden to Azure: the co-managed platform adjusts capacity, while product teams focus on the product.

What the 10 ms median doesn't say: the question of tail latency

Direct answer: OpenAI publishes no tail latency figures (p99, p999). Yet at the scale of a billion users, it is precisely the tails that define the perceived experience.

The analysis by ALFW is the most critical — and the most useful. Three caveats deserve to be raised:

1. The median always flatters. A p50 of 10 ms means that half of reads are slower. How much? 20 ms? 200 ms? Nobody knows outside of OpenAI.
2. More than 90% of requests are reads. The 12× gain therefore applies to the overwhelming majority of traffic, but writes — and their multi-region consistency — remain the blind spot of the communication.
3. No breakdown of the gains. Wide replication, built-in cache, Rust rewrite: three levers, one aggregate figure. It's impossible to know which one contributed the most — and therefore what is transferable elsewhere.

This is classic technical journalism: praise the result, demand the missing data. OpenAI has published a series that is remarkably detailed by industry standards — most players publish nothing — but the p99 remains in the black box. For a company selling reliability to a billion users, that's a defensible choice. It's not a figure you can cite as-is in an architecture review.

The business angle: what this means for inference costs and your own technical debt

Direct answer: the lessons from Habitat go well beyond OpenAI. Anyone running transactional storage behind an AI workload faces the same trade-offs — multi-tenancy, runtime language, QoS — with budgets on a completely different scale.

The cost of latency, line by line

Going from 120 ms to 10 ms isn't just an engineering feat: it's a line on the P&L. A 12× faster read means potentially fewer servers for the same load, fewer client-side retries, less preventive overprovisioning. Multiply by 70 million requests per second, around the clock, and the gains from the Rust rewrite — 6× CPU, 15× memory — translate into millions of dollars in annual infrastructure savings.

It's the same trade-off documented at other players. When DeepSeek open-sources its production inference engine, DeepInfer, under the Apache 2.0 license, the logic is identical: at AI scale, the efficiency of the inference runtime and storage is a competitive advantage that can be measured directly in dollars.

The specter of AI infrastructure depreciation

These efficiency gains come against a tighter backdrop. Gartner's projections on $3.5 trillion in AI infrastructure investments and the risk of datacenter depreciation raise the fundamental question: every 6× or 15× efficiency gain makes fleets sized for yesterday's needs obsolete a little faster.

For OpenAI, the equation is simple: every Habitat optimization reduces the marginal cost of the service and improves margins. For datacenter operators betting on indefinite demand growth, the Habitat story tells the opposite: demand doesn't grow faster than the efficiency of the software serving it. Technical debt, paid early — a Rust rewrite, a multi-tenancy overhaul — pays for itself in freed-up capacity.

Three transferable lessons for your stack

1. Measure the tail, not the median. If you only track p50, you don't see the users who are suffering. Demand p99 in your dashboards.
2. Explicit tiering beats overprovisioning. Classifying your requests by business priority and throttling the offline tier first costs a week of work. Overprovisioning by 40% costs 40% more every month.
3. AI as a migration tool is mature. Two engineers and Codex did what a traditional rewrite team would have taken years to accomplish. If you're porting critical Python to Rust, Go, or any other compiled language, the time to try assisted migration is now.

❌ Common Mistakes

Mistake 1: reading "12× faster" as a uniform gain

What goes wrong: headlines repeat the global p50 without context. The gain combines three distinct levers — replication, caching, Rust — with no breakdown, and half of all reads still sit above 10 ms.

The fix: always cite the median as a median, and look for tail data before basing an architecture decision on an aggregate figure.

Mistake 2: believing multi-tenancy is "free"

What goes wrong: container sharing cuts costs, but it requires protective mechanisms — hot key throttling, request tiering — without which a single noisy workload degrades everyone.

The fix: if you share infrastructure, invest in QoS first. Sharing without governance is a ticking time bomb in production.

Mistake 3: underestimating the organizational dimension of scaling

What goes wrong: the pre-provisioning failure described by OpenAI engineers — product teams forgetting to pre-scale and launch days spent in SEV war rooms — isn't a technical problem, it's a process problem.

The fix: co-managed serverless is an organizational answer. On your end, the equivalent is to automate sizing rather than rely on the vigilance of your teams.

Mistake 4: planning a Rust rewrite like a traditional team project

What goes wrong: budgeting a Python→Rust migration as a ten-person, two-year effort, when the OpenAI model — two engineers + generative AI + progressive cutover — delivered 95% of traffic in a single quarter.

The fix: pilot a migration assisted by today's agentic models (GPT-5.5, GPT-5.3 Codex) on a non-critical service, measure the actual ratio, then expand.

❓ Frequently Asked Questions

What is Habitat, exactly?

It's OpenAI's online storage platform, the layer that connects ChatGPT, Codex, and the API to the underlying storage systems: Azure Cosmos DB, caches, blob storage, and change-data-capture. It handles more than 70 million requests per second and stores over 500 petabytes across nearly 40 geographic regions (OpenAI, October 2026).

Why does the drop from 120 ms to 10 ms matter?

At the scale of 1 billion weekly users, every millisecond of latency is paid for in user satisfaction, abandonment rates, and server costs. A 12× improvement in read median, achieved in under a year, represents a major infrastructure saving and a noticeably more responsive experience.

How did two engineers rewrite such a critical service in Rust?

In the second quarter of 2026, two OpenAI engineers used Codex and GPT-5.5 to translate Habitat's Python code to Rust, with human validation and a gradual traffic cutover. The Rust version now handles 95% of production traffic, with a 6× improvement in CPU efficiency and a 15× reduction in memory consumption.

Is the 10 ms figure reliable?

It's a global median, not a maximum. OpenAI publishes no tail latency data (p99), and the gain aggregates three levers without any breakdown. It's a remarkable figure, but one to read with the methodological caveats raised by ALFW.

Is Azure Cosmos DB the only storage system behind ChatGPT?

No. Habitat relies on Azure Cosmos DB as its underlying transactional base, but also on caches, blob storage, and change-data-capture services feeding more than 100 Kafka topics. Cosmos DB is the centerpiece of online storage, not the only component.

What can a mid-sized company take away from this story?

Three things: request-priority QoS costs little and pays off handsomely; AI-assisted rewriting of critical code is now a credible option; and software efficiency — Rust, caches, targeted replication — is the most underexploited cost lever in AI architectures.

✅ Conclusion

Habitat tells two stories in one: that of an infrastructure that learned to share — co-managed containers, priority-based QoS, 12× on reads — and that of a new way of writing critical software, where two engineers and their AI models rewrite in a single quarter what traditional software engineering treated as a multi-year project. If you operate storage behind an AI workload, the lesson is the same at OpenAI as it is for you: pay down technical debt early, measure the tail, and let AI carry the tedious work. To go further on efficiency trade-offs at scale, dive into DeepInfer, DeepSeek's inference engine released under Apache 2.0 and the analysis of the $3.5 trillion of AI infrastructure according to Gartner and their depreciation risk.