Your LLM can return a 200 OK with someone else's data: Cloudflare and ChatGPT have leaked data this way

Your LLM Can Return a 200 OK With Someone Else's Data: Cloudflare and ChatGPT Both Leaked Data This Way

🔎 The Worst Bug of 2026 Doesn't Return Any Error

According to an analysis published by Towards AI on November 4, 2026, LLM APIs returned HTTP 200 responses containing other users' data. Two documented incidents, two major players: Cloudflare and ChatGPT. The common denominator isn't the model. It's the serving infrastructure.

The chilling detail: nothing "crashed." No 500 codes, no timeouts, no alerts in the dashboards. The status code claimed everything was fine. The content, however, belonged to someone else.

Why now? Because the race to million-token context windows — which has become the marketing standard for frontier models like Gemini 3.1 Pro or GPT-5.5 — has driven the adoption of disaggregated prefill/decode architectures with shared KV caches. Performance created the incident surface. And no one had anticipated the case where the two mix together.

Key takeaways

• LLM APIs returned HTTP 200s with cross-tenant data: other users' content, inside a "successful" response (Towards AI, November 4, 2026, as reported by LLM Marketcap).
• Root cause on the runtime side: an advertised context window of 1,048,576 tokens, but a prefill KV cache pool capped at 681,280 tokens. A bug in disaggregated prefill/decode mode let the handoff continue instead of returning an error.
• The decode node would then read stale cache pages left behind by earlier requests — and produce a fluent, plausible response that belonged to someone else.
• GeoCache (arXiv) proposes an elegant defense against cache extraction: key rotation using a session-specific operator unknown to the attacker.
• A five-point checklist for any API operator exposing an LLM, detailed below.

Recommended Tools

If you operate or audit an LLM infrastructure, here is the minimal toolkit for applying the checklist in this article.

What happened: 200 OK responses riddled with other users' data

According to Towards AI (November 4, 2026), two distinct incidents — one involving Cloudflare, the other ChatGPT — share the same signature: an HTTP 200 response, therefore treated as valid, whose content comes from other users of the platform.

"Cross-tenant" means the partition between tenants of the same infrastructure has failed. Your request, your session, your billing — but someone else's response. Conversation fragments, business contexts, personal data: what was supposed to stay walled off walks out the front door.

A classic breach leaves traces: an anomalous request, outbound volume, an exfiltration. Here, the data leaks inside a perfectly well-formed response, logged as a success, potentially cached, displayed, stored, or even fed back into downstream pipelines. Forensics starts with a major handicap: the system believes everything went fine.

Legally, this is the textbook scenario of a personal data breach within the meaning of Article 33 of the GDPR: notification to the supervisory authority within 72 hours, and informing the individuals concerned. It doesn't matter that the leak is a "bug". The data traveled between tenants.

A note of journalistic honesty: the publication does not detail the exact scope of each incident — number of requests affected, duration, data categories. That vagueness is itself a lesson. A leak that looks like a success can persist for a long time without anyone looking for it.

The mechanism: when the prefill/decode handoff happens in the dark

The bug isn't in the model or in the prompt: it's in the serving orchestration, at the precise moment when the request transitions from prefill to decode.

A mechanical refresher. Prefill processes the entire prompt in one pass and produces the KV cache — the attention tensors that "memorize" the context. Decode generates tokens one by one, building on that cache. For performance reasons, modern stacks disaggregate these two phases onto separate GPU nodes and transfer the cache from one to the other.

The KV cache is literally the model's working memory — this research on LLM working memory shows just how much this memory shapes reasoning. Here, it's this memory that leaked.

In the incidents described, the runtime advertised a context window of 1,048,576 tokens (2^20, marketing's round number), but the KV cache pool allocated to prefill was only 681,280 tokens — roughly 65% of the promise. Any prompt whose size fell into the zone between the two, between 681,280 and 1,048,576 tokens, should have been explicitly rejected.

That's where the bug kicks in. Instead of returning an error, the disaggregated prefill/decode mode let the request continue its handoff. The decode node would then read stale cache pages — left behind by earlier requests, by other tenants — and generate from that material. Result: HTTP 200, foreign content.

The worst part? The response is fluent, coherent, plausible. It's not noise: it's real text drawn from real conversations. Without canaries or provenance checks, a client can't distinguish it from a legitimate response.

Multi-tenancy and shared cache: a structural bomb, not an accident

If this bug could exist at all, it's because the entire economics of LLM serving rests on the aggressive sharing of resources.

GPUs are expensive, request volumes are exploding, so platforms multiplex thousands of tenants onto shared pools. And to save compute, caches reuse pages: prefix caching, which recognizes an already-processed prefix to avoid recomputing it, is sold as a major optimization. Latency halved, costs reduced.

But a shared resource is a shared failure domain. Cache reuse is like a hotel that doesn't change the sheets: as long as housekeeping is reliable, everything runs fast and well. The day housekeeping skips a step, the next guest sleeps in the previous guest's room — with their correspondence open on the table.

The race toward giant contexts mechanically compounds the risk. The larger the advertised window, the more massive the cache must be, the greater the pressure on the pool, the more complex the reuse and eviction heuristics become. A million tokens is also a million opportunities to grab the wrong page.

Crucial point: here, no attacker is required. An orchestration bug is enough. This is what distinguishes this incident from deliberate cache-extraction attacks — which also exist, and which GeoCache addresses. Both threats share the same root: the cache is an exposed surface, not an implementation detail.

GeoCache: Making Cache Extraction Mathematically Blind

In the face of cross-request extraction attacks, GeoCache (arXiv) applies a simple idea: rotate the victim's cache keys with a session-specific operator, unknown to the attacker.

The targeted attack scenario: an adversary probes the cache with candidate prompts, measures cosine similarity to detect matches, and gradually reconstructs the content of others' requests. An auxiliary-channel attack, patient and silent.

With rotation, every candidate produced by the attacker exhibits a cosine similarity close to zero against the rotated keys. The correct candidate becomes indistinguishable from an unrelated prompt: the signal the attacker is trying to read no longer exists. The attack doesn't become harder — it becomes blind.

Important clarification: GeoCache neutralizes deliberate cache extraction. It does not replace fixing the handoff bug described above, which is a runtime defect, not an auxiliary channel. The two defenses are complementary — and the checklist that follows covers both.

The defense checklist for an API that exposes an LLM

Five measures, all immediately implementable, that would have blocked the incidents described by Towards AI.

1. Isolate the KV cache per tenant and per request. No cache page should be reusable beyond its scope without an explicit guarantee. If you enable prefix caching, make sure it's partitioned by tenant, with verified namespaces — not assumed ones.

2. Validate context limits at the prefill/decode handoff. If the prompt exceeds the actual pool, fail explicitly: 413, 5xx, a clear message. Never silent degradation. The gap between the advertised window (1,048,576 tokens) and the actual pool (681,280 tokens) must produce an error, not a gray area.

3. Quotas and response pagination. Limit the size of what can come out of a single response. An attacker — or a bug — that can only extract paginated, capped fragments sees its blast radius shrink accordingly.

4. Detect prompt injections on the output side. Filtering input isn't enough: scan what the model returns. The prompt adware affair that turned Codex into a billboard proved it: an LLM's output is an attack vector in its own right.

5. Test cross-tenant contamination in your serving CI. Canary requests with marker payloads, issued from multiple tenants on the same infrastructure, with a binary assertion: no marker may appear outside its own session. If this test doesn't exist in your CI, your isolation is a belief, not a property.

You're consuming an LLM API: the client-side plan

You can't fix your provider's runtime. You can detect contamination, limit the damage, and choose your battles.

First line of defense: the per-session canary. Inject a unique marker into each session, log every response, and alert if a foreign marker appears.

Concretely, the approach comes down to three moves. First, generate a unique and unpredictable marker per session — for example a CANARY- prefix followed by a random hexadecimal string — and place it at the beginning of every prompt sent to the provider. Next, log the entirety of received responses in a raw log, with no filtering or overwriting. Finally, have these logs analyzed by a simple rule: if a canary marker other than the current session's appears in a response, cross-tenant contamination is suspected, and the alert must fire immediately. Detection is binary and requires no sophisticated tooling — a simple pattern search over the logs is enough. A canary outside its session is not a curiosity to investigate later: it's an incident.

Second line: treat all LLM output as untrusted input. This is even more critical with agents. When you code from your phone while Codex works on your machine, a contaminated response doesn't stay on the screen: it becomes commands executed on your machine. The choice of model for these workloads matters all the more — our comparison of LLMs for agents details the options.

Third line: classify your data before sending it. The sensitivity of what flows through these APIs is skyrocketing — ChatGPT now manages Pro subscribers' personal finances. When the LLM sees your bank transactions, a cross-tenant leak is no longer a technical incident: it's a personal financial data incident.

Fourth line, for truly sensitive workloads: step out of multi-tenancy. A self-hosted open-weights model — DeepSeek V4 Pro, Kimi K2.6 or GLM-5.1 — is single-tenant by construction: no shared cache, no neighbor. Our guide to installing an LLM locally and our selection of the best local LLMs are the starting point; to experiment, a small model on a VPS (Hostinger) is enough before investing in GPUs.

Finally, ask your providers the question in writing: how do you isolate the KV cache between tenants? What happens when a prompt exceeds the actual pool? A vendor who answers "we reuse the cache for performance" without ever uttering the word "isolation" has just given you their answer.

❌ Common Mistakes

Mistake 1: Treating a 200 OK as proof of integrity

The HTTP status measures transport, not the veracity of the content. The incidents described went unnoticed precisely because all monitoring was green. The solution: content-level checks — canaries, detection of unexpected PII, question-answer consistency verification.

Mistake 2: Disabling the cache "for security"

Cutting prefix caching off entirely wrecks latency and costs for a poorly targeted security gain. The right answer is per-tenant isolation and GeoCache-style key rotation, not abandoning an optimization that powers the economics of serving.

Mistake 3: Testing only the application layer

Teams test prompt injections in the application and never in the serving layer. Yet the bug documented by Towards AI lived beneath the application layer, in the prefill/decode handoff. Cross-tenant contamination tests must live in the infrastructure CI, not just in the product's CI.

❓ Frequently Asked Questions

Could my data have leaked if I use ChatGPT or Cloudflare?

It's possible if your requests went through the affected entry points during the bug window. The mechanism documented by Towards AI means that responses may have contained fragments of other sessions. The exact scope per provider remains unclear: monitor incident communications and demand written answers.

Are all LLM providers affected?

No, but any stack combining all three risk factors is potentially affected: disaggregated prefill/decode, KV cache shared across requests, and silent degradation at handoff. A single-tenant provider or one with no cache reuse doesn't have this exposure. Ask the three questions, in writing.

Should I stop sending sensitive data to LLM APIs?

No, but weave your data. Classify it: what is public can go to any provider, what is sensitive requires contractual isolation guarantees, detection canaries — or self-hosting. All-in and all-out are the two wrong answers.

Does self-hosting eliminate the risk?

It eliminates the cross-tenant risk by construction: a single tenant, no neighbor in the cache. But you inherit the bugs of the runtime you run — the same type of handoff defect can exist in your stack. Self-hosted doesn't mean risk-free: it means accountable.

How do I detect contamination in my own systems?

Three signals: a foreign canary in your responses, identifiers or PII you never sent, a response semantically disconnected from the prompt. Log everything, keep raw logs, and alert on deviations. Without raw logs, no forensics is possible after the fact.

✅ Conclusion

The accidentally contaminated 200 OK is the perfect nightmare of the LLM era: a bug that leaks data without ever looking like an error — and the checklist above (KV cache isolation, explicit failure on handoff, canaries in CI) is the bare minimum for any API that exposes a model. If you're choosing a provider this month, start with our comparison of the best LLMs — and ask about the KV cache before asking about the price.