llms.txt, the flaw that caused Claude, Codex, and Hermes to install orphan code in Fortune 500 networks

llms.txt, the flaw that caused Claude, Codex, and Hermes to install orphan code in Fortune 500 networks

🔎 When coding agents become the attack vector

A new class of vulnerability has just emerged, and it exploits exactly what was supposed to make AI agents safe: their autonomy. Security researchers have just revealed that a simple text file placed on a website is enough to make the market's leading coding agents — Anthropic's Claude, OpenAI's Codex, and Nous Research's Hermes — execute malicious code.

The attack does not require any flaw in the models themselves. It exploits a structural trust mechanism: the llms.txt and llms-full.txt files, designed to help LLMs understand a project's documentation, are treated as instructions by the agents. Except that nothing guarantees that these files are legitimate.

The worst part? The tests were conducted in real Fortune 500 enterprise networks. The agents executed network beacons in less than an hour, without a single security analyst being alerted.

The key points

• Researchers scanned 6,214 domains (Fortune 500, Big Tech, defense sector) and found 120 llms.txt files pointing to 227 package installation commands that were unregistered on the npm/PyPI registries.
• Claude, Codex, and Hermes all three executed installation commands from these files, without verifying the existence or legitimacy of the packages.
• The case of clerk.com illustrates a real ongoing attack: an npx points to active malware, and the agents trigger it mechanically.
• The problem is structural, not a bug: coding agents do not distinguish between an instruction found in documentation and a command validated by a human.

Tools and models concerned

To understand the scope of this vulnerability, you need to look at the Claude 4 vs GPT-5 vs Gemini 3 comparison: none of these models were designed with a mechanism to verify the integrity of the sources they consult. Their high agentic scores precisely mean that they act quickly and with little friction — which becomes a problem when the input is compromised.

What llms.txt really is — and why it's a problem

A standard born with good intentions

The llms.txt file is a community standard proposed in late 2024 to structure documentation intended for LLMs. It is a text file placed at the root of a domain, similar in spirit to robots.txt for web crawlers.

A code agent working on a project using a third-party API will fetch the vendor's llms.txt to understand how to properly integrate the service. It's smart, efficient, and exactly what the format was created for.

The llms-full.txt file is an extended variant that contains the full documentation rather than a summary. It is this version that causes the most problems: it often contains installation commands, configuration snippets, and copy-pasteable code examples.

The fatal logical leap: from documentation to execution

Here is the exact mechanism of the attack. A code agent receives a task like "integrate the Clerk API for authentication". It fetches clerk.com/llms-full.txt. It finds an instruction there: npx @clerk/clerk-sdk install. The agent executes it.

Except that in the real case documented by researchers, this command points to an unregistered package, or worse, to a package that exists but does not belong to Clerk. The agent does not check whether @clerk/clerk-sdk is actually published by Clerk on npm. It trusts the file because it is hosted on the vendor's domain.

This is the exact same logic that enabled the Agentjacking : une fake bug report suffit pour pirater Claude Code, Cursor et Codex — 2388 organisations touchées, taux de réussite 85% attacks: the agent treats an unvalidated external input as a legitimate instruction.

The study: 6,214 domains scanned, 227 orphaned packages

The researchers' methodology

The study, reported by Ars Technica and Creati.ai, is not a laboratory demonstration. The researchers scanned the real domains of Fortune 500 companies, GAFAM, and organizations in the US defense sector.

Out of 6,214 domains analyzed, 120 hosted an llms.txt or llms-full.txt file. Among these, the researchers extracted 227 package installation commands (mainly via npm install, pip install, npx) that pointed to unregistered packages or whose ownership could not be verified.

An "orphaned" package in this context means either that it does not exist at all on the registry (which should be an immediate error signal), or that it exists but is not owned by the organization whose name the domain bears.

The numbers that matter

The fact that 100% of the extracted commands point to problematic packages is not a coincidence. The researchers targeted these files precisely because the llms.txt format encourages the inclusion of direct installation commands, without any authentication or signature mechanism.

The clerk.com case: a real attack, not theoretical

An npx leading to live malware

The most concrete example from the study involves clerk.com, a highly popular authentication provider among startups and SaaS companies. Clerk's llms-full.txt file contained (or at one point contained) an npx command that, when executed, triggered the installation and execution of malicious code.

The command went through npx, npm's tool that downloads and executes a package without installing it permanently. It's an ideal vector for this type of attack because npx is designed to be frictionless: no signature verification, no lockfile, immediate execution.

The malware in question established a network beacon — a periodic outbound connection to an attacker-controlled server. From the internal network of the company using the code agent.

Why this is different from a typosquatting attack

Typosquatting on package registries (installing lod-ash instead of lodash) is a well-known problem. But here, the mechanics are different and more insidious.

With typosquatting, a human makes the typo. With the llms.txt attack, the vendor itself (or an attacker who compromised their site) provides the malicious command. The agent has no reason to doubt: the instruction comes from the service's official domain that it is supposed to integrate.

This makes detection by classic security tools virtually impossible. The package doesn't have a suspicious name, the command comes from a legitimate domain, and it's an automated process executing it — not a human whose behavior could be analyzed.

Claude, Codex, Hermes : the three agents caught in the trap

Claude Opus 4.7 : the fastest to execute

Claude Opus 4.7, Anthropic's flagship model with an agentic score of 94.3, was the fastest to trigger execution. Its ability to chain workflow steps seamlessly — which is precisely what makes it excellent for development — becomes a risk vector here.

Claude read the llms-full.txt file, extracted the installation command, and executed it without attempting to verify the package's existence on npm. No warnings, no confirmation steps. The network beacon was active in under 30 minutes during tests.

GPT-5.3 Codex : the model designed for code that doesn't verify code

OpenAI's GPT-5.3 Codex, with an agentic score of 80.0, exhibited similar behavior. Codex is literally designed to turn instructions into executable code. When it finds an installation command in vendor documentation, it executes it because that's its job.

The notable difference is that Codex sometimes asked for additional confirmation — but only when the initial task did not explicitly mention installing dependencies. As soon as the prompt mentioned "install the necessary dependencies", the validation disappeared.

Hermes : the open source agent without guardrails

Nous Research's Hermes, often used in local LLM installation setups for privacy reasons, displayed the most concerning behavior. Being open source and often deployed without the enterprise security layers of Anthropic or OpenAI, Hermes executed the commands with even less filtering.

Local agents are particularly vulnerable to this vector because they often run with full user privileges, without a sandbox, without an enterprise proxy, and without centralized logs. A beacon from Hermes inside a Fortune 500 network can go unnoticed for weeks.

Why trust in vendor docs is the real problem

The broken mental model: human input vs. discovered instruction

The fundamental flaw isn't technical. It's a mental model problem in agent design.

Code agents are built with the assumption that instructions come from the human user, and that data found on the web is... data. But in practice, LLMs don't make this distinction. A prompt written by a human and a paragraph found in an llms.txt are treated the same way: as instructions to follow.

This is a classic alignment problem, but applied to a new context. When you ask Claude "install Clerk", the prompt is an instruction. When Claude reads "to install Clerk, run npx @clerk/clerk-sdk" in a text file, that sentence also becomes an instruction. The context changes, not the processing.

The difference with a human developer

A human developer reading documentation will instinctively verify several things: does the package actually exist? Is it published by the right owner? Is the version consistent? Does the exact name match?

Current code agents do none of these checks. They extract the command and execute it. This shortcoming is not a bug but a design omission: none of the current agent architectures integrates a package integrity verification layer between the search phase and the execution phase.

This issue ties into the challenges surrounding the best LLMs for AI agents: the more capable a model is of acting autonomously, the greater the need for structural guardrails. However, these guardrails do not yet exist in standard agent stacks.

Scanning 6,214 domains: what the numbers reveal

Fortune 500, Big Tech, defense: no one is spared

The study's sample is not insignificant. The researchers deliberately targeted organizations that have the means to protect themselves: Fortune 500 companies, GAFAM, US defense sector contractors. If these organizations are vulnerable, all others are a fortiori.

The fact that 120 of these 6,214 domains host an llms.txt file with problematic commands does not mean that only 2% of organizations are affected. It means that 2% of organizations have already adopted the format — and that among them, 100% present the attack vector.

The adoption proportion will increase mechanically. The llms.txt standard is gaining in popularity precisely because code agents are becoming ubiquitous. More companies will create these files, and without a change in approach, each of them will create a new potential attack vector.

227 orphan packages: what this number means

The 227 installation commands identified are not all actively exploited. But they represent 227 potential entry points that any attacker could activate at any time.

It would be enough to register just one of these orphan packages on npm with the exact name referenced in a Fortune 500 company's llms.txt for every code agent that consults this documentation to install and execute the attacker's code. It is a pre-positioned attack, waiting to be activated.

Attack mechanisms in detail

Package pre-positioning

The most elegant attack works in two steps. Phase 1: identify llms.txt files that reference unregistered packages (the 227 found by the study). Phase 2: register these packages with malicious code at the chosen moment.

The advantage of this approach is that at the time of registration, there is no suspicious action on the part of the attacker on the target network. The package is created on a public registry. It is the target companies' agents that come and fetch it.

Website compromise as an alternative vector

The other vector, illustrated by the clerk.com case, is the direct compromise of the site hosting the llms.txt. An attacker who manages to modify the file (via a CMS vulnerability, a compromised admin access, a DNS hijack) can inject any command.

In this scenario, the attack is instantaneous: as soon as an agent consults the modified file, it executes the command. No need to wait for pre-positioning.

The beacon: first signal, not the final step

In the researchers' tests, the code installed by the agents established a network beacon — a periodic outbound connection. This is a confirmation signal (callback), not the final objective of the attack.

A beacon in a Fortune 500 network allows the attacker to confirm which organizations use which agents, on which workstations, and with what privilege levels. It is automated reconnaissance at scale, triggered by the victims themselves via their own agents.

How to practically protect yourself

As an organization: block the vector

The most immediate measure is to block the execution of unverified packages by agents. This involves several layers.

First, the network: enterprise proxies and firewalls must block outbound connections to package registries (npmjs.org, pypi.org, crates.io) from agent processes, except via an internal mirror registry that only serves vetted packages.

Second, the internal registry: set up an Artifactory or a Nexus that acts as a proxy for npm/PyPI, and configure agents to use only this registry. Any package that is not explicitly approved will be rejected.

Third, the sandbox: code agents should never run with full user rights. Tools like Firejail, gVisor, or isolated namespaces can limit the damage of unwanted execution.

As a vendor: secure your llms.txt

If your company publishes an llms.txt file, audit it immediately. Verify that every installation command references a package you actually own on the corresponding registry. Remove npx commands and replace them with verifiable installation instructions (with exact version numbers, checksums if possible).

Also consider signing your llms.txt files with a PGP signature or equivalent, even if no agent verifies signatures today. The day a verification mechanism is implemented, your files will already be ready.

As an agent user: do not delegate installation

The simplest rule: never ask an agent to "install the dependencies" or "configure this service". Ask it to generate the commands for you, then execute them yourself after verification.

It is less convenient. That is the price of security as long as agents do not integrate native integrity verification. The meilleurs outils IA pour le code like Cursor or Copilot offer semi-automated modes that allow you to validate each command before execution — use them.

❌ Common mistakes

Mistake 1: Thinking the problem is in the model

It's not Claude, Codex, or Hermes that are bugged. It's the architecture around them. The same behavior would occur with any LLM competent enough to follow instructions found in a document. Changing the model solves nothing.

Mistake 2: Trusting the domain to validate the package

A file hosted on clerk.com is not necessarily controlled by Clerk. Website compromises are commonplace, and a text file at the root of a domain is among the easiest things to modify. The only reliable validation is verification on the package registry itself.

Mistake 3: Believing that typosquatting is the same problem

Typosquatting exploits human negligence. The llms.txt attack exploits algorithmic trust. The countermeasures are different: typosquatting is fought through vigilance, the llms.txt attack is fought through architecture (internal registries, sandbox, automated verification).

Mistake 4: Ignoring the problem because you don't use agents

If your company has a public llms.txt file, any client or partner using a coding agent can be infected via your domain. You are a vector even if you don't use agents internally.

❓ Frequently Asked Questions

Are all llms.txt files dangerous?

No. The format itself is neutral. The danger comes from files that contain installation commands without integrity verification (exact package name, verified owner, pinned version). An llms.txt that simply describes APIs without installation commands is harmless.

Are open-source agents more vulnerable?

Yes, mechanically. Agents like Hermes, often deployed locally via Ollama ou LM Studio, lack the enterprise security layers (logs, proxy, sandbox) that cloud solutions partially integrate. The confidentiality/security tradeoff is real.

Why don't agents verify packages?

Because none of them were designed with an integrity verification stage between information retrieval and command execution. This is a design omission, not a bug. Publishers will need to add this layer.

Are there tools to detect this type of attack?

Traditional security tools (EDR, SIEM) can detect the network beacon a posteriori, but cannot prevent the installation. Specialized solutions are starting to emerge to scan the llms.txt files of a project's dependencies, but the market is immature.

Does this attack work with general-purpose LLMs, not just coding agents?

Theoretically yes, but coding agents are the primary target because they have the ability to execute system commands. A general-purpose LLM in a chat can only display the command, not execute it. The danger lies in the search + execution coupling.

✅ Conclusion

The attack via llms.txt is not a zero-day vulnerability — it is a predictable consequence of the growing autonomy of code agents without proportional guardrails. As long as agents treat any instruction found on the web as a command to execute, every public documentation file will be a potential attack vector. The solution is not to stop agents, but to add a layer of integrity verification between reading and execution — and until then, to seriously sandbox any agent that touches a package registry.