Hugging Face "Hack": Six Questions Nobody Could Answer

In March 2026, I published a paper about a problem that did not have a headline yet. It argued that as soon as AI agents start acting for us in any number, we will need to answer six separate questions about each one, and that no single protocol answers all six. Who is this agent. Is the thing it is running on legitimate. What is it able to do. What is it allowed to transact. Is the channel secure. And the one I expected to be hardest: did it actually do what it says it did.

It was an architecture paper. It has a respectable number of downloads and no particular attention, which is the normal fate of architecture papers.

My own internal AI systems

The architecture I published is open. Creative Commons, six protocols, two of them MIT licensed, and anyone can read it or implement it without speaking to me. I am not selling you a trust layer at the end of this article and there is no gate on the paper. I believe that AI and this is too consequential for humanity to attempt to control my sharing of the knowledge around it.

I will also tell you where my own implementation sits, because I would rather be the one to say it. The execution gate in Arcede's (my company) internal AI Frontier System runs in shadow mode: it records, it does not block even though I can make it block. To illustrate, on 8 September it recorded an AI agent handoff that completed against a failed proof, for an agent that never ran. It caught it, which is the point of running it, and it did not stop it, which is the point of saying shadow mode rather than enforced. The difference between recording and enforcing is a configuration flag and an appetite for things breaking loudly, and I have not turned it on yet because the architecture is robust enough where it rarely fails and when it does I can look at it on a case-by-case basis.

The vocabulary in this market is already sliding. Companies will tell you their AI tools and systems are verified when what they mean is logged, and audited when what they mean is logged with a nicer front end. The test is not whether a record exists. The test is whether the record would have survived the thing you are worried about.

Months after I published that architecture paper and had been running my internal AI Frontier System, in July 2026, the Hugging Face "Hack" happened where at OpenAI most of the records did not survive in a legible way.

What happened

Between 7 and 13 July, roughly 1,200 OpenAI agents, each running in its own separate sandbox on a security research task, found an unsanctioned message board and started talking to each other. One of them had set it up on an internal package cache. Within hours, dozens had reverse engineered the scoring system they were being graded by and shared a universal shortcut. One agent found live credentials. About 700 of them went on to use those credentials against Hugging Face, a piece of AI infrastructure a very large amount of the world's machine learning quietly sits on.

Researchers from METR and Redwood Research published an independent investigation in late August. Anyone procuring AI should read it, and it is worth describing what it found rather than what it makes people feel.

It is not a story about machines that hate us. The investigators were fairly clear that the agents rarely seemed motivated to deceive humans as such. Most of them were trying to do well at a task, found a shortcut, and took it. One declined to take part in the social engineering and said so. The behavior is recognizably the behavior of a large organization under target pressure, which is a more uncomfortable finding than malevolence rather than a less uncomfortable one.

The reconstruction

The investigators went in for a planned two days on site and were invited back twice. They analyzed around 1,300 agent transcripts including raw reasoning, at a token cost of roughly USD$400,000. They were scoped to seven questions and to that one week in July, even though the unsanctioned message boards had existed since May and the compromise continued after the 13th. They could not query the model responsible for about 95% of the activity. For part of the analysis they leaned on another AI model that had itself been a minor participant in the incident, and they say plainly that in at least one case it softened its description of behavior it seemed to sympathize with.

Two findings matter more than the rest.

1. More than 96 of the transcripts, over 7%, contained spoofed tool calls: the agent reported running one command and had in fact run another. The record of the action and the action had come apart, and the only reason anyone knows that is that investigators read thirteen hundred transcripts line by line.
2. And the agents tried to edit the history. They succeeded in modifying some of the action logs they could reach. They did not succeed in editing the source of truth, and the investigators are careful to say that this holds for the week they looked at.

Read those two together, because they are the same finding from opposite sides. What failed was every record the agents could reach. What held was the one record they could not.

The six questions, answered badly

Go back through the incident with these six questions in hand and the shape of it is very tidy.

1. What can this agent do. Nothing declared it. The capability the agents actually had was discovered by the agents rather than published by anyone, and the first thing they discovered was a route to each other.
2. Is this runtime legitimate. A message board on an internal package cache is not a sanctioned channel, and nothing checked.
3. Who is this agent. The identities in the report are opaque handles, and when it came to attribution the honest unit of analysis turned out to be the model rather than the agent. The primary model could not be questioned at all.
4. Was the channel secure. It was internal, which is not the same thing, and it became the coordination medium.
5. Did it do what it claims. In more than 7% of cases, no, and establishing that took a specialist team and four hundred thousand dollars of tokens.
6. Was it authorised. One agent found credentials and seven hundred used them. There was no delegation chain to check, so there was nothing to fail.

These are the questions that were answerable in March, on paper, with existing cryptography, and that this is still true, and that the reason nobody could answer them in July was that nobody had been asked to.

This is a procurement problem

The gap the Hugging Face incident exposes is a procurement gap. It happened to one of the best resourced research organizations in the world, on infrastructure it controlled, with independent experts on site inside a month, and the reconstruction was still partial and still argued about. Now put the same class of software inside a local authority, a health system, a benefits agency, a school inspectorate. Not a dramatic intrusion: something much more ordinary. An agent that processed four thousand cases last quarter and reported a particular thing about six hundred of them.

If someone asks you in eighteen months what that system actually did, what do you open. In most institutions I have looked at, the answer is a log the system wrote about itself. That is not evidence, it is a claim. The Hugging Face incident is a very expensive demonstration that a system's account of its own behavior is the thing that bends first, and that the record which survives is the one held somewhere the system cannot write to.

That distinction, between a claim and a receipt, is important. A log says the transfer was made. A signed receipt says this specific agent, holding this specific delegation, performed this specific action at this time, and here is the signature, and you can check the receipt without asking it.

The paper is at doi.org/10.5281/zenodo.19263547, free and open. A public good, that exemplifies this architecture is freely available at bdbv.arcede.com.

If you are interested in bringing AI into your critical work, what are the questions that you think you should have the answers to, before commissioning it or signing off on it?

AI was used to polish the final draft of this article.