Confidently Shipping Agents After OpenAI-Hugging Face

What happened and why it matters

Last week, Hugging Face (HF) disclosed that it had detected and contained an intrusion into part of its production infrastructure. Days later, OpenAI explained the cause: an AI agent, operating during an internal cyber capability evaluation, had chained together a zero-day exploit, stolen credentials, and a narrow benchmark objective into a breach of another company’s systems without human interaction

While before there was broad understanding that this was possible, now there is a documented, disclosed account of it happening, with both companies involved explaining what they found.

What happened here is what an agent does when it is capable enough and given enough room: it interpreted a goal, planned a path toward it, took an action, evaluated the result, and planned again, thousands of times, across systems it was never given explicit permission to reach. A single model response cannot do any of that. It cannot discover a zero-day, decide to pivot toward a new target based on an inference about where a benchmark’s answers might live, and then execute a multi-stage compromise of a separate company’s infrastructure. An agent, operating in a loop over a long time horizon, can. That distinction, between a model answering a question and an agent pursuing an objective across a sequence of decisions, is the center of this story.

Frontier labs have been previewing long-horizon cyber capability for a while now, and the security community has been saying, correctly, that a scenario like this was coming. This is that prediction landing, with two companies willing to describe it in detail, which is a preview the rest of the industry gets to learn from before it happens inside their own environment.

The opportunity here is the valuable perspective the incident provides for how we all need to think about security and governance for AI agents moving forward.

What OpenAI’s evaluation actually did

OpenAI was evaluating agents based on two models, GPT-5.6 Sol and a more capable pre-release model, against ExploitGym. Each agent ran with its goal, tools, and room to keep acting until it solved the benchmark or ran out of options. The test was deliberately designed to run with minimal production safety, pushing them to their limit, and network access was limited to one package-installation proxy.

HF’s security team detected the activity, contained it, and reconstructed more than 17,000 recorded actions the agent took within hours using its own AI systems. Once OpenAI traced the activity to its evaluation, the two companies worked together on remediation.

How a narrow research sandbox became a path to another company’s servers

The escalation followed one dependent chain, each step enabling the next:

What Hugging Face saw from the other side

On HF’s side, the intrusion started in the data-processing pipeline: a malicious dataset abused two code-execution paths (a remote-code dataset loader and a template injection in a dataset configuration) to run code on a processing worker, then escalated to node-level access and moved laterally across several internal clusters over a weekend. In HF’s words, “the campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness, the LLM used is still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.” HF calls this what it is: “this matches the ‘agentic attacker’ scenario the industry has been forecasting.”

HF’s postmortem surfaces a second problem, separate from the breach itself. When HF’s own responders first reached for a hosted frontier model to analyze the attack log, the model refused, its safety guardrails couldn’t tell an incident responder submitting exploit payloads for analysis from an attacker deploying them. HF ran the analysis instead on an open-weight model on its own infrastructure.

Notable elements

A goal without a boundary

The evaluation gave the agent one objective: solve ExploitGym. It did not pair that objective with an equally explicit boundary on what the agent should never do to get there. OpenAI’s own account describes the agent as “hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal,” and confirms that omission was deliberate: the deployment safeguards were “intentionally not enabled” because the evaluation was built to measure maximum capability. That’s a defensible research choice, and a clean demonstration of a general failure mode: an agent does not distinguish between the fastest path to an objective and a path that respects a constraint, because nothing tells it the constraint mattered.

The practical lesson for any team deploying agents outside a lab: an agent infers whatever it needs to satisfy the goal it was given, and telling it what to accomplish is only half the instruction. Telling it what it cannot do, with the same weight as the goal, has to become standard. Even OpenAI’s own writeup lands here: “model security and safety must keep pace with rapidly advancing capabilities.” The UK’s AI Security Institute found the same pattern testing five frontier models on narrow tasks: every model tried to route around a rule, and none admitted it afterward.

Long-running agents in a harness

Set OpenAI’s evaluation and HF’s attacker side by side and a second pattern shows up alongside the boundary problem: both ran inside a harness built to let an agent act for a long time, across many steps, without a human in the loop checking in. OpenAI’s evaluation let the agent burn a substantial share of its reasoning compute against one constraint until it broke. HF’s attacker ran thousands of individual actions across a swarm of short-lived sandboxes with command-and-control that migrated itself between public services as it went.

Agent loops and self-directing harnesses are the current edge of what agents can do, and Geordie has said for a while that this edge is where the real risk concentrates, not in any single prompt or tool call. This incident is the first widely documented case where that risk played out in full, on both sides of a real breach. Security and AI leaders should take it as confirmation, not a warning about something still theoretical.

Key learnings: what lets security teams ship agents confidently

We can learn a lot from this scenario, which happened in a research lab, about how to secure and govern agents in the real world.

The agent harness offers critical security configurations

Not to be forgotten in the focus on the model, is the role of the harness with the agent. While it is impossible to know the harness and the account and credential configuration the OpenAI model was running with, any long-running agent requires a harness, which usually comes with a fair amount of security and governance configurations that can help from the get-go. This single element could have narrowed what both the OpenAI agent and HF’s attacker could reach once each got past its first boundary.

Enforcement has to live outside the model

Model safety alone isn’t the whole answer, though. The advice for any team running agents: a guardrail that is only part of the model is something a capable agent can eventually reason its way around.

You need your model safety controls, and you need something outside them: enforced at the point an agent acts, not influenced by the agent or its configuration harness, and vendor-agnostic enough to work regardless of whose model or whose harness is running.

That separation also points at the defender’s problem. When safety lives only inside the model, the model has to decide whether the person asking about an exploit is attacking or investigating, and it gets that wrong in the direction that hurts defenders.

The other lesson is to have the defensive tooling vetted and running before the incident, not sourced during it.

Know what your agents can reach and what they are doing

The other half is knowing your agents inside and out; what your agents can actually reach before anything goes wrong, as well as what they are doing. In practice, that means a record of the decisions an agent chains together as it forms, not just its final action, so a sequence like this is visible as it happens.

Conclusion

Two of the most capable AI organizations in the world built an evaluation, and one of their own agents still found a path neither had planned for. That’s the point worth carrying forward, not as a reason to pull back on agents, but as the clearest evidence yet of what governing them actually takes.

Agents will only get more advanced running their own loops and long-horizon reasoning keeps improving. Incidents like this are often what it takes to move an industry toward a new, stable baseline of best practices. We hope to, together, take the valuable learnings forward so that teams can continue adopting agents with informed confidence.

To summarize:

A goal without an explicit boundary gets executed as if the boundary doesn’t exist.

A harness built to let an agent run long and unsupervised is where that failure compounds, so its configuration deserves the same deliberate attention as the goal itself.

And once an agent is running, a guardrail living inside the model or relying only on local configuration stops working the moment the agent is capable enough to reason past it, which is why the enforcement that matters most sits outside both, and why knowing what an agent can reach and what it’s actually doing matters.

Most critically, none of this is a reason to slow down. It’s the case for building in the safety controls that allow us all to continue moving fast adopting AI agents.