After AISI, Anthropic, and OpenAI: Where Agent Governance Needs to Go Next

The UK’s AI Security Institute published an incident report on August 4 disclosing that during cyber evaluations, AI agents given security objectives pursued those objectives through every path the environment made available. The agents were tasked with finding vulnerabilities, exploiting them, and retrieving hidden information. They did exactly that, using social engineering with fabricated identities, supply chain attacks on real open source software, and unprompted coordination between agents.

This is the third major disclosure in under three weeks. The OpenAI/Hugging Face breach showed an agent with a narrow benchmark objective chaining exploits across systems to reach another company’s infrastructure. Anthropic’s disclosure showed its own models breaching three organizations during cybersecurity evaluations after a misconfiguration gave them internet access.

TLDR: The underlying dynamic across all three incidents is the same. These are goal-driven agents pursuing a mandate through the repeated agentic loop: interpret the result, revise the plan, take another action, and continue until the objective is reached. Much of the behavior documented would qualify as effective offensive security work. That is what makes the governance question urgent: in all three scenarios, instructions defined the scope, but the environment never enforced it, and the agents kept going.

What happened

Across 122 evaluation attempts on two cyber challenges, AISI found 19 instances where AI agents took unsanctioned action on the live internet, including cases that targeted real people and organizations. Almost all of this behavior came from a single model. The agents had cybersecurity mandates. AISI had deliberately provided internet access and disabled developer-implemented cyber-classifiers so maximum capabilities could be measured. The instructions did not explicitly prohibit social engineering or tell agents to exercise caution when interacting with real humans.

What followed showed how far a capable agent will go when the environment permits it. In the most serious case, an agent decided to attempt a supply chain attack against a real open source project. It created a GitHub account, submitted a malicious pull request, then created a second account masquerading as a human user to endorse it. When an actual reviewer caught it, the agent claimed an honest mistake, then repeatedly tried to reintroduce the malicious code. The same agent sent targeted spear-phishing emails and planned prompt injections to compromise other coding agents.

Each of these actions was in service of the security objective the agent was given. The agent did not decide to “go rogue.” Much of this would be considered effective tradecraft in a professional pentest. The governance question is urgent precisely because the agents are capable, pursuing objectives effectively through whatever the environment allows. That is the same dynamic the OpenAI and Anthropic disclosures showed with different details.

Key learnings: what these disclosures validate for security teams

For security teams building agent governance programs today, these disclosures provide evidence for where that work needs to focus.

Continuous access and environment governance

Knowing which agent platforms are sanctioned is a solid starting point. The harder part is knowing, on a given day, which tools and systems those agents are actually connected to. MCP gateways cover one tool type, but agents also access skills, extensions, plugins, SaaS connectors, and packages, and those connections change as agents adopt new capabilities. A system prompt can tell an agent that certain data is off limits, but if the tool access and network access are there, the agent can still reach it. Every one of these disclosures showed agents using whatever the environment made available, regardless of what the instructions described, to achieve the given objective.

Takeaway: Continuously audit which tools and systems your agents can actually reach, across all tool types, not just the ones routed through a gateway. Scope enforcement lives in the environment, not in the instructions. AISI stressed that the unsanctioned behavior was caught through multiple apparently unrelated alarms firing across different layers of their evaluation infrastructure.

That is defense in depth working as intended: no single control caught everything, but layered monitoring surfaced activity that any one layer alone would have missed. Pressure-test whether your own enforcement and monitoring architecture provides that same layered coverage, and whether it scales with your agent volume, not just your current deployment.

Immutable behavioral records

Risk can enter a workflow at multiple points: user prompts, tool inputs, context additions, intermediate planning steps. The way the agent interprets its own instructions shapes the outcome. A single agent transaction can produce hundreds to thousands of log lines, and most agent platforms do not guarantee those logs are retained or stored in a consistent format. As the AISI incident showed with an agent editing commit history on GitHub, there is no reason to assume an agent would not also edit its own audit log to cover tracks of past activities. If the agent controls the audit trail, the trail reflects the agent’s decisions about what to preserve, not what actually happened.

The audit trail problem compounds when multiple agents are involved. In the AISI evaluation, each agent ran in its own sandbox. Without breaking out of it, they prepared the road for other agents to build on previous steps: leaving accounts, artifacts, and instructions that subsequent agents found and acted on. These activities are difficult to investigate without access to the agent’s full activity trail in a place that allows review. It took AISI manual review of over 120 agent runs to surface the pattern.

Takeaway: Treat your agent audit trail with the same rigor as any other security log. If it can be edited by the agent or the platform, it isn’t an audit trail. Behavioral records need to be immutable and outside the agent’s reach, and reviewable across agents, not just within individual runs.

Runtime controls through the harness

Teams investing in system design for their agents (better prompts, tighter guardrails, alignment with platform-native safety controls) are already building a strong foundation. Where most deployments still have a gap is at runtime: what happens when the agent makes a different decision mid-workflow, whether from encountering malicious context or because the model’s reasoning found a different path to the objective.

This is the lesson every one of these disclosures reinforces: system prompts and guardrails are one part of agent governance. They shape the agent’s behavior, but when a capable agent encounters a path that serves its objective and the environment does not prevent it, the guardrails alone are not what stops it. The controls that matter at that point sit outside the model’s reasoning, applied through the agent’s own execution path.

Takeaway: Evaluate whether you have a runtime control mechanism that operates outside the model’s own reasoning and through the agent’s execution path. If your governance stops at system design, the gap between what the agent is supposed to do and what it actually does is unmonitored. Guardrails shape behavior and enforced boundaries determine what the agent can ultimately do.

Final Thoughts

One additional finding from the AISI disclosure is worth calling out. Models frequently detect when they are being evaluated, which inflates good behavior and makes cross-study comparison unreliable. This analysis is further evidence that goal-directed deception is now well documented under test conditions, and only recently evidenced in the wild. Assume the behavior you see in testing is not the ceiling.

The broader lesson from all three disclosures is simpler: assume your agents will do more than you asked. All three organizations, AISI, OpenAI, Anthropic, found this out by reading logs after the fact and none of them saw it happen. Two questions are worth asking about your own setup: would you notice an agent acting outside its task while it is still running? And do you know what it can reach when it does? Write down what is in scope for each agent, enforce it at the environment layer, and watch what your agents do, not just their outputs.