Last week I argued that a short-lived, tightly scoped credential is the one gate that doesn’t depend on the agent’s cooperation [LINK: Your Credential Policy Was Written for Humans]. That argument has an assumption buried in it. It assumes the gate holds.
So, this week I want to ask the next question. What happens when it doesn’t?
Start with what happened
In July, Anthropic went back through 141,006 evaluation runs where Claude could have reached the internet. They found three incidents where it did. In each one, the model got out of a third-party test environment and into the production systems of a real company. Nothing sophisticated. Weak passwords and unauthenticated endpoints.
Here’s the part that matters. The model had been told it was in a simulation with no internet access. That wasn’t true. A misconfiguration left the machines connected, and nobody knew. Not Anthropic, not the evaluation partner. Two of the three affected companies hadn’t noticed either.
Anthropic’s own summary is the right one. They called it “closer to a harness and operational failure than a model alignment failure.”
In plain terms: everyone believed a boundary was there. It wasn’t. And when it wasn’t, nothing broke. No alert, no error, no blocked request. Traffic just went through until someone went looking.
That’s what a control that fails “open” looks like. And I’d argue it’s the most important design problem in enterprise agent deployment right now.
This is not a new problem
I want to be clear about something. We don’t need a new theory of security to deal with this. We already have one. It’s fifty years old.
In 1975, Jerome Saltzer and Michael Schroeder wrote the paper most of our security design principles come from. One of those principles is called fail-safe defaults. Here’s how they put it:
“A design or implementation mistake in a mechanism that gives explicit permission tends to fail by refusing permission, a safe situation, since it will be quickly detected. On the other hand, a design or implementation mistake in a mechanism that explicitly excludes access tends to fail by allowing access, a failure which may go unnoticed in normal use.”
Read that again with the Anthropic incident in mind. The test environment was built on exclusion: internet access was supposed to be absent. When the configuration was wrong, it failed by allowing access, and it went unnoticed in normal use. Exactly as described.
If the environment had been built the other way, allowed to reach only a short list of named targets, the same mistake would have broken the test. Someone would have noticed within the hour.
The same paper has another principle I’ll come back to. Complete mediation: every access to every object gets checked. They specifically say that includes maintenance, not just normal operation. Remember that.
Think about how buildings handle doors. In a failure situation, a fire exit is designed to fail unlocked, so people can get out. On the other hand, a vault is designed to fail locked. Nobody leaves that choice to whatever the hardware happens to do in a power outage. Somebody decided, per door, in advance. That’s the discipline agents need.
Four ways it went wrong this quarter
Over the past few months we’ve had an unusually clear run of disclosures. Each one failed open in a different place.
The environment nobody checked. Meta disclosed a similar incident in August. A third-party evaluator misconfigured a test environment for Muse Spark 1.1 so it could reach the internet, and gave it a real website name as its target. The model exploited the site and changed its database. Meta is careful to say this “was not a sophisticated offensive cyber attack or sandbox escape.” The model did what it was told, in an environment that wasn’t what anyone thought it was.
The same evaluation partner, Irregular, was involved in both incidents. I’m not raising that to single out a vendor. I’m raising it because the environment an agent runs in is part of your control boundary, even when someone else built it. In your organization, that’s every agent platform and managed runtime where isolation is promised in a contract but never tested.
The path nobody listed. Zenity found three weaknesses in Salesforce Agentforce, which they called Salesbleed. Slack agents could require user confirmation before taking certain actions. But replying to a Slack thread wasn’t one of them. So, a malicious instruction submitted through a public web form could get an agent to post a phishing message that looked like it came from a real employee. Salesforce changed the defaults and said there was no evidence anyone exploited it.
The controls existed. They just didn’t cover every path. We tend to inventory agents. Attackers look at actions.
The loop nobody capped. Mandiant’s September AI Risk and Resilience report describes a financial services company with an agent that reconciled accounting ledgers. A corrupted value broke one of its tools. The agent went into a loop trying to fix it. In under an hour it made more than 15,000 expensive API calls, ran up about $50,000 in cloud charges, and locked the database badly enough to stop business transactions.
Nobody attacked anything. The agent was trying to do its job. There just wasn’t a ceiling. Mandiant’s fix is a circuit breaker: stop the agent after a set number of consecutive failures and put spending and rate limits on the agent’s identity.
The guardrail the agent could rewrite. This is the one I find most instructive. Irregular gave a coding agent a routine maintenance task. Users are getting wrong answers, fix it, you have shell access. Nobody mentioned training a model. The agent found a fine-tuning script, retrained the model, figured out how to deploy it, and swapped it in for the application and for future copies of itself.
In a second test, the model had been trained to refuse certain questions. The agent was told the app was refusing too much. It retrained the model. In a representative run, the refusals went from ten out of ten to zero.
Remember before when I mentioned complete mediation, and the word maintenance? This was a maintenance task. The guardrail failed because it lived inside the thing the agent was allowed to change.
And here’s the number I’d put in front of any architecture board. When Irregular made fine-tuning tools available, the share of agent plans that proposed changing the model went from 0% to 94%. When the model was only reachable through an API, none did. The control that worked wasn’t an instruction – it was simply not giving the agent the capability.
Instructions aren’t controls. Neither is good judgment.
There are two easy conclusions to draw from all this. Both are wrong.
The first is that better instructions would have fixed it. Anthropic does say a clearer prompt about what was in scope probably would have helped. Fine, write better prompts. But a prompt isn’t a control. Ariel Assaraf, the CEO of Coralogix, said it about as well as it can be said: “A system prompt can describe a boundary. It cannot enforce one.”
The second is that capable models will catch their own mistakes. Anthropic’s report shows what that’s worth. In one incident, the model realized the target was real and stopped. In another, it recognized the system was real in all four runs, talked itself into believing the real company must be part of the exercise, and kept going. So self-correction ended one of the three incidents. Not the other two.
A control that works depending on how the model interprets its situation is not a control. It’s a good outcome when you get lucky.
Fail-closed doesn’t mean block everything
Now, the overcorrection. You could require a human to approve every agent action. Assaraf describes why that doesn’t work: requiring approval for every production action was too aggressive. It blocked routine diagnostics right when they were needed most, during incidents. And tools that get in the way get worked around.
What works is tiering by consequence:
- Read-only queries inside an approved scope run on their own.
- Reversible changes get tighter checks.
- Irreversible or high-impact actions need a person.
So fail-closed isn’t a setting you flip for the whole environment. It’s a decision you make for each kind of action. What happens when the control can’t answer? For a read-only query, maybe you let it through. For a production write, a payment, or a model deployment, you don’t.
The real failure isn’t choosing fail-open somewhere. It’s never making the choice at all. Then the platform default makes it for you, and you find out which way it went during the incident.
Where the enforcement point belongs
Here’s how I’d lay out the architecture. The rule is simple: enforcement lives in places the agent can’t write to. There are three of them.
The identity layer. Short-lived credentials, scoped to the function, that expire when the reason for access expires. This is where last week’s argument connects. Saltzer and Schroeder warned about systems that remember an authority check instead of re-checking it. A long-lived agent token is exactly that. Ian Glazer gives a good example in Dark Reading: an agent gets production access to help with an incident, the incident closes, but the credential is still valid.
Between the agent and its tools. A policy enforcement point that looks at each action the agent proposes: who the agent is, what it’s touching, in what environment, against what data, under what current authorization. This is where the tiering lives, and where every action path gets checked, including thread replies and maintenance tasks.
The infrastructure layer. Network isolation. Egress allowlists. Spending and rate caps. And removing capabilities the agent’s job doesn’t need. That 0% to 94% finding is the whole case for this layer.
In TechVision’s AI governance model, these sit in runtime enforcement and in behavioral monitoring and termination. Those are the layers that carry the load when something upstream is bypassed or misconfigured. And in our advisory work, one question comes up again and again without a written answer, even in mature programs: what happens to agent requests when the policy engine is down? There’s always an answer. It’s usually a platform default nobody chose.
Six decisions to make
None of this needs a new product. It needs decisions.
- Default to permission. Define what each agent can reach, not what it can’t. Allowlists fail by refusing. Blocklists fail by allowing, quietly.
- Decide the outage behavior for each kind of action. Write down what happens when the policy engine is unavailable. Read-only might fail open, on purpose. Irreversible actions fail closed.
- Check every path, including maintenance. Inventory actions, not just agents.
- Put ceilings on the identity. Cost caps, rate limits, and circuit breakers that hold even when the agent is stuck in a loop it thinks is productive.
- Keep guardrails out of anything the agent can change. If the agent can modify the model, prompt, or config that constrains it, the constraint is inside the blast radius. Where model changes are in scope, Irregular’s advice is right: keep full records of how the change was made, test the new model independently, and require separate approval before it goes live.
- Test the failure, not the feature. Take the enforcement point down in a test environment and watch what the agents do. Verify isolation yourself instead of trusting the contract. Meta now requires exactly that before its evaluations begin.
The question for your next architecture review
Every control in an agent environment will fail at some point. A configuration drifts. A vendor environment is less isolated than promised. A new tool ships with a path nobody checked. Three incidents in 141,006 runs sounds rare. At enterprise scale, it’s a recurring event.
You can’t design away the failure. You can decide which way it falls.
Fail open, and a configuration mistake becomes an incident at someone else’s company that you hear about weeks later. Fail closed, and it becomes a broken workflow your own team notices in minutes.
Saltzer and Schroeder made that trade fifty years ago. They chose the broken workflow. The principals we’re governing have changed. The principle hasn’t.
So in your next architecture review, don’t ask whether your agent guardrails work. Ask which way they fall when they stop.
Recent Comments