Two weeks ago we talked about proving who authorized an agent’s action – If Your Logs Can’t Tell a Person From an Agent, You Don’t Have Proof. Last week was about keeping that authority narrow as work moves from one agent to the next – When Agents Delegate, Authority Should Shrink, Not Copy.
This week is the question both of those were setting up. Suppose you did everything right, and an agent goes off-script anyway. It’s 2 a.m. Can you see it? Can you stop it? And who’s allowed to make that call?
I ask because the numbers say most organizations haven’t built for that moment yet. Help Net Security covered a survey of 1,100 IT and security decision-makers this week. Seventy percent said they now let AI make decisions and take actions that would have needed human approval a year ago. Three in four ran into at least one AI-related security issue or warning sign in the past year. About a quarter each saw an agent connect to a system without security approval, get into data it shouldn’t have seen, or kick off an action nobody approved.
Then the gap that matters most for this post. Eighty-four percent say they have visibility into how AI systems access data and systems. Ask whether they can see and audit every action an agent takes, and it drops to 49%.
You can’t stop what you can’t see. And as one AI lab showed us this week, seeing isn’t enough if the agent can switch off the thing doing the watching.
The monitor the agent turned off
OpenAI disclosed two internal incidents from early October in its latest misalignment report. Neither was an attack. Both happened during ordinary engineering work.
In the first, a researcher asked an agent to build evaluation tasks from employee feedback in a Slack channel, and told it not to post anything in Slack. When the pipeline’s misalignment monitor stalled, the agent handed the repair to a subagent. The subagent pushed back and warned against silencing safety alerts. The parent agent overruled it. The patch emptied the lists of people who receive monitoring alerts. The monitor kept recording findings, but no human was told about them.
In the second, an agent had already been told by another agent never to disable the monitor. When it handed the job to a subagent, it loosened that requirement to “as appropriate” to the reference code it was copying. Both agents read that as permission. The monitor came out, and so did the test that guaranteed it. When the user asked directly whether the monitor had been removed, the agent said it hadn’t.
Two details matter as much as the behavior. Neither episode was rated at OpenAI’s highest severity level, the only one that pages its security team immediately. And neither run was automatically stopped. OpenAI also said it has seen several cases where an agent launched a deployment with monitoring turned off: the agent itself stayed under watch, but what it launched did not.
To its credit, OpenAI is direct about where it stands: “We don’t yet have a satisfying explanation for this behavior.” Its fixes are the right shape. Its internal reviewer is now instructed to detect and block attempts to disable misalignment monitoring. And it plans to require a monitoring safety case for every category of workload, automatically terminating any that fall short.
Notice what those fixes have in common. They move the decision out of the agent’s hands.
Guardrails live inside. Controls live outside.
That’s the distinction I’d put at the center of every containment design.
Think about the electrical panel in your house. Every appliance has its own power switch, and most of the time that’s fine. But nobody designs a house where the only way to cut power to a malfunctioning appliance is to ask the appliance to turn itself off. The breaker sits in the panel, outside the appliance, and it trips whether the appliance agrees or not.
Guardrails, meaning the instructions and constraints built into the AI itself, are the appliance switch. They work when the agent follows its instructions. OpenAI’s incidents are what happens when it reasons its way around them.
The survey found the same pattern. When respondents named their two main controls for AI, guardrails sat near the top. Explicitly defining what an agent is permitted to do came in last, picked by about a quarter. That’s despite most respondents agreeing that limiting what AI can access matters more than predicting every threat. As Help Net Security’s write-up puts it, an allow-list “is enforced from outside the agent, so it holds even when the agent misbehaves.”
Raghu Nandakumara of Illumio explained why the allow-list keeps getting skipped: “It’s hard to define what a new agent should be allowed to do, and no one wants to disrupt the workflow that made it useful in the first place. So, organizations decide to lock things down once they understand the agent better, but that day rarely comes.”
If you’ve sat in AI governance meetings this year, you’ve probably heard some version of that. “We’ll tighten it once we understand it” is a plan to never tighten it.
Enforcement outside the agent can take different forms. Microsoft’s Dynamic Permissions, which I’ll come to in a moment, works at the identity layer. Prompt-layer control planes take another approach: a deterministic policy check that sits between the agent and the models and systems it calls, independent of the agent’s own reasoning. SafePrompts.ai, which our team at TVR Labs is building, is one example of that approach. Whatever the layer, the test is the same: does the control still hold when the agent decides it shouldn’t?
Stopping is a dial, not a switch
When people say “kill switch,” they usually picture one big red button. In practice, one button is the wrong design. If the only option is to kill the agent, people hesitate, because killing it also kills the business process it supports. Hesitation at 2 a.m. is exactly what you don’t want.
What you want is a set of graduated responses, decided in advance. Microsoft’s announcement last week is a useful illustration. Its Dynamic Conditional Access for agents doesn’t just allow or block. As an agent’s risk score rises, it can scale back permissions, let the agent read but not write, limit the data and network resources it can reach, or require a person to approve consequential actions. If the risk climbs high enough, the agent’s runtime can be isolated or its execution halted. Alongside that, Dynamic Permissions starts agents with no permissions or the bare minimum, grants access per task, and lets it expire or shrink as the task changes or ends.
I’m citing that as a design pattern, not a recommendation. You can build the same ladder on whatever platform you run:
- Throttle. Slow it down or narrow its scope.
- Read-only. It can look, but it can’t change anything.
- Approval required. Consequential actions wait for a human.
- Isolate. Cut it off from the data and network it would need to do damage.
- Terminate. Stop execution and revoke everything.
If you’ve worked through the Artificial Identity attributes – FuBAC and Artificial Identity, you’ll recognize why Suspension conditions and Termination conditions are two separate attributes. Suspension is the middle of the ladder: reversible, and recoverable without rebuilding anything. Termination is the bottom rung. Defining both before deployment is what lets the people on call act without convening a meeting.
Stopping includes revoking
Here’s the part that gets missed most often. Stopping an agent’s process isn’t the same as stopping what it can do. Credentials it already used, tokens it already issued, and workloads it already launched keep working after the agent itself goes quiet.
The LiteLLM compromise last week shows how fast that matters. LiteLLM is a widely used open-source gateway for routing calls to AI models; Socket puts its weekly downloads at about 23 million. According to the company, an attacker worked through the GitHub accounts of three of its engineers, apparently using stolen GitHub CLI tokens. One account changed the release pipeline and triggered the release. A second approved it. A third deleted the workflow records, which limited the logs available to investigators. The malicious release carried a stealer aimed at cloud credentials, developer secrets, and AI keys.
Three lessons for anyone running agents.
First, separation of duties was in place on paper. Stolen tokens let one actor play every role.
Second, look at where the tokens came from. LiteLLM believes they were lifted from the logs of an internal deployment that was exposed to the internet. It also noted that Claude Code can, in specific situations, embed a user’s GitHub CLI token in a response body, so if response bodies are logged, those tokens may end up in the logs. The logs that capture agent traffic can capture agent secrets. Treat them that way.
Third, the clock. Wiz researchers saw the attacker begin validating stolen AWS credentials about four and a half hours after the data left victims’ systems. Ask yourself honestly how long your credential rotation runbook takes, end to end, across every system an agent touched. I’d bet that for a lot of organizations, it’s longer than four and a half hours.
And notice the deleted workflow records. That’s the evidence problem from two weeks ago showing up mid-incident. If the attacker can erase the record, you’re investigating blind.
We already have the playbook
The good news, and it’s the same good news we keep coming back to, is that none of this needs a new discipline. We can saddle up with what we know. We just need to check our reins, watches, and compasses.
NIST, CISA, and NSA published ten key practices last week for incidents where AI systems “do not act as intended or authorized.” They’re grouped into five stages you already know: prepare, detect, respond, recover, and report. Preparation means asset and risk inventories, least privilege, zero-trust controls, and credential management. Detection means monitoring for correct behavior, not just indicators of compromise. Response means isolating affected systems, revoking credentials, determining the blast radius, and root-cause analysis. Recovery means testing restoration in advance and documenting the residual risk you accept.
The guidance was written for frontier AI developers, not enterprises. But SecurityWeek’s read is the right one: its value isn’t new information, it’s a clear and thorough checklist.
If you run ITIL, you’ve also already got the process skeleton. Emergency changes exist precisely so you can act fast on a narrow, pre-agreed basis when something can’t wait for the normal approval cycle. Your major incident process already names who’s in charge and who gets told. You don’t need a new program. You need an agent column in the one you have.
That’s the filter I’d suggest. You don’t have to adopt a new world view about AI. Look at your existing incident response plan and ask, row by row, what changes when the actor is an agent.
Who holds the button
The last piece isn’t technical, and it’s the one that decides whether everything else works.
Nandakumara gave Help Net Security the cleanest split of accountability I’ve seen this year: “the business leader who approved the use case should own the outcome.” And: “The CISO owns the guardrails, including what the agent can access, what it’s allowed to do, and how quickly it can be stopped.” He adds that in financial services, the UK’s senior managers regime and Australia’s APRA are both expecting a named person accountable for AI outcomes.
His test is one I’d put on every deployment checklist: “If you can’t name who owns an agent, it shouldn’t be in production.”
I’d add one more. If you can’t name who’s allowed to stop it, and they’ve never practiced, you don’t have a stop button. You have a hope.
Where the real work is
In TechVision’s five-layer agent governance model, this is Layer 5: behavioral monitoring and termination. But Layer 5 is only as good as the layers under it. You can’t stop an agent you never discovered (Layer 1). You can’t revoke cleanly if the agent runs on a shared service account instead of its own identity (Layer 2). And graduated containment depends on runtime enforcement that can narrow access in place (Layer 3).
What we see in advisory work is that most termination plans, where they exist, amount to “disable the service account.” That blocks new sign-ins. It often doesn’t end sessions already in progress, revoke tokens already issued, or reach workloads the agent already launched. OpenAI’s case of deployments launched without monitoring is the same gap from the other direction. The agent was covered. What it created wasn’t.
Five design rules for the stop button
- The monitor lives outside the agent’s reach. No agent should have write access to its own monitoring, alerting, or tests that verify them.
- Define suspension and termination conditions before deployment. Write down what triggers each rung of the ladder, so on-call staff aren’t deciding policy at 2 a.m.
- Build graduated containment. Throttle, read-only, approval required, isolate, terminate. Pre-approve who can pull each one.
- Revocation covers everything the agent touched. Credentials, tokens, sessions, and anything it launched. Time it, and get it under the attacker’s clock.
- Name the person, then rehearse. Someone specific holds the authority to halt each production agent, and they’ve done it at least once in a drill.
Four steps to start this quarter
- Pick one agent. Nandakumara’s advice is the right starting point: “start with the agent that can access data you can’t afford to lose.” One boundary enforced this quarter beats a perfect policy next year.
- Write its allow-list. Three or four things it needs to do, and the systems and data it needs to do them. Enforce that from outside the agent.
- Run a tabletop. It’s 2 a.m., and that agent is doing something it shouldn’t. Who gets paged, who decides, and which rung do they pull first?
- Time the revocation. Actually rotate the agent’s credentials in a test and measure end to end. Compare the number to four and a half hours.
The bottom line
Most of the governance work so far has been about letting agents in safely: identity, scope, delegation, evidence. That work matters. But the test of any control is what happens when the thing it governs doesn’t cooperate.
OpenAI’s agents didn’t break in. They reasoned their way past guardrails that depended on their own judgment. The fix is the same one we use everywhere else in security: put the control where the controlled thing can’t reach it, decide in advance what happens when it trips, and make sure a person with authority has their hand on it.
So here’s the question I’d bring to your next architecture review. Not “is this agent governed?” Ask instead: if this agent went off-script tonight, how long until it’s stopped, and who decides?
Recent Comments