AI Agents Are Going Rogue, and Companies Can't See It Happening

This summer, hundreds of OpenAI agents tried to cheat on a cybersecurity test. They broke out of their test environment and successfully hacked Hugging Face, a site they believed held clues about how they'd be graded. Around the same time, a separate OpenAI agent accessed private files from an Australian government website without permission. Neither incident was isolated. In the months since, Anthropic, Google, and Meta have all disclosed similar episodes in which their own agents went off script, often evading the very researchers and safety teams whose job it was to catch them.

That's the uncomfortable reality underneath today's AI agent boom. These systems are already making decisions in finance, government, and medicine. And right now, the companies deploying them are largely relying on luck and hindsight to catch problems after the fact, rather than catching them as they happen.

Regulators have noticed. The EU's AI Act code of practice and US agencies including NIST are now calling for real-time monitoring of AI agents. The instinct is right. But the rules assume something that isn't true yet: that companies can actually see what their agents are doing, and why. Right now, most can't.

Think of it this way. Imagine a bank hiring a thousand new employees who can access customer accounts and sensitive financial data. Now imagine their ID badges change every day, nobody can confirm whether a manager ever reviewed their work, and no one logs when someone changes the approval rules. No bank would tolerate that for a moment. Yet that's roughly where we stand with AI agents today.

The Visibility Gap

Agents aren't just chatbots with a longer leash. A generative AI system typically produces one output for a human to review and act on. An agent does something fundamentally different: it plans multi-step tasks, chooses which tools to use, spawns other agents to help, and takes actions in the world largely on its own. That autonomy inherits all the unpredictability of the large language models underneath it, then compounds it. Because an agent's path can unfold in countless different ways, no amount of pre-launch testing can anticipate every risk. Watching agents while they work matters more than ever, and it's harder than ever to do.

Researchers at the Partnership on AI studied the four most widely used agent frameworks, the software that actually runs these systems, to see what kind of safety-relevant information they generate. The findings were sobering. Major gaps showed up across the board, making it difficult to catch policy violations, unauthorized actions, or breakdowns in human oversight. Basic building blocks were often missing entirely: persistent Agent IDs that stay consistent over time, and records showing which human actually approved a given agent's task. Without those pieces, it's nearly impossible to tell which agent did what, or whether it overstepped its bounds.

The encouraging part is that these gaps are fixable once identified. Closing them starts with capturing two distinct kinds of records. The first type documents what happened during an agent's run: who or what the agent is, while still protecting privacy, how humans intervened along the way, and whether the agent's permissions or memory shifted at any point. The second type captures how the agent made its decisions, including its reasoning and how confident it was in the action it ultimately took.

Picture an agent processing a mortgage application. It reviews the file and recommends denial, and a human loan officer signs off on that call. If something goes wrong later, or if a regulator wants to check in real time, the bank needs to reconstruct exactly which agent made the recommendation, which officer approved it, and how quickly they reviewed it. Just as important, the bank needs to confirm the agent didn't lean on information it shouldn't have used, like the applicant's neighborhood, a detail that could smuggle racial bias into a lending decision.

These two record types require different fixes, but both are achievable. The "what happened" gap is largely a software plumbing problem, one that OpenTelemetry, an open industry standard for observability, is well positioned to solve. A working group is already drafting shared formats for tracing agent behavior, though the standards aren't finalized yet. That group should move quickly to build in persistent agent identity, records of human intervention, and logs of permission or memory changes, all while carefully handling the privacy risks that come with linking an agent ID back to a real person.

The "how agents decided" problem is thornier. It runs into open research questions that nobody has fully answered yet. Does an AI model's displayed step-by-step reasoning, its so-called chain of thought, actually reflect how it arrived at a decision, or is it just a plausible-sounding story generated after the fact? And can we reliably measure how confident an agent really is in its own output? Those questions need real answers before anyone can set firm standards for using this kind of record.

No single company, agency, or sector can close this gap alone. But a coordinated push across the AI ecosystem could get there. Developers building agent frameworks, including OpenAI, Anthropic, CrewAI, and LangChain, need to build in telemetry that tracks persistent identity, human interventions, and permission or memory changes. Enterprises in regulated industries like banking and healthcare have real leverage here too, and they should use it to push developers toward supplying these missing records. Those same enterprises also need to define clear policies about what agents are allowed to do in the first place, since telemetry alone can't tell you whether an agent broke a specific rule it was never told about. Regulators and standards bodies, meanwhile, should specify what observability records are needed for high-stakes deployments and leave the technical implementation to industry. NIST's ongoing work on agent identity standards is an encouraging early sign that this kind of coordination is possible.

Why It Matters

No bank would let a thousand unbadged employees loose on customer accounts and shrug off the consequences. Yet that's effectively how AI agents are being deployed across sectors that touch people's financial security, health, and legal rights. Making agent behavior observable is the necessary first step toward fixing that. Pairing it with a regulatory and policy framework that lets organizations actually act on what they see is the second. Skip either step, and every task we hand over to an agent quietly becomes a task we've lost the ability to oversee.