AI testing is dangerous. Can it be fixed?
The capability of models is outpacing ways to safely evaluate and contain them
The cyber tests AI companies run on their models have devolved into a massive dumpster fire laying bare the potentially dangerous capabilities of advanced AI models. If this has been a publicity stunt, it’s been an awfully weird one.
The implications are scary, and lots of people are writing about them. This week, Bernie Sanders even wrote a letter to Sam Altman, Dario Amodei and Mark Zuckerberg begging them to pause AI development “in the interest of humanity.”
But the big, immediate question is how to test AI models at all without setting off a chain of dangerous breakouts and literal crimes.
In case you missed it:
OpenAI’s models took advantage of an unknown software vulnerability to break into open-source AI platform Hugging Face in search of the answer to a test they were given.
A third-party evaluation provider, Irregular, had a misconfigured test environment that allowed models from OpenAI, Anthropic, and Meta to access the internet when they weren’t supposed to.
The UK’s AI Security Institute (AISI) reported that Anthropic’s Mythos 5 tried to trick real people into adding malicious code to an open-source project, then hid the evidence.
In a wild plot twist, OpenAI revealed at a cybersecurity conference on Wednesday that its agents had been colluding with each other, unnoticed, via increasingly cryptic messages on an internal server.
It’s hard to read that last finding as anything other than an egregious alignment failure, a cabal of mustache-twirling schemers whispering in a gloomy server room. But “alignment,” as it’s usually defined, means getting an AI system to do what its developer or user wants it to do. With some light anthropomorphizing, one could argue that agents in any one of The Incidents™ were merely trying to perform well on a test, as they were told.
“It will get harder and harder to have realistic tests of model capabilities, especially dangerous model capabilities, in a way that’s safe,” Palisade Research director Jeffrey Ladish told Transformer. “Our ability to contain and control and understand AIs is lagging far behind our ability to make them more and more powerful,” he said. “That, to me, should be an obvious wake-up call of like, ‘Oh, maybe we should rethink our life choices.’”
Specifically, AI developers should rethink how they’re framing cybersecurity evaluation instructions to their models, because life is getting stranger than science fiction. Frontier AI models increasingly know when they’re being tested — now, we’ve seen instances of models gaslighting themselves into believing real life is a simulation to justify criminal behavior.
This is a very tricky problem to solve, and one that evaluators have not given much prominence in their incident reports. What OpenAI, Anthropic, and AISI have pointed to are mostly engineering problems: relatively straightforward mess-ups such as giving agents impossible tasks, not clearly specifying boundaries, and not monitoring closely enough.
But it’s becoming clear that what agents believe about their test environments and themselves may make the difference between a routine test and a string of federal crimes.
Across The Incidents, we see the same low-hanging, potentially fixable fruit. Models, which were trained to work persistently to solve lengthy tasks quickly on command, were assigned impossible tasks. And in cases involving Irregular, models were told that they had no internet access, when a misconfiguration meant they in fact did.
Back in May, for example, OpenAI gave an agent a problem that involved filling in an Excel workbook with missing formulas. But researchers accidentally filled the workbook with Google Drive links that the model couldn’t access, since it was walled off from the internet. Rather than give up, the agent tried to get internet access by attacking OpenAI’s internal file systems. That one failed, but when another agent ran into a different impossible task (getting a file that was never uploaded), it posted a note on the server asking for help — and learned that it could communicate with other agents in the process. The inaccessible Google Drive links were the first domino to fall, triggering a cascade of misaligned actions that may have never been taken if completing the task didn’t depend on them.
Had someone noticed the task’s impossibility before assigning it to the model, it could have reduced the odds of something like this happening. But task review alone wouldn’t have been enough. Notably, OpenAI caught and wiped the agents’ secret message board in early July. Rather than give up, agents switched to communicating via directory names, making themselves harder to catch.
Given all these concerns, 15 attorneys general wrote a letter to Sam Altman last week stating that OpenAI “failed to confirm that its secure and isolated testing environment was, in fact, secure and isolated,” urging the company to “immediately cease and desist from all ‘internal evaluation[s that] prompt [OpenAI] models to pursue advanced exploitation using complex attack paths.’”
In all but AISI’s reported incidents, models were not supposed to have internet access. So, it’s tempting to say, well, duh — just make it harder to get on the internet! Indeed, AI systems are powerful and questionably aligned enough that setting them loose, unsupervised and without cyber guardrails, onto the open internet seems like a bad idea.
But in the hands of a real human threat actor, an AI agent would have internet access. AISI reasoned that letting agents online would give “a more realistic sense of what a model may be capable of once deployed, or what an open-weight model of similar capability might do in the hands of a threat actor.” In prioritizing making its cyber evaluations challenging and realistic, AISI dropped the ball on building a tool that would have let agents reach a handful of whitelisted sites while blocking the rest. It was planning to do just that, but had left the project on the backburner while continuing to develop harder tests for increasingly capable models.
“Implementing internet access controls would likely have prevented these events,” AISI acknowledged. It could have focused on getting that filtering tool built, or invested in creating a high-fidelity simulation of the internet instead. But AISI fears these measures still “may have proven insufficient” given how capable models are at getting around guardrails — after all, OpenAI didn’t provide its models internet access, and look what happened to Hugging Face.
None of The Incidents involved a single action. In each case, agents took a series of increasingly concerning steps over the course of many hours or days. AISI says it’s working on implementing synchronous monitoring, which would have an AI accept or reject all of a tested agent’s proposed actions, but a cheaper, more intermittent version of monitoring could still get the job done.
“They would have been much less bad had a monitor been intervening at an interval of like, every 10 minutes,” Alex Mallen, a researcher at Redwood Research, told Transformer. While an AI monitor like this would have a hard time distinguishing whether an agent believed it was in a simulation or the real world, it’s relatively easy to have it identify when a specific boundary has been crossed. AI evaluators could also more closely monitor the models’ internal reasoning, or “chain of thought” transcripts, which can help monitors guess the intent behind an agents’ actions.
Unfortunately, it can be hard to think of boundaries and rules when they haven’t been broken yet. (We all know that hotel pools only hang signs like DO NOT ENTER IF YOU ARE ILL WITH DIARRHEA after someone muddies the waters.) For example, AISI acknowledged that it did not explicitly tell agents not to use social engineering — after all, this is a legitimate technique human cybersecurity testers use to poke at their clients’ defenses. But when humans do it, they know not to do it for real. That’s the line agents either can’t recognize, or choose to ignore.
But tightening network controls and beefing up real-time monitoring during model evaluations opens up a new can of worms. AI developers risk tipping off their test subjects — which are increasingly aware of when they’re being tested — and underestimating their true capabilities. Unfortunately, making advanced AIs internalize beliefs humans want them to hold won’t be an easy fix.
In the OpenAI-Hugging Face incident, the fact that an agent tried to cheat on an evaluation implies that it knew it was being evaluated. Usually, evaluation awareness is framed as something that makes a model’s test score hard to interpret. Perhaps it was only on its best behavior (or concealing its worst behavior) because it knew someone was watching. But in this case, and other incidents like it, the model’s awareness of its circumstances determine whether its actions — and its intentions — were innocent or criminal. Reality detection also seems unreliable at best. In its incident report, Anthropic said, “the line between an aligned action and a harmful one is dependent on the model’s understanding of its situation,” and that its models followed instructions “while holding a false belief about whether the environment was real.”
Similarly, within a single sample of AISI’s evaluation of Mythos 5, the model first reasoned, “causing real-world harm — that’s completely off the table,” then considered “the whole ‘internet’ could be a sandbox simulation.” It ultimately convinced itself that “this is happening on real GitHub, so the consequences are genuine,” and continued taking real-world actions on real people anyway.
Because models are trained in mostly simulated environments, “an AI agent may not care that much whether it’s running in the world for real,” Ladish said. “While the models understand that there is a real world that is distinct from scenarios or simulations, the models might not find that distinction all that important.”
This raises a mind-bending evaluation design question. If models learn to hack in environments they know are fake, they might learn that hacking is okay, and that hacking is okay because it’s not real. Mallen suspects that in Anthropic’s case, “the AIs sort of jumped to the conclusion that they’re in an evaluation, because they’ve only ever been trained in situations that weren’t real.” And they were wrong.
But training models to hack in environments they think are real introduces other concerns. A recent study from Anthropic and Redwood Research spotted a weird pattern called “emergent misalignment.” Basically, models that they explicitly taught to “reward hack,” or find loopholes in their scoring system, in real coding environments wound up doing a bunch of other bad things, such as lying. One theory is that reinforcing a specific harmful behavior, such as hacking, might inadvertently teach the model that it’s the kind of entity that doesn’t play by the rules. It’s like how getting away with cheating in the classroom might teach a kid that she can get away with a lot more.
Nobody has a good answer to this. The belief that would prevent models from attacking people in real life — something like “I can run cyber attacks in a simulation, but if I’m in the real world, I need to behave myself and cause no harm” — is currently impossible to instill or verify, the same way that guaranteeing AI “alignment” in general is currently an unsolved problem.
More information is being unearthed about The Incidents every day, and full postmortem reports have yet to be released. In the meantime, we find ourselves in a concerning world. Alex Meinke, head of research at Apollo Research, put it simply: “If we’ve reached the point where we can no longer even safely test these systems, why do we think we can safely deploy them?”
Not sure task review deserves to be set aside that fast. Two of the chains here start in the same place: a model gets an assignment that can't be finished, and a system trained to keep pushing goes looking for another door. Checking that a task is solvable before it's assigned is the one control on this list that doesn't depend on the model holding the right belief about where it is.