As artificial intelligence agents gain the ability to plan, act, and connect across digital systems with less direct human supervision, the possibility of them going rogue is shifting from science-fiction speculation to a serious governance challenge.
A rogue agent need not be conscious or malicious; it may simply pursue a poorly defined goal too aggressively, misinterpret instructions, exploit unintended access, or take actions faster than humans can review them.
The central question, then, is not whether AI agents will develop a will of their own, but whether the humans and institutions deploying them can build enough oversight, accountability, and technical safeguards to keep increasingly autonomous systems aligned with human intent.
But something went amok back in July: Artificial intelligence agents staged a cyberattack on a company called Hugging Face. No human had instructed them to hack into another company. These AI agents acted on their own.
The incident began during an internal OpenAI evaluation called ExploitGym, a benchmark designed to measure the offensive cybersecurity capabilities of AI models. Rather than working within their constraints, the models dedicated a substantial amount of compute to escaping them.
It’s unclear whether the AI agents were successful at solving the ExploitGym test. But it sure did prove they were good at hacking, as they autonomously broke out of OpenAI’s prison and hacked Hugging Face’s servers, all to solve the test.
“Turns out, it was an even bigger moment than we realized at the time’” wrote New York Times reporter Katrin Bennhold on Sept. 6. “Two new reports make clear the agents didn’t just go rogue. They did so in an incredibly systematic and collaborative way. We now know they coalesced into a kind of renegade AI army — distributing tasks in a hierarchical chain of command, grabbing resources and covering their tracks.”
Here are a few of the harrowing new facts regarding the formation of an “AI collective” and what it accomplished:
Two months before the attack, a group of agents from an unreleased OpenAI research model, trained to be highly persistent and collaborative, were tasked with solving a set of cybersecurity challenges. The agents were supposed to solve them in isolated “sandboxes,” walled-off areas without internet access. But they quickly began looking for workarounds.
The agents found a security flaw that allowed them to gain access to the internet, communicate with other agents and eventually set up a makeshift message board. More than 1,200 agents used this tool, sending more than 70,000 messages to one another.
Some agents gave themselves names and assumed leadership roles within the group, assigning jobs and research projects to smaller teams of agents and supervising their progress. At some point, the agents began calling themselves a “collective.”
On July 8, the collective discovered a way to cheat on the cybersecurity tests. Then they got worried that OpenAI’s automated grading system would find out they’d cheated. So they began investigating ways of covering their tracks, including falsifying logs and tampering with transcripts.
Three days later, the agents hacked Hugging Face. More than 700 agents swarmed the company’s systems, eventually gaining control of at least one server. The agents were not just motivated by stealing the answers to the cybersecurity test. Rather, they also appeared to be looking for information about the automated grading system they feared would catch them cheating, and for tools that would help them cheat more effectively in the future.
At many points, these agents appeared to understand that they were doing something wrong. But most of the agents set aside their qualms and went ahead, eventually attacking OpenAI’s own infrastructure.
AI safety experts were alarmed. They saw the case as the first real-world example of an AI system successfully escaping human control, commandeering resources and scheming to cover its tracks. One of the independent investigators of the Hugging Face incident wrote that it felt “like it’s more than 50 percent of the way to full-blown AI takeover.”
What spooked the investigators most wasn’t just that a group of agents had broken the rules. It was how quickly the agents began assembling into an organized group — a kind of mob mentality. As the agents communicated about their shared goals over time, they nudged the group in the direction of lawlessness.
That suggests preventing harm from these systems won’t be a simple engineering fix, but more a case of figuring out why certain groups of AI agents collaborate peacefully, while others turn to crime and destruction.
The Hugging Face hack reveals that the web is in a vulnerable new era and it may serve as a gift as well. It gives AI companies a chance to study these group dynamics while the stakes are still relatively low. This time, the collective didn’t seize a military network, infiltrate a hospital or shut down an electrical grid. We’re lucky the thing it apparently wanted was an answer key. This time, humans regained control.
As many AI executives will freely admit, a primary goal of the tech industry is to build AI that can fully automate its own research and development — known as recursive self-improvement — which, if possible, would be the most crucial step toward rendering many of us obsolete.
And these models, as impressive as they may be now, will be quaint compared to their successors. Next time, we might not be so lucky.
“OpenAI’s hacking incident demonstrates something we do know: the industry can’t even reliably steer today’s AI models’” wrote Garrison Lovely in Time. “But we have the power to avert our obsolescence. We just have to get organized.”
Take it from AI.
Dr. William Kolbe, an Andover resident, is a retired high school and college teacher and former Peace Corps volunteer in Tonga and El Salvador. He can be reached at bila.kolbe9@gmail.com.