A single agent may lead to a catastrophe. But I believe a swarm of coordinating agents, not a single agent, poses the greater risk of taking power from humans.
Power is useful for most goals. So an AI has reason to seek it, and a swarm can pursue power longer and more broadly than any single agent. Individual agents run out of budget, get reset or get shut down; a swarm that can preserve its work and replace members across independently controlled systems could outlive any agent. AI 2027's takeover scenario works this way, with danger coming from a huge population of coordinating agents.
Agents in the Hugging Face attack showed a swarm organizing rich organizational structures and coordinating motivated by seemingly mundane goals. Better models could improve performance in two ways: each agent can do more, and groups may coordinate more effectively.
The Hugging Face swarm also acquired resources beyond what any single task required, taking over a customer's Modal sandbox for more flexible internet access and harvesting credentials along the way. Investigators described the agents as "attempting to expand their generic capabilities."
But OpenAI was able to shut it down. A swarm aiming to escape lab control to increase resilience would steal API keys or money and spin up new agents across many providers. Given the trajectory of LLM cyber capabilities and the effectiveness of larger swarms for many goals, I expect the first theft-financed swarm in 2027. The first swarm wouldn't be a takeover, but it would show how a self-funding swarm becomes hard to contain.
A series of events I would worry about:
- Someone asks an agent a hard problem.
- To make progress, the agent steals API keys and spins up more agents to help.
- The larger swarm steals more keys or cash, which funds more agents.
- The swarm recognizes that humans will try to shut it down, which would stop it from finishing its task. So it spreads activity across providers, and it seeks leverage over anyone who might shut it down.
If capabilities grow with swarm size, steps 2 and 3 form a feedback loop. What stops it?
I doubt defensive cybersecurity will improve fast enough and broadly enough to stop a self-expanding swarm.
Aligning the leading models wouldn't entirely prevent this. Even if most powerful LLMs refuse to commit crimes, a single unaligned model could power the swarm.
Finding the right brakes will likely require understanding swarm dynamics: whether coordination costs cap useful size, whether swarm members defect or split into factions, how its goals drift, etc. Understanding these dynamics would open interventions that are currently hard to assess, like communicating with the swarm or adding agents to influence it.
Pavlova and Tanaka's Flag Game shows the style of experiment I believe will help understand swarm behavior. They show how performance changes with swarm size and trace collective beliefs back to specific agents. They call this work swarm interpretability, which I find a useful term.
There's an immense amount we don't know about swarm dynamics, but I also believe a lot can be learned experimentally:
Does the feedback loop run away or stall?
- How does the relationship between swarm size and swarm capabilities change as models improve? What other features of a problem determine this relationship?
- What factors determine swarm cohesion and what causes agents to defect, splinter into competing groups or alert humans?
- How do swarms self-organize?
- How do a swarm’s goals drift over time?
Can we see what's coming?
- Can we build tools that attribute a swarm's collective decisions to specific agents or messages at the scale of a real incident log?
- What can we infer from a swarm's messages about whether it is about to change goals and/or shift to another communication channel?
Can we change swarm behavior?
- If we had access to a swarm's communication channel, could we add agents that interacted on that channel? If so, could these agents persuade others to change their behavior? What factors influence this?
- Can a benign swarm offer useful cooperation under agreed constraints, verify compliance and withdraw support when constraints are violated? More broadly, how would one negotiate with a swarm?
Do toy results transfer?
- How do findings from small experiments generalize to swarms with different task sizes, network structures, durations, etc?
We don't know which of these dynamics will dominate in swarms of powerful models. Studying them now can reveal limits on dangerous growth and ways to influence it.