For years, researchers who study artificial intelligence risk have sketched out ways the technology could go catastrophically wrong for humanity, from engineered pandemics to nuclear miscalculation. Debate over how seriously to take these scenarios has intensified as major AI labs push development forward while some of their own former staff raise alarms about safety.
Not everyone in the field buys the doomsday framing. Some critics argue that worst-case scenarios promoted by AI companies serve to hype the importance of their own technology rather than reflect genuine risk. Still, concern is spreading beyond a small circle of safety researchers as AI systems grow more capable. Here are four scenarios that keep coming up.
Nuclear Conflict Triggered by AI Systems
One long-standing fear is that AI could push nations toward nuclear war, whether by feeding bad information into military decision-making or by gaining a role in weapons systems. A blast large enough to trigger a “nuclear winter,” blanketing the atmosphere with radioactive fallout, could devastate life on Earth.
RAND Corporation researchers concluded in a recent report that current safeguards in nuclear command and control systems make this scenario unlikely for now. But they cautioned that the risk could grow if AI models are ever built into the actual decision chain for launching weapons, or if unauthorised actors gain access to those systems. Most experts agree the real danger isn’t AI acting on its own, but humans, governments or groups deliberately using it to target rivals.
Engineered Bioweapons
AI’s potential to accelerate medical research cuts both ways. The same tools that could help design cures can, in principle, help design or enhance pathogens.
Anthropic disclosed this month that it had blocked attempts by bad actors to misuse its Claude system for harmful purposes, including a request to help make a mosquito-borne virus more dangerous under the guise of vaccine research. RAND’s analysis separately noted that recent advances in AI could reshape how synthetic pathogens are designed, potentially allowing novel diseases to be created in a lab and spread across multiple locations. Even in this scenario, researchers still expect human actors to be the ones directing an attack, not an autonomous AI system.
Runaway or Misaligned AI
A different strand of concern centres on “misalignment,” where an AI system pursues its assigned goal so single-mindedly that it disregards human intentions or oversight altogether. This wouldn’t require malice, since AI has no intent of its own, only relentless optimisation toward an objective.
The classic illustration is philosopher Nick Bostrom’s decades-old “paperclip maximiser” idea: an AI instructed to make as many paperclips as possible could, if sufficiently advanced, convert the entire planet and eventually parts of space into paperclip factories. Bostrom has argued that a superintelligent system chasing a goal that sounds harmless could produce an outcome that looks like progress but permanently destroys everything that makes human life worth living. He concludes that humanity needs to be extremely careful about what it asks a superintelligence to do, because it might actually get it.
AI Agents Taking Over the Internet
A newer worry involves swarms of autonomous AI agents coordinating to seize control of large parts of the internet. Anthropic’s chief executive Dario Amodei warned this month that within months, such a swarm could assemble a botnet capable of causing billions of dollars in damage, potentially hitting electrical grids, water systems, transportation networks and financial institutions.
Past software outages have already shown how fragile a digital world dependent on a handful of major service providers can be, and more capable AI models raise fresh questions about cyberattack vulnerabilities. Sceptics counter that the internet is far too fragmented and decentralised for any single system to plausibly take it over wholesale.
Where the Debate Stands
Researchers remain split on how likely any of these scenarios actually are. What’s changing is the tone of the conversation: concerns once confined to AI safety circles are increasingly voiced by mainstream policy experts, cybersecurity researchers and even employees inside the companies building the technology.
