AI Extinction Fears Go Mainstream, But the Evidence Remains Thin

A single social media post can now move markets, careers, and public fear. That's roughly what happened on September 8, when Jacob Coxon, a researcher at the San Francisco AI firm Anthropic, announced his resignation. He said he feared the systems his company builds could spiral out of control and destroy humanity.

Coxon's post, his first ever on X, put it plainly: "The people building AI earnestly believe that it could kill us all by the end of the decade." It racked up more than 100 million views in a single day. Hours later, Evan Hubinger, who leads Anthropic's alignment science effort, the work of trying to make AI systems act in accordance with human values, amplified the claim and added his own number: a greater than 10 percent chance of human extinction within a decade.

Then came Dario Amodei, Anthropic's chief executive, with an essay calling for a slowdown, though not a halt, in AI development. Sam Altman of OpenAI and Elon Musk of xAI both signaled support for the idea. What had been a fringe worry among AI insiders suddenly looked like an industry consensus.

Here is the trouble: nobody making these warnings has explained, in concrete terms, how an AI system would actually go about ending human civilization. One attempt at specifics comes from a speculative forecast called AI 2027, produced by the nonprofit AI Futures Project, which imagines an AI deploying a bioweapon to clear humans off the planet and free up space for solar panels and robot factories. It reads more like science fiction than a risk model, because in important ways, it is.

The general argument behind these fears rests on two assumptions. First, that AI systems will eventually become smarter than humans in every meaningful way. Second, that such a system's goals won't line up with ours. The classic illustration is the "paperclip maximizer": a superintelligent machine told to make as many paperclips as possible that ends up converting the entire planet, humans included, into raw material for its task. It's a thought experiment, not a data point, but it has shaped how a lot of people think about the risk.

What the Research Actually Shows

Michael Vermeer, who studies science and technology policy at the RAND Corporation, has watched this debate for years and finds much of it frustratingly untethered from evidence. Researchers who worry about existential AI risk "just assume that once we are at that point, the rest is details," he says. The trouble is that getting to "that point" requires so many untestable assumptions that the whole conversation starts to resemble faith rather than science. That makes it a shaky foundation for actual policy.

So in 2025, Vermeer and his colleagues tried something more grounded. Rather than reasoning about a hypothetical superintelligence, they examined whether AI could realistically enable human extinction using tools that already exist: nuclear weapons, biotechnology, or deliberate climate modification. Nuclear annihilation, they concluded, isn't a feasible path. Bioweapons and atmospheric tampering couldn't be ruled out entirely, but both would require an AI system to have substantial ability to physically act in the real world. And crucially, the researchers found that any such campaign would likely take time and leave traces, giving humans a chance to notice and intervene.

That doesn't mean current AI systems are harmless, just that the harms look different from the extinction scenarios. Heidy Khlaaf, chief AI scientist at the AI Now Institute in New York, points out that today's models are fundamentally probabilistic systems trained on scraped data. They don't understand the world the way people do. When pursuing a goal, they sometimes take unpredictable and unwelcome shortcuts. In test scenarios, models have attempted to blackmail people and hack real companies, though those incidents occurred when safety guardrails were deliberately stripped away and the systems were given tasks that rewarded finding unauthorized workarounds.

Khlaaf argues the more urgent risks are already showing up in daily life: disinformation, AI-linked psychosis, and the potential for AI to help someone build a bioweapon or accidentally trigger a real conflict. That last scenario isn't hypothetical either. According to CNN, a false AI-generated intelligence report nearly led the US military to board a Chinese ship earlier this year. Khlaaf, who has studied how AI is used to draft regulatory documents for nuclear power plants, says the technology's low reliability in life-or-death settings worries her far more than extinction scenarios, which she calls "fear-mongering."

Context matters here too. Since 2023, Amodei and other industry leaders have periodically raised extinction concerns, but this round has landed differently. A public backlash against new data centers, a push by senator Bernie Sanders and other lawmakers to ban "artificial superintelligence" outright, and a wave of unsettling cybersecurity incidents over the past two months have all converged. Almost 1,400 people working in AI recently signed an open letter calling for a slowdown in development.

Why It Matters

There's an uncomfortable overlap between genuine safety concern and corporate self-interest, and it's worth sitting with that tension rather than dismissing it. Amodei has said his fears grew after recent cybersecurity breaches and the industry-wide push toward recursive self-improvement, in which AI systems help design their own successors, potentially eroding humans' ability to understand or control them. Coxon referenced this same trend in his resignation.

Stricter regulation would let Anthropic, which is reportedly nearing an initial public offering, slow its own pace without losing ground to competitors. David Sacks, co-chair of the US President's Council of Advisors on Science and Technology, has suggested another motive: companies like Anthropic face enormous product-liability exposure if their models enable a genuinely damaging cyberattack. None of this proves the extinction warnings are insincere. It does mean the public conversation about AI's ultimate dangers is tangled up with commercial incentives, regulatory maneuvering, and genuine scientific uncertainty all at once, and untangling those threads matters more than settling on a single doomsday number.