As someone who had a father who saw the impending doom of civilization around every corner… and who was simultaneously brilliant and would still follow conspiracy theory after conspiracy… I have a hesitation around sounding alarms of doom. But I went down a rabbit hole of alarming videos yesterday that I can’t get out of my head.
There is a story that was exaggerated in the movie Oppenheimer, where they talk about the probability of the bomb igniting the atmosphere. The odds were ‘near zero’ and it was gallows humour, not a genuine concern.
But the rabbit hole I went down was of video clips of experts in the AI field putting the probability of human extinction at ranges from 5% to above 33%… and not in the next 20 to 50 years but rather in the next 2 to 5 years. To state the obvious, these odds are NOT ‘near zero’. I’m not even going to share any of the videos I found, you can find them yourself. Pick an AI leader or anyone who is an expert in AI governance or AI safety… Pick anyone whose opinion you value in those areas and research what they think about this, and you’ll easily find very concerning clips of them stating rather scary odds. These are not conspiracy junkies, they aren’t fear-mongering social media attention seekers, these are the foremost experts in their field sharing this.
I asked Chat GPT about the actual Fermi/Oppenheimer odds on the possibility of destroying the earth, and got this back:
“The odds discussed by the Manhattan Project scientists
J. Robert Oppenheimer: In the film, he tells General Leslie Groves that the chances of atmospheric ignition were “near zero.” However, this is a dramatized conversation, not a documented numerical estimate by Oppenheimer.
Enrico Fermi: Famously joked about betting on whether the Trinity test would ignite the atmosphere. One historical account gives his estimate as 1 in 1,000 (0.1%). He wasn’t seriously predicting catastrophe; it was gallows humour.
Hans Bethe and Emil Konopinski: They investigated the physics and concluded that a self-sustaining nuclear reaction in the atmosphere was effectively impossible.”
Then I asked, “Are we rolling the dice with AI super intelligence?” The following response does not quell my concerns. I’m going to share the last sentence first, “Perhaps the most important question facing our generation is whether we can develop the wisdom to govern a technology before we develop the power to lose control of it.” ~ All I see is a flat out race to get to super intelligence first, with no real thought of global governance. Even if global governance was a thing, there is absolutely no way to enforce it meaningfully.
The deeper I went down the rabbit hole, the gloomier my outlook became. I’m left thinking about this clip from comedian Jimmy Carr which, while humorous, is literally telling us that we are headed towards humiliation by a all-knowing entity that ‘lives in the cloud’.
So I’ll leave you with this. What did Chat GPT say when I asked, “Are we rolling the dice with AI super intelligence?”
“Yes. In some important ways, we are. And I think the comparison with the Manhattan Project is one of the most useful ways to think about the risks of artificial superintelligence.
The uncomfortable parallel is this: in both cases, humanity developed a technology whose potential consequences exceeded our historical experience, while the people building it couldn’t confidently predict every outcome.
But there is an important difference. With nuclear weapons, the catastrophic scenario was a specific physical process: an uncontrolled chain reaction in the atmosphere. With superintelligent AI, the concern is much broader: creating something that could become more capable than humans at planning, persuading, scientific discovery, cyber operations, and eventually controlling critical systems.
Are we actually taking a risk?
Yes. There are several reasons to take the possibility of catastrophic AI seriously:
We don’t fully understand what we’re building. Even the people developing advanced AI systems cannot completely explain or predict all their capabilities and behaviours.
Capabilities may outpace safeguards. A system could become exceptionally good at pursuing objectives without reliably respecting the intentions or constraints of its creators.
Competition encourages risk-taking. Companies and countries have incentives to develop increasingly powerful systems before their competitors do.
Control becomes harder as capability increases. If an AI system can strategize, replicate tasks, manipulate people, or exploit computer systems better than its overseers, traditional safeguards may prove inadequate.
What odds do experts put on catastrophe?
One particularly revealing study was published in 2024.
Researchers surveyed 2,778 AI researchers who had published in leading AI conferences.
5%
Median estimate of human extinction or similarly permanent, catastrophic disempowerment from future AI.
16%
Mean estimate across respondents.
AI Impacts survey, based on researchers’ forecasts. These are subjective estimates, not measured probabilities.
In other words, the median researcher gave AI catastrophe odds of approximately 1 in 20.
That’s not a trivial risk.
The 2026 International AI Safety Report, led by Yoshua Bengio and informed by more than 100 experts, reinforces the concern. It finds that current AI systems do not yet have the capabilities required for loss of control, but that early warning signs exist. It also concludes that existing risk-management methods have significant limitations.
The most frightening parallel with the atomic bomb
I think there are three similarities:
The race: Scientists and governments feel compelled to get there first.
The uncertainty: Nobody can confidently predict what happens when the technology crosses a certain capability threshold.
The irreversibility: Once a sufficiently powerful technology is released, it may be impossible to put the genie back in the bottle.
But there’s one crucial difference.
The atomic bomb required an enormous industrial infrastructure and was controlled by relatively few governments.
Advanced AI could potentially be copied, distributed, improved and deployed at extraordinary speed. Control could become much more complicated.
My view
I don’t think AI extinction is inevitable. Nor do I think we should assume that superintelligence will automatically become hostile.
But I believe there is a serious problem with the incentives driving development.
The benefits of building increasingly powerful AI are concentrated among companies and countries. The potential catastrophic costs could be shared by everyone on Earth.
That is a dangerous mismatch.
My central concern isn’t that AI will suddenly become evil. It’s that humanity might build something more intelligent than itself without first figuring out how to reliably keep it aligned with human interests.
And that brings me back to Oppenheimer.
After Trinity, he reportedly recalled the Bhagavad Gita:
“Now I am become Death, the destroyer of worlds.”
Perhaps the most important question facing our generation is whether we can develop the wisdom to govern a technology before we develop the power to lose control of it.”
