From Zain Kahn’s ‘Superhuman newsletter‘:

Over 1,200 employees at frontier AI companies signed a letter warning of runaway development

“The open letter cautions that AI could advance “beyond our ability to understand or control the resulting systems,” and calls on the US government to deliberately pace progress at the international level, which would reduce competitive pressure on each lab. Signatories include chief scientists at OpenAI, Anthropic, and Meta. The scale of the signatory list — spanning companies and roles — is a signal of how seriously insiders are taking the risk.”

I grew up believing that a super-intelligent AI would be created in a lab. The fear was that it would somehow escape into the ‘real world’ and only then could it dominate the world. The whole idea of some sort of firewall to protect us just evaporated when LLM’s were trained and unleashed on the internet. We completely bypassed any sort of protection or safety net that would allow human intervention in order to prevent a rogue AI from running wild.

Worse yet, we have unleashed AI into a world where any one person with ill-intent can use AI with nefarious intentions. I’ve said before,

“I don’t fear the famous scientist, I fear the rogue, unhappy misfit who uses incredible technological advancements for nefarious reasons. The same technology that can make our lives easier, and create tremendous good in the world, can also be used with bad intentions. But there are differences between someone using a revolver for bad reasons and someone using a nuclear bomb for bad reasons. The problem we face in the future is that access to the equivalent harm of a nuclear bomb (or worse) will be something more and more people have access to. I don’t think this is something we can stop, and so as amazing as the technology is that we see today, my fear is that it could also be what leads to our demise as a species.”

Now 1,200 AI employees are heeding a warning that AI could advance “beyond our ability to understand or control the resulting systems.” That’s just AI systems acting on their own. That doesn’t include the danger of humans using these complicated and little understood intelligent tools with bad intentions. Sure we need to worry about AI running amok, but I think the far greater existential threat is a rogue government, or even a rogue lone wolf angry at the world, using AI in ways that we won’t be able to stop. Think of a cascading virus either electronic or environmentally contagious beyond our ability to stop it. Not created by scientists, but by a brilliant incel who feels cheated by the world.

We don’t need to fear rogue AI nearly as much as we need to fear rogue individuals armed with a super-intelligent AI. This is what really scares me.