An OpenAI Safety Insider Thought the Doomsday Warnings Were ‘Nuts’. Then He Saw How the Company Worked
An OpenAI Safety Insider Thought the Doomsday Warnings Were ‘Nuts’. Then He Saw How the Company Worked
After quitting OpenAI, a former safety leader issued a stark warning about the AI industry’s rush, citing a dangerous mix of money, fear, and human psychology.
EXPERT OPINION BY NICK HOBSON, MANAGING DIRECTOR NORTH AMERICA, INFLUENCE AT WORK, BEHAVIORAL SCIENCE ADVISER @NICKMHOBSON
Photo: Getty Images
David Robinson joined OpenAI in May 2023, the day after Sam Altman first testified before the Senate. He came in thinking that worries about AI wiping out humanity were a bit “nuts”. For the next three years his job, in part, was to explain, in writing, why OpenAI’s deployments were safe. He led the writing of the system card for GPT-6 Astra, the technical document that lays out a model’s risks and the safeguards around it.
He left OpenAI at the end of September, and last week he sat down with Ezra Klein to explain why. “This is nuts what’s happening. This is not right,” he told Klein, describing an industry building systems it can’t yet control.
Robinson arrived a skeptic of AI doom. What changed his mind was evidence, and most of that evidence is about how AI companies run themselves and the cultures around these powerful technologies.
A startup building something dangerous
Robinson’s central complaint is about tempo and structure. He told Klein that OpenAI, and the industry with it, still operates like a startup, more so than makes sense for what it now builds. In his view the models coming out of the frontier labs carry more capability and more risk than anything they shipped even six months ago.
The daily digest for entrepreneurs and business leaders
Featured Video
Quince Wants to Follow Amazon’s Playbook. Its Platform Could Transform It Into a Tech Giant.
The comparison he reaches for is nuclear power. A loss of control over these systems, he argued, could cause harm larger than a single reactor melting down, and the safety rigor should at least match the most dangerous things people already know how to do. Nuclear operators work under licensing, mandatory incident reporting, and clear chains of command. There are redundancies built in for obvious reasons.
Robinson described the opposite at OpenAI. It’s become a decentralized company where it often isn’t clear who has the authority to make a decision.
He was also unusually candid about why smart people, himself included, kept underestimating the risk. He named money, fear, and time pressure. On money, his point was that wealth gives people a strong reason to conclude that things are fine or will turn out fine.