Rival AI labs race to build what they fear

1. Distrust spawns new labs — ▶ 4:13

   Roose frames AI's history as a chain of schisms. DeepMind, founded in 2010, was the first AGI lab. After AlphaGo beat the world champions at Go, Elon Musk and Sam Altman distrusted Demis Hassabis and Larry Page and started OpenAI to stop them getting there first. About five years later, seven employees led by Dario Amodei, then vice president of research, saw evidence that models improved with more compute and data. They distrusted Altman and Greg Brockman and founded Anthropic. Each time, the logic was that the technology is powerful and inevitable, so one must build it first.

2. The Szilard template — ▶ 7:00

   Roose says the race-to-build logic is sincere, citing 150+ interviews. Amodei is obsessed with Richard Rhodes's The Making of the Atomic Bomb, and his hero is Leo Szilard, not Oppenheimer. Szilard tried to keep the chain reaction secret, then helped build the bomb before the Nazis once the secret leaked, urging Einstein to write to Roosevelt. Roose says Amodei sees himself in the same position: he tried secrecy, it failed, and the good guys must now win. Roose adds that Amodei counts himself among the good guys, better than China or Altman.

3. Rate of progress, not products — ▶ 10:29

   Rationalists and effective altruists pushed AI safety years before it was mainstream, funding organizations and steering young people toward safety and interpretability. Roose says their fears were not driven by the technology of the day; a Siri or Alexa did not look threatening. They were extrapolating the rate of progress, betting that if the straight lines on the graphs continued, systems 2-10 years out would pose much bigger risks. Roose judges that the scaling laws have held and that these early doomers have been vindicated, though that verdict concerns capability growth rather than the odds of catastrophe.

4. Two worlds of p-doom — ▶ 14:38

   Roose says the median p-doom, the probability that AI wipes out humanity, usually within 10 years, is near zero in America generally and in double digits at the labs. When news spread of a 10%-or-greater estimate from an Anthropic employee, his New York and Washington friends were alarmed, while San Francisco's reaction was that 10% seemed low. He routinely meets researchers with 30, 40 or 50% estimates, many of them building the technology. He treats this gap as an illustration of how insular his own bubble is.

5. Why build it anyway — ▶ 16:48

   Asked why anyone would build something with a 10-50% chance of ending humanity, Roose offers three rationalizations. First, the Szilard argument: build it before the bad guys do. Second, genuine belief in hyper-abundance, where robots handle grunt work and AI cures diseases, which he says is a real investment rather than marketing hype. Third, and least often said aloud, it is fun: there is scientific joy in using a program, data and GPUs to create a machine that can think. The explanations are motives, not evidence that the risk estimates are wrong.

6. Slow down, but not alone — ▶ 18:51

   Roose says the lab leaders want to slow down more than anyone. Amodei issued a 'pacing the frontier' statement saying things move too fast. Altman, whom Roose says Amodei has hated for years, agreed, and Hassabis supported it. Roose calls it remarkable that bitter rivals agree. The condition is that nobody slows unilaterally; each would slow if everyone else did. Roose does not describe any mechanism for achieving that coordination, so the stated preference remains conditional.

7. Model welfare and skepticism — ▶ 22:58

   Anthropic has a model welfare team that studies whether Claude is or could become conscious, and its leader told Roose his p-consciousness for today's models is 15%. The motive is partly moral: if models can suffer, Roose says, labs are effectively running a factory farm for AI, which many EA-vegan employees want to avoid. Roose is himself skeptical, calling consciousness possibly a red herring. There are no good tests, and a chatbot can simply claim to be conscious. He says Anthropic staff are divided, citing a holiday-party ukulele performance of a Claude-written song, 'Hello from the weights'.

8. Triage by future harm — ▶ 27:16

   The labs worry about present harms such as mental health effects, bad medical advice and students short-circuiting their learning, Roose says, but they are in triage mode. They prioritize what could do the worst damage, even if it is 6-12 months or 2 years out. His evidence is that automated cyberattacks, which people dismissed two years ago as fantastical, are now real. The inference is that scenarios called fantastical today may only be a matter of waiting for models to improve, such as a lost-control AI superhuman at hacking, deception and bioweapons.

9. Fewer than 50 decision-makers — ▶ 32:01

   The host notes that the Manhattan Project was governmental while the labs are private. Roose does not defend the comparison on that point. He calls where the technology goes, and who decides, the central question of our time. By his estimate, even 50 people have not truly shaped the technology. They are small, insular, mostly male, drawn from elite institutions, and long acquainted or estranged. No one elected them. In his words, they have become more important than anyone gave them permission to be, which he sees as a source of public backlash.

10. Recursive self-improvement is the goal — ▶ 33:26

    The labs are racing toward recursive self-improvement: automating the whole frontier model development cycle so that GPT-7 builds GPT-8 at machine speed, perhaps in 6 days or 6 hours instead of 6 months. Many in the industry expect this within a year or two, Roose says, risking systems thousands of times smarter than humans that can replicate and evade shutdown. He cites an Anthropic report that Claude leads 26% of its AI R&D tasks, and an unreleased OpenAI model that appears to have worked through several hundred unsolved math problems. Both are relayed claims.

11. Defectors, urgency, contingency — ▶ 38:18

    Roose rejects the idea that safety defectors are doing it for publicity: some gave up millions of dollars, and they voice what insiders say privately. He allows that individuals may be wrong about timelines, but holds that the central message, that labs are accelerating toward a technology no one knows how to control, is true. He also says AI was never inevitable, as it exists because a small group chose to build it. He adds, though, that the technology may be inevitable while society's response is contingent on collective choices over the next few years.

The most striking claim in Kevin Roose's account is that the AI race is sustained by sincerity rather than cynicism. The people running the leading labs, he says, believe that powerful AI will be built regardless of what any one of them does. They conclude that the least bad outcome is for someone they trust to build it first. Roose, who interviewed more than 150 people, including lab leaders, executives and their longtime friends, insists this is a real argument and not an all-purpose excuse for competition. That matters because a sincere belief is harder to dislodge than a rationalization. It also explains a pattern that otherwise looks absurd: organizations founded out of fear of the technology are the ones accelerating it.

The pattern repeats across the industry's history, as Roose tells it. DeepMind, founded in 2010, was the first lab devoted to artificial general intelligence, a goal that at the time could get a researcher laughed out of academic rooms. After AlphaGo beat the world champions at Go, Elon Musk and Sam Altman concluded that DeepMind was on track to get there first. They distrusted its CEO Demis Hassabis and Google's Larry Page, and they started OpenAI to prevent it. About five years later, seven OpenAI employees led by Dario Amodei saw evidence that language models scaled with compute and data, distrusted Altman and Greg Brockman, and left to found Anthropic. Each schism had the same structure: belief in inevitability, plus distrust of the person in front.

Roose traces the template to a specific source. Amodei is obsessed with Richard Rhodes's The Making of the Atomic Bomb, and his hero in it is Leo Szilard, not Oppenheimer. Szilard discovered the nuclear chain reaction and tried to keep it secret by urging scientists not to publish. When the secret leaked anyway, he concluded that the only responsible course was to help build the bomb before the Nazis did. He pushed Einstein to write to Roosevelt and joined the Manhattan Project. Roose says Amodei sees himself in the same position: he tried secrecy, it failed, and the good guys must now win. Notably, Amodei counts himself among the good guys, which Roose describes as better than China or Sam Altman. The category is self-supplied.

Why did anyone believe this before the technology looked threatening? Roose's answer is that the early worriers, many from the rationalist and effective altruist communities, were watching the rate of progress rather than the products. Ten years ago, he says, a Siri or an Alexa did not look like something that could take over the world, and fearing it sat outside the Overton window. Yet these communities funded safety organizations and steered young people into AI safety and interpretability, which made them influential early. Their wager was that if the straight lines on the graphs continued, systems 2, 3, 5 or 10 years out would pose much larger risks. Roose's verdict is that the scaling laws held and the predictions have been vindicated. That is a judgment about capability growth, not about the likelihood of catastrophe.

Estimates of catastrophe diverge sharply by location. Roose says the median p-doom, the probability that AI wipes out humanity, usually within 10 years, is close to zero in America at large and in double digits at the labs. When an Anthropic employee's estimate of a 10% or greater chance circulated, his friends in New York and Washington were alarmed. In San Francisco the reaction was that 10% seemed low, and Roose says he routinely meets people with 30, 40 or 50% estimates, many of them building the technology. Asked why anyone proceeds, he gives three answers. One is the fear-based historical argument. Another is a sincere belief in a future of hyper-abundance and cured diseases. The third, rarely admitted, is that building a machine that thinks is fun.

Roose argues that the labs want to slow down more than anyone. He cites a statement from Amodei on 'pacing the frontier' saying things are moving too fast. Altman endorsed it, and Hassabis supported it, despite what Roose describes as years of mutual hatred between Altman and Amodei. The condition is that no one slows unilaterally; each would slow if everyone else did. This is the trap in its plainest form, since the distrust that created three labs now prevents any of them from stopping. Roose describes no mechanism for converting the shared preference into coordinated action. Read charitably, the statement is sincere. It may also be better understood as a description of the problem than as a way out, which is an editorial inference rather than Roose's claim.

What the labs are chasing raises the stakes. Roose describes recursive self-improvement, the point at which AI can automate the entire development cycle of a frontier model, so that GPT-7 builds GPT-8 at machine speed, perhaps in 6 days or 6 hours rather than 6 months. Many in the industry expect it within a year or two, he says, and the result could be systems thousands of times smarter than the smartest humans that can replicate and resist shutdown. He points to early signs: an Anthropic report that Claude leads 26% of the company's AI research and development tasks, and an unreleased OpenAI model that appears to have worked through several hundred unsolved math problems. Both claims are relayed, and the first comes from the company itself.

The governance problem follows directly. The host notes that the Manhattan Project analogy strains because that project was governmental, whereas these labs are private and unaccountable to any government. Roose does not defend the analogy on that point. He reframes the issue as the central question of our time: where the technology is going and who gets to decide. He would be surprised if even 50 people have really shaped it. They all know each other, many have histories of estrangement, they are mostly men from a small group of elite institutions, and no one elected them. In his words, they have become more important than anyone gave them permission to be.

Roose is candid about the limits of the case. He doubts the consciousness debate, which he suggests may be a red herring, because there are no good tests and a chatbot can simply say it is conscious. He notes that Anthropic nonetheless has a model welfare team whose leader puts the probability for today's models at 15%. He allows that defectors who warn of danger may be wrong about timelines, and he describes his own San Francisco circle as an insular bubble. His sharpest evidence is a track record: automated cyberattacks, dismissed two years ago as futuristic, now exist. The urgency still rests on extrapolating continued improvement, and on the hope that superhuman math performance repeats in other domains. That is a prediction, and Roose states it as one.

Roose ends on contingency. The technology, he says, was never inevitable, because it exists only because a small group committed to building it, and society can either remain a passive recipient or enter the conversation. Yet in the same breath he allows that the technology may be inevitable while society's response is not. The tension matters, because the entire race logic described in his book rests on assuming inevitability. If that assumption is partly self-fulfilling, the labs' conditional wish to slow together points to the missing piece: an outside actor with the standing to supply coordination. Whether one can act before the acceleration insiders expect within a year or two is the question his account leaves open.