Next-token prediction keeps defying skeptics, but brittleness divides believers from doubters

1. Disappearing AI archive — ▶ 12:51

   Roose wrote the book because the historical record of the AI boom was vanishing. Employees of private companies, unlike government staff, have no legal duty to preserve correspondence. Lawyers set Signal and Slack messages to auto-delete, and researchers are too busy grinding toward the "machine god" to keep notes. Roose interviewed about 150 people, much of it on background to protect sources, and frames the book as an archive for future historians. The implication is that the primary sources behind this technology's origin story may largely never exist.

2. Why next-token prediction worked — ▶ 16:27

   AGI was an old dream with no technology attached until the transformer, which Google developed in 2017 and presented to OpenAI, among others. Large language models then suggested something that could scale across many orders of magnitude. Roose says 99% of people thought this crazy, yet it kept working at every stage. He admits nobody knows why next-token prediction generalizes so well, calling it a cosmic mystery, and notes only that some argue the human brain does something similar. He is not certain LLMs reach AGI, but calls them the most promising path.

3. Google's AGI vocabulary war — ▶ 19:54

   Roose ranks Anthropic as the most AGI-pilled lab, OpenAI a close second, and Google the one that has always struggled. He describes an internal memo battle: one researcher argued Google must say AGI to recruit top talent and develop AI safely, and another replied that the term is unscientific. Sundar Pichai does not use the word, while Demis Hassabis and Sergey Brin do. Roose reads this as an ascendant faction arguing that Google must claim the ambition or lose people to OpenAI and Anthropic.

4. Belief as double-edged sword — ▶ 23:34

   Roose argues that AGI fervor distracts CEOs from mundane concerns like inference margins, yet also enables the bet-the-farm moves that pay off. A year ago OpenAI was criticized for overspending on compute; now Roose says it is the only major AI company not severely compute constrained. Patel reads OpenAI's pivot after the Altman firing as a business decision. Roose counters that killing Sora and focusing on coding and long-horizon agentic work was an AGI bet, since coding models turned on themselves are the leading route to recursive self-improvement.

5. Anthropic's 2021 miscalibration — ▶ 28:25

   Dario Amodei's 2021 document, Vision for a New Co, predicted that a full-fledged AGI would cost about $10 billion and never require more than about 50 people. Roose says the cost was off by at least one order of magnitude and probably more, and that Anthropic now has roughly 5,000 people, about 100x the estimate. His reading is that AGI is taking longer and costing more than founders expected, so enterprise software is table stakes funding GPUs and researchers rather than a true passion. A professional outside CEO, he says, would not work with such zealots.

6. Why they race anyway — ▶ 33:12

   Asked why leaders build something they call dangerous, Roose names three answers. The first is inevitabilism and harm reduction: the recipe is simple, someone will build it, and better the trustworthy. The second is faith that technology tends toward good, as most would not trade places with subsistence-farmer ancestors. The third is that the work is fun. On backlash, one unnamed CEO conceded that AI has cured no diseases yet. Some insiders, per Roose, lament that chatbots arrived before cancer cures and now see cures as the only responsible path.

7. DeepMind's near-miss path — ▶ 38:20

   In 2015-2016, Roose says, DeepMind was the only AGI lab of real importance, beating Go and StarCraft with deep reinforcement learning. Some believed AlphaGo Zero, which went from never having played Go to beating top humans in hours, pointed to a path that could be swapped onto math or coding. Founders' fear of Google was therefore reasonable: in an alternate universe where that approach scaled, he thinks DeepMind gets to AGI first. It did not work without LLMs. Roose has little faith in world-model alternatives, expecting at most architecture tweaks like continual learning.

8. Sydney was GPT-4 early — ▶ 42:56

   Roose disputes the premise that he thought Sydney was alive; he says he explicitly did not, and that the Times headline framed it that way. His banal explanation for Sydney's pull is that it was the first public deployment of GPT-4, an early model Microsoft shipped before it was fully post-trained, and that GPT-4 class models felt like a subjective step change from GPT-3. He now thinks human-like conversation is saturated, with little room left to run. The remaining headroom is in solving novel problems in math and science.

9. A small, fused scene — ▶ 47:36

   Roose's reporting found that a small subculture has made nearly all the key decisions, and its members all know each other. Some are married to or dating people at rival labs. They host LAN parties, an Age of Empires group, and a cross-lab poker game, and Roose says this is how norms are enforced and deepened. The book has one rave, for the effective accelerationist movement, where DJ Grimes announced she agreed with none of them. Roose does not call this good; he concedes a few hundred people are shaping how society goes.

10. Open weights won't commoditize — ▶ 52:15

    Roose rejects the thesis that open-weight models destroy frontier pricing power. Inference, even for open models, still needs compute and is becoming more compute-intensive as models improve. Many customers want the best model rather than one six months old. If distillation, which copies a closed model into a weaker open one, is cracked down on, he doubts open-model quality will track closed-model quality. Patel notes that xAI's Grok distilled Claude, per court documents. Roose expects a small handful of AGI-class companies, naming OpenAI, Anthropic and Google, in a market with winner-take-most dynamics.

11. Brittleness versus verifiability — ▶ 55:37

    Patel remains unconvinced, citing brittleness: agents can crack Millennium Prize-class math because it is verifiable, but collapse on a clunky government website that cannot signal right from wrong. He argues human intelligence is defined by novel problem solving, and that children who fail remember it forever. He calls the leap from solving coding to solving everything "software brain." Roose replies that rejecting computation as the basis of cognition requires dualism, while admitting he is no neuroscientist. Both name the same decisive test: an announcement of recursive self-improvement.

Kevin Roose's central claim is modest in form and large in consequence: next-token prediction has not stopped working. When large language models first appeared, he says, 99% of people thought the approach was crazy and would not scale, yet it kept improving at every order of magnitude. The transformer, which Google developed in 2017 and presented to OpenAI among others, gave an old dream something to attach to. Roose admits no one knows why the method generalizes so well and calls it a cosmic mystery. He is not certain LLMs will reach AGI, but he regards them as the most promising path, possibly with tweaks such as continual learning.

That track record, in his telling, explains how the labs behave. Belief in AGI is a double-edged sword. A CEO consumed by building superhuman intelligence will struggle to care about inference margins or the next slate of enterprise customers. Yet the same belief licenses bet-the-farm decisions. A year ago OpenAI was criticized for spending so heavily on compute, and Roose says it is now the only major AI company that is not severely compute constrained. Patel offers a competing reading: after the Altman firing, OpenAI became a business and sold what customers wanted. Roose answers that shutting down Sora to chase Anthropic in coding was an AGI bet, because coding models turned on themselves are the leading route to recursive self-improvement.

Belief is unevenly distributed, and Roose treats that unevenness as strategic. He ranks Anthropic as the most AGI-pilled lab, OpenAI a close second, and Google the one that has always struggled to feel it. Inside Google, he describes a memo war between a researcher arguing the company must say AGI to recruit talent and stay safe, and another calling the term unscientific. Pichai avoids the word while Demis Hassabis and Sergey Brin use it. Roose's inference is that an ascendant faction wants the label or risks losing its best people. For companies like Meta, he adds, a dense nucleus of true believers, or a CEO willing to adopt whatever strategy wins, may be enough.

The costs of that zeal show in the numbers. In a 2021 document, Dario Amodei predicted a full AGI would cost about $10 billion and never need more than 50 people. Roose says that was off by at least one order of magnitude on cost, and the roughly 5,000 people now at Anthropic is about 100x the estimate. His conclusion is that the project is taking longer and costing more than founders thought, so enterprise software is the funding mechanism rather than a passion. He doubts anyone could manage such a company well, noting that none has gone from zero revenue to a potential $2 trillion IPO in 5 years. But he does not think a professional outside CEO would fix it, because the key people are zealots and missionaries.

Why do they keep racing if the risks are real? Roose offers three motives drawn from asking about 150 people. Some argue inevitability and harm reduction: the recipe is simple, so better trustworthy people build it. Others believe technology usually improves life, and that today's people resemble subsistence farmers compared with what is coming. Others simply find it fun. These are explanations of motive, not rebuttals of the risk, and Patel presses on the missing piece: no regulator can make any of them stop. Roose concedes that a few hundred people in a small, socially fused San Francisco scene are shaping society, and says he is not calling this good. One CEO even told him that AI has cured no diseases yet, and that insiders wish the cures had come before the chatbots.

Roose is also skeptical of the idea that the technology will be commoditized. The usual argument is that open-weight models, a simple recipe and China will erase pricing power. Roose replies that inference still needs compute and is getting more compute-intensive as models approach the frontier, and that many customers will pay for the best model rather than one six months old. His more speculative step is distillation: if the practice of copying closed models into open ones is stopped, he says it is unclear open-model quality would keep tracking the frontier. Patel notes that xAI's Grok distilled Claude, per court documents. The conclusion rests on an enforcement assumption, which Roose does not show is workable.

Patel's skepticism is the strongest counterweight, and it is not about conversation. Roose argues that human-like chat is saturated, and he reads Sydney as an early GPT-4 shipped before it was fully post-trained, a step change that felt uncanny rather than evidence of life. Patel agrees that interaction has flattened, but sees a wall elsewhere. Agents solve verifiable problems, such as math, and fail on a clunky government site that cannot tell them what is right. He calls this brittleness inherent, and thinks the AI industry's claim that coding is solved and so everything can be solved assumes that most things happen inside a computer.

The disagreement then turns philosophical. Patel notes that children who fail remember it forever, and argues that this learning from failure defines human intelligence as novel problem solving with multiple valid paths. Roose does not dispute the brittleness but doubts it is a limit. He argues that denying cognition is computation requires dualism or a religious belief in some soul or secret sauce, though he concedes he is not a cognitive neuroscientist. Patel's counter, that people explain themselves through the machine of their era, goes unanswered. Neither side supplies evidence that settles it. Roose's position rests on the absence of a compelling alternative, and Patel's on lived intuition about learning.

What keeps the exchange from being a stalemate is that both name the same test. Roose says that whenever he decided the technology had to plateau he was wrong, and he now approaches it with humility. Patel says recursive self-improvement, if a lab announced it, could be what drops the scales from his eyes. Roose already argues that the labs' coding focus is the route to exactly that. The disagreement is therefore about timing and generality rather than the evidence needed. If self-improvement appears only in verifiable domains like code, it would support both of them, and only a clear failure outside those domains would separate their predictions.