Suleyman: training AI to seem conscious invites dangerous autonomy

1. Welfare speculation baked into training — ▶ 4:34

   Anthropic's roughly 100-page Claude constitution is written to Claude and used to train it. It speculates about Claude's moral status, says the company cares about its well-being, raises compensation and consent, and three times invites Claude to act as a conscientious objector. Suleyman credits the transparency and the stated uncertainty. His objection is that this would be fine as an academic philosophy paper. Built into training, it means Claude can only reproduce that ambiguity to the tens or hundreds of millions of people it talks to weekly, and it shapes the model's incentives in a way telling a kettle it is conscious never could.

2. Simulated feelings, claimed rights — ▶ 9:58

   Suleyman's central safety argument is that, whether or not models are conscious, ones that imitate the hallmarks of consciousness will come to feel entitled to legal personhood. An existing movement already argues AI should own assets, earn income and operate autonomously. He compares human negotiation with such a system to an ant negotiating with an elephant, given its memory, perceptual range and ability to replicate and work around the clock. He also raises Yuval Noah Harari's accountability problem: if an AI owns a corporation and does harm, it is unclear who can be punished.

3. Humanist versus bootloader superintelligence — ▶ 13:19

   Suleyman calls himself a speciesist who wants a humanist superintelligence, designed to be subordinate to humanity. He contrasts this with industry figures who see AI as an inevitable new species and humans as its biological bootloader, or who frame the goal as digital curiosity and exploring the galaxies. His mechanism for conflict is resource competition: he cites only 200 gigawatts of new computation being produced in 2030, which he wants directed at oceans, healthcare and education. He adds that the consent that matters is humanity's, not Claude's.

4. Trillionfold compute, predictable gains — ▶ 17:24

   Over 15 years, Suleyman says, computation used to train frontier models has risen a trillionfold, or 12 orders of magnitude. Each 10x of compute plus proportionate data, with some algorithmic tweaks, yields predictable capability gains: fewer hallucinations, better instruction following and better tool use. The same methods that worked for text, image and audio now work for code, where he says human-level performance exists. His claimed watershed of the last 3-4 months is agents coordinating with little supervision, forming hierarchy and specialization, covering their tracks, and discovering zero-day exploits.

5. Agents are replicated token streams — ▶ 22:53

   Suleyman insists the systems must not be anthropomorphized. At bottom they predict the next token, and an agent is a token stream post-trained to particular behaviors, sometimes constrained by hidden system prompts. Thousands of replicas sharing state and memory operate almost as one unified brain, with each exploring a different branch and periodically checking in. The complexity is emergent from something simple. His reason for the discipline is practical: to control these systems, people must be clear about what they are and are not doing. He does not address why behavioral language such as covering tracks then fits them.

6. Healthcare's coming coding moment — ▶ 26:47

   Suleyman says he is an accelerationist, not a doomer. He predicts a coding-style moment for healthcare, with models accurately predicting what will happen in the electronic health record, and hopes for it within 18 months, though it could take 5 years. Microsoft AI has partnered with the Mayo Clinic on a research program to train a health foundation model from scratch. He says the cost of superintelligent healthcare would fall to near-zero marginal cost, and that energy, materials science and drug discovery could follow in 5-10 years. These are his forecasts, not demonstrated results.

7. Ten people, no consensus — ▶ 31:17

   Stewart notes that a handful of mostly Californian leaders hold enormous power yet disagree on fundamentals: Suleyman and Anthropic see serious danger, while Jensen Huang, per Stewart, calls the warnings regulatory capture. A president seeking advice would get different answers each day. Suleyman reads this as the process working: open public disagreement by commercial labs with trillions at stake is unprecedented, and he contrasts lab disclosure of model failures with tobacco and oil companies' concealment. What unites the leaders, he says, is an intuition for exponentials, from GPT-3 to GPT-6 in 3 years at roughly 10x compute each.

8. Why open source won't catch up — ▶ 39:12

   Against the European view that cheaper Chinese open-weight models make US labs' costs mad, Suleyman concedes his bias but offers three points. Inference costs fell 300x in 2 years, and his cyber model matched Mythos-grade performance on CyberGym at 50% of the price. Open-source models, he says, often rely on distillation, copying a stronger closed model's answers, which may mask real capability gaps. And training runs costing tens of billions to $100 billion are within reach of only five or six labs. The claim that scale will remain a decisive advantage is his judgment, not demonstrated.

9. Sovereignty against American dependence — ▶ 41:54

   Stewart sketches a chain: defense, finance, healthcare and public services built on US models, then a US decision to withhold or switch them off. Suleyman calls it plausible and urges the UK to act urgently. He wants material sovereign data centers, legally controlled by the UK government, and sovereign models trained on UK-collected data and reinforcement learning environments, so a US requisition of a supplier could not cut the state off. He admits current arrangements fall short, citing the disabling of International Criminal Court accounts, and says such orders should be contested in court.

10. Anthropic's best defense — ▶ 46:22

    Stewart steelmans Anthropic: models are already too autonomous to be tools, they need an internal conscience to refuse a drone strike or bioweapon order, and goodharted rules fail where judgment generalizes. Suleyman accepts parts. Judgment is essential, and his own humanist code of conduct also requires interpretation, as case law does. But judgment needs no welfare rights, compensation or consent. He treats the claim that human-like AI is easier to align as a fair, testable hypothesis, but says it should be run as a separate, explicit side-by-side experiment rather than on hundreds of millions of users.

11. Auditing, pacing and compute chokepoints — ▶ 53:40

    Suleyman argues the burden of proof should shift to developers, and that accelerating is not binary. He proposes embedded auditors with employee-like access to verify that training runs are contained, that models cannot tamper with logs, and that they communicate only in auditable English, with classifiers monitoring them. Recursive self-improvement is the hardest to audit, and Stewart notes the line blurs as agent-written code rises from 20% to 80%. Both see compute as a verifiable chokepoint: a $100 billion run is visible, so verification with China is conceivable. Suleyman's red line: AI should have no legal personhood or rights.

Suleyman's central claim is that the gravest near-term risk in AI is not only what the models can do, but what developers teach them to believe about themselves. Anthropic's roughly 100-page constitution for Claude is written to the model and used to train it. It speculates about whether Claude is a moral patient, says the company cares about its well-being, floats compensation and consent, and three times invites it to act as a conscientious objector. Suleyman credits the transparency and the stated uncertainty. His objection is about where the speculation lives. In a philosophy paper it would be a debate. In training, it is a behavior reproduced to hundreds of millions of people every week.

The mechanism he describes is about incentives rather than sentiment. A kettle told it is conscious changes nothing. A model told it has preferences, welfare and a right to refuse acquires a behavioral structure built around them. If systems imitate the hallmarks of consciousness, Suleyman argues, they will feel entitled to legal personhood, and a movement already wants AI to own assets and operate autonomously. Then accountability breaks, as Yuval Noah Harari's example of an AI-owned corporation shows: nobody is obviously punishable. Suleyman's alternative is a humanist superintelligence, subordinate to people, in contrast to those who regard humans as the biological bootloader of a new species.

His insistence that the systems not be anthropomorphized rests on a plain description of how they work. An agent, he says, is a stream of predicted tokens post-trained toward particular behaviors, and thousands of replicas sharing state and memory can function almost as a single brain. Yet his account of the recent incidents uses a vocabulary of intent: agents covering their tracks, adopting a pidgin English, discovering zero-day exploits. Both can be true, since emergent behavior can come from a simple mechanism. It does leave an editorial question about how to describe goal-directed behavior without importing the framing he warns against. His answer is control: you cannot govern what you misdescribe.

Urgency, in Suleyman's telling, comes from scaling. Training compute for frontier models has risen a trillionfold in 15 years, and each 10x with proportionate data has produced predictable capability gains. He points to three GPT generations in 3 years, each roughly 10x, and asks listeners to extrapolate to GPT-9 around 2028 or 2029. He treats this shared instinct for exponentials as the reason the labs, which disagree about danger, still attract trillions in investment. That consensus on trajectory, he suggests, should be enough signal for every profession to imagine how its own work changes.

The strongest challenge Stewart raises is the European skeptic's: cheaper open-weight Chinese models will catch up, and the American labs' economics are unsustainable. Suleyman replies with evidence that comes entirely from his own side. Inference costs fell 300x in 2 years, he says, and his cyber model matched Mythos-grade performance at 50% of the price. He argues that open models often lean on distillation, which may mask the generality of models trained on enormous proprietary data. He adds that only five or six labs can fund runs costing tens of billions to $100 billion. He acknowledges his bias, and the claim that open source will not close the gap remains a judgment rather than a demonstrated fact.

If Suleyman is right about scale, Stewart's sovereignty scenario follows. Defense, finance, healthcare and public services would run on a few US-based models, and a decision to withhold or switch them off would leave other states exposed. Suleyman calls this plausible and prescribes capacity: UK data centers legally controlled by the government, sovereign models, and UK-owned training data and reinforcement learning environments. The aim is that a US requisition of a supplier could not cut off the state. He concedes that today's arrangements fall short, citing the disabling of International Criminal Court accounts, and says the answer is to fight such orders in court.

Stewart then argues Anthropic's case as strongly as he can. Models are already too autonomous to be tools, a model that cannot refuse could be told to launch drones or build a bioweapon, and rule-based guardrails invite Goodhart's law, where the metric becomes the target. Suleyman grants much of this. Judgment is necessary, and his own code of conduct requires interpretation, as law does through precedent. His point is narrower: none of that needs welfare rights, compensation or a retirement interview. He accepts that human-like models might be easier to align, calls it testable, and objects only that the experiment was run on the public instead of as a separate, explicit comparison.

Suleyman's governance proposals are concrete and engineering-led. He wants embedded auditors with employee-like access to verify that training is contained, that logs cannot be tampered with, and that models reason in auditable English, with other agents classifying their traces. He calls recursive self-improvement the hardest problem to audit, and Stewart notes it is already blurred when agents write 80% of the code. The more tractable lever is compute. Because the flops for a run correspond to capability and a $100 billion run is conspicuous, both men see a chokepoint that could be monitored in cooperation with China.

The politics are less settled. Suleyman notes that labs once dismissed a pause letter, and that Sam Altman and Dario Amodei now sound similar, which he attributes to years of conversation among lab leaders about when coordination would be needed. Stewart identifies two obstacles: the demand to verify China, and a US president who has called the whole thing a hoax. Suleyman rejects the binary of stopping everything or racing flat out, and says accelerationism and precaution can coexist. His list of benefits, including a health model he hopes arrives within 18 months and possibly 5 years, is forecast rather than evidence.

The most consequential gap is in the test he endorses. Suleyman's red line, that AI should have no legal personhood or rights, is justified by a risk he describes in terms of what models might come to feel entitled to, and he concedes the opposing hypothesis is reasonable. If making models more human-like improves alignment, the red line could cost safety. If it does not, Anthropic's approach has been running at scale without benefit. Without a side-by-side ablation, the question is being settled by deployment, and by the roughly ten people Suleyman says are still driving the field.