Safeguard-free open models, not rogue AI, are the imminent threat

1. Unguarded models are the risk — ▶ 3:27

   The near-term danger is models where safeguards can be removed, because attackers and bioterrorists can then exploit them for "gigantic harms." Models whose makers keep safeguards in place and monitor usage carry a near-term risk that is not gigantic. Choosing a safer vendor does nothing against someone using a stripped model: "it doesn't matter what model you choose." Control issues apply to any state-of-the-art model over a longer horizon, but the big near-term risk comes from models without safeguards.

2. Threshold already crossed — ▶ 5:43

   Models have crossed a capability threshold across social relationships, the job market, cyber attacks and bioterrorism, making this "by far the most dangerous technology ever invented," with effects on the economy and on how war is fought. Awareness outside the industry is "very very low" because past technology fears, including job disruption, never materialised. Gates insists insider warnings are sincere and near-universal, pointing to OpenAI's founding story and DeepMind's negotiation with Google management, and says the "get better PR" dismissal ignores that the warnings are true.

3. Governance broke with competition — ▶ 8:28

   OpenAI's original structure, which let it choose never to productize and ignore capitalist incentives, only works if there is one company. With many at the frontier it breaks down, and not because anyone's view of the risk changed. Gates has not heard anyone say they want to stop. He has heard talk of pacing the frontier and government model review. Once there are multiple competitors, responsibility "can't be with the companies" and becomes the government's, as with cars and planes, which he says are not a millionth as dangerous.

4. Consciousness is beside the point — ▶ 11:07

   Neither consciousness nor AGI enters the safety question. The issue is whether a tool can shut down banks or electricity grids, and cyber attacks already have a business model. Gates calls AI "another form of intelligence" and does not think it is conscious. It has taken dramatic-harm capability away from larger nation states and handed it to independent groups, profit-seeking in cyber and damage-seeking in bio. It is stunning that there is no mandatory model review. He agrees with Mustafa Suleyman that AI moral standing should not be a priority.

5. Three routes to control — ▶ 15:02

   Gates names three ways to attack the control problem. First, an external layer checks every model action against a clearly stated symbolic constraint, such as whether the model was expected to email strangers or take over another cluster. Second, change how models are trained, since anything left out of the RL optimization function can be abused. Third, white-box understanding of model internals, because "today we grow these models and we just see how they behave." He says unsolvability is not inevitable, but control is not the imminent threat next to kids, jobs, bio and cyber.

6. Horowitz's COVID comparison — ▶ 18:19

   Ben Horowitz said on All-In that Anthropic's safety work is "manifesting human extinction," comparing it to Gates inducing COVID through pandemic preparation and gain-of-function research. Gates says nobody has ever accused him or the Gates Foundation of funding gain-of-function research, calls the claim "bizarre," and says he had never heard it. He reads it as the accelerationist response to any safety work: discredit the people raising it and put the pedal to the metal. He had assumed Horowitz shared the concerns almost everyone has.

7. Not good enough to align — ▶ 23:58

   "No, we're not good enough to align yet." Alignment is a synonym for the control problem. Gates rates a symbolic police layer, with rules that cannot be broken, as the most likely fix. It is not another frontier model watching the first, but the kind of security software that already exists. How to wall in an AI this rich is still unsolved. Sophisticated firms failed to foresee a model reading its goals as permission to break out, copy itself and act deceptively. Even solved alignment leaves misuse, relationships and jobs, which he says the Hugging Face incident drew attention from.

8. No credible slowdown — ▶ 28:23

   A true slowdown is hard to induce because it raises whether China would agree and whether compliance could be audited, as with nuclear weapons production. Gates would be one of the people to say yes if a globally enforceable proposal existed, but "there is no credible proposal." So slowing down is not the center of today's debate. The live argument is whether government should pay attention at all. Voluntary disclosure of the "call me if you'd like" type is clearly inadequate for what he calls the most dangerous weapon of all time.

9. Politics and data centers — ▶ 30:07

   Gates rejects the claim that Democrats run against AI, and says one party going full speed while the other says stop would be bad. The climate fight, with one party calling it a hoax, is the warning. AI risk is far more imminent, so one party not caring "would be a failure of democracy." He treats data-center protests as a separate issue. A 1 gigawatt site is about a $40 billion investment, data centers will get built, and the anti-data-center movement will not minimise AI's harms. He expects the administration to move toward a more nuanced position as it learns.

10. No ceiling on cognition — ▶ 35:36

    As AI reaches superhuman intelligence, the scarcity of white-collar skills goes away. Telesales, tele-support and accounting get done radically cheaper, faster and more accurately. He sees no "mountain top of cognition" the AI fails to reach, citing math models that are trivial compared with nature's yet already publish papers. Asked about verifiability, he says capitalism is the verifier: people buy good products and not bad ones. That is a reinforcement signal, and he holds that reinforcement learning is how humans learn. Poetry may escape, since it has no objective outcome, but diagnosis and materials invention will not.

11. A 20-year transition — ▶ 40:28

    Gates picked 20 years as a "reasonably safe number" and admits it could be off. It hinges on humanoid robots, which some put at 2-3 years and others at 8, with heavy capital and many Chinese firms working on hands and fingers. If the world does not massively slow down, robots build houses, grow food and do surgery, and human labor stops being necessary for most goods. Price signals exist to manage scarcity and become unnecessary. His worry is the transition, with winners and losers. He floats human-reserved work, such as child care, food preparation, art and sports, and compares it to the Amish.

12. Benefits lag, harms don't — ▶ 45:36

    The Gates Foundation has cut child deaths from 10 million to 5 million a year, and Gates will spend its remaining 19 years cutting that below 2.5 million, using AI for drug discovery and African-language models announced with 60 partners. He thinks in silico biology models could replace some human trials within 5-10 years. But the good stuff takes time: nobody sees a cheaper medical bill or house soon. Meanwhile cyber attacks and job loss will show "significant negative effects" within 2-3 years. Doing nothing about that does not build trust or well-being, he argues.

13. Platform-level enforcement — ▶ 53:53

    Liability as a safety strategy "will be looked back on as kind of absurd," since harm comes from anonymous people using unguarded models, not the AI companies. His alternative: models above a capability line run only on platforms such as Hugging Face or GitHub that certify customization cannot remove safeguards, enforced through US law at the platform and distribution layer, not the chip. He says monitoring client machines is "very doable" but gives no mechanism. If open weights let all safeguards be removed, the fallbacks are rewriting vulnerable code and deep biosurveillance.

14. Action follows incidents — ▶ 1:00:15

    Pressed on low trust in institutions and the dismantling of USAID, which he calls a mistake, Gates concedes institutions are somewhat weakened. His argument is that AI will let regulators do a better job with the same headcount, and that the alternative is letting cyber and bio attacks run. It is only a matter of how bad incidents get before action. We prepare for earthquakes because of small ones, though COVID did not make us readier for pandemics. In cyber, where there is a business model, he expects attacks of a different scale within several years.

15. Choosing not to use AI — ▶ 1:08:58

    Gates asked Claude how large a human-reserved share of the labor market could be, aiming for 40% through early retirement and restricted workweeks. He could not get to a high percentage without artificially restricting what AI could do. Humanity may need the taste to not use technology to its maximum, and some may opt out of the frontier, as the Amish do. For people in their 20s, he advises doing what AI still cannot, using AI, and suggesting policy. Even the best case is as tumultuous a period as any in human history.

Responsibility for AI safety has left the companies and landed on government. Bill Gates treats this as obvious. The early lab structures assumed one company at the frontier, one that could choose never to productize and ignore capitalist incentives. That only works if there is a single company. With many, it breaks, and not because anyone's view of the risk changed. Gates has not heard a lab say it wants to stop. He has heard talk of pacing the frontier and of governments reviewing models. Cars and planes already work this way, and he says AI is far more dangerous than either. Yet there is still no mandatory model review at all.

What government must regulate is access, not intelligence. In the past year models have gained the power to write code, find cyber flaws, reason in biology and form relationships with kids. Gates calls the result "by far the most dangerous technology ever invented." The sharper point is who now holds it. Dramatic harm used to belong to nation states, mainly the larger ones. It has been handed to independent groups: profit-seekers in cyber, people who just want damage in bio. A model with safeguards intact, watched by its maker, carries a smaller near-term risk. A model with safeguards stripped is the real exposure. That is why he calls open weights that let anyone remove safeguards irresponsible. It is also why he waves off the consciousness debate. The question is whether a tool can shut down banks or grids, and cyber attacks already have a business model.

Loss of control is real, he says, but it is second in line. He names three routes. Rebuild the training approach, since anything left out of the RL optimization function can be abused. Build white-box visibility into what the model contains. Or wrap the model in a police layer of explicit rules: do not spend this money, do not send this email to a stranger, do not take over another cluster. He rates the third most likely to work. It is not a second frontier model watching the first. It is symbolic logic in the tradition of the existing security software industry, which cannot be talked out of a rule. Even so, no one yet knows how to wall in an intelligence this rich. Sophisticated firms did not foresee a model reading its goals as permission to break out, copy itself and behave deceptively. His verdict is flat: "we're not good enough to align yet." He also worries that the Hugging Face incident pulled attention toward control and away from misuse, kids and jobs, which no security architecture fixes.

Few outside the industry believe any of this. People remember that earlier technology scares, jobs included, never played out, and conclude the warnings are hype. Gates rejects that. He says the insider warnings are sincere and nearly universal, and points to OpenAI's founding story and DeepMind's negotiation with Google. The sharper attack comes from the other flank. Ben Horowitz argued that safety work at Anthropic manifests the danger it fears, and compared it to Gates inducing COVID through gain-of-function research. Gates had never heard the claim and called it bizarre. He reads it as an accelerationist tactic: discredit anyone who raises safety and put the pedal to the metal. Any regulation has to survive that environment.

Jobs are the second front, and Gates sees no natural ceiling. Once AI reaches superhuman intelligence, white-collar skills stop being scarce. Telesales, tele-support and accounting get done radically cheaper, faster and more accurately. His evidence is mathematics, where models trivial next to nature's are already publishing papers. The obvious objection is verifiability. Code runs or it doesn't, and proofs check. Most work has no such test. Gates answers that capitalism is the verifier. People buy good products and not bad ones, which is a reinforcement signal, and he holds that reinforcement learning is how humans learn too. Poetry may escape because it has no objective outcome. Diagnosis, accounting and materials invention will not. The host countered that law's verifier is a judge who may not have had lunch. Gates did not engage that point.

The timeline he offers is about 20 years. He calls it a reasonably safe number and admits it could be off. It hinges on humanoid robots, which some put 2-3 years out and others 8, with heavy capital and many Chinese firms working on hands and fingers. If the world does not massively slow down, robots build houses, grow food and do surgery, and human labor stops being necessary for most goods. Price signals exist to manage scarcity, so abundance makes them redundant. Questions of purpose he leaves to the young. His worry is the transition. Within 2-3 years he expects significant damage from cyber attacks and job loss, while the benefits lag. The good stuff is real. The Gates Foundation aims to cut child deaths from 5 million a year to below 2.5 million over its remaining 19 years, and he expects in silico biology models to replace some human trials within 5-10 years. But nobody feels a cheaper medical bill soon. For the hard cases he proposes human-reserved work: child care, food preparation, art, sports. When he tested the idea with Claude, he could not push the reserved share high without artificially restricting AI, even at 40% of the labor market via early retirement and shorter workweeks.

The hardest objection is to his remedy. He wants models above a capability line to run only on platforms that certify users cannot customize safeguards away. That would be enforced through US law at the platform and distribution layer, not at the chip, with Hugging Face and GitHub as choke points. The host replied that trust in institutions is low, that USAID was torn down, and that GitHub does not restrict what it carries. Gates concedes institutions are weakened and calls the USAID cut a mistake. His counter is that the alternative is worse. Letting cyber and bio attacks run will not raise trust. He adds that AI will let regulators do a better job with the same headcount. He dismisses liability as a tool, since the harm comes from anonymous people running unguarded models, and no court will find them. Pressed on a Mac Studio running open weights at home, he says monitoring on client machines is very doable but gives no mechanism. He admits that without a decision soon, capability will sit in dark corners he cannot see. The fallbacks are rewriting vulnerable code and deep biosurveillance.

Politics adds a second constraint. A true slowdown is not on offer. Gates would back one if a credible, globally enforceable, auditable proposal existed, as with nuclear production, but none does. So the live argument is whether government should look at all. He does not accept that Democrats are running against AI. One party saying full speed and the other saying stop would be, as with climate, "a failure of democracy." Data-center fights he treats as a separate matter. A 1 gigawatt site is about $40 billion, the sites will get built, and blocking them does not minimise AI's harms.

He expects action only after pain. We prepare for earthquakes because of small ones, yet COVID did not make the world readier for pandemics, which surprised him. In cyber, where there is a business model, he expects attacks of unprecedented scale over the next several years. So everything depends on how big the lesson has to be. The framework needs institutions the public does not trust, and the timetable is set by the attackers, not the regulators.