Recorded 22 October 2026 · 42 min
Almost none of the universe is inside computers or on the internet, so a machine that understands physical phenomena has to be a robot. That is Sergey Levine's core claim, and the rest of the conversation hangs off it. Levine co-founded Physical Intelligence and is a professor and former Google robotics researcher. The host presses him on models, data and strategy.
The tension is between a promising endpoint and an unproven path. Once robots are useful, he says, their data will dwarf everything else. Getting there is a bootstrapping problem. The models need prior knowledge from LLMs, the right data, and deployments good enough to learn from. His answer is generality: solve robotics broadly rather than building for one robot or one task. He also admits what nobody knows, which is how the ultimate robot foundation model will work.
Grasping farm hit a wall — ▶ 2:53
Google's early bet was to fill a repurposed conference room with robots grasping objects around the clock, on the theory that experience would compound. It produced excellent grasping and nothing else. Other skills need knowledge as well as experience, and robots cannot discover knowledge from scratch, any more than an infant locked alone in a room would learn physical skills. The missing piece was prior knowledge, which the AI community had learned to extract from web data by training foundation models. Lots of robots in a room still has merit, but only when paired with that prior.
Pi 0 as outline — ▶ 7:22
A robot foundation model takes camera images and language instructions and outputs a continuous vector of roughly 30-50 numbers: end-effector positions, joint angles or motor torques. Training from scratch on robot data alone cannot supply the prior knowledge needed. An LLM is the right starting point because it is the complement of robot data: lots of knowledge, none of it grounded. Pi 0 was never meant to be the ultimate model. It was the outline, to be coloured in with chain-of-thought reasoning, video, human data and other sources, which is most of what the team has done since.
Reasoning and data in pi 0.5 — ▶ 10:14
Pi 0.5's generalization came from principles borrowed from other foundation models and mapped onto robotics. First, the right data, which is an economic, industrial and research problem at once. You usually cannot know what is right until you scale to some level. Second, internal reasoning. Told to clean a kitchen, the model breaks the job into thoughts such as pick up the dish, pick up the towel, wipe the spill, then grounds them into actions. Levine calls this "tremendously powerful" and says it is why coding agents plan before writing code.
Analogies hide in hindsight — ▶ 12:32
Asked whether Physical Intelligence can draft off frontier-lab research, Levine says the ideas are not actually the same. Analogies are more obvious in hindsight than in foresight. Prompt expansion was the biggest innovation in video models because detailed prompts have higher mutual information with the output. A robot cannot copy it, since it does not care about a dog's golden fur. It needs annotations tied to the behavior it must exhibit. Getting those right was key to zero-shot generalization in pi 0.7. The principle transfers, but the instantiation must be rediscovered.
Pi 0.7 goes zero-shot — ▶ 14:08
Pi 0.7 targets zero-shot generalization out of the box, mirroring the LLM path from fine-tuning bases to prompt engineering to instruction-tuned models. Robotics is not at "tell it anything" yet. But a robot can be coached in language to acquire new skills, because the model decomposes skills into steps and only the steps need supervision, not the actions. It also reasons in images. A generated picture of a finger on the power button beats "3 inches left of the edge." A separate image model, driven by the LLM, makes the picture. Integration is expected later, and a newer model is already in beta.
Attention to detail wins — ▶ 17:16
The key trait for a robotics researcher is attention to detail, because the stack is deep. A failure could be an unattached camera connector or a model that needs 32 billion parameters instead of 16 billion. His example is a model that fails not from the learning rate or dataset but because gripper fingers leave a gap when told to close, due to a loose connector. You go plug it in and test it yourself. Credentials are a poor proxy, and someone without a PhD can be hired. Signaling the right mindset is the hard part, and smart engineers tend to specialize.
The universe isn't online — ▶ 21:58
The universe is huge and, up to a rounding error, physical, so a machine that understands physical phenomena has to be a robot. The concrete version is that if robots become useful, as cars did, there will be lots of them, all trying things, failing and recording results. Even if only a fraction of economically relevant work goes to robots, their data dwarfs everything else, and it is "not even a close call." The asymptote looks extremely good. What remains is bootstrapping, which he calls "a good problem to have" because you know where you end up.
Cars are too specialized — ▶ 23:52
The host notes that Waymo runs roughly 5,000-6,000 cars and took 17 years to bootstrap. Levine says cars are too specialized. If driving were the only thing a child were allowed to do, acquiring physical skills would be very hard, and putting a 5-year-old behind the wheel is not a development strategy. Robots do more, so they have more to learn from, some tasks high stakes and some low. The low-stakes ones bootstrap the high-stakes ones, as in people. A car also cannot learn by playing around without being stopped by the cops or crashing, which makes RL hard.
Interventions, then RL — ▶ 26:31
Deployed robots at sufficient autonomy run under human oversight, and each intervention supplies exactly the data needed. That is a known strategy. The next step, working in the lab but not yet widely deployed, is learning from the robot's own experience. Interventions teach a robot to fix mistakes. If it drops an item in a logistics facility, a human puts it back, so it never learns to avoid the drop. Reinforcement learning fixes that clumsiness. It is a slower process, and it is enabled by the greater autonomy that interventions provide first.
Make the problem bigger — ▶ 28:48
When facing a hard intelligence problem, "making the problem as big as possible actually makes it easier." Early machine translation trained on paired English and Chinese sentences, and the better answer was to solve all NLP at once. Levine argues this applies more to robotics, since generality exposes more edge cases. Humanoid-only companies resemble 1950s IBM, which sold vertically integrated, room-sized computers with its own technicians and software. Computing later factorized into operating systems, hardware and software. He expects a physical intelligence layer to enable the same explosion of robot types and applications. He has nothing against humanoids.
Data over world models — ▶ 32:40
Whether there is still room for a new robot foundation model company depends on something nobody knows: how the ultimate model will work. Levine calls it an open science problem and says Yann LeCun's very different view could be right. On world models, he argues the most important thing is the data trajectory, from human-produced to intervention to autonomous data. The model type is a downstream choice. He still expects the ultimate model to include a world model, for reasoning below the semantic level, as in rock climbing. But he calls the world-model focus a bit distracting.
Robot data will dwarf every other source, which means the hard part is the climb, not the summit. Sergey Levine's argument is simple. If robots become useful, there will be lots of them, the way there are lots of cars. Each one tries things, succeeds, fails and records what happens around it. Even if only a fraction of economically relevant work goes to robots, the data they produce swamps everything else. Even the pessimistic case, thousands of robots instead of billions, yields enormous volumes at commercial scale. So the endpoint looks good. What remains is getting off the ground, and he calls that "a good problem to have". You know where you are going. You only have to do the engineering and industrial work to arrive.
The obvious objection is time. Waymo runs perhaps 5,000-6,000 cars, a small fleet, and its bootstrapping took 17 years. Levine's answer is that cars are too specialized. Make driving the only thing a child may do, and learning physical skills becomes very hard. A car also cannot learn by playing around, because it gets stopped by the cops or crashes. A robot does more, so it has more to learn from. Some of those tasks are high stakes and many are low stakes. The low-stakes ones bootstrap the high-stakes ones, as they do in people. Breadth is not the price of reaching scale. It is the shortcut.
The same logic drives his view of strategy. On a hard intelligence problem, "making the problem as big as possible actually makes it easier." Machine translation improved not by training on paired English and Chinese sentences but by solving all of NLP at once. Levine says no NLP researcher disputes this now. He thinks it applies more to robotics, because generality exposes the unusual situations and edge cases, and those matter more in the physical world. A company that collects only humanoid examples is making the translation-model mistake. Computing followed the same arc. In the 1950s and 60s IBM sold a vertically integrated, room-sized computer and supplied the technicians and software engineers too. Now operating systems, hardware and software are factorized. He expects robotics to do likewise, with a physical intelligence layer under many kinds of robots. He has nothing against humanoids. He just does not expect them to be the whole story.
Generality needs a source of prior knowledge, and that is where language models come in. His first attempt at scale, at Google, filled a repurposed conference room with robots grasping objects around the clock. The robots became excellent at grasping and nothing else. Other skills take knowledge as well as experience, and a robot cannot discover knowledge from scratch, any more than an infant locked alone in a room can. An LLM is the complement of robot data. It has lots of knowledge, and none of it is grounded. So the first model, pi 0, took camera images and language and produced a vector of roughly 30-50 continuous numbers, starting from a pretrained language backbone. Levine calls it an outline. Reasoning, video, human data and other sources were to be coloured in afterwards.
Borrowing from language models is not free, though. Analogies, he says, are more obvious in hindsight than in foresight. Pi 0.5 added internal reasoning. Told to clean a kitchen, it breaks the job into thoughts, pick up the dish, wipe the spill, and grounds each into actions. That is why coding agents work too. Pi 0.7 needed something less obvious. Video models gain from prompt expansion because a more detailed prompt has higher mutual information with the output. But a robot does not care about a dog's golden fur. It needs annotations tied to the behavior it must exhibit. Getting that right was a key ingredient in zero-shot generalization. A robot can now muddle through a kitchen appliance it has never used. It can be coached in language, with only the steps supervised rather than the actions. It also reasons in images. Saying the power button is 3 inches left of the edge is useless. A generated picture of a finger on the button is not.
Deployment is where the data climb starts to pay. A model with enough autonomy runs under human oversight, and each intervention supplies exactly the data it needs. That is now standard. Interventions teach a robot to fix mistakes, not to avoid them. If it drops an item in a logistics facility, a human puts it back, and the robot learns the fix but not how to stop dropping things. Reinforcement learning addresses that, turning a clumsy but autonomous robot into a good one. It is slower, and it is enabled by the autonomy that interventions provide first.
Here the confidence thins. Levine says the most important question is not the model but the data: where it comes from, and how the trajectory moves from human-produced data to interventions to autonomous experience. He finds the world-model focus a bit distracting for that reason, though he expects the ultimate model to include a world model. Rock climbing, he notes, is not reasoned about in semantic terms. Yet he says outright that the ultimate robot foundation model is an open science problem, that Yann LeCun holds a very different view, and that LeCun may be right. The self-improving loop that justifies the asymptote is not yet widely used in deployment. Waymo's 17 years remain the only timeline on the table. Underneath sits a people problem. Failures can hide anywhere in a deep stack, from a loose camera connector to a model that needs 32 billion parameters instead of 16 billion. So he hires for attention to detail over credentials, and admits there is no good way to signal it. How the model should work will be settled by the field. As he puts it, "there is an actual truth out there and somebody's right and somebody's wrong."
The open question. The bootstrapping problem is waved off as "a good problem to have", but no timeline is offered. Waymo's 17 years remain the only reference point on the table. The self-improvement loop that justifies the asymptote, robots learning from their own experience through reinforcement learning, works only in the lab so far. The prediction that the ultimate model will contain a world model, an LLM and diffusion action generation sits oddly beside the claim that the world-model focus is distracting. Watch for evidence that deployed fleets are improving from autonomous experience, not just from human interventions.
Watch the full video at https://www.youtube.com/watch?v=5X166Fead8o.
Comments ()