LLMs can't plan or foresee consequences, so world models win

1. Useful, not a path — ▶ 2:37

   LLMs underpin products LeCun uses himself, and he helped build open-source ones. His claim is narrow: they are "not a path towards human level or human like intelligence or even animal like intelligence." Language suits current architectures because it is discrete. Reality is "way more complicated": high-dimensional, continuous, noisy and messy. AMI, for advanced machine intelligence, carries the motto "AI for the real world". By the end of last year it was clear Meta was not the right place to pursue it, so he left to found AMI Labs.

2. Research versus product mode — ▶ 5:24

   Llama 1 came out of FAIR. In early 2023 Meta built the GenAI organization to productize it, producing Llama 2, 3 and 4. Llama 4 "was a bit of a disappointment", so Zuckerberg rebooted the organization. Meta then realized it had fallen behind and refocused on catching up. Exploratory research lost priority as a result. Zuckerberg and CTO Andrew Bosworth still backed the JEPA and world-model work, but the rest of the company was entirely LLM-focused. Most applications were in manufacturing, which Meta did not care about.

3. Planning is the gap — ▶ 8:22

   Vision-language-action models for robots are "pretty much now seen as a failure": unreliable and too hungry for data. A world model lets an agent predict the consequences of its own actions. With one, it plans by search and optimization over action sequences rather than predicting one action after another autoregressively. LeCun cannot imagine building an agentic system without this ability. LLMs lack both consequence prediction and planning, he says, because inference is next-token prediction, not search. Acting without predicting consequences is how people end up thinking you are an idiot.

4. Predict abstractly, not pixels — ▶ 11:19

   Nobody can predict how an uncapped water bottle will fall at pixel level, so mental models work at an abstract level. LeCun's epiphany came about five years ago. Every architecture that learned good image and video representations was non-generative, and generative ones had failed. VAEs and sparse autoencoders were weak. FAIR's large masked-autoencoder project was "very disappointing". Joint-embedding predictive architectures, where two encoders see different views and a predictor links their representations, worked much better. DINO v1 to v3 and V-JEPA are examples. Predicting pixels was "a losing proposition".

5. Data efficiency over scale — ▶ 16:25

   Impressive robot demos rely on imitation learning with enormous per-task data, which is expensive and brittle. A world model would plan new tasks with little or no training. A 17-year-old learns to drive in about 20 hours, yet millions of hours of driving data have not produced level-5 cars. Synthetic data is the wrong fix if you crack data efficiency. Labs lack incentive to try, since scaling keeps working and no one can afford to fall behind. Hence "digging the same trench", and AMI's Paris headquarters with a New York office instead of Silicon Valley.

6. Industrial control first — ▶ 22:03

   A domestic robot is "several years down the line". None of the many robot companies knows how to make machines smart enough to be useful, and even narrow manufacturing tasks work only through imitation learning. Nearer-term value is in complex systems: jet engines, chemical plants, power plants, manufacturing lines, patients, human cells. They are too complex for a few equations. A deep network learns their dynamics from data, and if it is action-conditioned it becomes a world model for optimal control. LeCun calls the number of industrial applications "mindboggling".

7. World domination, with caveats — ▶ 24:11

   Five years to "complete world domination" is a Linus Torvalds-style joke, but LeCun means it as the blueprint for future intelligent systems. LLMs keep a small place as a language interface. His systems think first, and talking and listening get added on top. Within a year he expects a general method for training hierarchical models across modalities. Within 12-18 months he expects demos of action-conditioned world models for robotics, industrial process control and maybe healthcare. The paradigm change will be "completely obvious" by early 2027, though that does not mean a solution by then.

8. Tapestry and sovereignty — ▶ 28:20

   If AI assistants mediate everyone's information diet, ones built in California or China will not fit users elsewhere. Their languages, cultures, values and politics are poorly represented. Tapestry proposes an open, fine-tunable foundation model trained federated-style. Contributors keep their data and exchange only parameter vectors, pulling each local vector toward a global consensus. The result would be as good as a model trained on all the world's data. Countries outside the US and China want sovereignty, so a frontier open platform could emerge by being trained on more and better data than proprietary systems.

9. Sun Microsystems redux — ▶ 34:54

   In 1996 the internet ran on Sun, HP and Dell with proprietary Unix and Windows NT. Linux wiped all of it out, and even Azure runs Linux. LeCun says OpenAI and Anthropic are "the Sun Microsystem and HPUX of yesterday", because platforms tend to become open. Openly available text data is already exhausted. Those companies now license copyrighted data or train on synthetic data, which gives open models room to catch up.

10. Language as the substrate — ▶ 36:34

    IMO gold and code success come from domains where language is the substrate of reasoning. LLMs are problem solvers, not creators of concepts and definitions, and programmers, not software architects. Humans move up a level of abstraction to decide what to build. Math and code work because search in token space can be checked by running the code or verifying the proof. That is inefficient and limited to verifiable domains. LeCun's bar is a zero-shot agent handed an unseen problem with no script. Without consequence prediction and planning, he says, it will fail.

11. They changed their minds — ▶ 40:53

    "I didn't change my mind. They changed their mind." LeCun says Geoffrey Hinton met GPT-4 in 2023 and reasoned that if 10 real neurons replace one backprop neuron, the cortex is only 1.6 billion units, close to GPT-4. LeCun rejects this and reads it as Hinton declaring victory. He says Hinton is now less vocal about danger. Yoshua Bengio's worry, in his reading, is inequality and bad use, not takeover. He rejects Anthropic's apocalyptic framing and suspects commercial motives for lobbying governments with it.

12. Unsafe by construction — ▶ 45:43

    LeCun calls current LLMs "intrinsically unsafe" and says it irritated some Meta colleagues. They cannot be stopped from hallucinating, and an agentic one may take an action whose outcome it never predicted. Behavior depends only on training, and the gap between training and test error guarantees some prompt where they fail. His alternative is objective-driven AI. The system optimizes a task cost function using a world model, plus low-level safety constraints it cannot violate by construction. He concedes the cost function or world model could still be wrong.

13. What FAIR got right — ▶ 53:46

    FAIR's wins were a top research lab, tools like PyTorch and a culture of openness. Innovation is a chain from blue-sky ideas through pushing them to beat records, then a company-scale engineering push, and that last step is where most firms fail. Meta handled it better than Xerox but missed steps. In 2023 GenAI took 60-70 scientists from FAIR and then, under short-term pressure, "became very conservative". Two Llama authors left to found Mistral. LeCun's recipe: hire the best people, give them the means, and get out of the way.

14. Why he really left — ▶ 1:02:07

    LeCun ran FAIR for four and a half years, then stepped down near age 58 to become chief AI scientist and return to research. He says he made zero technical contribution to Llama. His one input was arguing for open-sourcing Llama 2 in weekly 40-person debates lasting months. His 2022 vision paper rallied a team and won support from Zuckerberg and Bosworth, but the layers below did not see the point. Meta disbanded its robotics AI group, and FAIR was pushed to help with LLMs. Investors who had read the paper bought the story.

15. Preventing representation collapse — ▶ 1:14:42

    Predicting in representation space invites collapse: the encoder outputs a constant and prediction becomes trivial. LeCun's 1993 contrastive method does not scale with dimension. Distillation methods like DINO prevent collapse, but "we don't know why", and the minimized loss actually rises. V-JEPA uses them, and AMI is moving away. The new route is an explicit regularizer that maximizes encoder information content. Only upper bounds exist, so "we take an upper bound and we cross our fingers". SIGReg forces outputs toward an isotropic Gaussian. A small-scale world model paper with a former postdoc, now at Brown, is his recommended read.

Intelligence is a search for actions, and a language model does not search for actions. It produces the next token. Yann LeCun's whole technical case rests on that difference. An agent that acts in the world needs two things. It needs a way to predict what an action will do, and a way to find, by optimization, a sequence of actions that lands on a goal. LLMs have neither. Their inference is prediction of the next token, not search over consequences. LeCun is careful about the scope of the claim. LLMs underpin products he uses himself, and he helped build open-source ones. His claim is only that they are "not a path" to human-like or even animal-like intelligence.

Why did language work so well, then? Because it is easy. A language has a finite vocabulary, around 100,000 tokens. A network can output a probability distribution over all of them, sample one, shift it into the input and repeat. The real world offers no such luxury. It is high-dimensional, continuous, noisy and messy. LeCun's example is an open water bottle on a table. Push it at the bottom and it slides. Push it near the top and it flips. Nobody can predict where the water goes at pixel level, so human mental models work at an abstract level. That is the logic behind JEPA. Encode two views of the same thing, then predict the representation of one from the other.

The empirical record, as he tells it, points the same way. About five years ago he realized that every architecture that had learned good representations of images and video was non-generative. Variational autoencoders and sparse autoencoders gave mediocre results. FAIR's large masked-autoencoder project was "very disappointing". Joint-embedding methods like DINO and V-JEPA worked much better. Predicting pixels, he concluded, was a losing proposition.

The practical stakes show up in robotics. The demos are impressive, he says, but they are trained by imitation on enormous per-task data sets, which makes them expensive and brittle. A system with a world model could plan a new task without being trained on it. The comparison he keeps returning to is a 17-year-old learning to drive in about 20 hours. There are millions of hours of driving data in the world and still no level-5 car. If you crack that data efficiency, he says, you do not need synthetic data. He also writes off vision-language-action models as "pretty much now seen as a failure". The shorter-term market is industrial. Jet engines, chemical plants, manufacturing lines, patients and human cells are all too complex for a few equations. A learned, action-conditioned model of their dynamics would let you control them optimally.

The obvious objection is that LLMs are already succeeding at hard reasoning. They win IMO gold and sustain long coding tasks. LeCun's reply is that math and code are domains where language itself is the substrate of reasoning. LLMs are problem solvers, not creators of concepts and definitions. They are good programmers, not software architects. They also plan after a fashion, by searching token sequences. That works because the output can be checked: run the code, verify the proof. It is inefficient, and it only works where search in token space can be verified. He concedes the commercial point. If it works, it is fine. But he expects deployment to get harder use case by use case, with training data collected for each and no end to hallucination. His own bar is a zero-shot agent that handles a task it was never trained for. He adds a dig at benchmark gaming. When he names something an LLM cannot do, people type it into ChatGPT, it enters the fine-tuning set and the next version passes. That is training, he says, not new intelligence.

Safety follows from the same mechanism. LeCun calls current LLMs intrinsically unsafe, a view some Meta colleagues disliked. They cannot be stopped from hallucinating, and their behavior is set by training. Because training error never equals test error, there is always some prompt where they do something stupid. His alternative is objective-driven AI. The system imagines action sequences with its world model and picks the one that minimizes a task cost, subject to hard safety constraints. It can do nothing else, he says. He concedes the limits. The cost function can be inaccurate and so can the world model. The system can still err, but only within what it predicts.

This is also where he parts from the other two Turing Award winners. "I didn't change my mind. They changed their mind." Geoffrey Hinton, he says, met GPT-4 in 2023 and reasoned that if 10 real neurons stand in for one backprop neuron, the 16 billion neurons of the cortex shrink to 1.6 billion units. That is roughly where GPT-4 sits. LeCun rejects the argument and reads it as Hinton declaring victory and retiring to give talks. He now hears Hinton as quieter, having noticed that conceptual breakthroughs are still missing. Yoshua Bengio, he says, worries about inequality and misuse rather than takeover. Anthropic's apocalyptic framing he dismisses, partly as commercially convenient. That last claim is about motive, and he offers no evidence for it.

Organizations decide which architecture gets built, and LeCun's account of Meta shows how. Research and product are different modes. Llama started as a small FAIR project. Then GenAI took 60-70 scientists and, under short-term pressure, "became very conservative" and cut itself off from FAIR. Two Llama authors left to found Mistral. Llama 4 disappointed, Meta reorganized, and exploratory work lost priority. Meta also disbanded its robotics AI group. Most of his applications were in industries Meta did not care about. He corrects what he calls a popular misconception. After stepping down as FAIR director in 2018 he had no authority over what people worked on. He had zero technical contribution to Llama. His one input was to argue, in weekly 40-person debates lasting months, for open-sourcing Llama 2. His 2022 vision paper rallied people to his project, and investors who had read it bought the story when he raised money for AMI.

The same structural argument runs through his case for open platforms. Silicon Valley labs, he says, are all "digging the same trench". Scaling still works, so there is little reason to chase data efficiency, which is why AMI sits in Paris and New York. Tapestry applies the logic at national scale. If AI assistants mediate everyone's information diet, models built in California or China will misfit other languages, cultures and values. Countries outside the two powers want sovereignty. Contributors would train locally, keep their data, and exchange only parameter vectors that converge toward a consensus model. Platforms, he argues, tend to become open. In 1996 Sun, HP and Dell sold proprietary Unix and Windows NT, and Linux wiped them out. OpenAI and Anthropic, he says, are "the Sun Microsystem and HPUX of yesterday". Public text data is already exhausted, so closed labs are licensing copyrighted data or generating synthetic data.

The hardest problem in his own approach is one he describes with unusual candor. When a network predicts representations of representations, it can collapse to a constant output, making prediction trivial. His 1993 contrastive method does not scale with dimension. The distillation tricks used in DINO and V-JEPA do prevent collapse, but "we don't know why", and the loss they appear to minimize actually goes up. AMI is moving to explicit regularizers that maximize the information content of the encoder. Only upper bounds on information are measurable, so "we take an upper bound and we cross our fingers". One such regularizer, SIGReg, forces the outputs toward an isotropic Gaussian. It works at small scale so far.

So the bet is clear and so is its exposure. LeCun expects a general method for hierarchical models within a year, working demos in 12-18 months, and a field-wide recognition of the paradigm shift by early 2027. He then qualifies the last claim himself. Recognizing the shift does not mean a solution exists by then. The test he sets for LLMs, a zero-shot agent on an unseen task, is the same test his own world models have not yet passed at scale.