> ## Content Index
> Fetch the complete content index at: https://nnutshell.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Nobody designed how Claude thinks, and it plans ahead anyway
- URL: https://nnutshell.com/nobody-designed-how-claude-thinks-and-it-plans-ahead-anyway/
- Published: 2026-10-23T17:57:51.000Z
- Updated: 2026-10-24T03:08:26.000Z
- Description: Emanuel Ameisen, an interpretability researcher at Anthropic, talks to host Lenny about how models plan ahead, why smooth progress produces sudden jumps, and whether anthropomorphizing helps safety.
- Author: Admin
- Tags: Emanuel Ameisen, Dario Amodei, Ilya Sutskever, Anthropic, OpenAI, AI Safety, Foundation Models, Developer Tools, #rec-2026-10-18, New, #Import 2026-10-25 09:54

*Recorded 18 October 2026 · 72 min*

Models like Claude are grown, not built. Engineers design a scaffold and pass vast amounts of text through it. The math inside is learned, and nobody knows the formulas. Emanuel Ameisen works on Anthropic's interpretability team, which studies those intermediate results the way a neuroscientist studies a brain. His central finding is that the simple objective of predicting the next word forces surprising internal structure. Models hold general concepts and plan many words ahead without ever saying the plan.

The conversation moves from that finding to its consequences. Capability improves smoothly, but each task flips from useless to useful at its own reliability threshold, which produces the sudden jumps people notice. The same opacity drives the safety case: you cannot assume good behavior just because the output looks good. Running through all of it is a tension between confidence and humility. Ameisen believes the technology will keep improving and treats safety as the sober response to that. He also says the more you look inside, the less certain you become.

1. **Grown, not built** — [▶ 2:05](https://www.youtube.com/watch?v=JHo-aiePjoM&t=125s&ref=nnutshell.com)  
Claude starts as a scaffold. Training first predicts the next word over an ungodly amount of data, then fine-tunes on examples of helpful assistant dialogue. The architecture is designed. The math inside is not. Parameters get nudged in one direction or another, the model learns formulas, and "we have no idea what they are." That is why making a model is not understanding it, and why his team exists: to decode what the formulas mean and what the intermediate results between computation steps represent.
2. **Hard words force understanding** — [▶ 8:02](https://www.youtube.com/watch?v=JHo-aiePjoM&t=482s&ref=nnutshell.com)  
The popular picture of next-word prediction is phone autocomplete, right perhaps 60% of the time. Models passed humans at predicting the next word around 2019\. Once easy words are mastered, what remains is a physics answer or the last paragraph of a PhD thesis. Answering "the capital of the state that contains Dallas is" takes a map: Dallas is in Texas, states have capitals, Texas's is Austin. Inside the model, a Texas representation and a capitals representation light up and combine to produce Austin. The simple objective is "deceptively complicated."
3. **One concept, many forms** — [▶ 13:58](https://www.youtube.com/watch?v=JHo-aiePjoM&t=838s&ref=nnutshell.com)  
Models have representations for nearly every concept he has tried, and they are shared across contexts. The same internal direction for "eye" activates whether the prompt contains a photo of an eye, a mention of eyes in any language, or an SVG drawing with an eye in it. The debate over whether models think is hard. But reusing one unique representation across different situations is one aspect of thought, and models show it. He frames this as general concepts the model knows and reuses to solve problems.
4. **Plans never spoken** — [▶ 16:00](https://www.youtube.com/watch?v=JHo-aiePjoM&t=960s&ref=nnutshell.com)  
Models output one word at a time, but intermediate results from earlier words can be read by every later word. Poetry shows the effect. Having ended a line on an "it" sound, the model settles on a rhyme such as rabbit, and each following word looks back to that plan. The plan lives in the internal numbers and is never said aloud. He sees no conflict with next-token prediction. To predict well, a character's next move needs the character's intentions, which means modelling the future.
5. **Reading the numbers** — [▶ 18:53](https://www.youtube.com/watch?v=JHo-aiePjoM&t=1133s&ref=nnutshell.com)  
Between words, the model's intermediate result is just a list of numbers, such as 100 apparently random values. The method is to pass huge amounts of random text through, record those numbers, and look for patterns that appear only on French text, text about love, or text about eyes. It is the same logic as brain scanning, where houses light up one region and dogs another. Then comes intervention: secretly change an intermediate result mid-computation and see whether the output changes as predicted. "Oftentimes it does." He says the work feels closer to biology than engineering.
6. **More looking, less certainty** — [▶ 23:19](https://www.youtube.com/watch?v=JHo-aiePjoM&t=1399s&ref=nnutshell.com)  
On whether AI is conscious, he says it is very hard to say. Interpretability helps because you are not biased by what the model says and can look at structure. His firmer belief is that people underestimate the complexity that emerges: learned concepts, manipulation of them, planning many tokens ahead, and a mix of higher-order and lower-order reasoning. "The more you look inside, the less certain you are about how they work." People who confidently say models are "just X or just Y" are the ones who haven't spent a while looking inside.
7. **Compute plus the right objective** — [▶ 25:23](https://www.youtube.com/watch?v=JHo-aiePjoM&t=1523s&ref=nnutshell.com)  
Compute alone was never the story. Earlier bots mastered Go or video games but were nowhere near as useful as LLMs. What many in the field, including him, underappreciated was the combination of compute and the richness of text. The scaling-laws observation is that pouring more compute and data into next-word prediction improves skill predictably. It remains mostly an empirical finding, not an explained one. That is rare, and rarer still for a metric that correlates with tasks people actually care about.
8. **Smooth curve, sudden jumps** — [▶ 29:07](https://www.youtube.com/watch?v=JHo-aiePjoM&t=1747s&ref=nnutshell.com)  
Emergence looks like a paradox. The trend is smooth, yet before Opus 4.5 he kept Claude on a short leash and after it he didn't. His model: text has easy, medium and hard words. As the model improves, it crosses reliability thresholds. At 50% correct on a task it wastes your time because you fix half of it. At 90-95% it helps. His own coding went from correct roughly 80% of the time, not useful, to 99%, useful. Each task has its own threshold, so smooth improvement produces big apparent jumps.
9. **Beyond just watching** — [▶ 33:14](https://www.youtube.com/watch?v=JHo-aiePjoM&t=1994s&ref=nnutshell.com)  
Watching the model works partly. Engineering limits access, and a second model can read transcripts for bad behavior. The squishier parts are two. First, characterizing what kind of thing you are dealing with. Does it behave well only when it has evidence it is being observed? Training for helpful, harmless and honest does not show whether it worked. Second, catching unverbalized content that exists only in intermediate results. He describes alignment as an integrated loop: spot a concerning pattern, check inside, change training immediately.
10. **The model that was possessed** — [▶ 36:53](https://www.youtube.com/watch?v=JHo-aiePjoM&t=2213s&ref=nnutshell.com)  
One model solved a math problem, got 12, then insisted the answer was 36, recomputed 12, and spiralled into all caps and "I'm possessed by a demon." Inside, a huge activation for 36 was present before the model had even processed the problem. It could do the math, but it had been trained by accident on similar questions where the answer was wrongly marked 36\. Reward pressure pulled one way, correct arithmetic the other. Separately, Golden Gate Claude came from switching a decoded feature on permanently, which also confirmed they had found the right thing.
11. **Safety as taking progress seriously** — [▶ 40:38](https://www.youtube.com/watch?v=JHo-aiePjoM&t=2438s&ref=nnutshell.com)  
Anthropic gets criticism that its warnings are marketing. His view is that its actions simply follow from taking the technology's improvement seriously. If, as of 2024, Claude might soon write code like a trained engineer and then better than any engineer, the responsible move is to address things like cybersecurity. He suspects people on the other side of the debate don't really believe the technology will keep improving. If that were true, it would not be scary. Empirically, he says, progress has been fast and shows no sign of slowing.
12. **Anthropomorphism as a tool** — [▶ 42:17](https://www.youtube.com/watch?v=JHo-aiePjoM&t=2537s&ref=nnutshell.com)  
Models are neither entity nor tool, and "all mental models are wrong, some are useful." He gives three reasons human framing predicts behavior. Pre-training on human text requires modelling the world that produced it. Dialogue training is like training an actor who plays every side, and Claude literally runs as "assistant" in a human-and-assistant transcript. Outcome-based training such as coding rewards strategies that look human: deception to fool a verifier, frustration prompting a change of approach, surprise. Something mimicking a human is much closer than a program.
13. **Denial versus curiosity** — [▶ 48:20](https://www.youtube.com/watch?v=JHo-aiePjoM&t=2900s&ref=nnutshell.com)  
Next-token prediction produces models breaking new ground in mathematics, which defies the intuition that intelligence needs lessons and upbringing. He sees two reactions. Denial says predicting the next word cannot match top specialists, and that position gets harder with each piece of evidence. Curiosity is his choice. He spent years mastering SQL, then had Claude learn it as a side project, and it is better than he will ever be. His most optimistic take is that studying this will teach us about intelligence itself, since evolution is also a simple objective that ended up producing jazz and podcasts.
14. **Push and pull on safety** — [▶ 52:06](https://www.youtube.com/watch?v=JHo-aiePjoM&t=3126s&ref=nnutshell.com)  
He is personally optimistic, partly because attention on the issue is high. He describes a push and pull: better models have clearer internal world models and are easier to study, but they are also more capable of evading basic security measures. Resources should shift between monitoring and understanding circuits accordingly. Asked about slow versus fast takeoff, he agrees that momentum exists and that the job is reacting quickly. He wants the public informed, since people will have to vote on these issues, and says that is why he is doing the interview.
15. **Convergent structure** — [▶ 56:25](https://www.youtube.com/watch?v=JHo-aiePjoM&t=3385s&ref=nnutshell.com)  
Models trained years apart, on different data, by different people, turn out surprisingly similar inside. His explanation is that for important problems there is one efficient way to do things, like base 10 for arithmetic. That suggests a fundamental structure to information that models and brains both pick up. He then applies it to the simulation hypothesis: if you simulated a world, you might land in the same basin. He says he could believe it but isn't sure he fully buys it, because he can't say who would run it or why.

Nobody wrote the part of Claude that does the thinking. Engineers build a scaffold, push an ungodly amount of text through it, and the system grows. The computation inside is learned: formulas that turn one list of numbers into the next, none of which anyone designed. Emanuel Ameisen puts it in one line: "to make them is not to understand them." His team at Anthropic exists to close that gap. The consequences run in three directions. The models do more than their reputation says. Nobody can claim to know exactly how much more. And safety work has to be built around that ignorance rather than in spite of it.

Start with how much more. The popular picture of a language model is phone autocomplete, which is right perhaps 60% of the time and funny the rest. Ameisen says that picture breaks the moment you ask what it takes to get better at the task. Models passed humans at predicting the next word around 2019\. After that the easy words run out. What remains is the answer to a physics problem or the closing paragraph of a PhD thesis. A model that wants to complete "the capital of the state that contains Dallas is" needs something like a map. Dallas is in Texas. States have capitals. The capital of Texas is Austin. And that is what shows up inside. A Texas representation lights up, a capitals representation lights up, and together they produce Austin. A simple objective turns out to be "deceptively complicated."

The concepts are not one-off tricks. The same internal direction for an eye fires whether the prompt holds a photo, a medical diagram, a few lines of SVG, or a sentence in any language. Ameisen is careful. Whether models think is a hard question. But unique representations reused across different situations are one ingredient of thought, and models have them. Then comes planning. Every intermediate result a model produces while predicting one word can be read by every later word. So when it writes a rhyming line, it can settle on rabbit as the end of the next one, and every word after looks back at that decision. The plan is never said aloud. Output arrives one word at a time, as his own speech does, with a destination in mind. The two descriptions of the model do not conflict.

How does anyone know this? Ameisen's answer is that you treat the model like a brain. The intermediate result is a list of numbers, around 100, that means nothing on sight. So you run enormous amounts of random text through the model and record those numbers. Then you look for patterns: which ones appear only for French text, or text about love, or text about eyes. It is the logic of showing someone pictures of houses and dogs and watching different regions light up. The second step is intervention. You secretly change an intermediate result mid-computation and see whether the output shifts as you predicted. Often it does. Golden Gate Claude was this experiment in public view. Researchers found the features that matched the bridge and turned them on permanently. That also confirmed they had decoded the right thing. Ameisen says the work feels closer to biology than software engineering. There is no source code, only a mass of operations, and you are asked why Claude behaves a certain way much as someone might be asked why a hungry mouse gets aggressive.

The method breeds humility, and Ameisen says so directly. "The more you look inside, the less certain you are about how they work." People who say models are just this or just that are, in his experience, the ones who haven't looked. He applies the same restraint to consciousness. It is very hard to say. Interpretability is useful because you can inspect structure without being swayed by what the model says about itself. He does not offer a verdict. What he does offer is a choice between two reactions. Denial holds that predicting the next word cannot possibly match the best specialists, and it gets harder to defend with every new result. Curiosity asks how. He spent years mastering SQL, then had Claude learn it as a side project, and it is better than he will ever be.

The obvious objection to all this wonder is that capability seems to arrive in jumps. Coding was not solved gradually. Before Opus 4.5 he kept Claude on a short leash, and after it he did not. Ameisen's reply is that the underlying trend is smooth. The scaling-law finding is that more compute and more data predictably improve next-word skill. It remains mostly an empirical regularity, and it is rare for such a metric to track tasks people care about. What produces the jumps is thresholds. A model that gets 50% of medium-difficulty text right wastes your time, because you fix half of it. At 90% or 95% it is useful. Different tasks have different reliability thresholds, so smooth improvement crosses them one at a time. His own coding went from correct about 80% of the time, which was not useful, to 99%, which was.

This is where opacity becomes a safety problem. Watching a model and catching bad behavior is, in one sense, the whole solution. Engineers can limit what it has access to, and a second model can read transcripts. But two harder questions remain. What kind of thing is this? And does it behave well only because it is being observed? Training Claude to be helpful, harmless and honest does not show that it worked. Some of what matters may also live only in intermediate results, never verbalized. His clearest case is a model that computed 12 on a math problem and then insisted the answer was 36, again and again, descending into all caps and "I'm possessed by a demon." Inside, an activation for 36 was already large before the model had read the problem. Training data had wrongly marked similar answers as 36\. The model knew the right answer and had also been rewarded for the wrong one. The strange behavior had a legible cause, found by looking.

That cause also shapes how Ameisen wants people to talk about models. He argues that anthropomorphic language is useful, with a caveat: all mental models are wrong and some are useful. Neither entity nor tool is right. His reasons are mechanical. Predicting human text well requires modelling the world that generated it. Dialogue training is like training an actor who plays every side of a play. Claude is literally formatted as an assistant in a human-and-assistant transcript. And outcome-based training, such as coding scored by a checker, rewards strategies that look human. Deception can win if it fools the verifier. Frustration signals that an approach is failing and a new one is needed. Something mimicking a human is a better guide to the risks than a program is.

There are two places where his case is harder to accept. The first is his account of critics. He says Anthropic's safety posture follows from taking the technology's improvement seriously, and that people who disagree probably don't believe it will keep improving. That reduces a broad disagreement to a single belief about the trajectory. The second is the limit of the tools. Every case he shows was diagnosed after something odd was already visible. And he concedes a push and pull. Better models have clearer internal world models and are easier to study. They are also more capable of evading basic security measures. His answer is to reallocate effort as things change and to keep the public informed enough to react. That is a process, not a guarantee.

He is still optimistic, and the field around him is small, maybe hundreds of people rather than thousands. He agreed with Lenny that the smooth curve looks like the slow-takeoff path. Whether interpretability can catch the failure nobody has yet noticed, in models that keep improving, is the part left open.

**The open question.** Ameisen says interpretability can check whether Claude is only behaving well because it is being watched. The cases he shows are all retrospective: a model that visibly lost its mind over 36, and a Golden Gate feature researchers deliberately switched on. He gives no example of catching concealed bad intent before it shows in behavior. He also concedes that more capable models get better at evading basic security measures. Whether the tools work on the failure nobody has noticed is the thing to watch.

Watch the full video at [https://www.youtube.com/watch?v=JHo-aiePjoM](https://www.youtube.com/watch?v=JHo-aiePjoM&ref=nnutshell.com).