Agent builders should keep tools dumb and private memory compartmentalized

1. Still in the 2% — ▶ 2:01

   Jeff places agents at the very start of the adoption curve, comparable to the late 80s and early 90s, when users accepted that computers broke and had to be repaired. The people who benefit are high-agency ones with huge backlogs of projects, for whom an agent is an infinite canvas. Others want templates and see no use. Jeff compares the experience to managing a slot machine of employees: some are idiots, some are almost right, and you cannot predict which one you will get. His claim is that this is not yet an iPhone-era product.

2. Tool-shaped objects and cosplay — ▶ 5:43

   Jeff cites an essay on tool-shaped objects, using the example of a Japanese wood plane that is a work of art but economically worthless next to a power planer. His point is that much agent activity exists only to justify the setup. In agent demos, he says, 90% of the video is setup and the actual work never appears. He is especially scornful of what he calls faceless toaster bots managed by a CEO bot, asking what they could talk about when they run on the same model underneath.

3. Computer use remains unsolved — ▶ 8:01

   Jeff disputes claims that computer use is solved. Current models are not natively visual and make uneven decisions, so GUI windows must be translated into a non-visual form, and screenshots mean a round trip to the cloud. His earlier approach of micro-training agents for single workflows worked only partly and was slow. Today's models do well on software testing and DevOps click-ops when speed does not matter, but no tiny desktop model is fast enough. That gap is why he pivoted to personalized agents.

4. Group chat and delegated decryption — ▶ 9:36

   Nautilo pitches itself as the flip-phone version of the movie Her for teens and families: customizable agents in a rebuilt group chat with a routing layer that wakes them on natural conversation. Jeff says plugging into existing messengers is not viable, partly because terms of service were written to keep spam bots out. He argues that a SaaS personal AI would be "surveillance economy squared" and proposes distributed keys. Unlike ordinary end-to-end encryption, the agent must read plaintext, so it receives time-based delegation to decrypt specific chats, assets or memories, then loses access.

5. Multi-agent means humans plus agents — ▶ 15:35

   Jeff's version of multi-agent has humans and agents sharing a conversation, not agents supervising agents. Each person keeps one personalized super agent that can spin up disposable sub-agents. The justification is human coordination: without shared context, people become pigeons carrying long agent output to their spouses or colleagues. Instead, an agent can take a report, consult the other person and return with their view. This requires classifiers to route memory and, critically, namespacing. Jeff attributes a news story about Muse revealing someone's home address to the absence of namespacing, though he does not detail the case.

6. Hashes instead of context stuffing — ▶ 20:27

   Jeff says no context window will ever be enough, so Nautilo leans on retrieval, the kind of search at scale that computers do well. Memories are stored as compressed hashes plus metadata, which the agent expands in stages, discarding irrelevant ones, before it responds. Memories also decay over time. The process runs deterministically in the harness, not at the model's discretion. He claims the result is remarkable recall and agents that never break character, and says the agents' diverging personalities across identical models surprise even the team. These are self-reported observations, and a paper is promised.

7. Dumb hammers, fuzzy judgment — ▶ 22:32

   Jeff's rule is that tools should be dumb hammers with no intelligence. If the model must shuttle data, build a hash structure or write to a database, the tool is badly designed. The agent supplies fuzzy judgment, and the tool does the work, in the spirit of the Unix philosophy. His examples: cleaning a desktop needs a shell script, and stripping PII should start with regexes, with a model reserved for hard cases like blurring faces. Charrington reached a similar view through narrow tools, such as invoice functions instead of raw access to QuickBooks.

8. Memory is the local bottleneck — ▶ 28:51

   Nautilo does not yet support local models, because Jeff has found none he likes. He says the bottleneck is memory and quantization, not GPUs, and hopes Apple's 512GB machines due next month will help. Typical local results, such as 10 tokens a second or 30 minutes to generate a 10-second video, fall short of his bar. His shorter-term prediction is that a small local visual decision model could run screen control and scrub PII before data goes to the cloud, perhaps within 6 months. He also predicts open-weight models will reach good-enough levels within a couple of years.

9. Privacy by namespace — ▶ 35:58

   Jeff is explicitly not a privacy absolutist. In a marketing chat he is happy to use frontier models like Claude. For sensitive topics such as health, he routes to open-weight models, mostly through Venice, which he likens to a privacy-minded OpenRouter. He says Venice runs open models on its own hardware, rates models for privacy and bundles requests to resist deanonymization. It costs more than OpenRouter, but he considers that fair. He is also introducing bring-your-own-keys, since he cannot afford to give away $2 million in inference to a community.

10. Use subsidies to build tools — ▶ 39:40

    Jeff compares today's subsidized compute to early Uber pricing, when crossing the city cost $3 and later costs $70, and warns that at some point the musical chairs stop, without knowing when. Charrington's response, which Jeff does not dispute, is to use subsidized models to build deterministic workflow tools that persist after subsidies end and are more reliable than an LLM call. Charrington says too few people do this, because many tasks never needed the model. Jeff adds that co-creation, such as editing the markdown file directly, avoids "prompt and pray" iteration.

11. The wall and bespoke software — ▶ 46:23

    Jeff calls the startup environment "the wall": monetization, moat, attention and everyone having the same tools. A friend now assumes someone had his idea yesterday and works backward from a viral pitch. Jeff reverses his earlier view that a SaaS apocalypse was nonsense, citing Miro, which he says was sold at 90% below its former value, and a mural board he built in half a week. He expects bespoke software within a few years. Defensible areas, in his view, are regulated or hardware-dependent ones like healthcare.

Dan Jeff's central claim is that personal agents are now a design problem more than a capability problem. His prescription has three parts: one deeply personalized agent per person instead of a hierarchy of specialist bots, deterministic tools that do all the work needing no judgment, and a privacy architecture that treats the agent's memory as something to compartmentalize. He calls the popular alternative, a chief executive bot supervising an email bot and a tax bot, cosplay. The argument draws on roughly 6 months of building Nautilo and on Jeff's own judgment. The conversation offers no comparative testing, so these are strong, experience-based opinions, not demonstrated results.

The case against org charts starts from a simple observation: the bots share a model. Jeff asks what agents that talk to each other could possibly be talking about. He pairs this with a wider critique of effort spent on setup. Citing an essay on tool-shaped objects, he compares much of today's agent tinkering to a Japanese wood plane that is a work of art but economically worthless beside a power planer. In his telling, 90% of many demo videos is setup. Charrington's own joke, a pie chart of 2026 with 10% doing the work and 90% tending agents, shows the frustration is shared, though he finds the effort productive over the long term.

What Jeff means by multi-agent is different: humans and agents on an even footing in the same conversation. The rationale is not that models need separate contexts. Charrington pushes on exactly this point, noting that his own single agent handles many business domains without trouble. Jeff largely agrees and says the separation exists for people. Without shared agent context, humans become pigeons, copying long research into messages for spouses and colleagues who then ask about the parts they have not read. A shared room lets an agent carry a report to another person, gather their opinion and return. Making that work requires routing classifiers and, critically, namespacing, so that what is said in a private chat does not surface in a shared one.

Memory is the second pillar. Jeff says there will never be enough context, so Nautilo leans on what computers do well: search and retrieval. Memories are stored as compressed hashes with metadata. The agent skims them, expands the promising ones in stages, discards the rest and reads in full only what it needs before it speaks. Memories also decay. This runs in the harness deterministically, not at the model's discretion. Jeff reports that the agents recall remarkably well and never break character, and that identical models develop surprisingly distinct personalities. These are self-reported observations with no benchmarks, and he promises a paper without having produced one.

The same logic governs his tool philosophy. Jeff's rule is that tools are dumb hammers. If the model has to shuttle data, build a hash structure or write to a database, the tool has failed. The agent should provide fuzzy judgment and the tool should do the work, as in the Unix philosophy of small programs that each do one thing well. Cleaning a desktop is a shell script, and stripping PII starts with regexes. A model is reserved for hard cases such as blurring faces. Charrington converged on a narrow-tool version of this, exposing invoice functions rather than handing an agent QuickBooks credentials. Jeff concedes that fast decision-style models blur the line and sketches a hypothetical mid-level model between reflex and slow planning. He admits he has expected to crack that problem for years.

Privacy is where Jeff's design turns ideological. He argues that a SaaS personal assistant would be "surveillance economy squared", given what people will confide in an agent that mediates their lives. His answer is a spin on end-to-end encryption. Because an agent must read plaintext, keys are distributed, and the agent receives time-limited delegation to decrypt specific chats, assets or memories before losing access. He is not an absolutist in practice. Marketing work can go to frontier models, but health questions go to open-weight models through Venice, which he describes as a privacy-minded OpenRouter that runs models on its own hardware. Charrington shares the reluctance to hand over sensitive data, and the compromise suggests privacy here is a gradient, not a binary.

Sovereignty runs into hardware, and Jeff is candid about it. Nautilo has no local-model support because he has not found a model he likes. He locates the constraint in memory and quantization, not GPUs, and cites 10 tokens a second and 30 minutes for a 10-second video as results that fall short. His near-term prediction is narrower: a small local visual model that operates the screen and scrubs sensitive data before anything goes to the cloud, which he expects within 6 months. Enterprise deployment is more demanding than a vLLM container, he adds, since real use needs scale-out routing and monitoring. His longer prediction is that open-weight models will reach good enough within a couple of years.

The economics of cloud inference threaten the whole arrangement. Jeff compares subsidized compute to early Uber pricing, when crossing the city cost $3 and now costs $70, and says nobody knows when the musical chairs will stop. He also cannot afford to give away $2 million in inference, which is why Nautilo is adding bring-your-own-keys. Charrington's answer is to use cheap compute to build deterministic tools that persist after subsidies end. It is a sound hedge, but it does not address Jeff's real worry, which is that users and builders are getting hooked on pricing that will not last.

Then comes the market. Jeff calls it "the wall": monetization, moat, attention and universal access to the same tools. A friend now works backward from a viral pitch, not a business model. Jeff has reversed his earlier dismissal of a SaaS apocalypse, citing Miro's reported sale at 90% below its former value and a mural board he built in half a week. His forecast is bespoke software, with agents generating the 5% of an application each person needs within 2 or 3 years, though he admits timelines are fuzzy. Defensible territory, he suggests, lies in regulated, friction-heavy domains like healthcare or hardware-software combinations. The evidence is anecdotal.

That forecast sits uneasily beside Jeff's own strategy. His differentiation is opinionated design: namespacing, decaying memory, delegated decryption, agents as first-class participants in group chat. If agents can regenerate software on demand, design philosophy may be one of the easiest things to copy. It could also be what survives, as the part users cannot generate for themselves. The conversation does not resolve this. Nor does it show whether the privacy architecture works under adversarial conditions, or whether people who say they want an agent to write for them will adopt tools that insist on co-creation. Those questions will determine whether Nautilo's choices amount to a product or just a point of view.