> ## Content Index
> Fetch the complete content index at: https://nnutshell.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Modal's CEO: inference tokens commoditize, platform breadth endures
- URL: https://nnutshell.com/modals-ceo-inference-tokens-commoditize-platform-breadth-endures/
- Published: 2026-10-29T15:06:26.000Z
- Updated: 2026-10-29T15:06:26.000Z
- Description: Eric, founder and CEO of Modal, argues that inference serving is heading toward commodity pricing while durable advantage lies in multi-product infrastructure, even as compute stays chaotic and capital-hungry.
- Author: Admin
- Tags: Eric, Elon Musk, Modal, Nvidia, Anthropic, Cognition, Inference, NeoClouds, AI Infrastructure, Developer Tools, #rec-2026-10-28, New

*Recorded 28 October 2026 · 59 min*

Eric, founder and CEO of Modal, argues that serving inference tokens is a race to the bottom, and that durable advantage comes from a portfolio of infrastructure products (sandboxes, inference, training, batch) built on hard proprietary foundations. The claim carries weight because Modal, per the host, has crossed a $1 billion run rate roughly three years after a deck showing about $2 million.

The tension is between a market that rewards speed and one that lacks structure. Compute is sourced through WhatsApp groups and hotel-lobby introductions, neoclouds run on heavy leverage, and the agents that now write and consume software keep shifting the ground. Eric concedes he cannot name a next bet with sandboxes-scale potential, and the durability of a platform moat is argued more than evidenced.

1. **Compute is not fungible** — [▶ 2:18](https://www.youtube.com/watch?v=HJDekXH-h5c&t=138s&ref=nnutshell.com)  
Compute is sourced through late-night WhatsApp groups and personal introductions. Eric compared Nvidia's GTC to an open-air drug market for GPUs. Capacity exists if buyers accept higher prices and longer terms, and Modal closes two to three compute deals a week across 25 clouds. He doubts an exchange will work because location, disk, networking, CPU, memory and reliability all vary. Aggregation lets Modal pull failing capacity, including from former crypto miners running neoclouds, out of rotation automatically. Eric does not fully trust public GPU pricing, which diverges from what Modal pays.
2. **A portfolio assembled opportunistically** — [▶ 5:53](https://www.youtube.com/watch?v=HJDekXH-h5c&t=353s&ref=nnutshell.com)  
Modal began in 2021 as a serverless function product, and Stable Diffusion drove its first year of growth. Sandboxes launched in 2023, blew up in 2025 as models got good at writing code, grew almost 1,000x, and now account for about a third of revenue, a $300 million run rate on their own. LLM inference stayed on the sidelines until the DeepSeek moment in early 2025 and is also about a third of revenue. Modal has since launched dedicated and shared inference endpoints with token pricing, plus a training product.
3. **Hard infrastructure as moat** — [▶ 10:24](https://www.youtube.com/watch?v=HJDekXH-h5c&t=624s&ref=nnutshell.com)  
Eric calls refusing to avoid hard things a major source of alpha. Modal built its own file system, container runtime and scheduler, using neither Kubernetes nor Docker, with gVisor for isolation and KVM for some tasks. The payoff is fast container starts, CPU and GPU snapshotting that loads large models in seconds, and an RDMA peer-to-peer system built over the last six months. Eric says the result is very hard to replicate. That is his assessment; no competitive comparison was offered.
4. **Memory decides the chip** — [▶ 16:29](https://www.youtube.com/watch?v=HJDekXH-h5c&t=989s&ref=nnutshell.com)  
Nvidia remains king, and frontier LLMs need a full node, such as eight B300s. But voice, transcription, speech synthesis and bio workloads often run more cost-efficiently on Hoppers, A100s, L40S or RTX Pro 6000 cards, and the H100 is still Modal's most common GPU. Because workloads are memory-constrained, the usual heuristic is the cheapest GPU that fits the model. Modal wants to automate chip switching within a fixed dollar-per-token price, which would raise utilization. Eric expects models to keep growing with available memory.
5. **Tokens commoditize, platforms persist** — [▶ 21:49](https://www.youtube.com/watch?v=HJDekXH-h5c&t=1309s&ref=nnutshell.com)  
Eric expects plain token serving to become a race to the bottom, citing OpenRouter's commoditization of providers. Kernel and KV-cache optimizations still matter because so much money rides on them, but they are less durable because the tricks spread and get copied. His durable-advantage candidate is breadth: sandboxes, reinforcement learning, fine-tuning, batch and inference together, since customers do not want single-trick ponies. Cognition illustrates the pattern, using Modal for both inference and sandboxes for reinforcement learning rollouts. Modal also contributes heavily to vLLM and SGLang.
6. **Agents write code, seniors lead** — [▶ 27:24](https://www.youtube.com/watch?v=HJDekXH-h5c&t=1644s&ref=nnutshell.com)  
Eric estimates 95% of merged lines at Modal are written by agents to some extent, yet infrastructure's tail risks around reliability and scaling still require human supervision. Adoption surprised him: senior engineers use agents most, treating them like 10 interns. In one example, an agent given a POSIX test suite wrote a file system in about 3 days. Pressure is building on CI/CD, and he anticipates limits at 10x current commit volume. He expects engineer demand to keep rising, with new graduates facing a harder short term.
7. **Optimizing software for agents** — [▶ 30:48](https://www.youtube.com/watch?v=HJDekXH-h5c&t=1848s&ref=nnutshell.com)  
Agents want what developers want: fast feedback loops, clear errors, good documentation. The difference is deprecation. Because agents were trained on old code, removing features is harder, and Modal has lengthened its deprecation windows. It runs pipelines where LLMs continuously write Modal code to catch failures, and it tracks which tool Claude Code recommends for inference tasks, which Eric likens to search engine optimization. He sees growing incumbency: being in training data creates staying power, as with developers defaulting to React, which gives recommended tools a flywheel.
8. **Cheap asymmetric bets** — [▶ 33:54](https://www.youtube.com/watch?v=HJDekXH-h5c&t=2034s&ref=nnutshell.com)  
A sandboxes prototype took days, which Eric calls a low-downside, high-upside bet; after the 2025 surge Modal rewrote the system five times for scale. He cannot name an equivalent today. Larger, riskier bets include low-latency voice and real-time video infrastructure, a CDN for inference, mid-size model training and reinforcement learning, and VMs for Mac, Windows, Android and iPhone. On agent tooling he is cautious, since architectures evolve quickly and RAG would have been a bad bet two years ago. He favors durable infrastructure blocks over fashionable abstractions.
9. **Neoclouds are financing vehicles** — [▶ 43:12](https://www.youtube.com/watch?v=HJDekXH-h5c&t=2592s&ref=nnutshell.com)  
Eric calls neoclouds fine but undifferentiated businesses: capital-intensive lessors of hard assets whose high leverage makes equity value volatile. Residual GPU value has been higher than expected, which helps. He sides partly with Elon Musk on cost of capital, noting AWS, Azure and GCP are attached to cash-rich public companies. For neolabs, GPU commitments typically require about 30% upfront, so a $1 billion raise can secure roughly $3 billion of GPUs. That creates implicit leverage dependent on later fundraising, and some may pivot to reselling capacity.
10. **Post-training must become product** — [▶ 48:35](https://www.youtube.com/watch?v=HJDekXH-h5c&t=2915s&ref=nnutshell.com)  
Eric rejects the idea that every company will build its own model. Weighted by dollar revenue, he expects most models to be post-trained, but by company count most firms will not need it; large token consumers such as Cognition and Cursor do. Reinforcement learning as a service can lower cost and raise throughput, but parts of it resemble a service business. The central tension, he says, is whether post-training can be turned into a product that sells without bespoke work, which is what Modal is trying to build.
11. **Why GPU prices rise** — [▶ 55:56](https://www.youtube.com/watch?v=HJDekXH-h5c&t=3356s&ref=nnutshell.com)  
Eric reversed his GPU price forecast within 48 hours. After having an agent summarize Anthropic's S1, he concluded the company is far more profitable than he expected. His earlier view was that public-market discipline would slow training spend. Instead he sees no incentive for Anthropic to slow capacity acquisition, and assumes OpenAI faces similar dynamics. With the two being the largest buyers, he predicts GPU rental prices will be higher in 12 months. The inference rests on his reading of a summary, not on stated financials.

Serving tokens is a business Eric expects to be squeezed. The founder of Modal sees the straight-down-the-middle inference market as a race to the bottom, and he points to OpenRouter, which he says is already commoditizing the underlying providers. Better kernels and more efficient KV caching will matter for years, he says, because there is so much money in them. But those tricks spread and get copied. His alternative is breadth: one platform spanning sandboxes, inference, reinforcement learning, fine-tuning, batch processing and, more recently, training. Customers, in his phrase, do not want "single trick ponies." It is a coherent thesis, and Modal's own revenue mix gives it some support.

That mix was assembled less by plan than by positioning. Modal began in 2021 as a serverless function product, found its first wave of growth in Stable Diffusion hosting, and launched sandboxes in 2023 because Eric saw people using LLMs to generate code that would need isolated places to run. The bet was cheap: a prototype took days. When models got good at writing code in 2025, sandboxes grew almost 1,000x, and they now account for about a third of revenue, a $300 million run rate on their own. LLM inference is another third. Eric calls sandboxes a low-downside, high-upside bet, and admits he has no equivalent to point to today.

Beneath the products sits a market with little structure. Eric describes compute procurement as late-night WhatsApp groups and introductions through a sales executive's cousin, and he compares Nvidia's GTC to an open-air drug market for GPUs. Modal strikes two to three compute deals a week across 25 clouds. He does not expect an exchange to work, because compute varies in location, networking, disk, memory and reliability. That disorder is the product opportunity. Aggregating demand lets Modal pull failing neocloud capacity out of rotation and file tickets automatically, so customers do not have to think about capacity at all.

The second pillar is technical depth. Modal built its own file system, container runtime and scheduler, avoiding Kubernetes and Docker, and uses gVisor and KVM for isolation. Eric says the investment now pays off in fast starts, CPU and GPU snapshotting that loads large models in seconds, and an RDMA peer-to-peer system for moving data between nodes. He believes this is very hard to replicate, though he offers no competitive benchmark. It also explains a strategic choice: because Modal's advantage lies in the infrastructure layer, it contributes heavily to vLLM and SGLang rather than guarding its own engine. An editorial inference follows: commoditized engines are an input to Modal's moat, not a threat to it.

Hardware and model trends shape that infrastructure. Nvidia remains king, and frontier LLMs need a full node of eight B300s. But voice, transcription and bio workloads often run more cheaply on Hoppers, A100s and L40S or RTX Pro 6000 cards, and the H100 is still Modal's most common GPU. Because workloads are mostly memory-bound, the rule is to choose the cheapest GPU that fits. Eric wants Modal to automate that choice, switching chips beneath a fixed dollar-per-token price to raise utilization. He also expects models to keep growing, since labs train to fit available memory, which keeps a heterogeneous fleet in demand.

Agents appear in the argument twice, as builders and as customers. Eric estimates 95% of merged code at Modal is agent-written to some extent, yet infrastructure's tail risks still require human supervision. Senior engineers, not juniors, have adopted agents most aggressively, using them as 10 interns. As consumers, agents want what developers want, with one wrinkle: training data preserves deprecated APIs, so Modal has lengthened its deprecation windows and tracks which tools coding agents recommend, much as companies once tracked search rankings. Eric sees this producing incumbency, since software that enters training data gains staying power. The same force could cut against Modal's expansion into new categories.

The capital side of the market is where Eric is most openly skeptical. Neoclouds, he says, are fine businesses but financing vehicles that lease hard assets with high leverage, so their equity swings with GPU prices. He agrees partly with Elon Musk's point about cost of capital, noting the three major hyperscalers are tied to cash-rich public companies. For neolabs, GPU commitments typically require about 30% upfront, so a $1 billion raise can secure roughly $3 billion of hardware. That leaves them dependent on later fundraising. If it dries up, he expects some to pivot toward neocloud-style resale. He remains bullish on AI overall, but not on every lab becoming a trillion-dollar company.

His price forecast shows how quickly these judgments move. Eric expects GPU rental prices to be higher in 12 months, a view he adopted within 48 hours after having an agent summarize Anthropic's S1\. He concluded Anthropic is profitable to a degree he had not anticipated, and therefore has no reason to slow capacity purchases. He assumes OpenAI is in a similar position, and the two are the largest buyers. The reasoning is plausible but thin: it rests on one summarized document and on demand concentrated in two buyers. Eric's own reversal suggests how fragile such forecasts are, and his admission that few people can see three years out applies here too.

The breadth thesis also has limits that Eric acknowledges. Modal has about 150 engineers, and he calls prioritization the CEO's central task, sometimes pausing projects people are excited about. He worries post-training could turn into a service business unless it can be productized. Agent architectures, he says, are changing too fast to predict, which is why he favors durable primitives over fashionable abstractions, citing RAG as a bet that would have aged badly. What the conversation does not supply is evidence that breadth itself holds customers: no retention, margin or cross-sell figures appear. The argument rests on the pattern of Modal's growth, which is impressive but also consistent with riding a favorable wave.

The unresolved issue sits at the junction of his two themes. Modal's pitch is that customers need not think about capacity, because it absorbs higher prices and longer terms and manages supply in aggregate. Eric simultaneously predicts that capacity gets more expensive as profitable giants keep buying. If token serving is a race to the bottom and the input is getting costlier, margin must come from the platform layer, from the products customers use across the stack. Whether those products command enough premium to cover rising supply costs is the question Modal's strategy depends on, and the conversation leaves it open.

**The open question.** Modal's pitch is that customers should not think about capacity, because it absorbs higher prices and longer terms on their behalf. Yet Eric now forecasts rising GPU rental prices over the next 12 months, driven by profitable buyers with no reason to slow purchasing. Can a platform pass those costs through to customers while competing against token providers on price? And does breadth stay a moat if agent architectures keep resetting, as RAG did?

Watch the full video at [https://www.youtube.com/watch?v=HJDekXH-h5c](https://www.youtube.com/watch?v=HJDekXH-h5c&ref=nnutshell.com).