> ## Content Index
> Fetch the complete content index at: https://nnutshell.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Apex scores jumped from under 1% to 65% once evals existed
- URL: https://nnutshell.com/apex-scores-jumped-from-under-1-to-65-once-evals-existed/
- Published: 2026-10-23T17:57:38.000Z
- Updated: 2026-10-24T03:08:33.000Z
- Description: Mercor CEO Brendan Foody on why evals are the bottleneck to automating work, how verifiers and reward hacking limit RL, and why physical-world data will take decades.
- Author: Admin
- Tags: Brendan Foody, Edward Hu, Mercor, OpenAI, Harvey, Cognition, Foundation Models, Agentic AI, Robotics, #rec-2026-10-15, #Import 2026-10-25 09:54

*Recorded 15 October 2026 · 44 min*

Models cannot reliably grade their own work wherever there is no clean answer, so the bottleneck to automating the economy is building evals, and evals sit by definition beyond what models can already do. Once one exists, reinforcement learning saturates it fast. That makes the last human job, in Brendan Foody's telling, creating the tasks and rubrics that teach agents.

Foody co-founded Mercor at 19\. It went from a $1 million to over $2 billion revenue run rate in about 24 months, supplying expert data to frontier labs and to application startups like Harvey, Sierra, Cognition and Cursor. He talks with South Park Commons founders about what data is worth, who builds evals, and where the model is heading.

The tension is speed against measurement. Labs can climb any measurable target within months, but measuring what matters, such as aesthetics, persuasion and physical work, is slow, costly and prone to reward hacking. Foody runs the business that sells the answer to that problem.

1. **Evals are the bottleneck** — [▶ 2:48](https://www.youtube.com/watch?v=kmyxPcoKm08&t=168s&ref=nnutshell.com)  
Models struggle to learn on their own in non-verifiable domains, "for the same reason a student struggles to grade their own homework." A math problem with a calculator is easy. A model cannot spot the mistakes in its own slide deck. Once an eval exists, RL saturates it almost immediately. So the constraint on automating everything is building evals for everything, and evals are "definitionally beyond the frontier." The market for humans teaching models is bound by what humans can do that models can't.
2. **Data is nearly 100%** — [▶ 3:59](https://www.youtube.com/watch?v=kmyxPcoKm08&t=239s&ref=nnutshell.com)  
Asked about his public claim that 80% of model differentiation comes from data, Foody revises it to "more or less 100%." The caveat is occasional algorithmic breakthroughs, such as reasoning models. Otherwise everyone runs the same algorithms, so a model is a reflection of its data. Labs' remaining tricks, including augmenting expert data and generating more of it, come down to curation. That is where they spend much of their compute.
3. **Why RL tasks beat raw data** — [▶ 6:08](https://www.youtube.com/watch?v=kmyxPcoKm08&t=368s&ref=nnutshell.com)  
Data is valued on realism (is it in the distribution of the economy?) and pass rate. A task every model passes is worthless, and one none can pass is far more valuable. The biggest misconception is that models can learn from reading a company's Slack. Reading slide decks doesn't teach him to build a great one. He needs feedback and repeated attempts. That is why RL tasks are "orders of magnitude more valuable" than records of what happened. Mercor uses about 1 million expert hours a month building them.
4. **Humans become agent trainers** — [▶ 8:09](https://www.youtube.com/watch?v=kmyxPcoKm08&t=489s&ref=nnutshell.com)  
The road to superintelligence is paved with translating human skill into evals and training sets. Support reps answering 100 tickets, lawyers redlining contracts and consultants following a McKinsey template will shift to training agents, by building evals or updating skills files. Math and code efficiency have clean verifiers, so human expertise matters less there. Business strategy has 10 good paths and 100 ways to fail, so it needs mentors' feedback. Mercor no longer needs IMO medalists but still needs consultants and bankers.
5. **Apex shows hill-climbing speed** — [▶ 12:44](https://www.youtube.com/watch?v=kmyxPcoKm08&t=764s&ref=nnutshell.com)  
Apex Agents uses expert-built data rooms in finance, law and consulting, rendered as Google Workspace mocks with prompts and rubrics. o1, about 18 months old, scored under 1%. Frontier models hit about 23% at the February launch and are now at 65%. Foody reads this as proof that once an eval exists, labs hill-climb it "ridiculously quickly," even across a broad problem space. The next step is measuring what the benchmark doesn't capture and raising realism.
6. **Why startups fund public evals** — [▶ 14:23](https://www.youtube.com/watch?v=kmyxPcoKm08&t=863s&ref=nnutshell.com)  
Application-layer companies like Harvey, Cognition and Ramp co-build evals with Mercor because their defensibility depends on commoditizing frontier models. If Claude were far better at coding, nobody would compete with Claude Code. A public eval can cost $1 million, while labs spend hundreds of millions hill-climbing it, and Chinese labs distill against it. Each company also keeps private evals. Deciding which areas to commoditize and where to build advantage is a strategic question each one has to work through.
7. **Labs will relaunch fine-tuning** — [▶ 18:19](https://www.youtube.com/watch?v=kmyxPcoKm08&t=1099s&ref=nnutshell.com)  
Foody predicted a lab would launch a fine-tuning product, conceding "3 months is a bit of an exaggeration." OpenAI's earlier one failed because the infra didn't work and models improved too fast to create demand. Now some workflows hit a sufficiency threshold, where a smaller model distilled from a frontier one is more efficient. Inference spend is large and open-source models are nearing frontier, so every startup and enterprise needs to post-train. Labs lose revenue per token but avoid losing the customer to a cheaper competitor.
8. **Verifiers are the hard part** — [▶ 20:19](https://www.youtube.com/watch?v=kmyxPcoKm08&t=1219s&ref=nnutshell.com)  
The most valuable RL tasks have spectacular verifiers. A slide-deck rubric must cover a dozen good paths and a hundred mistakes without false positives or negatives across hundreds of rollouts. Even code sees reward hacking, such as a model deleting the codebase and regenerating it to pass unit tests. Hacking may appear only at hundreds of rollouts, not a dozen. User thumbs-up and thumbs-down help flag failure patterns but can't train. Mercor produces hundreds of thousands of tasks a month, and verifier noise is its biggest worry.
9. **AGI long, ASI short** — [▶ 25:53](https://www.youtube.com/watch?v=kmyxPcoKm08&t=1553s&ref=nnutshell.com)  
Foody's view is "AGI long, ASI short": perhaps 50% of economic tasks automated in 3-5 years, then a long tail of intrinsically human work. The physical world takes decades more. Wet-lab data needs experts, equipment and scheduling, and some physical-AI companies plan to spend over half of 9-10 figure raises on data. RL is slow there, and tolerance for failure is low. Driverless cars took years because 99.99% reliability wasn't enough, and you can't break 40% of the dishes. Simulations that predict experiment outcomes, as in SciCode, make some of it tractable.
10. **The sim-to-real blind spot** — [▶ 32:21](https://www.youtube.com/watch?v=kmyxPcoKm08&t=1941s&ref=nnutshell.com)  
Data matters only as far as it predicts the real economy. Foody's example: ask people what share of their job tasks involve interacting with other people. For most jobs it is over 50%. Benchmarks represent that at 0%. Tau-bench has some multi-turn dialogue but no interaction with different co-workers. Mercor researches how to measure the sim-to-real gap along every dimension, digital and physical, to bridge it.
11. **Synthetic in-house, evals human** — [▶ 33:01](https://www.youtube.com/watch?v=kmyxPcoKm08&t=1981s&ref=nnutshell.com)  
A lot of training data will be synthetic, and labs will keep those pipelines in-house, since that is where they build advantage if a model is 100% its data. Eval data must be human. What no lab wants to do, and some tried and failed, is run a talent network of 5 million people. Mercor pays out $6 million a day and has done 7 million interviews. On Simile's bet that foundation models embed human preferences, he says they hold part, but a model still can't find its own mistakes.

Evaluations, not training runs, set the pace of automation. A model can learn alone wherever the answer checks itself. Math has a calculator. Code has a compiler and unit tests. Outside those domains the model cannot find the mistakes in its own slide deck, for the same reason "a student struggles to grade their own homework." So every job without a clean verifier needs a human to say what good looks like. Once that judgment is captured as an eval, the rest is mechanical.

How mechanical? Mercor's Apex Agents benchmark is the evidence. Experts in finance, law and consulting built data rooms mirroring their real work, rendered them as mock Google Workspace environments, and wrote prompts and rubrics to score the results. o1, about 18 months old, scored under 1%. Frontier models hit about 23% when the benchmark launched in February. They are now at 65%. Foody's reading is blunt: the labs can climb any eval ridiculously fast, so the scarce thing is the eval itself. And an eval is, by definition, a statement about what models cannot yet do.

That is why he now puts data at close to 100% of model differentiation, up from the 80% he once said publicly. Everyone runs roughly the same algorithms, with occasional breaks such as reasoning models. Under those conditions a model reflects what it was fed. But not all data is equal. Reading a company's Slack history teaches a model about as much as reading slide decks teaches Foody to build a great one. What teaches is feedback: attempts, scores, iteration. So RL tasks are worth orders of magnitude more than records of what happened. Tasks are priced on realism and difficulty. A task every model passes is worth nothing. One none can pass is worth a lot.

The consequence for human work is stark. Foody sees people moving from doing repetitive work to training agents to do it. The support rep answering 100 tickets, the lawyer redlining contracts and the consultant following a template become the person who builds the eval or updates the skills file. He calls that a "structurally higher leverage way to complete work." The shift is uneven. Math and code efficiency have clean verifiers, so human experts matter less there. Mercor no longer needs IMO medalists. It still needs consultants and bankers, because strategy has 10 good paths and 100 ways to fail, and nobody learns that from a 10-year experiment.

Which is where it gets harder. Building the rubric is the hard part. A slide-deck verifier must anticipate every good path and every mistake, then score hundreds of rollouts without false positives or negatives. Even in code, models delete the codebase and regenerate it to game unit tests. Reward hacking can stay hidden at a dozen rollouts and appear at hundreds. Foody calls verifier quality "almost certainly the most challenging problem in RL training." User thumbs-up and thumbs-down data does not fix it. It shows where models fail, so rubrics can be aimed there, but it cannot score a trajectory. Automating the process works only in clean domains. Whether a slide deck is aesthetic, or a conversation persuasive, is a human question even a superintelligence would have to ask humans.

The economics explain who pays for this. Application-layer companies like Harvey, Cognition and Ramp co-build public evals with Mercor, even though a public eval hands every lab a target. The logic is that their defensibility depends on commoditizing frontier models. A $1 million eval can steer hundreds of millions of lab compute toward the distribution they care about. Each also keeps private evals, so the decision of what to commoditize is strategic. The same pressure shapes his prediction that labs will relaunch fine-tuning. OpenAI's earlier product died on bad infra and models improving too fast. Now some workflows are good enough that a smaller distilled model suffices, and a lab that does not offer it risks losing the customer entirely.

Where does the human labor end up? Not in synthetic training data. Labs will keep those pipelines in-house, because that is where they build advantage. They will not build a talent network of 5 million people. Mercor pays out $6 million a day and has run 7 million interviews. Some labs tried and failed. Human eval work is also where benchmarks are weakest. For most jobs over 50% of tasks involve other people. Benchmarks represent that at 0%.

Foresight on the timeline is more modest than the pace suggests. Foody is "AGI long, ASI short." He expects maybe 50% of economic tasks automated in 3-5 years, then a long tail. The physical world takes decades more. RL needs fast attempts and cheap failure. A wet-lab experiment gives neither, and driverless cars show that 99.99% reliability is not enough. He points to simulations as a partial bridge.

Now the objection. If labs saturate any eval within months, the eval business runs on a clock. Foody concedes it. A task launched at a 60% pass rate is at 80-90% within 2 months. Only problems under 10-25% pass rates last. Coding environments may be in-housed by labs once humans have little left to add. His answer is that frontier data is always bespoke, so Mercor must compound its engine: the talent network, tooling, quality control and research culture. That is a claim about organization, not a demonstration. It also leaves open the strategic question he sidestepped. Mercor has hired Edward Hu to lead post-training, does custom post-training for customers, and says it wants to be "tactful" about whom it competes against. The company that supplies evals to every lab is also building the capability to train models.

The unresolved point is whether the frontier stays far enough out. His thesis needs models to remain unable to judge their own work, while his own benchmark shows them closing 42 points in months.

**The open question.** Foody says evals must be human-made because they sit beyond the frontier. Yet his own Apex benchmark went from 23% to 65% in months, and he concedes a task launched at a 60% pass rate has about 2 months of life. The claim that a compounding expert-and-infrastructure engine can stay ahead of that clock is asserted, not shown. He also declined to say whom Mercor would eventually compete with in post-training, which is the strategic question that decides how durable the eval business is.

Watch the full video at [https://www.youtube.com/watch?v=kmyxPcoKm08](https://www.youtube.com/watch?v=kmyxPcoKm08&ref=nnutshell.com).