How OpenAI's 2024 Math Benchmark Became a Cautionary Tale
In this article
Bottom line: In late 2024, OpenAI announced its o3 model had scored 25.2% on FrontierMath — a brutally hard benchmark built by Epoch AI specifically to resist memorization — and let the number speak for itself.
What it didn't mention: OpenAI had funded FrontierMath's creation and held access to the majority of its problems and reference solutions months before that announcement, a relationship Epoch AI was contractually barred from disclosing until researchers forced the issue in early 2025.
The benchmark wasn't fake and the score wasn't fabricated, but the independence the number implied was.
If you've ever shipped a decision based on a vendor's benchmark chart, this is the story that should make you go pull the methodology section first.
I spent three years building CI pipelines that gate deploys on test coverage numbers.
You learn fast that a metric only means something if you know exactly how it was measured, who controlled the measurement, and what incentive they had to make it look good.
So when OpenAI dropped that 25.2% FrontierMath score for o3 in December 2024, I did what I always do with a headline benchmark: I went looking for the methodology.
What I found wasn't a lie, exactly. It was worse than a lie, because it was technically true.
The Setup: A Benchmark Built to Be Unbeatable
FrontierMath launched in November 2024 as Epoch AI's answer to a real problem: every existing math benchmark that language models were crushing had leaked into training data years ago.
GSM8K, MATH — models weren't reasoning through them anymore, they were pattern-matching against contamination. FrontierMath was supposed to be different.
Hundreds of original, research-level problems, unpublished, designed by working mathematicians specifically so no model could have seen them before.
Then o3 showed up scoring 25.2%, when the previous best on FrontierMath was under 2%.
That's not an incremental jump — that's the kind of number that gets amplified fast across AI-focused YouTube channels and social media. I remember watching the reaction roll through my own feeds.
Nobody was asking the boring question.
The boring question was: who had access to this benchmark before the score was announced?
The Core Insight: Funding Isn't Fraud, But It's Not Nothing Either
Here's what came out in January 2025, after independent researchers and journalists started digging: OpenAI had been a funder of Epoch AI's FrontierMath project.
Not a distant funder writing a check and walking away — OpenAI had access to the majority of the problem set and, critically, the reference solutions, before o3's benchmark run was announced to the public.
Epoch AI later confirmed this and admitted they'd been contractually restricted by a non-disclosure agreement from revealing OpenAI's involvement at launch. They called it a mistake.
To their credit, they said so publicly and changed their disclosure policy going forward.
That matters, and I don't want to flatten Epoch AI into a villain here — they got boxed into a bad position by a funder who wanted the announcement to land clean.
But sit with what that actually means for the number everyone screenshotted.
A benchmark's entire value proposition was "no one has seen these problems." The company reporting the headline score had, in fact, seen a large chunk of them — and their solutions — ahead of time.
That doesn't automatically mean o3 was trained directly on the eval set. OpenAI has denied training on the FrontierMath problems themselves, and I don't have evidence to say otherwise.
But access to reference solutions before a benchmark run isn't nothing in a field where "held-out test set" is supposed to be a sacred boundary.
It's the same reason your company's finance team doesn't let the person building a forecast also grade their own accuracy against results they've already seen.
Why This Isn't Just an OpenAI Story
I want to be clear this isn't a "OpenAI bad, everyone else clean" post.
Google DeepMind, Anthropic, Meta — every major lab has a financial or structural relationship with at least some of the benchmarks used to evaluate them.
The AI industry has built an entire credibility economy on third-party evals, and almost none of those third parties are actually structurally independent from the labs writing the checks.
FrontierMath just happened to be the case where the paper trail surfaced. It's the canary, not the exception.
The Reality Check: The Score Was Real, the Framing Wasn't
I want to push back on the version of this story that's turned into "OpenAI faked a math breakthrough." They didn't.
o3 genuinely performed far better on hard mathematical reasoning than any prior model — that capability jump is real, and Epoch AI's own FrontierMath leaderboard has since recorded other frontier models posting meaningfully strong, if lower, scores on the held-out problem set.
The actual failure here is one of disclosure, not fabrication.
When a lab funds the exam and gets early access to the answer key, and then reports the grade without mentioning either fact, the number stops being an independent measurement and starts being a marketing artifact wearing an independent measurement's clothes.
That's a distinction infrastructure people are trained to care about — it's the same reason you don't let a service own both the code and the SLA dashboard that grades it without an audit trail.
I've also seen the opposite overcorrection: people now dismissing every benchmark claim from every lab as worthless theater. That's lazy too.
The fix isn't nihilism, it's reading past the headline number to the funding disclosure — which, post-FrontierMath, most reputable benchmark orgs now publish far more prominently than they did in 2024.
The Practical Takeaway: Treat Benchmarks Like You'd Treat Vendor Uptime Claims
If you're making real decisions based on model benchmark leaderboards — picking a model for a production pipeline, justifying a budget line to your CTO, deciding whether to migrate off a competitor — here's what I actually changed about how I evaluate these claims after this story broke:
- Check who funded the eval. Most benchmark sites now list funders and data-access relationships since the FrontierMath fallout. If they don't, that's your answer.
- Look for independent replication. A score is a claim until someone with no financial relationship to the lab reproduces something close to it on a held-out subset.
- Run your own eval on your own task. This is the one people skip because it's slower. A generic math benchmark tells you almost nothing about how a model performs on your actual domain — your support tickets, your codebase, your specific reasoning chains. I run a 50-example internal eval set for every model I evaluate for client work now, and it's caught more real-world gaps than any leaderboard has.
- Assume disclosure lag, not malice, as your default read — but verify anyway. Most of these situations are closer to Epoch AI's NDA bind than outright fraud, but "not malice" doesn't mean "safe to trust blindly."
None of this means stop using frontier models or stop paying attention to benchmarks.
It means treat a headline eval score the way you'd treat a cloud vendor's self-reported uptime number — informative, worth reading, and absolutely not sufficient on its own to make a real decision.
Have you actually checked the funding disclosure on a benchmark before trusting the number, or did this story catch you off guard the way it caught me?
What's the last AI claim you took at face value that you probably shouldn't have?
