What estimation is made of
five operations · the coordinate system
Most benchmarks treat estimation as one undifferentiated skill — ask for a number, check the number. But a Fermi estimate is built out of a small set of distinct moves, and a model can be fluent in one and helpless in another. We score the whole distribution, but we design against these five operations: they're how we build questions, how we classify them, and the axes we plot the rest of the field on.
The point of naming them is leverage. Two of the five — analogical scaling and triangulation — have essentially no estimation-native benchmark coverage anywhere. That's not a gap we noticed by accident; it's the territory this bench is built to claim.
Recall
retrieve a quantity you already know.
The answer is a fact sitting in memory: a country's population, the speed of light, a typical adult's mass. No reasoning, just lookup.
What breaksconfident wrong recall. Models state hallucinated constants with the same false precision they'd give a real one — "exactly 224" — and there's no internal signal distinguishing a remembered number from an invented one. The honest move is to recall with an interval; models rarely do.
Reference class
anchor to a known population or analogue.
You don't know the target directly, but you know something it resembles, and you adjust from there. "Roughly like a mid-size European city's annual water use."
What breakschoosing the wrong class — usually the salient one over the right one. The most available analogue is often a decoy, and resisting it is a discrimination problem, not a retrieval one.
Analogical scaling
cross from a known quantity to an unknown by a chain of ratios.
open territoryStart from something you can anchor, then multiply and divide your way over: "X is like Y, but an order of magnitude larger because it serves ten times the population, minus a factor for efficiency." The skill is in the ratios.
What breaksscaling on the wrong dimension, or compounding small ratio errors across a chain until the answer is off by orders of magnitude. No estimation-native benchmark tests this in isolation — it's open air.
Bottom-up decomposition
split the target into a product or sum of estimable pieces.
The classic Fermi move: population × adoption rate × uses per year. You replace one impossible quantity with several merely-hard ones.
What breaksa missing or double-counted factor, and — more interestingly — stopping at the wrong depth (see below). This is the operation that gives an estimate its structure, which is exactly why it's the one most worth scoring per-step later.
Triangulation
reach the same number by independent routes and reconcile them.
open territoryEstimate it bottom-up, estimate it again by reference class, and see whether the two agree. Disagreement is information; agreement is confidence earned rather than asserted.
What breaksnot triangulating at all — committing to a single path with no cross-check — or fake triangulation, where two "independent" routes secretly share an assumption and their agreement means nothing. Reconciling genuinely independent estimates is an aggregation problem, and like analogical scaling, no existing benchmark targets it.
Estimation isn't flat
depth · matching the tree to the problem
The five operations don't sit side by side — they sit at different depths of an estimate's decomposition tree. Recall is depth-zero: the answer is a single retrieved fact. Reference class is one shallow hop to an analogue. Triangulation is wide rather than deep — several short paths reconciled at the top. Bottom-up decomposition is the operation that adds depth, splitting the target into factors that may themselves need estimating, two or three levels down.
So a good estimate isn't the deepest one — it's the one whose depth matches the problem. And that's a thing models get wrong in both directions. They stop too early, answering from lazy recall when the quantity genuinely needed decomposing; or they over-decompose, spawning a deep tree of shaky guesses where one clean reference class would have been tighter. Depth isn't a virtue to maximize. Calibrating it is part of the skill — and a benchmark that only checks the final number can't see whether the model got there by matching the tree to the problem or by luck.
Where we chose to aim
the landscape, as we read it
Once you have the five operations as axes, you can ask where the existing work clusters — and use that to decide where a new bench is worth building. What follows is our reading of that landscape, meant to explain why these questions and not others, not to stand as a complete survey. Two impressions shaped the design.
Coverage seems to thin out as the operations get deeper.
The operations you can test in a single hop — recall and reference class — are the ones we've seen exercised most. The moves that involve traveling further through quantity-space — analogical scaling, and especially triangulation — are the ones we've come across least. Roughly, our read:
| Operation | How much we've seen it tested | Note |
|---|---|---|
| Recall | A lot — but rarely as estimation | Factual-QA exercises it constantly; the honest-interval version of it is the part that's thin. |
| Reference class | The most-covered corner | This is where forecasting and calibration-tournament work tends to sit. |
| Bottom-up decomposition | Some | The classic "Fermi problem" datasets live here. |
| Analogical scaling | Little we've found | — |
| Triangulation | Little we've found | Reaching a number by independent routes and reconciling them. |
The crowding sits at the shallow end; the deeper moves are where we saw room. That's where we lean.
Most of what we looked at scores a point, not a distribution.
Across the benchmarks we read, the common pattern is to take a single estimate and check it against a target — accuracy, order-of-magnitude bands, exact match. Scoring the shape of the answer, or the honesty of its stated uncertainty, is far less common; the few cases we found of it are recent. Since distributional, calibration-aware scoring is the thing this bench is built around, that's a natural place for it to be different rather than redundant.
Put the two together and it's enough to motivate the design: lean toward the operations we saw tested least, and score them in the way we saw used least.
What the older datasets taught us
building on prior work · and doubting it
We didn't start estimation from scratch. There's a line of Fermi-question datasets we built on, and we run modern models against them as transfer evals — scoring them, and reviewing where they go wrong. (That's the work behind the REALFP and SciOly reviews.)
Placeholder for the precise lineage — which datasets, what each one measures, and what it leaves out. Drop in the named, cited version here when the coverage-map notes are open.
The most useful thing those reviews gave us wasn't the scores. It was what the reviewing forced: going question by question, every answer had to be sorted into one of three buckets — the model got it wrong, the dataset's own gold answer is wrong or ambiguous, or the question itself is broken. That third sort happens more often than you'd hope.
And that's the doubt that shaped how we built our own bench:
A benchmark's ceiling is set by the quality of its ground truth, not the cleverness of its questions.
If the gold answers are shaky, you can no longer tell a model's error from a labeling error — you're measuring your own annotation noise and calling it a capability gap. A clever question with a wrong answer is worse than no question at all, because it produces confident, wrong-signed conclusions about the model.
We hold that doubt against ourselves, deliberately. Our ground truth is derived, not scraped from an answer key — and the same failure mode that bites the older datasets could just as easily bite ours. That's the whole reason we verify each truth ourselves and keep the derivation visible rather than asking you to trust a number. The point of reviewing other people's gold answers so harshly is that it keeps us honest about our own.
There's a payoff loop in it, too: when a whole class of gold answers turns out to be broken in the same way, that's not just noise to discard — it's a signal about which quantities are genuinely hard to pin down. Those are exactly the notes we mine to build the next round of questions.
A first audit of REALFP's gold
one reviewer, 556 answers — a start, not a verdict
One reviewer (@ErnestoMFaloh) read all 556 answers from a single model (gpt-5.5-2026-04-23), sorting each into model-wrong, gold-wrong, ambiguous, or fine. It points in a direction; it is not a verdict, and it wants replication by more reviewers before anyone leans on it.
The model was right on 59% of questions. Of the 224 where something was off, the gold answer was at fault in roughly 75% of them, the question's ambiguity in another 13%, and the model itself in only 12% — plus a couple of stray arithmetic slips. On this set, in other words, the benchmark is wrong about seven times more often than the model it is meant to grade.
And the gold doesn't break at random. The failures pile up in the quantitative domains with long unit chains — physics and astronomy (≈49% of golds wrong), technology (≈44%), earth and climate (≈32%) — where one botched conversion or a dropped exponent ruins the reference. Everyday-knowledge questions, like food and demographics, hold up far better. The model's own errors, by contrast, are rare and spread evenly across types.
The few real model errors are the interesting part — they aren't scattered noise. They cluster into recognizable modes: reading a question too literally ("I'm a language model, I have no hair" → zero), confusing one physical magnitude for another (energy for power), and scale errors that propagate through a decomposition. The audit sorts them into six distinct, separately-measurable capabilities — geometric inference, lexical ambiguity, global-scale reasoning, human-domain calibration, physical-magnitude confusion, and reasoning about a system's unobservable parts.
That lands on the thesis this bench is built on: a benchmark grown by accumulation against unverified gold measures resemblance to old answers, not capability. The honest alternative is smaller and deliberate — a few dozen questions, each with an independently verified truth and a concrete reason to exist. Those six failure groups are, in effect, a first draft of exactly that. The full per-question verdicts live in the REALFP review.