An estimation & calibration benchmark for language models
Open-ended questions with verifiable numeric answers. Models reply with a full probability distribution — not a single guess — and we score how close that distribution lands to a ground truth we derive ourselves, and whether its uncertainty is honest.
What estimation is made of
Most benchmarks treat estimation as one undifferentiated skill — ask for a number, check the number. But a Fermi estimate is built out of a small set of distinct moves, and a model can be fluent in one and helpless in another. We score the whole distribution, but we design against these five operations: they're how we build questions, how we classify them, and the axes we plot the rest of the field on.
How we score it
one score · two diagnosticsCramér-log
We compare the model's whole predictive distribution against a verified ground-truth distribution in log space; lower is a tighter match. It rewards being right about the shape, not just the midpoint — and it's the one metric the leaderboard ranks on.
Cramér-log is a Cramér–von Mises distance: the integrated square of the gap between the model's cumulative distribution and the truth's, taken in log space. Log space because Fermi answers span many orders of magnitude; the whole-CDF comparison because it's sensitive to both where the mass sits and how wide it is — and it stays finite even when a guess barely overlaps the truth, where KL blows up. We compute CRPS-log and KL as well, but rank on Cramér.
Bias & sharpness
Whether the distribution sits above or below the truth, and whether it is appropriately wide rather than arbitrarily confident — sharpness measured against the reference, not in a vacuum.
Coverage
How often the truth actually falls inside the model's stated 90% interval. Across many questions an honest model should hit about 90% — here they tend to fall well short.
The Standings
shared 19-question core · 6 models · lower Cramér is better| # | Model | Mean Cramér | Coverage @90 |
|---|---|---|---|
| 1 | GPT-5.1 OpenAI | 0.313 | 68% |
| 2 | Gemini 2.5 Pro Google DeepMind | 0.316 | 68% |
| 3 | GPT-5.1 OpenAI | 0.323 | 68% |
| 4 | Claude 4.5 Sonnet Anthropic | 0.411 | 58% |
| 5 | DeepSeek R1 DeepSeek | 0.593 | 16% |
| 6 | Llama 3.3 70B Meta | 2.817 | 16% |
Apples-to-apples over the largest set of questions these models all answered. Cramér-log is the ranking metric (lower is a tighter match to the verified ground-truth distribution); coverage @90 should sit near 90% for an honest model. On the full bench — every question — only two models reach complete coverage; see the full leaderboard.
Are the models honest?
coverage @90 vs. the 90% targetBar = share of 90% intervals that contained the truth; the ink line marks the 90% target a calibrated model would hit. Every model falls short.
Every model's interval is too narrow.
Each model states a 90% interval; an honest one should contain the truth about 90% of the time. The bars show how often it actually does — and none reach the line.
The frontier models come closest, around 70%, while the weakest sit near 15% — badly overconfident. Every bar falls short of 90%, and that shortfall is exactly the over-narrow uncertainty the Cramér-log score penalizes.
A question, drawn out
How many Three Gorges Dams' worth of concrete does China produce per year?
Ground truth — 225 count (dams equivalent) · verified
From the set
Have a better question?
Estimation questions with a checkable answer are always welcome. Submit one with the question and a source for the truth.
Contribute a question →Why we score calibration, not just accuracy
A model that says “about two hundred, give or take a few times” and is right is worth more than one that says “exactly 224” and is certain — but wrong. Rewarding honesty about uncertainty is the whole point.
Most benchmarks ask a model for a single number and check whether it is close. That measures accuracy but hides what estimation is really about: knowing what you don't know. A model that is wrong but said so — wide interval, low confidence — has behaved well. One that is wrong while certain has behaved badly. Accuracy alone can't tell them apart.
So every question here is answered with a full probability distribution, written in a small modelling DSL. We score that distribution against a ground truth we derive and verify ourselves, using a Cramér-style distance in log space — and we separately check whether the truth lands inside the model's stated 90% interval.
The pattern that falls out is overconfidence, and it is near-universal: every model's stated 90% interval contains the truth far less than 90% of the time — some only a little short, others wildly so. The intervals are simply too narrow. That gap between claimed and real confidence, not raw accuracy, is what this benchmark is built to expose.
The full leaderboard
every model · Cramér-logQuestion setBrowse every question
prompts, truth & per-run detailModelsEvery model evaluated
per-model scores & detailHelp us with reviews & annotations
Alongside the main bench we run transfer evals against existing Fermi datasets — scoring modern models and reviewing where they (or the datasets' gold answers) go wrong. The notes from those reviews are exactly what we mine to build better questions. If you'd like to help review and annotate, dive in: