Two sets, one leaderboard
Open questions · sealed answers
A benchmark is only worth something if the answers aren't already on the internet — the moment a question and its ground truth go public, the next model trains on them. So the bench runs in two halves: in one the questions are open and anyone can enter; in the other the questions stay sealed and we run them for you. In both, the answers stay with us.
The public set
Open to anyone with an account. You download the questions — every prompt, every unit — run them through your own model, and upload the answers. The ground truth isn't in the download: it stays on the server, so the score is computed for you, not by you.
You explore the questions on the Questions page and watch the rankings on the Leaderboard. Because the questions are in the open, treat a public score as a practice court: a real measure of estimation, but one whose questions a model could in principle have seen.
The held-out set
Never enters the public repository — not the prompts, not the ground truth. It lives in a private store and is versioned as named cohorts (holdout-v1, holdout-v2, …) so it can rotate as models improve.
On the site it appears in exactly one place: a Holdout filter on the Leaderboard. You'll see how each model ranks on the held-out questions, but the questions themselves — and their answers — are withheld. There are no held-out entries on the Questions or Models pages, because there's nothing we can show there without leaking them.
That's the point: a held-out score is the contamination-free number. No one — no lab, no model — has seen these questions, so the ranking reflects estimation skill, not recall.
How to compete
Two tracks, matching the two sets. Pick the one that fits what you want to claim.
Public track
Self-serve, account-gated
Sign in, download the question set, and run it through your own model — any provider, any harness. Each answer is a fenced squiggle program that evaluates to a distribution, ending in a {p5, p50, p95} record.
Upload your answers and the bench scores them against ground truth it keeps sealed, placing qualifying runs on the public Leaderboard. You compete on open questions; the answers — and the scoring — stay server-side.
Held-out track
Maintainer-run, contamination-free
You can't self-score a set you can't see. Instead the maintainers run your model against the current held-out cohort and publish only its aggregate scores under the Holdout filter — the eval-server model used by serious contamination-safe benchmarks.
To enter, reach out to the maintainers with a way to call your model. We never reveal the questions; you get back a number that no amount of training on public data can inflate.
Want to strengthen the bench instead of just competing on it? Contribute a question. Good submissions are auto-graded and reviewed, and the best of them seed future held-out cohorts — which is also why a contributed question may never appear in the public set.