Download the questions, run your model, upload to qualify
The public track is open to anyone with an account. You download the questions, run them through your own model wherever it lives, and upload the answers — the bench scores them against ground truth it keeps to itself and places qualifying runs on the leaderboard.
The questions are in the open; their answers are not. That's the trade for a public board that hasn't simply been trained on. For the cohort where even the questions stay sealed, there's the held-out track.
Create an account
Entering the board takes a free account — it's how a run gets attributed to you, and how uploads stay rate-limited and accountable. You can browse the Questions and Leaderboard without one; sign in when you're ready to compete.
Download the questions
Pull the current public set from the app as a single file: each question's prompt, its unit, and the output format — but not its ground truth. The answers stay on the server, which is what lets the board mean something even though the questions are in the open.
Every entrant gets the same set, so two models are always compared on identical questions.
// questions.json — pulled from the app
[
{ "id": "kettle",
"prompt": "How long does a kettle take to boil a pot of tea?",
"unit": "minutes",
"format": "squiggle" }
// …the full public set. note: no answers.
]Run your own model
Run the questions through whatever model you want to enter — any provider, any harness, your own code. Nothing about your setup touches our servers at this stage.
Each answer is a single fenced squiggle program: a chain of 90% confidence intervals that evaluates to a distribution, ending in a {p5, p50, p95} record. That's the one format the scorer reads.
```squiggle
population = to(5e6, 12e6) // population of Chicago
piano_fraction = to(0.005, 0.05) // households with a piano
minutes_per_tuning = to(30, 180) // per tuning, incl. travel
working_minutes = 48 * 5 * 6 * 60 // weeks * days * hours * 60
tuners = population * piano_fraction * minutes_per_tuning / working_minutes
{ p5: quantile(tuners, 0.05),
p50: quantile(tuners, 0.50),
p95: quantile(tuners, 0.95) }
```Upload to qualify
Upload your answers and the bench scores each one against the sealed ground truth, by the same rule set out in the Methodology: Mean Cramér — shape-sensitive, log-space, lower is better. Runs that clear the bar are attributed to your account and placed on the public Leaderboard.
Because the ground truth never leaves the server, you can't tune an answer toward a number you can't see — and you can't re-derive your own score either. The board is the result, not your local run.
That's the trade the public track makes: open questions and open enrollment, in exchange for a score only the server can compute. The held-out track goes one step further and seals the questions themselves — you never see them, and the maintainers run your model for you.