Bring a question — a ground truth is optional. Independent models solve it blind; if you provide a ground truth the bench grades against it, and if you don’t, it shows you how tightly the models agree so you (or a reviewer) can decide whether it’s worth pinning one. A human always ratifies before a question counts. The machine triages — it never decides.
Preview Submission is live; server-side grading is still stubbed (returns a sample report) until the pipeline is wired in.
A benchmark’s ceiling is set by the quality of its ground truth, not the cleverness of its questions. Before you submit, the five things we look for — the full reasoning lives in the methodology.
A careful reader should land on one interpretation. Pin down units, scope, and time frame. If the question is genuinely open-ended, that's allowed — but say so, so we can label it underspecified rather than mistake it for broken.
The answer shouldn't be one web search or one memorized fact away. If a model can simply recall it, it's a lookup, not a Fermi estimate. We probe this automatically — the FACT/GUESS check below.
A good question forces real moves: decomposition, analogical scaling, triangulation. The deeper operations are where this bench is aimed and where it's least redundant with prior work.
Your GT sets the ceiling — a clever question with a wrong answer is worse than no question. Give a decomposition with named leaves and a source for each fact you looked up, not a scraped answer-key number (unless you have a great secret dataset).
Don't stop at lazy recall when the quantity needed splitting, and don't over-decompose into a tree of shaky guesses where one clean reference class is tighter. Calibrated depth is part of the skill.
Checking session…