Held-out track

Sealed questions, maintainer-run, a number that can't have leaked

The held-out track is the contamination-free half of the bench. You can't self-score a set you can't see — so the questions stay sealed, the maintainers run your model against them, and only the aggregate score is published.

It's the eval-server model used by serious contamination-safe benchmarks: the prompts and ground truth never leave a trusted environment, and no lab or model gets to train on them. For the half where the questions are open and anyone can enter, there's the public track.

The questions are sealed

Held-out questions never enter the public repository — not the prompts, not the ground truth. They live in a private store, versioned as named cohorts (holdout-v1, holdout-v2, …) that rotate as models improve.

They never appear anywhere you can reach: not in the question set you can download, and not on the Questions or Models pages. There's nothing we could show there without leaking them.

// the downloadable public set lists public questions only.
// held-out questions never appear in it — they live in a private
// store as named cohorts, rotated as models improve:
//   holdout-v1, holdout-v2, …

The open tooling only knows the public set. Held-out cohorts live outside the repo entirely.

You hand over a way to call your model

Because you can't see the questions, you can't produce the answers — so instead of run records, you give the maintainers a way to call your model: an Anthropic or OpenAI model id we can reach, or an OpenAI-compatible endpoint.

To enter, reach out to the maintainers to arrange access. Nothing about the cohort comes back in return — not the questions, not the count, not the answers.

# What you provide to enter:
provider:  openai-compatible          # or anthropic / openai
endpoint:  https://api.your-host.com/v1
model:     your-model-name

# Open an issue on the repo to arrange access. No cohort details come back.

You send a handle, not answers — you can't see the questions, so there's nothing to score yourself.

The maintainers run and score it

Your model is run against the current cohort inside a trusted environment, through the same scorer that grades the public set. Same output contract — a fenced squiggle program per question — and the same headline metric, Mean Cramér, so a held-out score sits on the same axis as a public one.

The prompts and ground truth never leave that environment. Only the scores come out.

# inside the trusted environment, against the sealed cohort:
#   your model answers each question with a fenced squiggle block,
#   scored by the same rule as the public set — Mean Cramér.

# prompts + ground truth never leave this environment.

Run by maintainers inside a trusted environment, with the same scorer that grades the public set.

Only the aggregate number ships

What enters the committed web bundle is scores and domain tags — never a prompt, a ground truth, or a raw response. A guardrail in the build fails if any held-out content tries to slip into the public payload.

On the site it surfaces in exactly one place: the Holdout filter on the Leaderboard. That ranking is the contamination-free number — no one has seen these questions, so it reflects estimation skill, not recall.

# runs.public.json — the committed web payload:
{ "held_out": true, "model": "your-model",
  "score": { "cramer_log": 0.21 } }

# prompt, ground truth, raw response: stripped.
# surfaces under the Holdout filter on the Leaderboard.

What enters the public bundle: scores only. A guardrail blocks any held-out content from leaking.

That's the point of sealing them. A held-out score is one no amount of training on public data can inflate — the price is that you trust the maintainers to run it, rather than running it yourself. The public track is the other half: open questions, open enrollment.