Official Task Variations (2/2): Add arena-hard, alpaca eval and mt-bench official tasks

[kargibora]

Problem

JudgeArena already supports general pairwise evaluation, but official benchmarks do not all use the same judging rules.

AlpacaEval, Arena-Hard, and MT-Bench differ in their datasets, baseline answers, prompts, answer-order rules, parsers, and scoring methods. Keeping these choices outside the task definition makes it difficult to see what a run does and easy to mix incompatible settings.

This PR adds explicit official task variants. Each variant keeps its standard settings in task YAML and selects the required prompt, parser, and scorer through the shared benchmark pipeline.

Before and after

Area	Before	After
Task selection	General JudgeArena task variants	Explicit official task names
Defaults	Benchmark rules were spread across runtime code and constants	Task YAML owns the dataset, baseline, prompt preset, order policy, judge settings, and scorer
Parsing	Pairwise scores used one general output format	Arena-Hard verdicts and AlpacaEval tokens are parsed by their own registered parsers
Scoring	Pairwise results used the general win-rate scorer	Official variants select benchmark-specific scoring
Dataset loading	Official files and baselines required benchmark-specific runtime handling	Dataset adapters load pinned sources and expose a common pairwise format

Added tasks

Task	Main behavior
alpaca-eval-2.0-official	Uses the pinned AlpacaEval instruction set and GPT-4 Turbo baseline, randomized answer order, single-token judging, logprob-weighted preferences, and length-controlled win rate
arena-hard-v0.1-official	Uses the official v0.1 baseline, Arena-Hard verdict format, both answer orders, and weighted scoring
arena-hard-v2.0-official	Adds category-specific baselines and prompts, both answer orders, per-category results, and weighted scoring
mt-bench-official	Uses the pinned FastChat data and references, category generation temperatures, reference-aware judging, and both answer orders

How it works

official task YAML
    -> selects the dataset adapter
    -> loads the pinned examples and baseline answers
    -> selects the prompt preset and parser
    -> runs the existing benchmark runner
    -> selects the task scorer
    -> writes the normal JudgeArena artifacts and report

The shared pairwise runner remains responsible for generation and judging. Task-specific code is limited to the parts that are genuinely different:

• Dataset adapters normalize official inputs and baseline outputs.
• Prompt presets contain the official judge instructions.
• Parsers convert each official output format into a preference.
• Scorers implement Arena-Hard weighted results and AlpacaEval length-controlled win rate.
• The scorer resolver gives the runner one clear lookup path instead of task-name branches.

Existing non-official tasks keep their current defaults and behavior. Runtime overrides are still supported where the task allows them. A run with a different judge or overridden settings remains useful, but it should not be treated as an exact reproduction of the official leaderboard.

Example

judgearena \
  --task arena-hard-v2.0-official \
  --model.name OpenRouter/<candidate-model> \
  --judge.model OpenRouter/<judge-model> \
  --generation.n_instructions 10 \
  --run.result_folder smoke-results/arena-hard-official

The task YAML supplies the official baseline, prompt preset, answer-order policy, judge limits, and scorer. The command only needs to choose the candidate model, judge, sample size, and output path.

Tests

The tests cover the benchmark rules that can change results:

• Official task loading and defaults
• Category-specific Arena-Hard prompts and baselines
• Arena-Hard verdict parsing, answer-order handling, weighted scoring, missing orientations, and category results
• AlpacaEval token parsing, logprob weighting, annotations, and length-controlled scoring
• Pinned dataset and reference loading

Notes

Most of the LoC coming from the scorer implementations and prompts that are added, not the complexity of how they are added.

[kargibora]

I refained from the ELO_SCORERS[x] call, because resolve_elo_scorer(x) is much cleaner and customazible

[kargibora]

Should we or not add this requirement (we already did with this PR). The reason: alpaca-eval already establishes most of the code for calculating the results and it is not easy, we just convert our format to theirs and get the result by calling it. Take a look at `official_annotations``

[kargibora]

I did not want to reuse judgearena_tables as it has some column normalization. However I can change the adapter and re-use the existing tables if its better.

[ErlisLushtaku]

q: this only implements upstream's non-style-controlled path (decisive verdicts weighted 3x, bootstrap resample the mean).

The official leaderboard for hard_prompt/coding/math uses the style-controlled path instead. arena-hard-v2.0-official.yaml uses this scorer for all four categories, so for three of them the result isn't consistent with the official leaderboard number.

Should we add the style-controlled BT fit for arena-hard's official variant, or relabel the yaml/scoring_details so it's explicit this reproduces the default CLI path and not the official-configuration number?