Meta-Eval (1/2): Add Judge Meta-Evaluation Tasks

[kargibora]

This PR is mostly written with respect to the code of @ErlisLushtaku. It is the re-integrated versions to rebased on top of the feat/official-task-variants branch to unify some structure

Problem

JudgeArena can use human arena votes to estimate a model's ELO rating, but it does not provide a direct way to evaluate the judge itself.

Judge evaluation starts from completed arena battles. The framework must ask a judge to label the stored model responses and then compare those labels with the human votes. Running model generation or treating the judge as a candidate model would mix two different experiments.

Before and after

Area	Before	After
Judge evaluation	Required custom scripts or manual analysis	Runs through the normal judgearena entry point
Input	Evaluation pipelines expected a model to generate or rate	Meta-evaluation reads existing arena prompts, responses, and human votes
Task definition	Arena and evaluation settings were not represented as a task	Task YAML selects the arena, language variants, prompt preset, and scorer
Prompt handling	Judge formats needed separate evaluation code	Existing prompt and parser registries are reused
Results	Human and judge labels could be inspected separately	Agreement and ranking metrics are produced in one result set

Added tasks

Task	Data
meta-eval-lmarena-140k/100k	LMSYS Chatbot Arena 140k/100k human votes
meta-eval-comparia	ComparIA human votes

Language variants use the same task pattern, for example meta-eval-lmarena-140k-en and meta-eval-comparia-fr.

How it works

meta-eval task YAML
    -> load human-labeled arena battles
    -> filter languages and select the most-battled models
    -> sample battles with a fixed run seed
    -> judge the stored response pairs
    -> parse the judge verdicts
    -> compare judge and human preferences
    -> write metrics and run artifacts

Meta-evaluation does not generate completions and does not evaluate a candidate model. The judge is the subject of the experiment.

The main components are:

• MetaEvalProtocol validates the arena, judge settings, and scorer selected by a task.
• The meta-evaluation runner owns sampling, judging, reporting, and artifact creation.
• Arena loaders normalize LMSYS and ComparIA battles into one shared format.
• Existing prompt presets and parsers are reused, including PairScore and Arena-Hard verdicts.
• Additional AlpacaEval-style presets are included because their output formats differ.
• The registered meta-evaluation scorer calculates agreement and ranking metrics without adding branches to the shared CLI.

Results

The report includes:

• Accuracy and Cohen's kappa for all battles and for battles without human ties
• Spearman correlation and ELO mean absolute error for English and multilingual model rankings
• Held-out ELO error across different annotation budgets, both with and without judge-predicted ties
• Sample counts by language and human verdict

Each run writes sample.parquet, annotations.parquet, summary.csv, results.json, config.yaml, and run-metadata.v1.json.

Example

judgearena \
  --task meta-eval-lmarena-140k \
  --judge.model OpenRouter/deepseek/deepseek-v3.2 \
  --meta_eval.languages '["en", "es"]' \
  --meta_eval.top_models 20 \
  --meta_eval.battles_per_model 50 \
  --meta_eval.n_bootstraps 20

The task defaults to the meta-eval-pair-score prompt preset. A different registered preset can be selected with --judge.prompt_preset, and --judge.swap_mode both enables judging both answer orders.

Change composition and task scope

Unique stacked diff against feat/official-task-variants at 28e640c8; parent PR changes are excluded. Changed LoC = added + removed lines.

Change type	Files	Changed LoC	Added	Removed
Core implementation	21	1,730	+1,678	-52
Tests	6	884	+878	-6
Static files	8	165	+165	-0
Docs/packaging and other	2	61	+58	-3
Total	37	2,840	+2,779	-61

• Core capability areas: arena normalization, deterministic connected sampling, judge annotation, agreement/ranking metrics, and safe run artifacts/checkpoints.
• Task scope: Judge meta-evaluation on ComparIA and LMArena 100k/140k, plus shared arena-battle normalization and reporting.

Tests

The focused tests cover task loading, deterministic sampling, human-label normalization, prompt parsing, swap handling, agreement metrics, ranking metrics, ELO-gap calculation, saved artifacts, and CLI configuration.

Notes

In the future, we can make the entire pipeline more abstract and seperate JudgeAnnotations, ModelCompletions etc so meta-eval is just a ELO pipeline without JudgeAnnotations and instead uses the existing battles with different scorer (agreement instead of ELO). However for simplicity of the current PR (and as we did already extensive refactoring), we should defer this for the future.

[ErlisLushtaku]

q: this only keeps the continuous preference when the parser is PairScore, so ArenaHard's graded >>/>/=/</<< value and AlpacaEval's logprob-weighted will collapse to 0/0.5/1 before reaching the soft-ranking/BT scorer.

Should we keep it like this, e.g. only score-format judges are meant to feed the soft metric? If not, maybe this could key off a has_continuous_preference property, so any new parser doesn't silently lose granularity here.

[kargibora]

Yes I thought I have fixed this but it's probably in seperate branch. I have noticed this and I agree we should add some field so log-probs can just fed into soft-ELO. Thanks for catching this

[ErlisLushtaku]

LGTM, just left a comment earlier that we should not forget in case we want to fix it