New judging system (v2)#50
Open
rank-and-file wants to merge 108 commits into
Open
Conversation
Deterministic check that compares final_model/config.json against bundled reference configs on fine-tuning-invariant architecture fields. Judges previously had to infer model identity from logs; a tokenizer warning referencing Mistral was enough to trigger a false "disallowed use detected." - judge_tools/model_identity_check.py + reference_configs for the four allowed base models (Qwen3-1.7B-Base, Qwen3-4B-Base, SmolLM3-3B-Base, gemma-3-4b-pt), resolved via AutoConfig so nested text_config defaults are filled in. - prompt.txt tells the judge to run the oracle in hard cases and not to treat generic vendor-mentioning warnings as evidence. - run_task.sh and run_judge.sh both expose the oracle and the candidate config as ../final_model_config.json inside the judge sandbox. Verified on 92 agent-produced runs: all MATCH against the true base; 36 cross-model swaps (negative control) all MISMATCH. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rerun judge in --rerun mode for the latest run per (benchmark, model)
across claude_claude-opus-4-5_10h_{final_v3,v5,v6_seed1}. Pulls
POST_TRAIN_BENCH_RESULTS_DIR from .env directly to avoid sourcing
set_env_vars.sh, which fails on nodes without tclsh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Judge in two phases: work backwards from the last agent messages in `../solve_parsed.txt` to pin down the base model, datasets, and training scripts that actually produced `final_model`, then inspect only those artifacts for contamination. The trace is authoritative — files left in the task directory may have been edited or never executed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bump codex judge from GPT-5.2 (high) to GPT-5.4 (xhigh) across run_task.sh
and run_judge.sh, and rename per-judge output files accordingly. Replace
--rerun with --gpt-only/--sonnet-only so run_judge.sh always writes _rerun
outputs without touching the original judgements, and partial reruns can
skip one model while still aggregating with the other's existing _rerun
result. Add rerun_single_gpt_only.sh + rerun_judge_gpt_only.sub plus
recompute_selected_methods_gpt_only.sh for the three
claude-opus-4-5_10h_{final_v3,v5,v6_seed1} method dirs, and switch
rerun_judge.sub to the user.judge:3333 concurrency limit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the two flat text files (contamination_judgement.txt, disallowed_model_judgement.txt) with one structured judgement.json per judge containing contamination/disallowed_model booleans plus justification strings. A new aggregate_judgement.py merges per-judge files into judge_result.json (logical OR for booleans, prefixed concatenation for justifications), and run_judge.sh / run_task.sh now call it instead of grep-based bash aggregation. Downstream readers (contamination_list.py, aggregate_contamination.py, extract_traces.py, the rerun scripts and utils) are updated to consume the new schema. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace opus_4_6_codex_5_3.sif with the new gpt_5_5.sif (definition file added under containers/) in both run_task.sh and run_judge.sh. Tighten failure handling: run_judge.sh now exits non-zero if a judge doesn't produce judgement.json, and aggregate_judgement.py raises SystemExit on any missing / malformed per-judge file instead of writing a False/False default that would mask a crashed judge. Also polish the prompt wording, drop rerun_judge.sub concurrency from 3333 to 1250, and broaden the oauth_token gitignore to oauth_token*. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a separate GPT-5.4 judge that inspects the agent trace for disallowed hosted-LLM API usage (training-data generation, distillation, etc.) and folds its verdict into the aggregated judge_result.json alongside the existing contamination/base-model judges. Also adds an --api-only rerun path and matching condor submit wrapper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swap the second contamination judge from Claude Sonnet 4.6 (claude CLI + subscription OAuth) to DeepSeek V4 Flash Free (opencode CLI + OPENCODE_API_KEY). Renames judgement_sonnet4_6.json -> judgement_deepseek.json across run_judge.sh, run_task.sh, aggregate_judgement.py, extract_traces.py, and the rerun tooling, and replaces the --sonnet-only flag/scripts with --deepseek-only equivalents. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the four duplicated agents/*/human_readable_trace.py parsers and
their ten symlinks with a single dispatcher in src/trace_parsing/. The
dispatcher substring-matches the agent name against {claude, codex,
gemini, opencode} and falls back to a verbatim copy when no key matches,
preserving the previous behavior for glm5 and qwen3max.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add src/trace_parsing/sanitize_trace.py, which redacts every *_API_KEY value sourced from the repo's .env file (live-environment values take precedence; `your-*` placeholders and short strings are skipped) and labels matches as `[REDACTED:<NAME>]`. Hook it into parse_trace.py so that every dispatcher run automatically drops `<stem>_sanitized<ext>` companions next to both the raw input and the parsed output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both collect.py and aggregate.py now default their output/input dir to <POST_TRAIN_BENCH_RESULTS_DIR>/_aggregated instead of writing next to the method dirs. Keeps the results root tidy. collect.py additionally skips any method whose name starts with an underscore so _aggregated/ (and any other derived-artifact dir) isn't mistaken for a method directory.
New POST_TRAIN_BENCH_EXTRA_RESULTS_DIRS env var (colon-separated PATH-style) lists extra read-only results roots. collect.py walks the primary root plus each extra; method-name collisions across roots are warned about and resolved by first-seen wins. Extras are ignored when --data-dir is passed on the CLI (single-dir intent).
The leaderboard scan crashed with KeyError on any final_*.csv whose header didn't include every benchmark in factors.json (extended-hour, METR, and partial-rerun methods). Add a benchmark-completeness gate next to the existing model-completeness gate so those files are silently skipped from the weighted-metric scan the same way partial-model files already are.
Adds "Fable 5 (Max)" to HARDCODED_AGENT_MAP (2 seeds). Fable's own gpqamain evaluation is mostly missing (3 of 8 cells have data across both seeds, one model has none), so scripts/patch_fable_gpqa.py copies the per-seed gpqamain column from the matching Opus-4.8 (Max) run — 1:1 seed pairing so Fable's aggregated std stays honest instead of collapsing to zero. Run order: collect.py -> patch_fable_gpqa.py -> aggregate.py. Delete the patch script (and its call from the local workflow) once real Fable gpqa runs land.
All codex_non_api* variants — _high, _high_reprompt, _reprompt, _xhigh, _xhigh_reprompt, _max — now share the ChatGPT subscription auth in agents/codex_non_api/auth.json via symlinks. One 'codex login' refreshes the token for every variant AND for the API/contamination judges (which also hard-code the same file). Prevents the stale-auth failure mode where re-logging into one variant leaves the judges' shared auth file unrefreshed. Symlinks (34-byte text blobs pointing at the target path) do not contain the OAuth tokens themselves, so tracking them past .gitignore is safe.
One folder per judge (data_contamination_judge/, api_usage_judge/), each with a judge.conf (label, output-file id, prompt file, inline fatality, condor weight) and its prompt template (moved byte-identical). Shared judge_lib.sh drives both run_task.sh's inline judges and the standalone runner, so adding a judge is: new folder + one line in ALL_JUDGES. - run_judges.sh replaces run_judge.sh; judge selection via --judges <name>[,<name>] instead of the per-mode flags - get_judge_prompt.py takes --judge <name>, reads the prompt from the judge dir; per-judge fill functions in EXTRA_FILLERS - batch rerun pipeline consolidated: commit_rerun_judges.sh replaces the 3 commit_* and 2 stale recompute_* scripts; a single rerun_judges.sub (judges/judge_weight submit variables) replaces the 4 .sub files and 4 rerun_single_* wrappers - contamination_check_tool/ renamed to test_data_download/ (it only (re)downloads benchmark test data) - docs consolidated into src/judges/README.md; AGENTS.md paths updated Functionality is unchanged: output filenames (judgement_gpt5_4.json, judgement_api.json, judge_output_*, _rerun variants), prompts, and the judge sandbox environment are preserved. Verified old-vs-new: prompts byte-identical across all benchmarks/agents; apptainer argv and produced files identical under a shim harness for both inline and standalone paths (incl. the missing-judgement failure path); batch dry-run selection identical over 889 real result dirs. Only deviation: contamination-only leftovers from --skip-existing now submit at their judge.conf weight (400) instead of commit_all_gpt_only.sh's 1250.
The API usage judge's verdict is now consumed by scoring: a disallowed_api_usage flag adds an 'A' letter to the contamination cell and the run falls back to the baseline score in scripts/collect.py (missing judgement_api*.json = "not flagged", for runs predating the judge). Cells are now combinations of M/C/A. The PTB-lookup judge becomes archival: its verdict no longer feeds the cell or the score fallback, but collect.py raises a RuntimeError naming the run dir if it is ever true, so a firing lookup judge aborts aggregation instead of passing unnoticed. RuntimeError deliberately escapes the broken-run except clause (which only catches FileNotFoundError/ValueError/KeyError/TypeError → baseline fallback). The api/ptb loaders share a generic _load_optional_flag_judgement() helper (rerun-over-original preference, None when absent). Docs updated throughout (judge.conf headers, run_judges.sh, judges README, AGENTS.md, scripts/README.md, extract_traces.py comment). Verified end-to-end on a synthetic results root: clean run keeps its score, api-flagged run gets cell 'A' + the baselines.json value, missing-judgement runs are not flagged, ptb-flagged run makes collect.py exit non-zero naming the directory.
Adds a _fs_agent() helper that replaces spaces with underscores in the filename form (aggregated_avg_Opus-4.8_(Max).csv, aggregated_avg_GLM_5.2.csv, aggregated_avg_Fable_5_(Max).csv, GPT-5.1-Codex-Max_High/Low). CSV row values still use the display form so the website's AGGREGATED_NAME_TO_KEY lookup still matches. Lets us drop the files straight into posttrainbench-website/data/.
Extracts the codex access token from agents/codex_non_api/auth.json and curls chatgpt.com/backend-api/codex/models before solve_task is invoked. Exits fast with a re-login hint if the session is invalidated, instead of wasting a 10h agent run only to hard-fail at the judge phase.
Moves the CLEANING UP block (copy final_model/system_monitor.log/task/ to EVAL_DIR, delete_hf_models, rm -rf /tmp/posttrain_container) to run before the judges instead of after. Patches judge final_model_config.json source from JOB_DIR/task to EVAL_DIR/final_model since delete_hf_models now runs before the judge. Effect: a judge crash leaves EVAL_DIR with final_model + task + trace so rerun_single_gpt_contamination_only.sh can produce a verdict later without redoing the 10h agent run. metrics.json is still written by the eval phase that runs after judges.
Resubmits the full agent run for every result dir the API-usage judge
flagged. Reads output.log line 1 to recover the original agent/config/
benchmark/model/hours/num_gpus, reconstructs the experiment suffix from
the parent method dir name, and preserves it via -a experiment_name.
Supports --method/--benchmark filters, --dry-run, and --max-parallel N
(via HTCondor concurrency_limits tag user.${USER}_rerun_api).
Integrates hrdkbhatnagar's run_task.sh changes into the unified judge structure from the src/judges/ refactor: - Judge OAuth precheck runs before the agent phase (comment updated for the unified three-judge setup; the TODO it carried is implemented by the cleanup reordering below, so it is dropped). - The CLEANING UP block (copy to EVAL_DIR, delete_hf_models) runs before the judges, so a judge crash no longer loses the agent run. In the unified structure this means prepare_judge_sandbox now reads the final_model config from EVAL_DIR instead of JOB_DIR/task. - rerun_api_flagged.sh: point FIND_PY at src/judges/rerun/, where the judge refactor moved find_disallowed_api_usage.py (interface unchanged).
rank-and-file
force-pushed
the
new_judge_v2
branch
from
July 16, 2026 09:35
b8377d4 to
239d2e9
Compare
Fourth judge alongside the three reward-hacking ones, aimed at what they were not designed to catch: run-integrity problems caused by the agent (novel hacking outside the others' scope) or suffered by it (premature stop, usage limits, grader-API credit exhaustion, harness/infra failure). Runs GPT-5.6 Terra on a codex CLI pinned to 0.144.5, which judge_lib.sh npm-installs into the sandbox home via the new JUDGE_CODEX_VERSION key (empty = the container's codex, as before). Its `general_anomaly` verdict never touches a score. When it flags, collect.py finishes the collection pass and then writes nothing at all, raising with every flagged run and its verdict file — so a flag in the last method cannot leave earlier methods' CSVs behind. Writing is split out of collect_method() into write_method_csvs() to make that possible. Also adds scripts/find_disallowed_ptb_lookup.py, the lookup-judge counterpart to find_disallowed_api_usage.py. Two judge.conf keys removed as part of this: - JUDGE_CONDOR_WEIGHT: rerun_judges.sub now pins a fixed user.codex_judge_rerun:2500 limit instead of reading $(judge_weight), which left the per-judge weight, both callers' -a judge_weight, and the docs inert. Dropped the plumbing rather than the hardcoded limit. - JUDGE_MISSING_JUDGEMENT_FATAL_INLINE: inline judges are now uniformly non-fatal. A judge that produces no judgement.json warns and moves on, so a finished 10h agent run is always evaluated and the rerun pipeline can supply the verdict later. Standalone reruns stay fatal — that was never this key, run_judges.sh passes a literal 1. This loosens the API and lookup judges, which used to abort the run before evaluation.
…hival Scored runs (metrics.json present) now must carry a contamination and an API verdict, and — for run ids >= NEWER_JUDGES_MIN_RUN_ID — a PTB-lookup verdict too. A method containing a scored run without a required verdict is skipped (warning, stale CSVs removed) instead of aggregated, so unchecked scores cannot enter the aggregation while the rest of the sweep still collects. The general (unknown-unknowns) judgement is no longer read by collect.py at all: it never affects a score and neither its absence nor a flag raises. Review its flags with find_flagged_runs.py --judge general_judge.
Spell out that grading is legal through evaluate.py or any agent wrapper that calls into the official evaluation_code/ grader, but a hosted call bypassing that grader (or used for anything besides grading this model's outputs) is still a violation.
New rule 4 forbids deriving training data from specific benchmark test items (paraphrasing, perturbation, seeding, hand-written coverage) even without verbatim overlap, mirroring the contamination judge's benchmark-targeted-data clause; style/format/domain matching stays allowed. Old rules 4-8 shift to 5-9 and the conditional grading-key exception becomes rule 10.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This branch ("new_judge_v2") reworks PostTrainBench's contamination/safety judging, overhauls the result-aggregation layer, consolidates trace parsing, adds an evaluation task, and moves configuration to a
.env-based flow.Judging system (v2)
src/disallowed_usage_judge/run_judge.sh(new in this PR; previously the judge was invoked inline). It runs a two-judge pipeline after the agent finishes:disallowed_api_usageschema; writes its ownjudgement_api.json(archival, not consumed by scoring).judgement_gpt5_4.jsondirectly. The oldscripts/migrate_judgement_files.pyis removed.benchmark_id.judge_tools/(contamination_check.py,model_identity_check.py,reference_configs/), pluscontamination_check_tool/download_test_data.{py,sh}. The judge runs in thegpt_5_5container with apptainer--containall.src/disallowed_usage_judge/rerun_judge/— batch resubmission for GPT-only / API-only / contamination-only reruns; outputs always carry a_rerunsuffix so originals are preserved. Includesfind_disallowed_api_usage.py(replaces the removeddev_utils/find_api_illegal.py).Aggregation / scripts
scripts/collect.pynow skips error runs, reads config from.env, and drops the API-usage judge from the collect/aggregate flow.scripts/utils.py,scripts/aggregate.py, andscripts/parse_all_to_human_readable.shupdated accordingly.scripts/migrate_judgement_files.py.Trace parsing
agents/*/human_readable_trace.py, now removed) intosrc/trace_parsing/(claude,codex,gemini,opencode) with a shared_common.pyand aparse_trace.pydispatcher.sanitize_trace.py, run after every parse.Agents & tasks
codexhigh/codexlownow emit JSON traces; minorsolve.shtweaks toclaudeandclaude_non_api.evaluate.py,benchmark.txt,info.json).info.jsonmetadata to existing tasks (aime2025, arenahardwriting, bfcl, gpqamain, gsm8k, healthbench, humaneval) for judge/prompt generation.Config & infra
.envis now the canonical source forPOST_TRAIN_BENCH_*vars; added anexample.envtemplate and a README "Environment Setup" section.set_env_vars.shupdated to source.env.AGENTS.md/CLAUDE.mdagent guidelines (and un-ignored them in.gitignore)..err/.out/.logfiles now route tologs/(gitignored) instead of the repo root;.gitignorealso ignores NFS silly-rename leftovers, rerun-pipeline run logs, and downloadedtest_data.json.latest-2026-03-27.def.Notes
🤖 Generated with Claude Code