Add WILD-raw item-level adapter (every_eval_ever/adapters/wild) by borgr · Pull Request #203 · evaleval/every_eval_ever

borgr · 2026-07-20T20:29:08Z

What / source

Adds every_eval_ever/adapters/wild/ — an adapter for WILD-raw
(kensho/WILD-raw, arXiv:2604.01418): item-level eval responses for 65 models ×
27 benchmarks (~7.5M (model, item) rows), run by Kensho with Inspect AI.

Grain. One aggregate EvaluationLog per (model, benchmark) — the overall
accuracy (wild.<task>) plus one result per subtask (wild.<task>.<subtask>), each
a continuous [0,1] mean of the binary item scores with an analytic proportion
standard_error + num_samples. A benchmark with ≤1 distinct subtask emits only the
overall (no byte-identical duplicate leaf). Every result carries the registry's
canonical accuracy metric id; the task lives in evaluation_name, so the
cross-source accuracy join stays whole. --include-instances additionally writes the
per-item <uuid>_samples.jsonl sidecar linked from detailed_evaluation_results.

source_type=evaluation_run, evaluator_relationship=third_party,
eval_library=inspect_ai (paper-confirmed). source_data names each benchmark's
own dataset repo (mmlu→cais/mmlu, arc_*→allenai/ai2_arc,
finance_fundamentals→kensho/bizbench, …; each verified on HF), not WILD-raw —
WILD-raw is the results source and lives in source_metadata.
Instances: input.raw = the prompt turns only (no answer leak, so it hashes
identically across models); output.raw = the assistant turn(s), i.e. the model's
full generation, empty when the row has none; the scorer's parsed answer goes to
answer_attribution.extracted_value, with source naming where it came from and
extraction_method = the real Inspect scorer. sample_hash uses the shared
cross-adapter recipe. Instances link to the finest-grain result only — every item
belongs to exactly one subtask, so also linking the overall would duplicate ~7.5M
rows for no new information (reference/instance-level.md sanctions leaf-only with
a comment, which the code carries).
Reads parquet from HF in bounded record batches, and publishes through the
shared publish_evaluation_logs: sidecars are staged into a temporary datastore
tree, re-validated and re-checksummed, and nothing is left behind if any log fails.
A populated output directory is an error rather than a second copy of every
evaluation_id; --replace-existing supersedes only the prior copies of the
(model, benchmark) pairs this run rewrites — matched on the logical evaluation_id
read out of each file, and removed only once the replacements are published — so a
run whose input covers just some of a model's benchmarks leaves the rest of that
model's directory intact, and one that fails partway leaves the previous refresh
whole.
Rows whose score is not a usable 0/1, and rows with incomplete token usage, are
handled rather than silently absorbed: the former enter
adapter_reports/wild_failures.json and leave the denominator (the command then
exits non-zero), the latter are omitted from token_usage and from the token means
instead of being measured as zero.

Review lane

Fast — no conflicts, scoped (tests / one adapter / one file), verified by me,
under ~1000 hand-written lines
Needs a human — design change, cross-package, large, refactor, material
change in outcome, or large agent-authored change

Scoped to one adapter and structurally additive — publication reuses
every_eval_ever/converters/common/publication.py and helpers.io as they already
exist — but it is over the Fast lane on size: a 786-line module, 143 lines of README
and 425 lines of tests. Four review rounds have run on it.

Checklist

python -m every_eval_ever validate clean — no warnings — at the final
data/wild/<dev>/<model>/ path, on the whole conversion: 1,755 aggregates,
0 invalid, 0 errors, 0 warnings, semantic checks on
every unconvertible source row is in adapter_reports/, and the command exits
non-zero (SourceConversionResult.raise_if_incomplete, report written before
publication so a publication error cannot take the accounting with it)
offline unit tests added (20, up from 6) + full pytest tests green
(409 passed, 20 skipped; 431 with current main merged in)
ruff check clean
destination and selection are validated before anything is read: --output-dir
must be the data/wild collection directory publication writes into, and
--models requires at least one value and fails when the source matches none
of them rather than publishing nothing successfully
model ids are the dataset's HF-form ids, and the publisher comes from
helpers.developer — 15 of the 65 ids carry no namespace, so a prefix split
cannot name one; benchmark canonicalization is a registry follow-up (enumerated
in the adapter README), not a blocker here
content spot-checked: no answer leak in input.raw, instances not
double-counted across overall+leaf, evaluation_id keyed on the pinned source
commit date (stable across reruns; no now() fallback)

Decisions & coverage

Decision / where: publish per log rather than in one batch — publish().
Chose / instead of: one publish_evaluation_logs call for all ~1,755 logs.
Why: the shared publisher buffers a batch's bytes before creating any file, so one
call would hold the entire sidecar corpus in memory. Per-log calls bound that to one
model×benchmark, and publish() removes anything earlier logs created if a later
one fails, so the run still leaves the whole conversion or nothing.
Confidence: med. General? yes — any item-level source large enough to matter
hits this; a streaming/batched mode in the shared publisher would remove the need
for each adapter to solve it.
Decision / where: instances link to the leaf subtask only — make_instance.
Chose / instead of: also emitting them under the benchmark overall (~2× rows).
Confidence: high (explicitly one of the two sanctioned options). General? no.
Decision / where: a local --parquet run now requires --evaluation-timestamp.
Chose / instead of: time.time(). evaluation_id is keyed on this value, so a
now() fallback gives identical reruns different logical identities.
Confidence: high. General? yes — the same trap exists in any adapter whose id
includes a timestamp it cannot read from the source.
Decision / where: a remote run stops when it cannot resolve a concrete commit.
Chose / instead of: reading the mutable main ref (which would let the aggregate
and instance passes read different data). --revision <sha> pins it explicitly.
If the metadata lookup fails with a --revision in hand, that value is accepted
only when it is a 40-character SHA — a branch or tag can still move between the two
passes — and since the lookup was also the source of the commit date,
--evaluation-timestamp then becomes required rather than guessed.
Confidence: high. General? yes.

Coverage: the adapter converts every item row it reads. Verified end to end on the
whole source, pinned to 06af2cc48c351a9c67b9caec4dff7d1c43d451dd: 7,237,945 source
rows → 1,755 logs across 65 models and 17 publishers, 0 dropped, 0 failures, in 68 s.
All 1,755 validate clean at their datastore paths with semantic checks on, and produce
no warnings under the registry-alias check in #230 either. A full-corpus scan of shard 0
found 0 rows with a missing score and 0 with missing token counts, so the score and
token guards are defensive rather than a repair of observed corruption.

Operator asked about policy calls? Re-hosting large data: instance sidecars stay
behind --include-instances and the companion datastore PR ships aggregates only
(1,755 records) — WILD-raw is already public, so the ~7.5M item rows are not
re-hosted by default. New canonical benchmark ids (squad, paws, chembench,
finance_fundamentals, pre_flight) and the arc_easy/arc_challenge →
AI2-ARC aliases go through the registry, not this PR.

Companion work (paired): data
https://huggingface.co/datasets/evaleval/EEE_datastore/discussions/174 · registry
benchmark ids — not yet opened; the seven aliases/canonicals are enumerated in
wild/README.md under Benchmark
canonicalization. This PR does not depend on it: evaluation_name is
wild.<task>[.<subtask>] and source_data carries each benchmark's dataset repo, so
the registry change is additive.

The datastore PR is still a draft, and its 1,755 records predate this review round
on six counts: schema_version 0.2.2, per-task metric ids
(wild.winogrande.accuracy), no deployment_type/model_availability (so every record
there fails the current validator), no pinned dataset_revision, no
n_items_with_token_usage, and 405 records filed under data/wild/unknown/ — the
flat-id bug, published. It is regenerated in full from this adapter before the set
merges; evaluation_id is unchanged by any of the fixes, so logical_identity()
supersedes each wild/<model>/<benchmark> pair rather than adding a second copy.

Converts kensho/WILD-raw (65 models x 27 benchmarks, ~7.5M item responses, run with Inspect AI) to Every Eval Ever: one aggregate EvaluationLog per (model, benchmark) with overall + per-subtask accuracy (binary items -> proportion with analytic standard error), and optional per-item instance _samples.jsonl (--include-instances) linked via detailed_evaluation_results. Streams parquet from HF per row group (never the full 7GB). source_type=evaluation_run, eval_library= inspect_ai; source_data points at each benchmark's own dataset repo; input.raw is the prompt only (assistant turn -> output.raw); sample_hash for cross-model joins. Adds README + a utils/README.md row + tests/test_wild_adapter.py (3 tests).

The core test matrix installs no extras; pyarrow (used by the adapter/tests) only arrives via --all-extras, so a top-level import broke core collection. Guard with pytest.importorskip like the inspect/helm adapter tests.

mrshu

This is really nice work. The per-row-group streaming and the docstrings walking through each mapping decision made it easy to follow. I left a few small suggestions inline, all optional and nothing blocking. Thanks for putting this together!

mrshu · 2026-07-21T12:19:16Z

+    subtask = row['subtask'] if row['subtask'] not in (None, '') else '_'
+    # attach to the leaf result when the benchmark is split by subtask, else the
+    # single overall result (matches build_log's dedup so the FK always resolves).
+    if multi_subtask:


One thought here: since each multi-subtask item feeds both the overall and the leaf result, it might be nice to also emit an instance record for the overall one, since instance_level_types.py:170,174 leans that way, so the overall result isn't left without linked instances. It does roughly double the instance rows for these benchmarks, so leaving it as is would be totally reasonable too. Maybe just a short comment noting the choice either way?

mrshu · 2026-07-21T12:19:16Z

+def resolve_eval_timestamp(override: str | None) -> str:
+    """When the evaluation was RUN. We don't have per-run times, so use the WILD
+    dataset's HF lastModified as a stable proxy (override with --evaluation-timestamp)."""
+    if override:
+        return str(override)
+    try:
+        from huggingface_hub import HfApi
+        info = HfApi().dataset_info(HF_REPO_ID)
+        if info.lastModified:
+            dt = info.lastModified
+            if dt.tzinfo is None:
+                dt = dt.replace(tzinfo=timezone.utc)
+            return repr(dt.timestamp())
+    except Exception:  # noqa: BLE001 - fall back to now
+        pass
+    return str(time.time())


Small suggestion: since this timestamp feeds into evaluation_id, it could be nice to pin a commit SHA for remote runs so reruns stay idempotent even if the HF lookup hiccups, and maybe log a note rather than quietly falling back to time.time(). The same pin would also keep the two passes over main (aggregate and instances) on the same revision. Definitely a nice to have, not urgent.

mrshu · 2026-07-21T12:19:16Z

+from pathlib import Path
+from typing import Iterator
+
+import pyarrow.parquet as pq


Tiny one: pyarrow is imported at import time but isn't a declared dependency, though the importorskip guard handles CI nicely. Might be worth a wild extra or a README note about --all-extras, just so someone running the adapter fresh gets a clear signal. Totally up to you.

mrshu · 2026-07-21T12:19:16Z

+def _scorer_name(scores_json: str | None) -> str:
+    """The Inspect scorer that produced the item score = the key of the scores
+    JSON (e.g. 'match', 'choice', 'model_graded_qa')."""
+    if scores_json:
+        try:
+            keys = list(json.loads(scores_json).keys())
+            if keys:
+                return str(keys[0])
+        except (ValueError, TypeError, AttributeError):
+            pass
+    return 'unknown'
+
+
+def _raw_output(scores_json: str | None, extracted: str) -> str:
+    """The model's full generation lives in the scorer output (`scores.<scorer>.answer`);
+    fall back to the extracted `answer` if absent."""
+    if scores_json:
+        try:
+            for scorer in json.loads(scores_json).values():
+                if isinstance(scorer, dict) and scorer.get('answer'):
+                    return str(scorer['answer'])
+        except (ValueError, TypeError, AttributeError):
+            pass
+    return str(extracted or '')


Minor thought: _scorer_name grabs the first key while _raw_output scans all the values, so if scores ever has more than one key the two could point at different scorers. If it's always single-key then this is a non-issue. Might be worth a quick confirm, and maybe a note reconciling output.raw (which comes from scores.answer) with the docstring that mentions the assistant turn.

mrshu · 2026-07-21T12:19:16Z

+    overall = next(r for r in log["evaluation_results"] if r["evaluation_name"] == "wild.mmlu")
+    assert overall["metric_config"]["score_type"] == "continuous"
+    assert (overall["metric_config"]["min_score"], overall["metric_config"]["max_score"]) == (0.0, 1.0)
+    assert abs(overall["score_details"]["score"] - 2 / 3) < 1e-9


Could be nice to also assert the standard_error here (sqrt((2/3)(1/3)/6) for this fixture) so the SE math has a little regression guard. Optional, since the formula is pretty straightforward.

@mrshu

…y, deps, SE guard) Addresses @mrshu's review on evaleval#203 (all optional, none blocking): - Idempotency: pin a concrete kensho/WILD-raw commit SHA up front (resolve main once), reuse it across BOTH the aggregate and instance passes, derive evaluation_timestamp from that commit's date, record dataset_revision in source_metadata, and add --revision to reproduce a past snapshot. Warn loudly instead of silently falling back to now() (which made evaluation_id non-idempotent). - Scorer consistency: read the scorer name AND output.raw from the SAME scorer (_primary_scorer) so they can't diverge if `scores` ever has >1 key. - Deps: declare pyarrow as a `wild` extra (it was undeclared) + a README install note, so a fresh run fails clearly instead of relying on a transitive dep. - Test: assert the analytic standard_error sqrt(p(1-p)/n) as a regression guard. - Instances: document that instances attach to the finest-grain result (leaf); the overall roll-up is intentionally not re-emitted (it is the union of the leaves). Verified: ruff clean; pytest tests/test_wild_adapter.py green; live smoke (shard 0, --include-instances) validates 125/125 with a real pinned commit SHA in every record.

borgr · 2026-07-21T14:26:34Z

Thanks @mrshu — really appreciate the careful read. Pushed a commit addressing all five:

Idempotency (2): resolve & pin a concrete kensho/WILD-raw commit SHA up front, reuse it across both the aggregate and instance passes, derive evaluation_timestamp from that commit's date, record dataset_revision in source_metadata, and add --revision to reproduce a past snapshot. It now warns loudly instead of silently falling back to time.time().
Scorer consistency (4): a single _primary_scorer() reads the scorer name and output.raw from the same scorer, so they can't diverge if scores ever has >1 key; reconciled the docstring.
pyarrow (3): it was undeclared — added a wild extra (in all) + a README install note, so a fresh run fails clearly rather than relying on a transitive dep.
SE (5): added an analytic standard_error = sqrt(p(1-p)/n) regression assertion.
Overall vs. leaf instances (1): kept instances attached to the finest-grain (leaf) result and did not also emit them for the overall roll-up, with a comment explaining why. This matches the house pattern — lm_eval's mmlu group gets an aggregate result but no instances, and HELM keys instances 1:1 per scenario — since the overall is exactly the union of the leaves, so re-emitting would duplicate every item and risk double-counting. Happy to switch to emitting overall instances too if you'd prefer FK-completeness over the ~2× rows.

Verified: ruff clean, pytest tests/test_wild_adapter.py green, and a live smoke run (shard 0, --include-instances) validates 125/125 with a real pinned commit SHA in every record.

mrshu

⚒️ review-anvil report

Review decision: COMMENT — The adapter is a strong, well-tested addition, and a few data-contract edges would benefit from follow-up.
Result: One confirmed high-priority mapping issue and two medium-priority repeatability issues remain.
Scope: This PR adds a streaming WILD-raw adapter for aggregate and item-level Inspect AI results.
Checks: 2 concerns checked; 2 confirmed, 0 ruled out, 0 set aside, 0 lowered in priority.
Second check: targeted, 2 reviewers; 5 kept, 4 fix paths clarified, 0 set aside, 0 removed.

Earlier review comments

Earlier review comments (5 items)

Still open: Overall instance linkage remains a useful suggestion. Multi-subtask items still link only to leaf results.
Still present at the current head: Revision fallback still uses mutable main when SHA lookup fails. The old anchor is outdated.
Fixed: The pyarrow dependency is now declared in the wild extra and lockfile.
Fixed: Scorer selection now keeps the scorer name and answer together.
Fixed: Standard-error coverage now checks the analytic value and sample count.

What I noticed

[high] instance output utils/wild/adapter.py:361 — output.raw stores Score.answer, which Inspect defines as an extracted answer. WILD-raw keeps the complete response in assistant turns, but the adapter discards those turns. Rows where extraction shortens the response lose the full model output. (RAVF001; inline)
[medium] repeat runs utils/wild/adapter.py:473 — Stable evaluation IDs do not make writes idempotent. Each run creates a new UUID aggregate and sidecar, so a reused output directory can collect duplicate logical evaluations. (RAVF002; inline)
[medium] local provenance utils/wild/adapter.py:421 — Local parquet runs have no remote revision, but the output records main. They also use the current time without an explicit evaluation timestamp, so the provenance and identity change between runs. (RAVF005; inline)

Things to try

[high] instance output — The last non-empty assistant turn can populate output.raw. A missing assistant turn can remain explicit instead of labeling extracted scorer text as a complete response. (RAVF001)
[medium] repeat runs — An adapter-owned output set can replace the aggregate and sidecar together. The sidecar must be rebuilt rather than appended. (RAVF002)
[medium] local provenance — Unknown local revisions can stay unset. An explicit evaluation timestamp can keep local runs stable without inventing remote provenance. (RAVF005)

Run details

Target: PR #203 (add-wild-adapter, 7 files, +775/-1)
Rounds: 1/1 completed; adaptive off; material findings remained
Mix: 3 codex-exec
Focus: correctness, maintainability, simplicity, production impact, and constructive suggestion-oriented language
Earlier review comments: 5 comments; 2 still present and 3 fixed
Finding counts: 0 critical, 1 high, 4 medium, 0 low, 0 nit
Checks: concerns=2; confirmed=2/ruled-out=0/set-aside=0/lowered=0
Second check: targeted; reviewers=2; kept=5/clarified=4/set-aside=0/removed=0; approval changed no
Set aside: 0 items

Reviewed with review-anvil.

mrshu · 2026-07-29T18:35:11Z

+        sample_hash=sample_hash,
+        interaction_type=InteractionType.single_turn,
+        input=Input(raw=prompt, reference=reference),
+        output=Output(raw=[raw_out]),


[high] instance output — output.raw stores the extracted scorer answer

_primary_scorer() reads Score.answer, and make_instance() stores it as the complete response. Inspect defines this value as an extracted answer. WILD-raw keeps the full response in assistant turns, but _prompt_from_conversation() discards those turns.

The last non-empty assistant turn can populate output.raw. A missing assistant turn can stay explicit, and a fixture with different full and extracted answers would cover the mapping.

mrshu · 2026-07-29T18:35:11Z

+    for (model, task), subs in groups.items():
+        log, developer, model_slug = build_log(model, task, subs, eval_ts,
+                                               retrieved_ts, revision)
+        path = save_evaluation_log(log, args.output_dir, developer, model_slug)


[medium] repeat runs — Reusing an output directory creates duplicate logical evaluations

save_evaluation_log() creates a new UUID filename on every call. Instance mode also creates another full sidecar. Stable evaluation_id values therefore do not make the filesystem output idempotent.

An adapter-owned aggregate and sidecar can be replaced as one set. The sidecar needs a fresh write because its current append mode would otherwise duplicate rows.

mrshu · 2026-07-29T18:35:11Z

+    revision. Warns loudly rather than silently degrading to a mutable ref."""
+    if parquet:                       # local files: no remote revision to pin
+        return None, None
+    try:


[medium] local provenance — Local parquet input is recorded as remote main

resolve_source_revision() returns no revision for local files. build_log() then records revision or HF_REVISION, while the timestamp path falls back to the current time. The same local input can therefore report false remote provenance and get a new identity on each run.

Unknown local revisions can stay unset. Requiring an explicit evaluation timestamp would keep durable local runs stable.

- sample_hash: use the canonical cross-adapter recipe (sha256 over canonical JSON of {"raw","reference"}) instead of a bespoke string concat, so items join across adapters for the same example - output.raw: emit the model's full generation (assistant turn) rather than the scorer's parsed answer; split conversation into prompt vs generation (prompt is user+system only -> answer-free, hashes across models) - provenance: don't stamp dataset_revision='main' for local --parquet runs (a false remote-provenance claim); record a local-source marker instead

borgr · 2026-07-30T21:33:50Z

Thanks for the review — pushed a commit addressing these:

sample_hash — replaced the bespoke concat with the canonical cross-adapter recipe (sha256 over canonical JSON of {"raw","reference"}), so items join with other adapters for the same example.
output.raw — now the model's FULL generation (assistant turn), not the scorer's parsed answer; conversation split into prompt (user+system → input.raw, answer-free) vs generation (assistant → output.raw). Extracted answer stays in answer_attribution.
Local-run provenance — no longer stamps dataset_revision='main' for --parquet runs (a false remote-provenance claim); records a local-source marker, and only stamps dataset_revision for a pinned remote read.

mrshu

⚒️ review-anvil report

Review decision: COMMENT — The advertised instance mode still fails after writing partial output.
Result: One high and ten medium concerns remain. Five earlier concerns are fixed.
Scope: This PR adds a WILD-raw aggregate and optional item-level adapter.
Checks: 11 concerns checked; 8 confirmed and 3 narrowed.
Second check: targeted, 2 reviewers; 11 kept, 5 fix paths clarified, 0 set aside, 0 removed.

Earlier review comments

Earlier review comments (14 items)

The dependency declaration, scorer pairing, standard-error test, and false local main label are fixed.
Missing-response handling, revision pinning, repeat publication, local identity, and overall-instance linkage remain present.
Existing discussions remain the place for those carry-forward items. This review does not create duplicate inline threads.

What I noticed

RAV-RUN2-R1-F001 [high] instance-publication utils/wild/adapter.py:528 — Instance mode passes a sidecar basename where the schema requires a canonical datastore path. Construction fails after aggregate and sidecar files are written. (inline)
RAV-RUN2-R1-F002 [medium] documentation utils/wild/README.md:32 — The smoke command passes a directory to a file-only validator. Its output root also cannot satisfy datastore path checks. (inline)
RAV-RUN2-R1-F003 [medium] memory-use utils/wild/adapter.py:142 — Capped instance runs still materialize all selected columns for a complete row group before the cap is checked. (inline)
RAV-RUN2-R1-F004 [medium] data-validation utils/wild/adapter.py:157 — Missing scores become incorrect answers and stay in aggregate denominators. Missing token data becomes measured zero instead of being omitted. (inline)
RAV-RUN2-R1-F005 [medium] metric-identity utils/wild/adapter.py:210 — Each task receives a different metric ID even though every result is canonical accuracy. Cross-source joins by metric ID become fragmented. (inline)
RAV-RUN2-R1-F006 [medium] instance-linkage utils/wild/adapter.py:367 — Multi-subtask rows contribute to overall and leaf results, but instances link only to leaves. This conflicts with the one-record-per-contributing-result guidance. (earlier comment remains open)
RAV-RUN2-R1-F007 [medium] instance-output utils/wild/adapter.py:383 — When no assistant response exists, the adapter substitutes an extracted answer into output.raw. Scorer data is then labeled as complete model output. (earlier comment remains open)
RAV-RUN2-R1-F008 [medium] revision-pinning utils/wild/adapter.py:471 — When metadata lookup fails without --revision, both remote passes continue against mutable main. (earlier comment remains open)
RAV-RUN2-R1-F009 [medium] local-provenance utils/wild/adapter.py:478 — Local runs without --evaluation-timestamp key evaluation_id on current time. Identical reruns receive different logical identities. (earlier comment remains open)
RAV-RUN2-R1-F010 [medium] repeatability utils/wild/adapter.py:510 — Reruns create fresh UUID files for stable evaluation IDs. Incremental writes also leave partial batches when later work fails. (earlier comment remains open)
RAV-RUN2-R1-F011 [medium] cli utils/wild/adapter.py:546 — A bare --parquet flag yields an empty list. The adapter silently starts the full remote 15-shard conversion. (inline)

ID legend: RUN is the observed PR review run, R is the immutable origin round, F is a finding, and P is a plan.

Things to try (10 items)

[high] publication — Complete output can stage before publication. Existing logical output can be rejected until replacement is atomic. Sidecar validation and hashing must stay streaming for WILD's size. (RAV-RUN2-R1-P001; covers RAV-RUN2-R1-F001, RAV-RUN2-R1-F010)
[medium] documentation — Smoke output can use the required data/<collection> path. The validator can receive a quoted fixed-depth file glob. (RAV-RUN2-R1-P002; covers RAV-RUN2-R1-F002)
[medium] memory-use — Bounded record batches can flush and hash incrementally. (RAV-RUN2-R1-P003; covers RAV-RUN2-R1-F003)
[medium] data-validation — Rows with invalid scores can enter the failure report. Incomplete token usage can be omitted, with token means based only on complete values. (RAV-RUN2-R1-P004; covers RAV-RUN2-R1-F004)
[medium] metric-identity — accuracy can remain the metric ID while evaluation_name distinguishes tasks. (RAV-RUN2-R1-P005; covers RAV-RUN2-R1-F005)
[medium] instance-linkage — Overall-leaf pairs can emit as one unit. The cap can stop before a pair that does not fit. (RAV-RUN2-R1-P006; covers RAV-RUN2-R1-F006)
[medium] instance-output — Missing assistant output can remain empty. Attribution can name the actual answer or scorer source. (RAV-RUN2-R1-P007; covers RAV-RUN2-R1-F007)
[medium] revision-pinning — Default remote runs can stop when a concrete revision cannot resolve. Verified immutable overrides can remain usable. (RAV-RUN2-R1-P008; covers RAV-RUN2-R1-F008)
[medium] local-provenance — Local parquet runs can require an explicit evaluation timestamp. (RAV-RUN2-R1-P009; covers RAV-RUN2-R1-F009)
[medium] cli — --parquet can require at least one path. (RAV-RUN2-R1-P010; covers RAV-RUN2-R1-F011)

Run details

Target: PR #203 at 6c8e3b20a4fbff2d332cab3ccbe544bac618f87a (7 files, +855/-1)
Run ordinal: 2
Rounds: 1/1 completed; adaptive off; material findings remain
Mix: 3 codex-exec
Focus: correctness, maintainability, simplicity, production blast-radius, and constructive optional suggestions
Earlier review comments: 14 ledger entries; fixed and carry-forward items revalidated
Finding counts: 0 critical, 1 high, 10 medium, 0 low, 0 nit
Checks: concerns=11; confirmed=8/ruled-out=0/set-aside=0/narrowed=3
Second check: targeted; reviewers=2; kept=11/clarified=5/set-aside=0/removed=0; approval changed no
Fixes applied: 0 (review-only)

Reviewed with review-anvil.

mrshu · 2026-08-06T04:33:46Z

+        for key, count in counts.items():
+            if not count:
+                continue
+            path = agg_paths[key]


RAV-RUN2-R1-F001 [high] instance-publication — Instance mode passes a sidecar basename where the schema requires a canonical datastore path. Construction fails after aggregate and sidecar files are written.

Complete output can stage before publication. Existing logical output can be rejected until replacement is atomic, with sidecar validation kept streaming.

mrshu · 2026-08-06T04:33:46Z

+
+```bash
+uv run python -m utils.wild.adapter --output-dir /tmp/eee-wild --limit-shards 1
+uv run python -m every_eval_ever validate /tmp/eee-wild


RAV-RUN2-R1-F002 [medium] documentation — This smoke command passes a directory to a file-only validator. Its output root also cannot satisfy datastore path checks.

Smoke output can use the required data/<collection> path, and validation can receive a quoted fixed-depth file glob.

mrshu · 2026-08-06T04:33:46Z

+        with opener() as fh:
+            pf = pq.ParquetFile(fh)
+            for rg in range(pf.num_row_groups):
+                tbl = pf.read_row_group(rg, columns=columns)


RAV-RUN2-R1-F003 [medium] memory-use — Capped instance runs still materialize all selected columns for a complete row group before the cap is checked. The cap does not bound this allocation.

Bounded record batches can flush and hash incrementally.

mrshu · 2026-08-06T04:33:46Z

+    in_tok: int = 0
+    out_tok: int = 0
+
+    def add(self, score, in_t, out_t):


RAV-RUN2-R1-F004 [medium] data-validation — Missing scores become incorrect answers and stay in aggregate denominators. Missing token data becomes measured zero instead of being omitted.

Invalid scores can enter the failure report. Incomplete token usage can be omitted, with token means based only on complete values.

mrshu · 2026-08-06T04:33:46Z

+    return EvaluationResult(
+        evaluation_result_id=rid,
+        evaluation_name=name,
+        source_data=_source_data(task, agg.n, subtask),


RAV-RUN2-R1-F005 [medium] metric-identity — Each task receives a different metric ID even though every result is canonical accuracy. Cross-source joins by metric ID become fragmented.

accuracy can remain the metric ID while evaluation_name distinguishes tasks.

mrshu · 2026-08-06T04:33:46Z

+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description='Convert kensho/WILD-raw to Every Eval Ever.')
+    p.add_argument('--parquet', nargs='*', default=None,


RAV-RUN2-R1-F011 [medium] cli — A bare --parquet flag yields an empty list. The adapter treats that as no local input and silently starts the full remote 15-shard conversion.

The option can require at least one path.

Relocates the adapter to every_eval_ever/adapters/wild/ for the post-evaleval#218 layout, moves its README row to the adapters table, and relocks the wild extra.

Publication (F001, F010, P001): stage every sidecar into a temporary datastore tree, link it with the canonical repository-relative path, and publish through the shared publish_evaluation_logs, which re-validates and re-checksums the staged bytes and rolls back what it created. Publication is per log so peak memory stays at one model x benchmark, and files created by earlier logs are removed if a later one fails. A populated output directory is now an error rather than a second copy of every evaluation_id; --replace-existing removes the previous set, and only once the replacement is fully staged. Memory (F003): read bounded record batches instead of whole row groups. A WILD shard is a single 500,000-row row group, so --max-instances previously still materialized every selected column for all of them. Data validation (F004): a row whose score is not a usable 0/1 enters the failure report and leaves the aggregate instead of counting as a wrong answer; incomplete token usage is omitted rather than measured as zero, and token means cover only the rows that carried usage. The run exits non-zero after publishing whatever was valid. Metric identity (F005): every result keeps the canonical `accuracy` metric id; evaluation_name distinguishes the tasks. Instance output (F007): output.raw holds the assistant turns only, empty when there are none, and answer_attribution.source names where the parsed answer came from. Provenance (F008, F009): a remote run stops rather than falling back to the mutable main ref, and a local run without --evaluation-timestamp stops rather than keying evaluation_id on now(). CLI (F011): --parquet requires at least one path. Docs (F002): smoke commands use a data/<collection> output path and a quoted fixed-depth file glob.

mrshu

⚒️ review-anvil report

Review decision: COMMENT — This pass is comment-only, and replacement publication still risks losing valid prior output.
Result: The adapter now resolves most earlier review items well. Four focused suggestions remain for publication, pinning, and CLI safety.
Scope: Add a streaming WILD-raw adapter for aggregate and optional item-level Inspect AI records.
Checks: 4 concerns checked; 3 confirmed and 1 ruled out.
Second check: 2 reviewers checked 4 findings; all 4 stayed, and 3 fix paths were clarified.

Earlier review comments (14 grouped concerns)

Still present: replacement publication (RAV-RUN2-R1-F010) and metadata-failure pinning (RAV-RUN2-R1-F008) remain relevant at the moved adapter path.
No longer relevant: instance linkage (RAV-RUN2-R1-F006) was checked against the current schema guidance. The documented leaf-only choice is permitted.
Fixed: sidecar paths, scorer/output semantics, local provenance, memory bounds, score and token validation, canonical metric identity, --parquet, dependency packaging, tests, and README validation commands.

What I noticed

ID	Priority	Topic	Code location	What I noticed
RAV-RUN2-R1-F010	high	publication	`every_eval_ever/adapters/wild/adapter.py:660`	Replacement deletes every prior file for selected model directories before new publication. Partial inputs can remove absent benchmarks, and later failures cannot restore deleted files.
RAV-RUN2-R1-F008	medium	revision pinning	`every_eval_ever/adapters/wild/adapter.py:516`	After metadata failure, an explicit symbolic revision can reach both remote passes when an evaluation timestamp is also supplied.
RAV-RUN3-R1-F001	medium	models CLI	`every_eval_ever/adapters/wild/adapter.py:678`	A bare `--models` disables filtering. A fully unmatched filter exits successfully after publishing zero logs. (inline)
RAV-RUN3-R1-F002	medium	output path	`every_eval_ever/adapters/wild/adapter.py:608`	An arbitrary output path loses its final component, so publication and replacement can target a different `wild` directory. (inline)

Things to try

[high] publication — New output can be staged and checked while prior adapter-owned files remain recoverable. Replacement can use logical evaluation identity and preserve unrelated tasks. (RAV-RUN3-R1-P001; covers RAV-RUN2-R1-F010)
[medium] revision pinning — Symbolic refs can resolve once to info.sha. A metadata-failure fallback can accept only a full SHA with an explicit timestamp. (RAV-RUN2-R1-P008; covers RAV-RUN2-R1-F008)
[medium] models CLI — --models can require one value. A selection with no source matches can fail before stale-file discovery or publication. (RAV-RUN3-R1-P002; covers RAV-RUN3-R1-F001)
[medium] output path — The documented data/wild shape can be validated before metadata lookup, reads, or replacement scans. (RAV-RUN3-R1-P003; covers RAV-RUN3-R1-F002)

Run details

Target: PR #203 (add-wild-adapter, 7 files, +1137/-1) at ed99266c74958f79249a2d0ac68618c0b76b7f8b
Run ordinal: 3
Rounds: 1/1 completed; adaptive off in review-only mode
Mix: 3 codex-exec
Focus: correctness, maintainability, simplicity, production blast radius, and constructive suggestions
Earlier review comments: 14 grouped concerns; 2 still present, 1 ruled out, and 11 fixed
Finding counts: 0 critical, 1 high, 3 medium, 0 low, 0 nit
Checks: concerns=4; confirmed=3, ruled-out=1
Second check: targeted; reviewers=2; kept=4, clarified=3, removed=0; approval unchanged

Reviewed with review-anvil.

mrshu · 2026-08-07T16:52:49Z

+    p.add_argument('--output-dir', type=Path, default=Path(DEFAULT_OUTPUT_DIR))
+    p.add_argument('--limit-shards', type=int, default=None,
+                   help='Only read the first N shards (for smoke runs).')
+    p.add_argument('--models', nargs='*', default=None, help='Filter to these model ids.')


RAV-RUN3-R1-F001 [medium] cli — A bare --models parses as an empty list, which disables filtering and starts the full source run. A nonempty filter with no matches instead exits successfully with zero logs.

Requiring one or more values would block the first path. A no-match check can stop before existing-output discovery or publication.

Both fixed in a5bef76. --models is nargs='+', so a bare flag is an argument error instead of [] → no filtering → a full remote conversion. An entirely unmatched selection raises straight after aggregation — before the replacement scan and before publication — naming the ids, since nothing would be published and that is a typo rather than an empty refresh; a partial match warns and publishes what matched. test_models_filter_matching_nothing_publishes_nothing, test_bare_models_flag_is_an_error. Full reply

mrshu · 2026-08-07T16:52:49Z

+        file_uuids.append(str(uuid.uuid4()))
+
+    # Checked before the instance pass so a rejected rerun costs nothing.
+    stale = existing_records(args.output_dir.parent, logs)


RAV-RUN3-R1-F002 [medium] output-path — The code passes only args.output_dir.parent to lookup and publication, then derives wild below it. An arbitrary path such as /tmp/intended therefore targets /tmp/wild while messages name the requested path.

Early validation of the documented data/wild shape would keep lookup, replacement, and status output on one destination.

Fixed in a5bef76. resolve_base_output_dir() is now the first statement in run() — ahead of the metadata lookup, any parquet read, and the replacement scan — and requires --output-dir to end in the collection directory publication writes into. Your /tmp/i... case is exactly it: the run would have published into <parent>/wild/ while every message named the directory asked for, and the replacement scan would have read that other directory too. test_output_dir_must_be_the_collection_directory also asserts the rejected run wrote nothing. Full reply

borgr · 2026-08-08T03:33:28Z

Thanks — this was a useful pass. All eleven findings are addressed in
ed99266, except F006, which I'm declining with a reason below. CI is green on
that head (core / locked, core / loose, full / locked, full / loose).

P001 / F001 / F010 — publication

Rewritten to go through the shared machinery rather than writing sidecars in place:

F001 (sidecar link was a basename; the link was built after the files
existed). The link is now the canonical repository-relative path from
helpers.io.datastore_repo_file_path — data/wild/<developer>/<model>/<uuid>_samples.jsonl
— and nothing lands in the datastore until the whole conversion is staged.
run() writes every sidecar into a TemporaryDirectory, attaches
DetailedEvaluationResults (path + sha256 + row count), and only then calls
converters/common/publication.publish_evaluation_logs, which re-validates each
log, re-checksums the staged bytes and refuses to overwrite. A test pins the exact
file_path string and the checksum against the published file's bytes.
F010 (fresh-uuid reruns silently duplicate; partial batches on failure).
existing_records() now looks for anything already published for the models this
run will write, and a populated target is an error naming
--replace-existing, checked before the expensive instance pass so a rejected
rerun costs nothing. Stale files are unlinked only once the replacement is fully
staged, and publish() removes anything earlier logs created if a later one
fails. Two tests: the rerun rejection (output byte-identical after the refusal,
then replaced-not-accumulated with --replace-existing), and a monkeypatched
mid-batch failure asserting no *.json* survives.

One deliberate divergence in how: I publish per log rather than one batched
call. publish_evaluation_logs buffers every artifact's bytes in
_PreparedArtifact.content before creating any file, so a single call over ~1,700
logs would hold the entire sidecar corpus in memory. Per-log calls bound that to one
model×benchmark, with cross-log rollback in publish(). A streaming or chunked mode
in the shared publisher would be the better long-term fix — happy to open that
separately if you want it there rather than here.

F003 — memory

Reads are now bounded by iter_batches(..., batch_size=BATCH_ROWS) (20,000 rows)
instead of per row group, which matters because each WILD shard is a single
500,000-row row group — --max-instances 100 previously still allocated 500k rows.
test_iter_batches_bounds_rows_per_read asserts the batch sizes. The sidecar sha256
is accumulated over the bytes as they are appended, so no sidecar is ever re-read or
held whole (~7.5M instance rows at full scale).

F004 — score and token handling

A row whose score is not a usable binary value no longer becomes a wrong answer:
it leaves the numerator and the denominator, is skipped in the sidecar, and is
named in adapter_reports/wild_failures.json via SourceConversionResult; the
command then exits non-zero (raise_if_incomplete). The report is written before
publication so a publication error can't take the accounting with it.
Incomplete token usage is omitted rather than measured as zero: no token_usage on
that instance, and the row is out of the aggregate means, with
n_items_with_token_usage published alongside them as the divisor.

Worth stating plainly: this is a guard, not a repair of observed corruption. I
scanned shard 0 in full — 0 rows with a null score, 0 with null token counts — so
on today's snapshot these paths don't fire. I still think they belong in, because the
old behaviour turned a future null into a silently plausible wrong number rather than
an error, but you should read the change as defensive.

F005 — metric identity

Every result now carries the registry's canonical accuracy
(was wild.<task>.accuracy, which fragmented the cross-source join). The task
identity stays in evaluation_name (wild.<task>[.<subtask>]) and source_data,
which is where it belongs. Pinned by an assertion that the whole metric_id set on a
log is exactly {"accuracy"}.

F006 — instance linkage → declining, with a comment in the code

Instances still attach to the finest-grain result only. The skill's
reference/instance-level.md names leaf-only-with-a-comment as one of its two
sanctioned options ("either attach instances to the overall too (≈doubles rows) or
leave a comment that leaf-only is intentional"), and here the trade is lopsided: in WILD every item belongs to
exactly one subtask, so re-emitting each instance under the benchmark overall
would duplicate ~7.5M rows and add no information a GROUP BY can't recover. The
rationale is stated in make_instance so the choice is visible where the code makes
it. If you'd rather the repo standardise on both-grain linkage, say so and I'll change
it — but I'd rather that be a repo-wide decision than a WILD-only one.

The rest

#
F002	docs	The smoke command passed a directory to a file-only validator. Now a quoted fixed-depth glob: `validate '/tmp/eee-wild/data/wild///.json'`, verified against the real output (30 files, 0 invalid / 0 errors / 0 warnings).
F007	instance output	`output.raw` is the assistant turn(s) — the model's full generation — with the `[extracted]` fallback removed, so a row with no assistant turn is an empty list rather than the scorer's parsed answer wearing the generation's clothes. The parsed value stays in `answer_attribution.extracted_value`, with `source` naming where it came from (`answer` / `scores.<scorer>.answer`) and `extraction_method` = the real Inspect scorer. A test asserts the generation differs from the extracted value.
F008	revision pinning	A remote run that cannot resolve `main` to a concrete commit now stops instead of reading the mutable ref (which would let the aggregate and instance passes see different data). `--revision <sha>` pins it explicitly.
F009	local provenance	`--evaluation-timestamp` is now required for local `--parquet` runs: `evaluation_id` is keyed on it, so the `now()` fallback gave identical reruns different logical identities. A local run also records a local marker rather than a false `dataset_revision`.
F011	cli	`--parquet` is `nargs='+'`, so a bare `--parquet` is an argument error rather than a silent fall-through to a full remote conversion.

Verification

pytest tests → 401 passed, 20 skipped; tests/test_wild_adapter.py 6 → 12
tests; ruff check clean.
Live end-to-end on shard 1 pinned to 06af2cc48c351a9c67b9caec4dff7d1c43d451dd:
111,353 source rows → 27 aggregates + 3 sidecars (2,000 instance rows at the cap),
0 failures, and validate on the published tree returns 0 invalid, 0 errors, 0
warnings across 30 files. Sidecar links are full data/wild/... paths; a rerun
without --replace-existing is refused.
I also trimmed the module's prose: comments+docstrings went 21.9% → 14.9% of lines
(peer adapters sit at 2.5–10.2%). What's left is one- to three-line invariant
statements, each tied to one of the findings above.

One thing this breaks downstream

The companion datastore submission
(discussions/174,
still a draft, 1,755 records) was generated by the pre-review adapter: its records
carry the old per-task metric ids (wild.winogrande.accuracy), no
n_items_with_token_usage, and no sidecar link. I'll regenerate it from this adapter
before the set is merged. evaluation_id is unaffected by these fixes, so the
regeneration replaces data/wild/ rather than adding a second copy of every model.

…leting Replacement keyed on logical evaluation identity (the evaluation_id without its timestamp) read from each prior file, instead of globbing every file in the selected model directories: a run over one benchmark no longer removes the other benchmarks that share the model's directory. The removal also moves after publication, so a run that fails partway leaves the previous refresh whole. A metadata-lookup failure now accepts an explicit --revision only when it is a 40-character SHA; a branch or tag can move between the aggregate and instance passes, so it cannot stand in for the pin the lookup failed to produce. --output-dir must end in the collection directory, validated before any lookup, read or replacement scan. --models takes nargs='+' so a bare flag errors rather than silently converting every model, and a selection the source cannot match at all is an error rather than an empty success.

borgr · 2026-08-08T11:00:25Z

Run 3's four findings are fixed in a5bef76 (CI green on all four jobs). Two of them
are carry-forwards I reported as done in
my previous comment
and hadn't actually finished — F010 and F008 — so this starts by correcting that:
you re-raised both against the exact head I claimed them fixed on, and you were right
on both.

RAV-RUN2-R1-F010 · publication — replacement could delete what it wasn't replacing

What the code actually did:

stale = existing_records(args.output_dir.parent, logs)   # dir.glob('*.json*') per model
...
for path in stale:
    path.unlink()
published = publish(logs, file_uuids, args.output_dir.parent, staging_root)

Both halves of your finding follow from those three lines. The scan was keyed on the
directory, which names only the model — and a model's directory holds every
benchmark that model was evaluated on, so an input covering fewer of them (a
--limit-shards smoke run, a subset of local --parquet shards) deleted the
benchmarks it had nothing to republish. And the unlink ran before publish, so a
failure in publication left the directory emptier than it started with nothing to
restore from. My earlier comment described the order I intended ("unlinked only once
the replacement is fully staged") rather than the one that was written; staged is not
published.

Fixed along the fix path you gave — logical evaluation identity, unrelated tasks
preserved:

logical_identity() drops the trailing commit-date segment of an evaluation_id,
leaving wild/<model>/<benchmark>. Replacement keys on that, so re-pinning the
dataset still supersedes the same pair's earlier copy even though its id changed.
superseded_records() opens each candidate and reads its own evaluation_id
instead of matching on the path, so the set contains only files this run
republishes; each aggregate brings its _samples.jsonl sidecar with it. A file
whose evaluation_id cannot be read is warned about and left in place — the
adapter does not delete something it cannot identify.
The unlink loop moved after publish(). Prior files go only once their
replacements exist on disk, and publish()'s existing rollback removes whatever a
failed run created — so the tree holds one complete set at every moment: the
previous refresh if this run dies, this one if it succeeds.

Two tests: test_replacement_supersedes_only_the_benchmarks_it_rewrites (publish
mmlu for a model, then republish arc_challenge for the same model with
--replace-existing; the mmlu aggregate and sidecar survive, arc_challenge is
replaced rather than accumulated) and
test_a_failed_replacement_leaves_the_previous_refresh_in_place (monkeypatched
mid-batch failure during a replacement; the directory listing afterwards is identical
to before it ran).

RAV-RUN2-R1-F008 · revision pinning — a symbolic ref could outlive the failure

Also correct, and also mis-reported by me: I fixed the no-override branch — which
now stops rather than reading mutable main — and left the override branch accepting
whatever it was handed, --revision main included. With --evaluation-timestamp
supplying the date the failed lookup would have given, the run then proceeded with a
moving ref across both passes, which is the exact thing the strict branch exists to
prevent.

The metadata-failure fallback now accepts an override only when it is 40 hex
characters. A SHA is already the pin, so the only thing the failure costs is the
commit date — and losing that makes resolve_eval_timestamp require
--evaluation-timestamp rather than guess, which is the "with an explicit timestamp"
half of your fix path, enforced rather than assumed. Anything symbolic raises with the
reason and the instruction to pass the SHA.
test_symbolic_revision_cannot_stand_in_for_a_failed_pin covers all three branches.

RAV-RUN3-R1-F001 · models CLI

Both parts, in the order you asked for them. --models is nargs='+', so a bare flag
is an argument error rather than [] → no filtering → a full 15-shard remote
conversion. And a selection the source matches none of now raises immediately
after aggregation — before the replacement scan and before any publication — naming
the ids, because nothing would be published and that is a typo, not an empty refresh.
A partial match warns and publishes what matched, since those rows are real.
test_models_filter_matching_nothing_publishes_nothing and
test_bare_models_flag_is_an_error.

RAV-RUN3-R1-F002 · output path

resolve_base_output_dir() is the first statement in run() — ahead of the metadata
lookup, any parquet read, and the replacement scan, as the fix path asks.
--output-dir has to end in the collection directory publication writes into
(wild); anything else exits naming the shape to pass. That closes the case you
described: /tmp/x/foo would have published into /tmp/x/wild/ while every message
said foo, and the replacement scan would have read that other directory too.
test_output_dir_must_be_the_collection_directory also asserts the rejected run wrote
nothing.

Verification

pytest tests → 407 passed, 20 skipped (tests/test_wild_adapter.py 12 → 18
tests); ruff check clean; CI green on a5bef76. The datastore submission
(discussions/174,
still a draft) is regenerated from this adapter before the coordinated set merges —
it predates this round as well as the last one.

15 of WILD's 65 model ids carry no namespace ("gpt-4o", "nova-pro", "llama-3.1-8b"). build_log derived the developer from the id prefix, so those became 'unknown', which the datastore path helper rejects — the full 65-model run aborted after aggregating all 1,755 groups and published nothing. Delegate to every_eval_ever.helpers.developer.get_developer, which keeps the prefix for a namespaced id and pattern-matches a flat one, and stop the run naming the offending ids if any still resolve to nothing. Two tests cover the four publishers the flat ids resolve to and the refusal; _task_rows is shared with the existing single-task fixture.

borgr · 2026-08-09T11:41:23Z

Ran the adapter at full scale for the first time and it did not work. Fixed in
eb785624a (CI green on all four jobs). Re-checked the companion datastore submission
too, and what I told you about it in the two comments above was incomplete.

The full run published nothing

build_log derived the publisher from the model id's prefix:

developer = model.split('/')[0] if '/' in model else 'unknown'

15 of WILD's 65 model ids carry no namespace — gpt-4o, nova-pro,
llama-3.1-8b, claude-3.5-sonnet. Those became developer='unknown', which
helpers.io.require_identity refuses as a datastore path component, so the run
aggregated all 1,755 (model, benchmark) groups over 7.2M item rows and then died on
the first of them without writing a file.

Not a regression — git log -L puts that line in the adapter's first commit. It
survived every verification round because each round was a single-shard smoke run, and
the shard I used holds only 01-ai/Yi-*. Every id I ever pushed through publication
was namespaced. The "0 invalid, 0 errors, 0 warnings across 30 files" in my earlier
comments is true and was never evidence about this.

The fix delegates to helpers.developer.get_developer — prefix for a namespaced id, so
the other 50 models are unchanged, pattern match for a flat one — and the run now stops
naming any id that still resolves to nothing, rather than letting a placeholder reach
the path helper, whose own refusal (model_info.developer must be known) does not say
which model it was. The 15 resolve to:

publisher	records	ids
`openai`	135	`gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`, `gpt-4o`, `gpt-4o-mini`
`anthropic`	108	`claude-3-haiku`, `claude-3-sonnet`, `claude-3.5-haiku`, `claude-3.5-sonnet`
`meta`	108	`llama-3.1-70b`, `llama-3.1-8b`, `llama-3.2-1b`, `llama-3.2-3b`
`amazon`	54	`nova-lite`, `nova-pro`

test_model_ids_without_a_namespace_still_name_a_publisher pins the four directories
and validates each record; test_a_model_that_names_no_publisher_is_named_in_the_error
pins the refusal and that it wrote nothing.

First full-scale run

7,237,945 item rows → 1,755 aggregates across 65 models and 17 publishers in 68 s,
pinned to 06af2cc48c351a9c67b9caec4dff7d1c43d451dd. All 1,755 pass
validate_file(..., run_semantic_checks=True) at their real datastore paths, with
available_files and read_repo_file wired to the tree so the companion checks
resolve. They also produce zero warnings under #230's registry-alias check — all 17
directory names agree with the eval-card-registry.

Correcting the record on discussions/174

I called it "generated by the pre-review adapter" and named three missing fields. It is
worse than that, and my first re-count of it was wrong in the other direction: a
truncated tree listing had me believing it held 921 files across 6 publishers. It holds
all 1,755, across six differences:

	discussions/174	this adapter
`schema_version`	`0.2.2`	`0.3.0`
`metric_id`	`wild.<task>.accuracy`	`accuracy` (F005)
`deployment_type` / `model_availability`	absent	`unknown`
`dataset_revision`	absent	pinned SHA (F008/F009)
`n_items_with_token_usage`	absent	present
publisher directory	405 records under `data/wild/unknown/`	the four above

The missing deployment axes mean every record there fails the current validator — the
same two fields that fail 84% of the published datastore (#242). And the 405 under
unknown/ are this bug, published: the old code wrote that directory itself, before
the publication rework routed writes through require_identity, which is why today it
crashes where it used to file a placeholder.

So the regeneration replaces all 1,755 rather than patching the 405 — every record
differs. evaluation_id is unaffected by any of this, so logical_identity() still
supersedes each wild/<model>/<benchmark> pair rather than adding a second copy.

One thing it does not change: the draft is aggregate-only, and so is the
regeneration. Instance sidecars are ~7.5M rows at full scale, so whether the submission
should carry them is a separate call rather than something I'd fold into this — say the
word if you want them and I'll size it. Either way it stays a draft until this merges.

It is a usage note, not a registry follow-up.

mrshu · 2026-08-10T23:17:31Z

⚒️ review-anvil report

Review decision: COMMENT — This run is comment-only; one earlier high-priority publication issue remains.
Result: The latest revision resolves the other canonical findings. Replacement can still discard the last complete refresh or leave mixed generations.
Scope: Add the WILD-raw aggregate and optional instance-sidecar adapter with safe repeat publication.
Checks: The verifier could not fetch the pinned head. I confirmed both paths directly from the GitHub file at fc12d2e.
Second check: Targeted, 2 reviewers; the finding stayed high and the recovery plan was simplified.

Earlier review comments (13 canonical items)

RAV-RUN2-R1-F010 [high] publication replacement — Still present. Source failures can replace valid output, and retirement failure can leave mixed generations.
RAV-RUN2-R1-F001 through F009 and F011 are fixed or resolved by the documented leaf-only policy.
RAV-RUN3-R1-F001 and F002 are fixed.
Legacy dependency, scorer-alignment, output, provenance, and standard-error test notes are fixed.

What I noticed

RAV-RUN2-R1-F010 [high] publication replacement every_eval_ever/adapters/wild/adapter.py:731 — run() publishes new logs and removes old files before raise_if_incomplete(). A source-row failure can therefore replace a complete refresh before the command exits non-zero. Old files are also removed one at a time after publisher rollback ends. A removal failure can leave both generations in canonical paths.

Things to try

[high] publication replacement — Incomplete conversion results can stop before canonical publication. Complete replacements can quarantine old aggregate and sidecar files on the same filesystem. A retirement or publication failure can restore them and remove new files. Quarantine cleanup can remain best-effort after the canonical transition commits. (RAV-RUN4-R1-P001; covers RAV-RUN2-R1-F010)

Run details

Target: PR Add WILD-raw item-level adapter (every_eval_ever/adapters/wild) #203 at fc12d2ec65c4160a045b9e4413749d2e2538277b (7 files, +1365/-1)
Run ordinal: 4
Rounds: 1/1 completed; adaptive off
Mix: 3 codex-exec
Focus: correctness, maintainability, simplicity, production blast radius, converter content checks, and constructive suggestions
Earlier review comments: 13 canonical items; 1 still present, 12 fixed or resolved
Finding counts: 0 critical, 1 high, 0 medium, 0 low, 0 nit
Checks: 1 concern checked; independently confirmed after verifier access failed
Second check: targeted; reviewers=2; kept=1; clarified=1; removed=0
Fixes applied: 0 (review-only)

Reviewed with review-anvil.

borgr mentioned this pull request Jul 20, 2026

seed(benchmarks): resolve WILD benchmark slugs (+ AI2-ARC/RACE aliases, 5 new canonicals) evaleval/eval-card-registry#46

Merged

borgr mentioned this pull request Jul 20, 2026

seed(models): add 19 BenchPress model-slug aliases (gate-verified) evaleval/eval-card-registry#39

Merged

mrshu reviewed Jul 21, 2026

View reviewed changes

chore(wild): relock uv.lock for the new wild extra (fixes locked CI)

2213d53

mrshu reviewed Jul 29, 2026

View reviewed changes

mrshu reviewed Aug 6, 2026

View reviewed changes

borgr added 2 commits August 7, 2026 10:30

Merge main into add-wild-adapter

db974bc

Relocates the adapter to every_eval_ever/adapters/wild/ for the post-evaleval#218 layout, moves its README row to the adapters table, and relocks the wild extra.

mrshu reviewed Aug 7, 2026

View reviewed changes

borgr changed the title ~~Add WILD-raw item-level adapter (utils/wild)~~ Add WILD-raw item-level adapter (every_eval_ever/adapters/wild) Aug 8, 2026

borgr added 2 commits August 8, 2026 01:44

wild README: name the runs whose input covers only some benchmarks

a5bef76

This was referenced Aug 8, 2026

Add Open Medical-LLM Leaderboard adapter #204

Open

Add BenchPress score-matrix adapter #197

Open

borgr added 2 commits August 10, 2026 08:50

Merge origin/main into add-wild-adapter

52c1741

Move the instance-read cost note out of the canonicalization list

fc12d2e

It is a usage note, not a registry follow-up.

Uh oh!

Conversation

borgr commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What / source

Review lane

Checklist

Decisions & coverage

Uh oh!

mrshu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

borgr commented Jul 21, 2026

Uh oh!

mrshu left a comment

Choose a reason for hiding this comment

⚒️ review-anvil report

Earlier review comments

What I noticed

Things to try

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

borgr commented Jul 30, 2026

Uh oh!

mrshu left a comment

Choose a reason for hiding this comment

⚒️ review-anvil report

Earlier review comments

What I noticed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrshu left a comment

Choose a reason for hiding this comment

⚒️ review-anvil report

What I noticed

Things to try

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

borgr commented Aug 8, 2026

P001 / F001 / F010 — publication

F003 — memory

F004 — score and token handling

F005 — metric identity

F006 — instance linkage → declining, with a comment in the code

The rest

Verification

One thing this breaks downstream

borgr commented Jul 20, 2026 •

edited

Loading