Add BenchPress score-matrix adapter

[borgr]

What / source

Converts the BenchPress score matrix (microsoft/benchpress-score-matrix) into
aggregate EEE records under data/benchpress/.

BenchPress is an aggregator: it re-reports scores scraped from provider blogs,
tech reports, model cards, leaderboards and third-party aggregators, each cell
carrying its own citation (reference_url) and provenance (source_type). So it is
handled like llm_stats — source_type=documentation, source_role=aggregator, and
logs split by evaluator_relationship.

Adapter lives at every_eval_ever/adapters/benchpress/
(rebased onto the post-#218 layout; utils/ no longer exists).

Review lane

• Fast
• Needs a human — evaluator_relationship is a provenance claim on every
  record, and this adapter takes it from who published the citation, not from
  what kind of document it is. first_party needs a provider-authored
  source_type and a citation on a domain carrying the model provider's
  name, which gives {other: 2,654, third_party: 980, first_party: 841} scores
  across {third_party: 139, other: 138, first_party: 49} logs. Worth a
  maintainer's eyes: the evidence rule is the substance of the change, not the
  mapping table.

Design agreed in: n/a — the mapping questions are answered inline in the review
thread below.

Checklist

• python -m every_eval_ever validate clean at the final
  data/benchpress/<dev>/<model>/ path — 326 files, 0 invalid, 0 errors,
  0 warnings, exit 0
• every source row is accounted for individually in
  adapter_reports/benchpress_failures.json — unconvertible rows with a
  non-zero exit, excluded rows itemized without one. The report is written on
  every run, before publication, and swapped in atomically, so no earlier run's
  report can be read as this one's and an interrupted write cannot truncate a
  complete one.
• offline unit tests added (24, in
  tests/test_benchpress_adapter.py);
  full pytest tests green (435 passed, 20 skipped)
• ruff check clean
• model/benchmark ids resolve in the registry — not yet. model_info.id is
  <provider-slug>/<benchpress-slug> and the raw slug is kept in
  additional_details.benchpress_model_id; ids become canonical downstream via
  the eval-card-registry, same as the other aggregator adapters.
• content spot-checked against the live snapshot: citations resolve to the
  documents they claim, no benchmark is double-counted across relationship
  splits, evaluation_id is stable for a given (relationship, model, snapshot)

Decisions & coverage

• Decision / where: evaluator_relationship, relationship_from_score +
  _provider_publishes.
  Chose: third_party for an independent source_type; first_party only for a
  provider-authored source_type whose citation is hosted on a domain carrying
  the scored model's provider name (openai.com, cdn.amazon.science,
  moonshotai.github.io, x.ai); other otherwise.
  Instead of: trusting source_type alone (the first version), or trusting it when
  the citation covers exactly one provider (the second).
  Why: source_type records what kind of document a citation is, not who
  published it, and the one-provider heuristic conflated "cited for one provider's
  models" with "published by that provider". The live snapshot falsifies it in both
  directions: a Google gemini-2.5-flash score cited to
  Qwen's model card was
  labelled first_party, while
  openai.com/index/introducing-gpt-5-5
  — 46 OpenAI cells plus 8 competitor cells on one page — was demoted wholesale
  because the page tabulates competitors. A domain has an owner, so it names a
  publisher; a path on a shared host does not (the org in huggingface.co/Qwen/…
  is path, and storage.googleapis.com serves anyone's bucket, which is why its
  68 cells stay other). Every one of the 12 hosts that now yields first_party
  is the provider's own. The rule under-claims rather than guesses: a provider
  publishing on a hostname that does not spell its name
  (lf3-static.bytednsdoc.com for a ByteDance model) is other. On that same
  page, BenchPress already tags the competitor cells source_type=third_party
  itself, so they come out third_party — the export made the distinction and the
  old heuristic discarded it. No score value changes and no cell is dropped.
  Confidence: high. General? yes — any aggregator carrying per-cell citations
  has this problem, and document type is never document ownership.

• Decision / where: duplicate results, make_logs grouping.
  Chose: key each log's results by evaluation_result_id; an identical repeat
  collapses, a second differing result for the same benchmark is a reported
  failure.
  Instead of: the de-dup pass this adapter had, which kept whichever row came
  first and dropped the other with no entry anywhere.
  Why: evaluation_result_id is the schema's "recommended deterministic join key",
  so one log carrying two results for one benchmark would break the join — but a
  row that cannot be represented is a row to report, not to lose, and the silent
  version made total_source_records reconcile against a count that had quietly
  shrunk. Zero live rows hit either path (4,903 − 410 − 18 = 4,475 exactly), so
  this closes a latent hole rather than repairing observed loss. Giving the id a
  citation suffix was the alternative; it would change the join key on all 4,475
  cells to fix a zero-occurrence case, or make identity depend on whether a
  sibling row exists.
  Confidence: high. General? yes — the same silent de-dup shape is easy to write
  in any adapter that groups cells into logs.

• Decision / where: row filtering, ACCEPTED_AUDIT_STATUSES.
  Chose: export only audit_status in {verified, verified_third_party}.
  Instead of: exporting every row.
  Why: BenchPress excludes dropped / needs_review / flagged from its own
  canonical matrix, so re-publishing them would assert more than the source does.
  These are recorded as exclusions (not failures — a policy exclusion must not fail
  the run); --include-unaccepted exports them.
  Confidence: high. General? no.

• Decision / where: score-vs-declared-range disagreement, _within_bounds.
  Chose: report the row as unconvertible.
  Instead of: rescaling, or widening MetricConfig to fit the value.
  Why: the export mixes scales inside a single benchmark — mt_bench_101 declares
  1–10 and carries values up to 90.2, fleurs WER declares [0,1] and carries up to
  86.4, creative_writing_v3 Elo declares [1000,2000] and carries 87.5. The record
  cannot state both, and a rescale would be a guess about which cells are on which
  scale.
  Confidence: high. General? yes — worth a note in the conversion skill.

• Decision / where: snapshot pinning, resolve_revision / fetch_payload.
  Chose: resolve whatever was asked for — the default tip, a branch, a tag — to
  one commit sha, read all four files at it, and record that sha as
  benchpress_dataset_revision; --revision replays a snapshot. A --revision
  that is already a full sha is used as given.
  Instead of: reading main per file, or splicing a supplied symbol straight into
  the four file URLs.
  Why: main moves, and four files are four requests, so a run could otherwise mix
  revisions — and a supplied branch or tag has exactly the same problem, plus it
  would be recorded as provenance that can later point somewhere else.
  metadata.json alone was not enough — the manifest can lag the CSVs. Resolving a
  full sha would only cost a request, since it cannot move.
  Confidence: high. General? yes (already the pattern in vectara).

• Decision / where: per-row failure boundary, _parse_scores / make_logs.
  Chose: keep the score as the source wrote it and parse it inside a per-row
  boundary that catches only what bad source data raises (ValueError,
  TypeError, pydantic ValidationError).
  Instead of: parsing during CSV read (the previous behaviour), or catching
  Exception per row.
  Why: parsing at read time happened before SourceConversionResult existed, so one
  unusable cell ended the run without recording the row it came from — the failure
  mode the accounting is meant to prevent. Catching broadly would file adapter bugs
  as bad source data, so anything else still surfaces as a crash. total_records
  stays the full source row count, so nothing is silently dropped to make the
  numbers reconcile. The descriptive numerics (params_total_M, num_problems)
  only ever reach a record as text via _str_map, so they keep unparseable text
  rather than failing.
  Confidence: high. General? yes — every adapter that parses before it can
  record has this.

• Decision / where: when the accounting report is written, write_conversion_report.
  Chose: unconditionally, atomically, and before publication. The non-zero exit
  stays conditional on failures.
  Instead of: only when a row failed (the previous behaviour).
  Why: an exclusions-only run printed a count and preserved nothing, and a clean run
  left an earlier run's report standing to be read as its own. It accounts for the
  conversion, so a publication error is when it is most worth having on disk.
  Confidence: high. General? yes — a stale ledger reads as current in any adapter
  that writes one. It is adapter-local in the same shape as Add Open Medical-LLM Leaderboard adapter #204's; the shared home
  is helpers/io.py, proposed as one follow-up covering both rather than slipped
  into either PR.

• Decision / where: publication, export_logs.
  Chose: the shared save_evaluation_logs, deleting the adapter's own writer.
  Why: that writer emitted bare Infinity, which strict_json_loads now rejects,
  so its output was no longer loadable; the shared path writes the JSON string
  "Infinity" and batches publication so a late failure leaves no partial tree.
  Confidence: high. General? no — it was this adapter's bug.

Coverage: 4,903 source scores → 326 logs / 4,475 scores. 410 excluded
(BenchPress's own non-accepted rows: 342 dropped, 61 needs_review, 7 flagged) ·
18 failed (7 rows whose model or benchmark id is absent from the export, 11 rows
outside their benchmark's declared range) — each itemized with its own row and
reason in adapter_reports/benchpress_failures.json, the failures with a non-zero
exit. No caps, no sampling. Re-verified live on this head (both a fresh fetch and
an --input-json replay, which agree): 4,903 → 326 logs / 4,475 scores, validate
all 326 passed, and the same 18 failures and 410 exclusions as before — no new
failure of any kind, including the new duplicate check.

Operator asked about policy calls? Yes, three, all resolved above and in the
review thread: the first_party provenance claim (changed), the 410-row policy
exclusion (kept, made visible), and the mixed-scale rows (rejected rather than
guessed at).

[Erotemic]

GPT 5.6 Review (3m 49s)

Findings

High — Do not identify tau-bench Pass^k as pass_at_k

make_metric_config() labels every pass_N result with:

metric_id='tau_bench.pass_at_k'

at utils/tau_bench/adapter.py:389.

These are Pass^k reliability metrics, not pass@k metrics. Pass^k measures repeated consistency—all k trials must succeed—whereas pass@k measures whether at least one of k attempts succeeds. They move in opposite directions as k increases. Tau-bench explicitly introduced Pass^k as its reliability metric. ([arXiv]1)

This is more than a display-name issue: metric_id is EEE’s stable join/deduplication identity, so downstream consumers can combine these scores with semantically incompatible pass@k results.

Fix: use a distinct identifier such as tau_bench.pass_pow_k or tau_bench.pass_hat_k, and retain {"k": k} as the parameter. Do not use pass_at_k.

---

High — Current voice submissions silently lose their interaction-quality metrics

make_results() reads only the top-level results object and emits pass_1–pass_4 plus cost. It never reads submission["interaction_metrics"].

Current tau-bench voice submissions contain per-domain latency, responsiveness, interruption, and selectivity metrics. The upstream submission model explicitly defines this field, and voice submission preparation computes it as part of the published result. A current published submission already contains this panel.

Because the adapter fetches the mutable upstream main branch and advertises voice support, merging it now would silently discard most of the voice-specific leaderboard measurements.

Fix: emit EvaluationResult entries for each per-domain interaction metric, with correct units and directionality:

• latency metrics: seconds, lower is better;
• response/yield/selectivity rates: proportion, higher is better;
• agent interruption rate: lower is better;
• preserve backing counts and interaction-metric version/config in details.

---

Medium — Score parsing accepts values prohibited by the source schema

parse_score() accepts anything convertible by float(), without checking finiteness or metric bounds. Consequently, values such as True, "125", -3, NaN, or Infinity can enter exported records.

The upstream schema requires Pass^k scores to be numeric and between 0 and 100, and costs to be nonnegative. EEE’s MetricConfig.min_score and max_score describe the range but do not validate ScoreDetails.score, so the contradictory record can still pass EEE model validation.

Fix: make parsing metric-aware, reject booleans and non-finite floats, enforce 0 <= pass_score <= 100 and cost >= 0, or validate each submission against the upstream submission schema before conversion.

Verdict

I would request changes for the Pass^k identity and missing voice metrics. The PR is also 13 commits behind current main, and its GitHub Actions runs contain no jobs because they remain in action_required, so there is no current CI execution to rely on.

[mrshu]

⚒️ review-anvil report

Review decision: COMMENT — The adapter is well documented and tested, and four data-selection or portability details would benefit from follow-up.
Result: One high-priority audit-status issue and three medium-priority provenance or serialization issues remain.
Scope: This PR adds a BenchPress score-matrix aggregator with citations, bounds, and snapshot metadata.
Checks: 3 concerns checked; 0 broad claims kept, 3 narrowed, 0 ruled out, 0 set aside.
Second check: targeted, 2 reviewers; 4 kept, 4 fix paths clarified, 0 set aside, 0 removed.

Earlier review comments

None.

What I noticed

• [high] data selection utils/benchpress/adapter.py:432 — The adapter exports every numeric row from scores_all, including rows outside BenchPress's accepted audit statuses. BenchPress currently reduces 4,903 audit rows to 4,493 accepted rows before canonical processing. (RAVF001; inline)
• [medium] evaluator relationship utils/benchpress/adapter.py:69 — tech_report always maps to first_party, although document type does not establish who evaluated the model. Cross-provider report rows can receive incorrect relationship provenance. (RAVF002; inline)
• [medium] JSON portability utils/benchpress/adapter.py:494 — Logs with inferred unbounded metrics contain bare Infinity tokens. Python accepts them, but standards-compliant JSON parsers reject the entire file. (RAVF003; inline)
• [medium] snapshot provenance utils/benchpress/adapter.py:223 — Metadata and three CSVs are fetched independently from mutable main. Metadata errors are ignored, so one run can mix revisions or lose its version anchor while still succeeding. (RAVF004; inline)

Things to try

Things to try (4 items)

• [high] data selection — Keeping scores_all while admitting only verified and verified_third_party rows would fix the narrow issue without changing canonical-selection scope. (RAVF001)
• [medium] evaluator relationship — Explicit report ownership can be compared with model provider. When ownership is absent, other avoids guessing from an arXiv or publisher host. (RAVF002)
• [medium] JSON portability — A shared schema representation for unbounded values can keep the normal writer and strict JSON compatibility. Until then, unsupported unbounded results can be counted and skipped. (RAVF003)
• [medium] snapshot provenance — One resolved Hugging Face dataset revision can pin all four live reads. Its revision can stay separate from the manifest's upstream source commit. (RAVF004)

Run details

• Target: PR #197 (add-benchpress-adapter, 5 files, +744/-0)
• Rounds: 1/1 completed; adaptive off; one reviewer returned empty output
• Mix: 3 codex-exec; 2 succeeded and 1 failed the empty-output check
• Focus: metric semantics, provenance, serialization, maintainability, and constructive suggestion-oriented language
• Earlier review comments: none
• Finding counts: 0 critical, 1 high, 3 medium, 0 low, 0 nit
• Checks: concerns=3; narrowed=3/ruled-out=0/set-aside=0
• Second check: targeted; reviewers=2; kept=4/clarified=4/set-aside=0/removed=0; approval changed no
• Set aside: 0 items

---

Reviewed with review-anvil.

[mrshu]

[high] data selection — Rows outside BenchPress's accepted audit statuses become normal results

The adapter reads scores_all and accepts every linked numeric row. BenchPress applies a verified or verified_third_party status filter before its canonical pipeline; preserving audit_status only as metadata does not enforce that boundary.

Keeping scores_all while applying the accepted-status allowlist would fix this narrow issue without changing representative or fill semantics.

[mrshu]

[medium] evaluator relationship — Document type alone does not prove first-party evaluation

Every tech_report row maps to first_party, but BenchPress permits a report from one provider to contain scores for another provider's model. The relationship field describes the evaluator's relationship to the model.

Explicit report ownership can be compared with the model provider. When that signal is absent, other avoids guessing from a publisher or arXiv URL.

[mrshu]

[medium] JSON portability — Unbounded metrics produce non-standard Infinity tokens

allow_nan=True writes bare Infinity or -Infinity. Python's loader accepts these tokens, but RFC-compliant JSON parsers reject affected files.

A shared schema representation for unbounded values can preserve the meaning and the standard writer. An adapter-local wire convention would create a second serialization policy.

[mrshu]

[medium] snapshot provenance — One export can combine files from different revisions

Metadata and the three CSVs are separate reads from mutable main. Metadata errors become an empty object, after which the adapter uses the current time and omits its promised version anchor.

Resolving one Hugging Face dataset revision can pin every live read. The dataset revision and manifest source_git_commit can remain separate provenance fields.

[borgr]

Thanks — all four are fixed, and running the conversion against the live snapshot
turned up a fifth. The adapter has also moved to
every_eval_ever/adapters/benchpress/ now that #218 removed utils/.

RAVF001 (high, audit status) — fixed as suggested. scores_all stays the input;
only audit_status in {verified, verified_third_party} is exported. Your 4,903 →
4,493 count is exact — I get verified 3,784 + verified_third_party 709 = 4,493,
with 342 dropped, 61 needs_review, 7 flagged. Those 410 are recorded as
exclusions rather than failures: BenchPress rejecting a row is its policy, not a
conversion error, so it must not fail the run. --include-unaccepted exports them.

One thing that came out of checking this, in case it's useful elsewhere:
audit_status == 'verified_third_party' ⟺ source_type == 'third_party_aggregator'
exactly, 709/709. So audit status carries no independent signal about who ran the
eval — it only says whether BenchPress accepts the row.

RAVF002 (relationship) — fixed, and it was broader than the finding. You scoped
it to tech_report; model_card and official_blog have the same defect, and by
volume they're worse:

source_type	rows called first_party	from a multi-provider citation
model_card	1,196	1,069
official_blog	1,157	437
tech_report	1,123	1,013

2,519 of 3,476. The concrete case: Claude scores cited to
arxiv.org/abs/2412.19437 — that's DeepSeek's V3 report, whose comparison table
BenchPress scrapes cell by cell. Calling those first_party says Anthropic reported
them.

Your "when ownership is absent, other avoids guessing" is what I implemented, with
citation breadth standing in for the absent ownership field: a provider-authored
document type is first_party only when every score citing that URL belongs to one
provider. Otherwise other. Outcome:

before: first_party 3,476 · third_party ~1,017
after:  other 2,550 · third_party 991 · first_party 945

That's a material change in what the same input produces, so I've moved the PR to
the Needs a human lane.

RAVF003 (bare Infinity) — fixed by deletion. The shared representation you
suggested already landed on main: an unbounded MetricConfig bound is the JSON
string "Infinity", and strict_json_loads now rejects the bare token. So this
adapter's local write_log (with allow_nan=True) wasn't just non-portable, it was
producing records the current validator rejects. It's gone; publication goes through
the shared save_evaluation_logs, which also batches so a late failure leaves no
partial tree. No need for the count-and-skip fallback.

RAVF004 (snapshot provenance) — fixed as suggested. The dataset commit sha is
resolved once and all four files are read at it, kept separate from the manifest's
source_git_commit as you noted:
source_metadata.additional_details.benchpress_dataset_revision. --revision <sha>
replays a snapshot. The swallowed metadata error is gone too — a missing sha now
raises and names --revision.

Fifth issue, not in the review: the export mixes score scales inside one
benchmark. With the four fixes in, 11 of 332 records failed validation.
canonical_setting.range is the benchmark's documented scale, but the cells don't
all honour it:

• mt_bench_101 declares [1, 10], carries 8.59 … 90.2
• fleurs (WER) declares [0, 1], carries 0.08 … 86.4
• creative_writing_v3 (Elo) declares [1000, 2000], carries 87.5

A record can't state both, and rescaling would be a guess about which cells are on
which scale — so these are reported as unconvertible source rows in
adapter_reports/benchpress_failures.json with the disagreement named, and the
command exits non-zero. Happy to revisit if BenchPress documents a per-cell scale.

Verification. 4,903 source scores → 332 logs / 4,475 scores; 410 excluded, 18
failed (7 unresolvable model/benchmark id + the 11 above). validate over all 332 at
data/benchpress/<dev>/<model>/: 0 invalid, 0 errors, 0 warnings, exit 0. Full
pytest tests: 400 passed, 20 skipped. ruff check clean. Tests added for each
fix, including that a two-provider citation is not first_party, that a dropped
row is an exclusion and not a failure, and that a published unbounded bound round-trips
through the strict parser.

[mrshu]

⚒️ review-anvil report

Review decision: COMMENT — This pass is comment-only, and one high-priority row-accounting gap remains.
Result: The adapter now handles the earlier selection, JSON, and default snapshot concerns well. Four focused suggestions remain.
Scope: Add a BenchPress score-matrix adapter with snapshot pinning, provenance policy, range checks, and complete source-row accounting.
Checks: 3 concerns checked; all 3 confirmed.
Second check: 2 reviewers checked 4 findings; all 4 stayed, and 3 fix paths were clarified.

Earlier review comments

• Fixed: accepted audit-status selection and strict JSON publication now follow the intended source and shared writer.
• Fixed: the default fetch resolves one immutable dataset SHA before reading all four files.
• Still present: the earlier evaluator-relationship concern remains because citation breadth does not identify document ownership.

What I noticed

ID	Priority	Topic	Code location	What I noticed
RAV-RUN2-R1-F001	high	failure accounting	every_eval_ever/adapters/benchpress/adapter.py:214	One malformed score or row-level conversion error can stop the full run before that source row is recorded. (inline)
RAV-RUN2-R1-F002	medium	snapshot provenance	every_eval_ever/adapters/benchpress/adapter.py:246	The default path is pinned, but an explicit symbolic --revision can still mix snapshots and records the symbol instead of a SHA. (inline)
RAV-RUN2-R1-F003	medium	evaluator relationship	every_eval_ever/adapters/benchpress/adapter.py:299	A one-provider citation does not prove that the document owner matches the scored model provider.
RAV-RUN2-R1-F004	medium	exclusion accounting	every_eval_ever/adapters/benchpress/adapter.py:641	An exclusions-only run prints a count but does not preserve each excluded row and reason in the report. (inline)

Things to try

Suggested next steps (4 items)

• [high] failure accounting — Expected parsing and schema errors can stay inside a per-row boundary. Unexpected program errors can remain visible. (RAV-RUN2-R1-P001; covers RAV-RUN2-R1-F001)
• [medium] snapshot provenance — Every supplied reference can resolve once to a SHA, preserving current branch and tag support. (RAV-RUN2-R1-P002; covers RAV-RUN2-R1-F002)
• [medium] evaluator relationship — Ownership-unknown documents can use other until the source exposes comparable publisher evidence. (RAV-RUN2-R1-P003; covers RAV-RUN2-R1-F003)
• [medium] exclusion accounting — The shared report can be saved for failures or exclusions. The exit remains nonzero only for failures. (RAV-RUN2-R1-P004; covers RAV-RUN2-R1-F004)

Run details

• Target: PR #197 (add-benchpress-adapter, 5 files, +942/-0) at 3d624b536f87ffd74d3f135bf43c7044df673cdc
• Run ordinal: 2
• Rounds: 1/1 completed; adaptive off in review-only mode
• Mix: 3 codex-exec; 2 completed, 1 failed after DNS transport retries
• Focus: correctness, maintainability, simplicity, production blast radius, and constructive suggestions
• Earlier review comments: 4 grouped concerns; 3 fixed and 1 still present
• Finding counts: 0 critical, 1 high, 3 medium, 0 low, 0 nit
• Checks: concerns=3; confirmed=3
• Second check: targeted; reviewers=2; kept=4, clarified=3, removed=0; approval unchanged

---

Reviewed with review-anvil.

[mrshu]

RAV-RUN2-R1-F001 [high] failure-accounting — Score parsing happens before SourceConversionResult exists, and later row construction has no row-level error boundary. One malformed source value can stop every valid row without a source-row failure record.

A narrow boundary can catch expected parsing and schema errors. Unexpected program or infrastructure errors can remain visible.

[mrshu]

RAV-RUN2-R1-F002 [medium] snapshot-provenance — The default fetch resolves one SHA, but a supplied symbolic revision bypasses that step. A moving branch can resolve differently across four requests, while metadata records only the symbol.

Resolving every supplied reference once would keep branch and tag support while making all file URLs reproducible.

[mrshu]

RAV-RUN2-R1-F004 [medium] exclusion-accounting — The report is saved only when result.failures is nonempty. An exclusions-only run succeeds but loses each excluded source reference and reason.

The shared report already supports exclusions. It can be saved for either collection while the nonzero exit stays conditional on failures.

[borgr]

All four findings are fixed in c8b0f7c40. Three follow the fix path as written;
F003 ends up somewhere neither of us proposed, and that is the substantive part
below. Chasing it also turned up a fifth defect of the same kind as F001/F004,
fixed here too.

Re-verified live on this head, by a fresh fetch and an --input-json replay that
agree: 4,903 source scores → 326 logs / 4,475 scores, validate all 326
passed, and the same 18 failures / 410 exclusions as before — no new failure of any
kind. pytest tests 413 passed, 20 skipped (this adapter's file 11 → 24 tests);
ruff check clean.

RAV-RUN2-R1-F001 · failure accounting — one bad cell ended the run

Correct, and it was the failure mode the accounting exists to prevent. _parse_scores
called float() while reading the CSV, which is before SourceConversionResult
exists — so an unusable cell raised out of parsing and no row was recorded, not even
the one at fault.

The score now stays as the source wrote it and is parsed inside make_logs, where the
row is still in hand. The boundary is deliberately narrow, per P001: it catches
ValueError, TypeError and pydantic's ValidationError — what unusable source data
raises — and nothing else, so a bug in this adapter still surfaces as a crash rather
than being filed as bad upstream data. total_records stays len(payload['scores']),
so nothing is silently dropped to make the report reconcile.

Two details are judgement calls rather than mechanics:

• The id columns stay strict. r['model_id'] is left as a subscript.
  DictReader gives every key for every row, so that only raises when the header
  lacks the column — every row missing an id means the export changed shape, which
  should stop the run, not produce 4,900 identical row failures.
• The descriptive numerics went the other way. params_total_M,
  params_active_M and num_problems only ever reach a record as text through
  _str_map, and no score depends on them, so _optional_number keeps unparseable
  text ('30 (I+II)') rather than failing a row over a value that would have been
  stringified anyway.

Four tests: a junk score and a NaN each fail only their own row (both source_refs,
both reasons, the rejected value present in the report, total_records == 9, and the
surviving logs still validate); a schema-rejected field is attributed to its row; a
monkeypatched RuntimeError propagates; the descriptive text survives to
model_info.additional_details.

RAV-RUN2-R1-F002 · snapshot provenance — a symbolic --revision mixed snapshots

Also correct, and worse than mixing: the symbol was spliced straight into the four
file URLs and recorded as benchpress_dataset_revision, so the provenance on every
record was a ref that can later point somewhere else.

resolve_revision now resolves whatever it is asked for — the default tip, a branch,
a tag — to one commit sha before any file is read, which keeps branch and tag support
as P002 requires. A --revision that is already a full 40-hex sha is used as given,
since it cannot move and resolving it would only cost a request. Refs containing /
are URL-quoted, so refs/pr/2 reaches the API as refs%2Fpr%2F2 instead of a 404.
Three tests cover the three paths (symbolic resolves and no file is read at the
symbol; a pinned sha triggers no API call; the slash case is escaped).

RAV-RUN2-R1-F003 · evaluator relationship — the claim now needs a publisher

Your diagnosis is exactly right: citation breadth measures which providers' models a
document was cited for, never who published it. Two live checks before choosing a
fix, because P003's "until the source exposes comparable publisher evidence" turns on
whether such evidence exists at all:

• Breadth is falsified in the over-claiming direction. A Google
  gemini-2.5-flash score cited to
  Qwen's model card had
  breadth 1 and was labelled first_party.
• And in the under-claiming direction, which I had not expected.
  openai.com/index/introducing-gpt-5-5
  carries 46 OpenAI cells and 8 competitor cells, so breadth demoted the whole page —
  including OpenAI's own scores on OpenAI's own domain. BenchPress had already tagged
  those 8 competitor cells source_type=third_party itself. The export made the
  distinction and the heuristic threw it away.

So the evidence is there, but it is the citation's domain, not its type or its
breadth: a domain has an owner. first_party now needs a provider-authored
source_type and a hostname label matching the scored model's provider name —
openai.com, cdn.amazon.science, moonshotai.github.io, x.ai. A shared host
names nobody, whatever its path says: the org in huggingface.co/Qwen/… sits in the
path, and storage.googleapis.com serves anyone's bucket, so its 68 cells stay
other. No provider→domain table, so nothing to hand-maintain and nothing to guess —
and the rule under-claims where it cannot tell: a provider publishing on a hostname
that does not spell its name (lf3-static.bytednsdoc.com for a ByteDance model) is
other.

Live result: 841 cells across 49 logs are first_party, from exactly 12 hosts,
and every one of the 12 is the provider's own domain —

cells	host	developer
476	openai.com	openai
120	www.anthropic.com	anthropic
99	deepmind.google	google
28	cohere.com	cohere
26	www-cdn.anthropic.com	anthropic
25	ai.meta.com	meta
17	cdn.amazon.science	amazon
15	mistral.ai	mistral
15	moonshotai.github.io	moonshot-ai
15	x.ai	xai
4	anthropic.com	anthropic
1	cdn.openai.com	openai

Full distribution {other: 2,654, third_party: 980, first_party: 841} across
{third_party: 139, other: 138, first_party: 49} logs, from
{other: 2,550, third_party: 980, first_party: 945} / {third_party: 139, other: 119, first_party: 74} before. 2,614 published cells carry a provider-authored type whose
publisher the export does not identify, and all of them are other. No score value changes and
no cell is dropped.

Three tests: the split, a unit test on the host rule (including the two shared-host
negatives), and one asserting a provider-authored type on a shared host is not a
first_party claim.

Fifth defect, found while fixing F003 — a silent de-dup

Not in your ledger, and the same shape as F001/F004: a de-dup pass collapsed a log's
results by evaluation_result_id, keeping whichever row arrived first and dropping the
other with no entry anywhere — not a failure, not an exclusion. The report would
still reconcile against total_source_records, because the loss happened after
counting.

evaluation_result_id is the schema's "recommended deterministic join key", so one log
genuinely cannot carry two results for one benchmark. Now: an identical repeat still
collapses (it carries nothing new), and a second differing result is reported as a
failure naming the citation already in the log. Giving the id a citation suffix was the
alternative, but it would change the join key on all 4,475 cells to fix a case that
does not occur — or make identity depend on whether a sibling row exists, which is the
mistake #203/#204 were pulled up on. Zero live rows hit either path
(4,903 − 410 − 18 = 4,475 exactly), so this closes a hole rather than repairing
observed loss. test_a_second_result_for_one_benchmark_is_reported_not_dropped pins
both halves.

RAV-RUN2-R1-F004 · exclusion accounting — one fix, one deliberate divergence

Both defects are real and fixed: the write was conditional on result.failures, so an
exclusions-only run printed a count and preserved nothing, and a clean run left the
previous run's report standing to be read as its own. write_conversion_report now
writes on every run and swaps the file in with os.replace, so an interrupted write
cannot leave a truncated report where a complete one was. The exit stays non-zero for
failures only, as P004 specifies — an excluded row is BenchPress's own audit decision.

The divergence is when: the report is written before publication, not after. It
accounts for the conversion, so a publication error is precisely when it is most worth
having on disk. test_the_report_survives_a_publication_error makes export_logs
raise and asserts the failures are on disk anyway. Two more tests cover the defects you
named: an exclusions-only run exits 0 and itemizes all seven excluded rows with
reasons, and a seeded stale report is replaced with no partial file left beside it.
Happy to move the write after publication if you would rather have it gated.

This is the same shape as the fix in #204, and deliberately not factored out: a shared
helpers/io.py helper would couple the two PRs into a merge order and needs
cross-package agreement per AGENTS.md. It stays adapter-local in both, and I will
propose the shared home as one follow-up covering both once these land.

---

One housekeeping note: the "GPT 5.6 Review" comment above is about tau-bench —
utils/tau_bench/adapter.py:389, Pass^k labelled pass_at_k — and does not apply
here. This PR touches only the five adapters/benchpress/ + test files and never that
path. It looks like it was meant for #192, so the high finding on it should not be read
as outstanding on this branch.

[mrshu]

⚒️ review-anvil report

Review decision: COMMENT — The earlier findings are addressed, and seven focused suggestions can strengthen the new adapter.
Result: Two high-priority and five medium-priority findings remain.
Scope: Add a BenchPress score-matrix adapter with stable identity, correct metadata, complete row accounting, and safe publication.
Checks: 7 concerns checked and confirmed.
Second check: 2 reviewers kept all 7 findings and clarified 5 fix paths.

Earlier review comments

Earlier review comments (23 ledger entries)

• Audit-status filtering, expected row failures, immutable revision reads, evaluator ownership, strict JSON publication, and exclusion-report persistence are fixed.
• Old utils/benchpress anchors are no longer relevant after the adapter move.
• The findings below are distinct residual cases on the current head.

What I noticed

ID	Priority	Topic	Code location	What I noticed
RAV-RUN3-R1-F001	high	data selection	every_eval_ever/adapters/benchpress/adapter.py:557	A missing audit_status column turns every row into a normal exclusion. The run can publish no logs and still exit successfully.
RAV-RUN3-R1-F002	high	stable identity	every_eval_ever/adapters/benchpress/adapter.py:628	evaluation_id omits the immutable dataset revision. A replay can use current time, while a lagging manifest time can collide across revisions.
RAV-RUN3-R1-F003	medium	model metadata	every_eval_ever/adapters/benchpress/adapter.py:352	BenchPress supplies open_weights, but the typed model_availability field remains unknown.
RAV-RUN3-R1-F004	medium	metric semantics	every_eval_ever/adapters/benchpress/adapter.py:470	Every metric receives a benchmark-specific .score ID. Known global metrics such as percent-correct results cannot join on canonical identity and scale.
RAV-RUN3-R1-F005	medium	exclusion accounting	every_eval_ever/adapters/benchpress/adapter.py:559	Exclusions omit the source row. Repeated model and benchmark references cannot be distinguished in the report.
RAV-RUN3-R1-F006	medium	evaluation metadata	every_eval_ever/adapters/benchpress/adapter.py:631	The log names BenchPress as the evaluation library although BenchPress aggregates results from varied harnesses.
RAV-RUN3-R1-F007	medium	duplicate accounting	every_eval_ever/adapters/benchpress/adapter.py:603	Equal duplicate rows collapse to one result without a failure or exclusion entry. The source-row ledger cannot reconcile the duplicate.

Things to try

Suggested changes (7 items)

• [high] data selection — Missing or blank status values can become row failures before policy filtering. A fresh report and non-zero exit can preserve the incompatible export evidence. (RAV-RUN3-R1-P001; covers RAV-RUN3-R1-F001)
• [high] stable identity — The raw model, relationship, and required immutable dataset revision can define the ID. Generated and retrieved times can stay as metadata. (RAV-RUN3-R1-P002; covers RAV-RUN3-R1-F002)
• [medium] model metadata — Recognized source booleans can map to open_weights or closed_weights. Unknown tokens can remain typed unknown, with raw text preserved. (RAV-RUN3-R1-P003; covers RAV-RUN3-R1-F003)
• [medium] metric semantics — A pinned mapping can canonicalize only metrics with established semantics. Score, bounds, unit, and direction can convert together while ambiguous ratings remain namespaced. (RAV-RUN3-R1-P004; covers RAV-RUN3-R1-F004)
• [medium] exclusion accounting — Each exclusion can retain its complete score row. Repeated references then preserve distinct scores, citations, and statuses. (RAV-RUN3-R1-P005; covers RAV-RUN3-R1-F005)
• [medium] evaluation metadata — The log-level library can be set only when every result has the same recognized harness. Mixed groups can use unknown while preserving each result claim. (RAV-RUN3-R1-P006; covers RAV-RUN3-R1-F006)
• [medium] duplicate accounting — Exact source-cell duplicates can keep one emitted result and add exclusions for later rows. Differing rows with the same result ID can remain failures. (RAV-RUN3-R1-P007; covers RAV-RUN3-R1-F007)

Run details

• Target: PR #197 at c3d16fe138396b433a9393ab61a7ad1391b546e0
• Run ordinal: 3
• Rounds: 1/1 completed; adaptive off; material findings
• Mix: 3 codex-exec; one reviewer could not access the exact head, and two completed substantive reviews
• Focus: correctness, maintainability, simplicity, production blast-radius, and constructive optional suggestions
• Earlier review comments: 23 ledger entries; all fixed or no longer relevant
• Finding counts: 0 critical, 2 high, 5 medium, 0 low, 0 nit
• Checks: concerns=7; confirmed=7; set-aside=0
• Second check: targeted; reviewers=2; kept=7; clarified=5 plans; removed=0; approval unchanged
• Set aside: 0 items

---

Reviewed with review-anvil.

[mrshu]

RAV-RUN3-R1-F001 [high] data-selection — A missing audit_status column can produce a successful empty run

_parse_scores() reads this field with get(). This branch then treats every absent value as a normal exclusion. With all rows excluded, no failure reaches raise_if_incomplete().

Missing or blank status values can become row failures before policy filtering. A fresh report and non-zero exit would preserve the incompatible export evidence.

[mrshu]

RAV-RUN3-R1-F002 [high] stable-identity — The ID does not include the immutable score-matrix revision

This timestamp can fall back to conversion time, and the source manifest can lag the pinned CSV revision. Replays can change identity, while distinct revisions can collide.

The raw model, relationship, and required dataset_revision can define the ID. Generated and retrieved times can stay as metadata.

[mrshu]

RAV-RUN3-R1-F003 [medium] model-metadata — Typed availability remains unknown despite source evidence

The source provides open_weights values, and this code keeps them only as raw additional detail. ModelInfo then fills model_availability with unknown.

Recognized booleans can map to open_weights or closed_weights. Unknown tokens can remain typed unknown, with their raw text preserved.

[mrshu]

RAV-RUN3-R1-F004 [medium] metric-semantics — Known global metrics receive benchmark-specific IDs and source scales

The AIME fixture declares percent correct, but this path emits benchpress.aime-2025.score on a 0–100 scale. That result cannot join canonical accuracy records.

A pinned mapping can canonicalize metrics only when source metadata proves their meaning. Score, bounds, unit, and direction can convert together; ambiguous ratings can remain namespaced.

[mrshu]

RAV-RUN3-R1-F005 [medium] exclusion-accounting — Repeated exclusion references lose row-level detail

This exclusion stores only the model and benchmark reference. The fixture contains repeated references, so the report cannot distinguish their scores, citations, or status values.

Each exclusion can retain its complete score row through the existing source_record field.

[mrshu]

RAV-RUN3-R1-F006 [medium] evaluation-metadata — The aggregator is recorded as the evaluation harness

BenchPress re-reports scores from varied harnesses, which the results already retain. This log-level value states that BenchPress ran every evaluation in the group.

A common harness can populate this field only when every result agrees. Mixed groups can use unknown while keeping each result's source claim.

[mrshu]

RAV-RUN3-R1-F007 [medium] duplicate-accounting — Equal duplicate rows disappear from the source ledger

This branch keeps one result and silently skips the later equal row. total_records still includes both, but neither failures nor exclusions identifies the duplicate.

Exact source-cell duplicates can remain deduplicated while later rows become exclusions with their complete source_record.