feat(sbd-v6): declare the axis a session graded on, and grade the conc sweep on it too

[luochen-amd]

Reopens the work from #1480, rebased onto current main. That PR was closed by request with its commits kept on the branch; this is that branch, brought forward. One conflict, an import line in conc_sweep.py, resolved by keeping both sides.

I re-checked the three defects it claims to fix against today's main before reopening, because #1380 and #1445 were both invalidated by main moving underneath them. All three are still live -- see Still current at the end.

Why

Nothing in session_breakdown.json said which axis its gains were on. An AgentX replay is ranked on the slow-tail interactivity percentile (e2e_norm_intvty_p90) with total throughput held as a guard; a synthetic run is ranked on output throughput alone. Every throughput field in the breakdown is the output axis by construction, and benchmark_mode reaches reports/final.json but never this document.

A consumer therefore had no way to keep an AgentX session out of a synthetic leaderboard. It would sort one against the other and every number would look plausible -- on the canonical corpus the two axes differ by roughly two orders of magnitude.

The two blocks, and why they are two

"metadata": {
  "grading": {
    "benchmark_mode": "agentx",
    "objective": "e2e_norm_intvty_p90",
    "tput_guard": {"enabled": true, "noise_pct": 3.5}
  }
}

metadata.grading is what the session was configured to rank on. outcome.validation.graded_on is what the run actually decided its last promotion on, read off that promotion. The two differ whenever a comparison could not supply the axis pair and fell back to output.

Neither resolves the other, deliberately. Publishing the configured axis in outcome would put an interactivity label on an output number; resolving metadata from outcome would erase what the session asked for. There is a test on each direction.

Recorded, not re-derived

Resolution reads HYPERLOOM_PERF_METRIC / HYPERLOOM_PERF_NOISE_PCT. Every later reader of those is somewhere the environment is not evidence: a resume is a new process, a re-baseline runs in a subprocess, and CLOSE frequently drives this export from one that never inherited them. So the axis and its band are resolved once at seed and recorded in SharedState.grading, beside benchmark_mode and agentx_epoch, which live there for the same reason.

This fixes a live defect rather than only serving the export. resolve_graded_comparison re-derives on every comparison, and the recorded band now reaches passes_intvty_gate and passes_tput_guard -- both already accept noise_pct, and neither call site passes it, so a lost HYPERLOOM_PERF_NOISE_PCT silently widens a configured 3.5% guard band back to the 5% default. The KEEP/REVERT rule has to be the one the session started with.

Sessions seeded before the field derive as before and report a null band: the band they actually applied was never recorded, and today's default is not evidence of it.

The conc sweep, grading on the same axis

The sweep resolved its own axis through graded_metric_key, which diverges from the KEEP path two ways now that KEEP reads the axis recorded at seed.

A resume whose shell lost HYPERLOOM_PERF_METRIC would redraw the curve on output throughput while the promotions in the same session were decided on interactivity. And graded_metric_key has no scriptable carve-out where intvty_serving_grading_enabled does, so an image framework -- which reports no interactivity axis at all -- would rank every rung on a field it never measures and report the whole sweep as failed. Both call sites now go through resolved_grading.

Each pair also carries the throughput axis beside the objective, and whether it held within the session's band. Reported, not enforced: InferenceX publishes a 2-D frontier with no fixed interactivity target, so a rung that traded throughput for interactivity moved along that frontier rather than violating a constraint -- and drawing that frontier is what a sweep is for. Gating here would drop half the curve. best_conc stays the best rung on the objective alone, with best_conc_guard_holds saying whether the session's own KEEP rule would have taken it.

The pair columns are renamed off throughput accordingly: baseline_tput / optimized_tput held an interactivity percentile whenever the session graded on one. They are now *_value, with result.metric naming the axis. Also fixes V6ConcSweepPoint declaring intvty_p90 where the recorded curve has always written e2e_norm_intvty_p90.

The axes themselves

outcome.baseline.perf, outcome.final.perf and outcome.validation.perf carry the four AgentX axes, recorded where the measurement is taken rather than read off state.baseline_perf and current_best at export. A second source for one baseline is a second answer to the same question, and a revalidation moves the cumulative gain without re-promoting the recipe, so current_best can be a later measurement than the gain published beside it.

An unmeasured axis is an explicit null. Absent would be indistinguishable from an axis the framework failed to report, and zero reads as "measured, and it was zero", so a synthetic run publishes four nulls. TPOT p90 was already parsed out of the benchmark report and dropped at five call sites; it now lands.

Degrade is a finding, not a field

An adoption that fell off the configured axis is reported through validation.notes, naming the count and the reason. Its contribution sits in the same sum as the axis-graded ones, so the total is not single-axis and the reader has to be told.

The reason is recorded on the adoption, because that is the only comparison that can degrade. The whole-stack validation is recorded only after the orchestrator found the comparison comparable, so a degrade_reason on that row would never be non-empty -- a field that reads as protection the document does not have.

Still current

Verified against main at ff7c235e4 before reopening:

Claim	State on main today
resolve_graded_comparison re-derives the axis from the environment on every call	Still does -- shared_state.py calls intvty_serving_grading_enabled(...) per comparison
The configured noise band never reaches the guards	Still true -- both call sites pass no noise_pct, so both fall back to parse_intvty_noise_pct()
The conc sweep resolves its axis separately, with no scriptable carve-out	Still true -- conc_sweep.py calls graded_metric_key at two sites; it has no scriptable parameter

Rollout

Additive to the wire shape. throughput_tok_s_per_gpu, output_throughput and every other existing key keep their output-axis meaning, so nothing downstream has to change to keep working.

Cortex has no column for the interactivity or total axes yet, so the immediate value is that an AgentX session becomes identifiable -- the prerequisite for keeping it off a leaderboard it is not comparable on.

Test

25 cases in test_sbd_v6_grading.py: both axes at seed, the scriptable carve-out, recorded-beats-environment in both directions, the null band on a legacy session, the block surviving the recorder and the overlay end to end, the metadata/outcome separation locked in both directions, the axis fallback for a session that adopted but never validated, explicit nulls, and the degrade finding. Plus the conc-sweep cases for the renamed pair columns and the guard column.

ruff check is clean on src/hyperloom after the rebase. The suites here import fcntl transitively, so they are POSIX-only and I could not run them on Windows -- CI is the check that matters for this one, and I will follow up on anything it turns up.