feat(cli): results matrix — behavior × arm comparison table - #274
Open
changliu2 wants to merge 1 commit into
Open
feat(cli): results matrix — behavior × arm comparison table#274changliu2 wants to merge 1 commit into
changliu2 wants to merge 1 commit into
Conversation
Adds `assert-ai results matrix` rendering a 2D pivot (rows=behavior, cols=arm/variant, cells=metric rate) over multiple runs, for the N-arms × M-behaviors comparison used by the 1-behavior-per-yaml pattern. Pivots on each run's config.yaml behavior.name and an arm label derived from the run id. Includes --suite auto-expand, --json, and tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3
changliu2
requested review from
AaronAspinwall123,
jakepresent and
tangym
as code owners
July 22, 2026 21:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
assert-ai results matrix, a 2D comparison view for multi-run eval results. Companion to the bank-manager one-behavior-per-yaml configs in #262 — it renders the full N-failure-modes × M-arms grid that the viewer's single-suite compare view can't.What changed
assert-ai results matrix SUITE/RUN ...under theresultsgroup.--suite <SUITE>auto-expands all scored runs in a suite; pass several to span multiple behavior-suites in one table.--metric(defaultpolicy_violation),--json,--no-color,--results-dir.config.yamlbehavior.name(falls back to manifest, then suite id).baseline → prompted → acsthen any others alphabetically (deterministic).-in tables andnullin JSON.Example
Demo output (rendered from real runs)
Validation