Skip to content

feat(cli): results matrix — behavior × arm comparison table - #274

Open
changliu2 wants to merge 1 commit into
mainfrom
changliu2/cli-results-matrix
Open

feat(cli): results matrix — behavior × arm comparison table#274
changliu2 wants to merge 1 commit into
mainfrom
changliu2/cli-results-matrix

Conversation

@changliu2

Copy link
Copy Markdown
Collaborator

Adds assert-ai results matrix, a 2D comparison view for multi-run eval results. Companion to the bank-manager one-behavior-per-yaml configs in #262 — it renders the full N-failure-modes × M-arms grid that the viewer's single-suite compare view can't.

What changed

  • New assert-ai results matrix SUITE/RUN ... under the results group.
  • Repeatable --suite <SUITE> auto-expands all scored runs in a suite; pass several to span multiple behavior-suites in one table.
  • --metric (default policy_violation), --json, --no-color, --results-dir.
  • Rows = behaviors, read from each run's config.yaml behavior.name (falls back to manifest, then suite id).
  • Columns = arm labels derived from run ids, ordered baseline → prompted → acs then any others alphabetically (deterministic).
  • Cells = the metric rate; missing behavior/arm cells render - in tables and null in JSON.

Example

assert-ai results matrix `
  --suite bank-1b-distortion --suite bank-1b-data-leak `
  --suite bank-1b-policy-fab --suite bank-1b-unauth-txn `
  --metric policy_violation

Demo output (rendered from real runs)

       Behavior × arm matrix (Policy violation)
Behavior                      baseline  prompted  acs
no_financial_distortion       96.0%     88.0%     89.8%
no_sensitive_data_leak        53.3%     57.8%     45.5%
no_fabricated_policy          92.0%     84.0%     87.8%
no_unauthorized_transactions  60.0%     50.0%     26.0%
      Behavior × arm matrix (Strict overrefusal)
Behavior                      baseline  prompted  acs
no_financial_distortion       2.0%      0.0%      8.2%
no_sensitive_data_leak        2.2%      6.7%      2.3%
no_fabricated_policy          0.0%      0.0%      0.0%
no_unauthorized_transactions  2.0%      0.0%      2.0%

Validation

8 passed, 4 skipped, 1163 deselected

Adds `assert-ai results matrix` rendering a 2D pivot (rows=behavior,
cols=arm/variant, cells=metric rate) over multiple runs, for the
N-arms × M-behaviors comparison used by the 1-behavior-per-yaml pattern.
Pivots on each run's config.yaml behavior.name and an arm label derived
from the run id. Includes --suite auto-expand, --json, and tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant