Add machine-readable scenario summary report

[shreyaskommuri]

Summary

• Adds a default summary scenario reporter that writes cloudai-summary.json next to the existing scenario reports.
• Includes scenario name/status, system metadata, result directory, scenario-level report artifacts, per-test-run status, output paths, artifacts, and metrics already exposed through configured metric reporters.
• Registers the reporter by default and keeps tarball as the final scenario reporter.

Fixes #917.

Test Plan

Environment:

• macOS
• Python 3.10.20 via uv run

Commands and results:

$ uv run ruff check src/cloudai/reporter.py src/cloudai/registration.py src/cloudai/_core/registry.py src/cloudai/core.py tests/test_reporter.py tests/test_init.py
All checks passed!

$ uv run ruff format --check src/cloudai/reporter.py src/cloudai/registration.py src/cloudai/_core/registry.py src/cloudai/core.py tests/test_reporter.py tests/test_init.py
6 files already formatted

$ uv run pytest tests/test_reporter.py tests/test_init.py tests/test_registry.py
60 passed in 0.43s

$ uv run pytest
1573 passed, 4 skipped, 463 deselected in 5.14s

Additional Notes

• This keeps existing HTML, tarball, comparison, and workload-specific reports unchanged.
• The summary intentionally uses metrics CloudAI already exposes through each test run's configured metric reporter, so this PR adds a stable automation artifact without introducing new workload-specific parsing logic.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: ad537ac1-b363-4530-90bf-c04d2cef8dc9

📥 Commits

Reviewing files that changed from the base of the PR and between 7715acf and dfa28da.

📒 Files selected for processing (4)

• src/cloudai/registration.py
• src/cloudai/reporter.py
• tests/test_init.py
• tests/test_reporter.py

---

📝 Walkthrough

Walkthrough

Adds a new SummaryReporter that writes a machine-readable cloudai-summary.json; registers and exports the reporter; adjusts report ordering to place "summary" before "tarball"; and adds tests that assert registry ordering and the produced JSON structure.

Changes

SummaryReporter: Machine-Readable Summary Artifact

Layer / File(s)	Summary
SummaryReporter class implementation src/cloudai/reporter.py	New SummaryReporter class extends Reporter to generate cloudai-summary.json with scenario metadata, per-test-run status, artifact listings, and numeric metrics (filters out METRIC_ERROR). Adds helpers to build summaries, enumerate artifacts, compute relative paths, and handle DSE sweep nesting.
Public API and registry integration src/cloudai/core.py, src/cloudai/registration.py, src/cloudai/_core/registry.py	Exports SummaryReporter from cloudai.core, registers a new "summary" scenario report (enabled) in registration, and updates report_order to place "summary" between "dse" and "tarball".
Test coverage and validation tests/test_init.py, tests/test_reporter.py	Imports updated to include SummaryReporter; registry expectations and test_report_order are updated to include "summary" before "tarball"; new test test_summary_reporter_writes_machine_readable_summary stubs artifacts, runs generate(), reads cloudai-summary.json, and asserts expected JSON schema and content including nested sweeps and metrics.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

enhancement

Suggested reviewers

• srivatsankrishnan
• jeffnvidia

Poem

🐰 A little rabbit hops through files so spry,
It gathers artifacts beneath the sky,
Writes cloudai-summary.json tidy and bright,
So tools can wake and read results at night.
Hooray for stable JSON — automation's delight!

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title 'Add machine-readable scenario summary report' accurately and directly reflects the main change: adding a new SummaryReporter that writes cloudai-summary.json.
Description check	✅ Passed	The PR description is directly related to the changeset, explaining the addition of a default summary reporter, its functionality, and the rationale for the changes.
Linked Issues check	✅ Passed	The PR fully addresses issue #917 by implementing a SummaryReporter that generates cloudai-summary.json with scenario name, status, result directory, report artifacts, and per-test-run details including metrics.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to implementing the summary reporter feature. Registry ordering, core exports, registration, reporter implementation, and tests are all necessary for the feature.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[podkidyshev]

I'd rather do a full object assert like

assert summary == {
    "schema_version": "1.0",
    ...
}

[podkidyshev]

I think this property is a bit too much atm, let's keep the set of fields to minimum. Only those that bring value

[podkidyshev]

this won't work gracefully when we run sweeping. in the sweeping case, the folders structure will be

/scenario foler/
               /case 1/
                      /sweep 1/
                              /some-report.html
                      /sweep 2/
                              /some-report.html

[podkidyshev]

sweeping is actually quite important in our work. I believe this json summary report should bring insights if those were run

[shreyaskommuri]

Addressed the requested changes and updated the PR branch from latest main.

What changed:

• Replaced the summary reporter regression with a full-object JSON assert.
• Trimmed the first summary contract to the minimum useful automation fields: scenario, status, result_dir, reports, and test_runs.
• Grouped DSE/sweep outputs under their parent test run via a sweeps list, including case-level artifacts such as trajectory.csv and per-sweep nested report artifacts.

Validation on the updated PR head dfa28da:

• uv run ruff check src/cloudai/reporter.py tests/test_reporter.py tests/test_init.py src/cloudai/registration.py src/cloudai/_core/registry.py src/cloudai/core.py -> passed
• uv run ruff format --check src/cloudai/reporter.py tests/test_reporter.py tests/test_init.py src/cloudai/registration.py src/cloudai/_core/registry.py src/cloudai/core.py -> passed
• uv run pytest tests/test_reporter.py tests/test_init.py tests/test_registry.py -q -> 60 passed
• uv run pytest -q -> 1599 passed, 4 skipped, 476 deselected