ci: hourly Quartz consumer workflow to test against latest ready nightly by zichguan-amd · Pull Request #505 · ROCm/rocm-examples

zichguan-amd · 2026-08-25T15:34:31Z

Summary

Adds a standalone nightly workflow that tests rocm-examples against the latest READY ROCm nightly reported by Quartz (ROCm's CI/CD data hub), running exactly once per new version.

Gate: resolves Quartz release-nightly/latest.json and requires the Linux ROCm build to be success. It deliberately does not gate on overall_status, which folds in every pipeline/phase and is routinely red from unrelated test failures.
Dedup: a lookup-only actions/cache marker keyed on (rocm_version, build_date) ensures each version is tested once, not on every poll.
Fail-safe: an in-progress build or an unreachable Quartz yields resolved=false and the run does nothing — doubling as a Quartz availability canary.

Files

.github/quartz/read_status_json.py — vendored byte-for-byte from ROCm/Quartz (scripts/consumer, schema v2) with a provenance header so it can be re-synced on schema bumps.
.github/quartz/consume_status.py — the consumer/gate: an unsupported schema major is a hard failure (not a soft skip); requires linux.rocm.build == success; follows the latest.json symlink pointer; emits resolved/rocm_version/build_date/source/wheels_url/tarballs_url to $GITHUB_OUTPUT.
.github/quartz/tests/test_consume_status.py — unit tests for the gate/resolve logic (good / not-ready / unavailable / bad-schema-major-exits / real-bug-propagates / absent platform).
.github/workflows/quartz_test.yml — polls every 2h cron "30 1-11/2 * * *" (01:30–11:30 UTC, around the ~02:30 build-green time) + workflow_dispatch (force). Jobs: gate (consume + cache dedup + matrix), test (reusable build), record (marker on pass only), report (dynamic skip titles).
.github/workflows/build-rocm-examples-reusable.yml — new optional rocm_version input that pins the exact nightly for both whl (==<ver>) and tarball (exact filename) installs, plus wheels_url/tarballs_url inputs so the install index/base come straight from Quartz's published URLs; all empty preserves today's latest-index behavior, so existing callers are unaffected.
.github/build_tools/configure_ci.py — new opt-in --exclude-preinstalled flag that drops preinstalled-only images (the version-pinned stable image) from the matrix. Defaults off, so ci_nightly.yml is unaffected.

Design decisions

Record on pass only — a green run is the sole conclusive "tested." Any failure (transient infra or a real build/test break) leaves the version unrecorded so the next poll re-tests it, rather than masking it until the 7-day cache eviction. Retry is bounded: resolve() always prefers the newest ready build, so a genuinely broken nightly is superseded within the morning poll window, not retried hourly.
Matrix = full nightly minus preinstalled images — 4 multi-arch distros × {gfx1100, gfx1151} × {whl, tarball} = 16 legs. The version-pinned stable image is excluded (via configure_ci.py --exclude-preinstalled) because its ROCm version is baked in and unrelated to the nightly we pin.
Poll window — every 2h from 01:30–11:30 UTC. The Linux ROCm build starts ~00:02 and goes green ~02:24–03:09 on a normal night, so 01:30 sees it in progress (cheap skip), 03:30 usually catches it green, and the morning tail absorbs publish lag. A morning finish drops only the rare evening re-run, which self-heals when the next nightly supersedes it.
Cache eviction — the 7-day marker eviction is accepted (a stale week means one harmless re-run).
repository_dispatch (Quartz pushing on each build, removing polling+dedup) is deferred as a follow-up.

Test plan

consume_status.py unit tests pass (python3 -m unittest discover -s .github/quartz/tests)
Consumer resolves 10.1.0a20260821 (build=success) against the live Quartz document
configure_ci.py --exclude-preinstalled drops the preinstalled image and keeps distros/distro_map in sync; both pinned tarballs return HTTP 200
Both workflow YAMLs parse
workflow_dispatch run: confirm gate resolves, matrix pins the exact version, and record writes the marker
Second immediate run shows the skip (already-tested) report

🤖 Generated with Claude Code

… nightly Adds a standalone workflow that polls Quartz (ROCm's CI/CD data hub) for the latest READY ROCm nightly and runs the full example matrix exactly once per new version. Gates on the Linux ROCm build being success (not overall_status), dedups via a cache marker keyed on (rocm_version, build_date), and is fail-safe when Quartz is unreachable. Polls every 2h 01:30-11:30 UTC around the ~02:30 build-green time. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

Adds a pull_request trigger so the Quartz workflow runs on this PR for end-to-end validation. The schedule only fires on the default branch, so this is the only way to exercise the gate/test/record path pre-merge. Revert this commit before merging. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

HereThereBeDragons · 2026-08-25T20:22:55Z

+          else
+            echo "is_new=true" >> "$GITHUB_OUTPUT"
+          fi
+


from my understanding the dedup caching part of the record: step must follow here.
i dont know how long your test needs but you dont want to trigger a 2nd test run on the same build just because the first one is still in_progress

This should prevent it, new jobs are queued and won't cancel the existing one, and when the running job finishes it would have cached the marker so the following jobs can exit accordingly.

# Never let a new poll cancel an in-flight test. concurrency: group: quartz-test cancel-in-progress: false

but our jobs won't take long to finish anyway, usually ~30min, 2h interval is more than enough.

HereThereBeDragons · 2026-08-25T20:27:05Z

        }

+    # Keep the distros list in sync with the (possibly filtered) map.
+    distro_keys = list(distro_map_out.keys())


we also have rpm/deb support via quartz if interested?

Will definitly expand on that as a follow up

…e polish Fixes the gate silently failing on the live Quartz feed: release-nightly/ latest.json is a symlink and raw.githubusercontent serves the target path (e.g. "20260826/status.json") as text, not the document, so json parsing raised and the run took the "unavailable" path. consume_status now follows that pointer and resolves the real status.json. Review comments addressed: - pin actions/checkout and actions/cache to commit SHAs - report job distinguishes "Quartz unavailable" from "no ready build" - resolve() returns a Resolution NamedTuple with a return annotation - --latest-only uses BooleanOptionalAction and is mutually exclusive with --source - drop redundant exclude_preinstalled local in configure_ci Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

Addresses review B: the whl/tarball index URLs in the reusable workflow were outdated (rocm.nightlies.amd.com/{whl,tarball}-multi-arch). consume_status now emits the exact wheels/tarballs base URLs from the Quartz doc (summary.linux.urls), the gate forwards them, and the reusable workflow uses them for the Quartz path. Non-Quartz callers fall back to the refreshed defaults (nightly.repo.amd.com/rocm/whl-next and /rocm/core/tarball). Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

Previously the record job wrote the "tested" marker on any non-cancelled result, so a version that failed to even install (wheel-index outage, tarball 404, runner hiccup) was permanently marked tested and never retried until the 7-day cache eviction. Record only on success so a failed run leaves the version unrecorded and the next poll re-tests it. Retry is bounded: a version is revisited only while it remains Quartz's latest-ready build, and the next nightly supersedes it. Addresses review comment D. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

A dynamic job name renders as raw YAML in the Actions UI when the job is skipped -- the common case, since report only runs on a no-op poll. Give the job a static "Report skip reason" name and move the outcome expression onto the Report step's name instead: it surfaces the outcome (already tested / Quartz unavailable / no ready build) when the job runs, and a skipped job renders no steps so nothing leaks. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

HereThereBeDragons

looks good aside from the question about the gating.

do you want first the changes to land in quartz or you want to merge this here and then upgrade it at a later point? (also goes along with the question of the sparse checkout)

HereThereBeDragons · 2026-08-28T15:02:31Z

these are all good changes but i think this is something quartz should provide and fix in the read_status_json.py.

Upstream Zichuan's #505 work into Quartz

Tier 1 — bug

scripts/consumer/read_status_json.py — make load_status() follow the latest.json symlink (retry on JSONDecodeError -> fetch resolved target)

scripts/consumer/tests/read_status_json_test.py — test: pointer body resolves; malformed JSON still raises

Tier 2 — robustness (in the example)

docs/status-json/example_consume_status.py — wrap load_status in try/except (availability canary, emit ready=false not crash)

docs/status-json/example_consume_status.py — schema-version guard ("2.")

Tier 3 — docs

docs/status-json/README.md + tutorial.md — one line: latest.json is a symlink, load_status follows it

After it lands

ci: hourly Quartz consumer workflow to test against latest ready nightly #505 drops _read_pointer/load_latest, calls load_status directly

pr: ROCm/Quartz#91

HereThereBeDragons · 2026-08-28T15:48:18Z

      - name: Summary
        run: |
-          echo "Recorded ROCm ${{ needs.gate.outputs.rocm_version }} (${{ needs.gate.outputs.build_date }}) as tested; result=${{ needs.test.result }}." >> "$GITHUB_STEP_SUMMARY"
+          echo "Recorded ROCm ${{ needs.gate.outputs.rocm_version }} (${{ needs.gate.outputs.build_date }}) as tested (passed)." >> "$GITHUB_STEP_SUMMARY"


how is success defined here? just that build-rocm-examples-reusable ran independent of the build/test results?
if not you might have multiple runs for the same rocm nightlies because rocm-examples has a bug and is legitly failing

Conditioned on needs.test.result == 'success' which requires the entire matrix of ./.github/workflows/build-rocm-examples-reusable.yml being green. So the ctests and Makefile testing must build and pass.

HereThereBeDragons · 2026-08-28T15:50:44Z

+      tarballs_url: ${{ needs.gate.outputs.tarballs_url }}
    secrets: inherit

  record:


coming back to the discussion here where it belongs:

if you really only need 30min and check quartz in a 2h intervall i guess it will be ok.

but if you see any problems you probably want an additional "in_progress" gate with a timestamp where you wait and if at that time the "successful-gate" is still from the day before and the "in_progress" gate is 4h+ old, you ignore it and trigger another build-rocm-examples-reusable.yml

having some more thoughts about it in ROCm/Quartz#91, the concurrency group should capture it. however in case downstream of the workflows you have some workflow_dispatch calls you will also need to guard them

…xcept Pin all quartz_test.yml runners to ubuntu-24.04. In consume_status.py, follow the Quartz example_consume_status.py split: transient fetch failures (OSError/ValueError) soft-skip as "unavailable", but an unsupported schema major now fails loudly (sys.exit) instead of silently reporting "not ready". Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

… dedup, and an interactive dry-run (#91) ## Summary Upstreams the findings from ROCm/rocm-examples#505 into Quartz's own `status.json` consumer docs and helper, and hardens the example against the failure modes a real downstream poller hits. The core bug: `latest.json` and `prerelease/latest.json` are git symlinks, and raw.githubusercontent.com serves a symlink as its **target path** (a one-line body like `20260707/status.json`), not the file it points to. A plain fetch of `latest.json` therefore returns that path, and parsing it as JSON fails. This broke the most obvious way to consume the "latest" endpoint. ## Read helper (`scripts/consumer/read_status_json.py`) - `load_status` now follows the symlink pointer transparently: on a JSON decode failure it checks whether the body is a bare `<date>/status.json` pointer, resolves it against the source URL with `urljoin`, and fetches the real document once. Anything that is not clearly a pointer re-raises the original error, so genuine malformed JSON still surfaces. Local paths follow symlinks natively and never take this fallback. - Added `read_status_json_test.py` covering the pointer-resolution path. ## Poll workflow and consumer example (the larger change) The workflow side is where most of the work went, because a naive poller either re-triggers on every poll or double-processes a build: - **Deduplication** via `actions/cache` keyed on `(rocm_version, build_date)`. `restore` is lookup-only (no download) and only probes; the marker is `save`d **only after** the React step succeeds, so a failed run is retried on the next poll rather than being marked done. - **Concurrency**: a static `concurrency` group serializes overlapping polls and lets an in-flight run finish. This is what makes a single save-after marker sufficient even when the work outlives the poll interval: React always writes its marker before the next poll starts. - **Fire-and-forget caveat**: a `concurrency` group only serializes runs of the same workflow. If the poll dispatches a separate long-running workflow, that run is independent and the poll's serialization does not extend to it. The tutorial documents the fix (move the marker into the dispatched workflow, key its own concurrency per build) with a worked `my_build.yml` sample. The Python example (`example_consume_status.py`) was also hardened: - A transient fetch failure (status.json momentarily unavailable around a release) reports `ready=false` and exits 0 so the next poll retries. - An **unsupported schema major** is treated as permanent and fails loudly (`sys.exit`, non-zero): retrying cannot fix it, and a new major can move the fields the accessors read, so continuing would risk silently misreading the document. - The pip dry-run step now echoes the command and streams pip's output live, so the wait is visible, and targets a smaller device package (`device-gfx1150`) to keep the example quick. - Added `example_consume_status_test.py` (9 tests) covering the gate, the outputs glue, and the ready / not-ready / schema-fail branches. ## Docs - README documents the `latest.json` pointer behavior and how to resolve it by hand if you fetch it yourself. - tutorial.md gains the "Dispatching a separate long-running workflow" section and the per-workflow marker guidance. ## Test plan - [x] `python3 -m unittest discover -s docs/status-json/tests -p '*_test.py'` - [x] `python3 -m unittest discover -s scripts/consumer/tests -p '*_test.py'` - [x] `python3 docs/status-json/example_consume_status.py` against the live `latest.json` endpoint (confirms the pointer fix end to end)

zichguan-amd · 2026-09-02T15:48:30Z

Closing in favor of #512

@zichguan-amd

…tly (v2) (#512) ## Summary Adds a standalone nightly workflow that tests rocm-examples against the **latest READY** ROCm nightly reported by [Quartz](https://github.com/ROCm/Quartz) (ROCm's CI/CD data hub), running the expensive matrix **once per new version on success** — a passing version is recorded and skipped thereafter, while a failing one is re-tested on later polls until it passes or the next nightly supersedes it. This is the sparse-checkout revision of #505: instead of vendoring Quartz's reader, the workflow reuses it directly from ROCm/Quartz at a pinned commit (unblocked by ROCm/Quartz#91). Co-authored with @zichguan-amd. - **Gate:** resolves Quartz `release-nightly/latest.json` and requires the Linux **ROCm build** to be `success`. It deliberately does *not* gate on `overall_status`, which folds in every pipeline/phase and is routinely red from unrelated test failures. - **Dedup:** a lookup-only `actions/cache` marker keyed on `(rocm_version, build_date)`, written **only on a green run**, skips a version once it has passed; a failing version stays unmarked and is re-tested on later polls (bounded — see *Record on pass only*). - **Fail-safe:** an in-progress build or an unreachable Quartz yields `resolved=false` and the run does nothing — doubling as a Quartz availability canary. ### Files - `read_status_json.py` is **not vendored**. The `gate` job sparse-checks out the single file from ROCm/Quartz (`scripts/consumer/read_status_json.py`) at pinned commit `910cc21e` (the ROCm/Quartz#91 merge, which ships the schema-v2 reader that follows the `latest.json` symlink) and puts it on `PYTHONPATH`. Bump the SHA to pick up reader fixes or a schema-major update. - `.github/quartz/consume_status.py` — the consumer/gate: imports Quartz's reader (via `PYTHONPATH`) and calls `load_status`, which follows the `latest.json` symlink pointer internally; an unsupported schema **major** is a hard failure (not a soft skip); requires `linux.rocm.build == success`; emits `resolved/rocm_version/build_date/source/wheels_url/tarballs_url` to `$GITHUB_OUTPUT`. - `.github/quartz/tests/test_consume_status.py` — unit tests for the gate/resolve logic (good / not-ready / unavailable / bad-schema-major-exits / real-bug-propagates / absent platform). Requires `PYTHONPATH` pointed at a Quartz checkout (see the module docstring: `git clone --depth 1 https://github.com/ROCm/Quartz /tmp/quartz`). - `.github/workflows/quartz_test.yml` — polls every 2h `cron "30 1-11/2 * * *"` (01:30–11:30 UTC, around the ~02:30 build-green time) + `workflow_dispatch` (`force`). Jobs: **gate** (sparse-checkout reader + consume + cache dedup + matrix), **test** (reusable build), **record** (marker on pass only), **report** (dynamic skip titles). - `.github/workflows/build-rocm-examples-reusable.yml` — new optional `rocm_version` input that pins the exact nightly for both whl (`==<ver>`) and tarball (exact filename) installs, plus `wheels_url`/`tarballs_url` inputs so the install index/base come straight from Quartz's published URLs; all empty preserves today's latest-index behavior, so existing callers are unaffected. - `.github/build_tools/configure_ci.py` — new opt-in `--exclude-preinstalled` flag that drops preinstalled-only images (the version-pinned stable image) from the matrix. Defaults off, so `ci_nightly.yml` is unaffected. ### Design decisions - **Reuse the Quartz reader via sparse checkout, don't vendor it** — a single-file sparse checkout at a pinned SHA keeps the reader byte-identical to upstream with no copy to re-sync on schema bumps; the pin makes reader updates an explicit SHA bump. `SUPPORTED_SCHEMA_MAJOR` in the consumer and the pinned SHA must be bumped together on a schema-major change. - **Record on pass only** — a green run is the sole conclusive "tested." Any failure (transient infra *or* a real build/test break) leaves the version unrecorded so the next poll re-tests it, rather than masking it until the 7-day cache eviction. Retry is bounded: `resolve()` always prefers the newest ready build, so a genuinely broken nightly is superseded within the morning poll window, not retried hourly. - **Matrix = full nightly minus preinstalled images** — 4 multi-arch distros × {gfx1100, gfx1151} × {whl, tarball} = 16 legs. The version-pinned stable image is excluded (via `configure_ci.py --exclude-preinstalled`) because its ROCm version is baked in and unrelated to the nightly we pin. - **Poll window** — every 2h from 01:30–11:30 UTC. The Linux ROCm build starts ~00:02 and goes green ~02:24–03:09 on a normal night, so 01:30 sees it in progress (cheap skip), 03:30 usually catches it green, and the morning tail absorbs publish lag. A morning finish drops only the rare evening re-run, which self-heals when the next nightly supersedes it. - **Cache eviction** — the 7-day marker eviction is accepted (a stale week means one harmless re-run). - **`repository_dispatch`** (Quartz pushing on each build, removing polling+dedup) is deferred as a follow-up. ## Test plan - [x] `consume_status.py` unit tests pass (`PYTHONPATH=<quartz>/scripts/consumer python3 -m unittest discover -s .github/quartz/tests`) - [x] Consumer resolves the live Quartz `latest.json` (build=success) and emits the pinned version + install URLs - [x] `configure_ci.py --exclude-preinstalled` drops the preinstalled image and keeps `distros`/`distro_map` in sync - [x] Both workflow YAMLs parse - [x] Triggered run: gate resolves and the matrix pins the exact nightly (observed — gate resolved `10.1.0a20260902`; whl leg installed `rocm==10.1.0a20260902`) - [ ] `record` writes the dedup marker on a **green** run (blocked: current nightly fails to build due to TheRock `libhipcxx`/`libomp` packaging, so no green run yet) - [ ] Second immediate run shows the **skip** (already-tested) report --------- Co-authored-by: zichguan-amd <zichuan.guan@amd.com> Co-authored-by: Claude Opus 4 <noreply@anthropic.com>

zichguan-amd requested a review from a team as a code owner August 25, 2026 15:34

zichguan-amd force-pushed the zichguan/quartz branch from a6b3a17 to 9f3f9e7 Compare August 25, 2026 16:03

zichguan-amd force-pushed the zichguan/quartz branch from 9f3f9e7 to 35b7298 Compare August 25, 2026 16:38

HereThereBeDragons reviewed Aug 25, 2026

View reviewed changes

cgoea reviewed Aug 26, 2026

View reviewed changes

Comment thread .github/workflows/quartz_test.yml Outdated

zichguan-amd and others added 4 commits August 26, 2026 10:32

HereThereBeDragons reviewed Aug 28, 2026

View reviewed changes

HereThereBeDragons mentioned this pull request Sep 1, 2026

[Consumer] Harden the status.json consumer example: pointer fix, poll dedup, and an interactive dry-run ROCm/Quartz#91

Merged

3 tasks

HereThereBeDragons reviewed Sep 1, 2026

View reviewed changes

Comment thread .github/workflows/quartz_test.yml Outdated

Comment thread .github/quartz/consume_status.py Outdated

Comment thread .github/quartz/consume_status.py Outdated

HereThereBeDragons mentioned this pull request Sep 2, 2026

ci: hourly Quartz consumer workflow to test against latest ready nightly (v2) #512

Merged

7 tasks

zichguan-amd closed this Sep 2, 2026

zichguan-amd deleted the zichguan/quartz branch September 2, 2026 15:51

Uh oh!

Conversation

zichguan-amd commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Files

Design decisions

Test plan

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HereThereBeDragons left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Upstream Zichuan's #505 work into Quartz

Tier 1 — bug

Tier 2 — robustness (in the example)

Tier 3 — docs

After it lands

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zichguan-amd commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zichguan-amd commented Aug 25, 2026 •

edited

Loading