Skip to content

[ROCm] Dual-mode AMD pipeline generator for CI/release revamp - #820

Merged
khluu merged 15 commits into
mainfrom
akaratza_revamp_ci_deployment
Oct 27, 2026
Merged

[ROCm] Dual-mode AMD pipeline generator for CI/release revamp#820
khluu merged 15 commits into
mainfrom
akaratza_revamp_ci_deployment

Conversation

@AndreasKaratzas

@AndreasKaratzas AndreasKaratzas commented Oct 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Companion PR: vllm-project/vllm#61939 (the vLLM-side build scripts and Dockerfiles — must merge together or with ci-infra first: the vLLM PR deletes the scripts this repo's current template runs)

Add new-layout support to the AMD pipeline generator, gated on the vLLM checkout having .buildkite/scripts/rocm/build.sh (the revamped build entrypoint introduced in the companion vLLM PR). Old checkouts and forks without that file continue to render the existing pipeline unchanged.

New-layout behaviour (when build.sh exists)

bootstrap-amd.sh:

  • Computes and exports ROCM_CI_REPO (trust rule: rocm/vllm-ci iff non-PR build of vllm-project/vllm main and not AITER nightly; else rocm/vllm-dev, where vLLM's build.sh prefixes tags and caches with pr<N>- and trusted builds never read).
  • Parses hardware_tests/amd.yaml build steps and passes them as amd_build_steps JSON to minijinja-cli.
  • Renders the template with ci_repo and amd_build_steps variables.

test-template-amd.j2:

  • Renders refresh-rocm-base-amd + image-build-amd build steps on queue amd-cpu with ROCM_CI_REPO in env.
  • $ROCM_CI_REPO is kept as a literal in concurrency_group (Buildkite interpolates it at pipeline upload time, not at template render time).
  • Pod image for GPU test steps becomes $ROCM_CI_REPO:build-$BUILDKITE_BUILD_ID.
  • Workspace emptyDir removed (new image already contains /vllm-workspace).
  • verify-native-ci-base-amd dependency removed from GPU test step depends_on for new-layout builds.

Gated per layout (kept for old checkouts, not rendered for the new one):

  • DinD: on the new layout every AMD GPU job is a native pod, in the template and in the Python generator (dind is ignored there).
  • The DinD env block (DOCKER_IMAGE_NAME, VLLM_CI_BASE_IMAGE: rocm/vllm-dev:ci_base, VLLM_CI_FALLBACK_IMAGE).
  • The artifact/workspace env (VLLM_CI_ARTIFACT_STEP, VLLM_CI_ARTIFACT_CHECKSUM_GLOB, VLLM_CI_USE_ARTIFACTS, VLLM_CI_ARTIFACT_GLOB, VLLM_CI_WORKSPACE, VLLM_CI_REQUIRE_WORKSPACE_MOUNT): the per-build image is the artifact.
  • Multi-node steps (num_nodes >= 2) fail with a clear message on the new layout: its pods are single-node.

Added:

  • RDMA /dev/infiniband hostPath volume + mount in amd_native_k8s_plugin (template only), gated on a step.rdma field no step sets today.
  • A change to run-amd-test.sh or to the pytest plugins it loads (.buildkite/scripts/hardware_ci/amd_ci_*) unblocks every native AMD job.

ci_selector:

  • tools/vllm-rocm/ is AMD-exclusive: a therock_wheels.py change (a ROCm stack key input) runs the AMD jobs, not every pipeline.
  • A release-only file keeps the generator's run_all escalation in its claim.
  • Tests read the ROCm base step of either layout (ensure-ci-base-amd / refresh-rocm-base-amd).

amd.py / buildkite_step.py:

  • rocm_build_layout(), rocm_ci_repo(), get_amd_pod_image() — new-layout trust rule, env, concurrency group.
  • Pod image and VLLM_CI_BASE_IMAGE point at the per-build image; no workspace volume.

What is unchanged for old-layout checkouts

Everything. The {% if amd_build_steps %} gate in the template means old-layout checkouts (forks, pre-merge branches) render identically to today. The generator's Python path also keeps the old behaviour when rocm_build_layout() returns false.

Files changed

13 files, +488 / −62 vs origin/main (the table lists the main ones).

File Change
buildkite/bootstrap-amd.sh Trust rule + amd_build_steps JSON render
buildkite/test-template-amd.j2 New-layout build steps block; DinD removal; RDMA volume; stale artifact env removal
buildkite/pipeline_generator/amd.py rocm_build_layout(), rocm_ci_repo(), get_amd_pod_image()
buildkite/pipeline_generator/buildkite_step.py New-layout pod image + env; DinD cleanup
buildkite/tests/test_amd_template.py Both layouts; trust cases; file gating; AITER nightly
buildkite/tests/pipeline_generator/test_amd.py New-layout: trust, env, concurrency, queue, pod image, volumes
buildkite/tests/test_aiter_nightly_template.py Bootstrap variable changes
README.md Dual-layout and trust note; the ROCm release builds in vLLM's release pipeline, nothing to set up here
buildkite/ci_selector/tests/test_classify.py The ROCm release runs CI's build.sh, so a release ref may also be a live step's script: the test now checks such a file keeps its live steps' claim instead of assuming no overlap

AI assistance (Claude Code) was used to draft this change. Every line was reviewed by the submitter.

@AndreasKaratzas
[ROCm] Dual-mode AMD pipeline generator for CI revamp
4ec71ae 
Add new-layout support gated on the vLLM checkout having
.buildkite/scripts/rocm/build.sh (the revamped build entrypoint).
Old checkouts continue to render the existing pipeline unchanged.

New layout:
- bootstrap-amd.sh computes ROCM_CI_REPO (trusted: rocm/vllm-ci iff
  non-PR build of vllm-project/vllm and not AITER nightly) and passes
  amd_build_steps JSON to the Jinja template.
- Template renders refresh-rocm-base-amd + image-build-amd build steps
  on amd-cpu queue; pod image becomes $ROCM_CI_REPO:build-$BUILD_ID;
  workspace emptyDir and verify-native-ci-base-amd dep are dropped.
- amd.py / buildkite_step.py updated to match: trust rule, env,
  concurrency group with $ROCM_CI_REPO literal for Buildkite runtime
  interpolation, timeout 540 for base.

Fix: template was erroneously Jinja-substituting $ROCM_CI_REPO in
concurrency_group; it must remain a literal for Buildkite to expand.

Tests: 295/295 pass (up from 59 in test_amd_template.py alone).

Co-Authored-By: Claude <noreply@anthropic.com>

Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas
[ROCm] Drop DinD dead code; RDMA pod volume; unblock run-amd-test.sh …
00bacf9 
…changes

All GPU steps are now native (pod-based) — the DinD branch is dead code:
- Remove step.dind == false guards from depends_on, agent_tags, k8s plugin,
  and env rendering; all steps unconditionally get the native plugin and env
- Remove DinD env block (DOCKER_BUILDKIT, DOCKER_IMAGE_NAME, VLLM_CI_BASE_IMAGE
  pointing at the deleted ci_base, VLLM_CI_FALLBACK_IMAGE)
- Keep verify-native-ci-base-amd dependency for old-layout non-gated steps
  (the old CI path still uses it)
- run-amd-test.sh file-diff now unblocks any step that calls it, not just
  steps where dind == false
- Add RDMA /dev/infiniband hostPath volume + volumeMount in pod spec,
  gated on step.rdma field (defaults false); replaces the deleted
  /dev/infiniband check in run-amd-test.sh's docker run args

ci-infra: 295/295 tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>

Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas
[ROCm] Remove stale DinD artifact env vars from GPU step template
f0e2160 
In the pod-based CI model the vLLM wheel is baked into the CI image
at /opt/vllm-wheels/. The image is the artifact — there is no separate
vllm-rocm-install.tar.gz Buildkite artifact to download or verify.

Remove from GPU test step env:
  VLLM_CI_ARTIFACT_STEP, VLLM_CI_ARTIFACT_CHECKSUM_GLOB
  VLLM_CI_USE_ARTIFACTS, VLLM_CI_ARTIFACT_GLOB
  VLLM_CI_WORKSPACE, VLLM_CI_REQUIRE_WORKSPACE_MOUNT
  (VLLM_CI_BASE_IMAGE, HF_HOME, HF_HUB_*, PYTORCH_ROCM_ARCH kept)

run-amd-test.sh no longer reads these vars, so the template change
has no observable effect on running jobs. ci-infra tests: 295/295.

Co-Authored-By: Claude <noreply@anthropic.com>

Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas AndreasKaratzas mentioned this pull request Oct 15, 2026
Closed
@AndreasKaratzas
AndreasKaratzas force-pushed the akaratza_revamp_ci_deployment branch from 95c4030 to f0e2160 Compare October 16, 2026 02:06
@AndreasKaratzas AndreasKaratzas mentioned this pull request Oct 16, 2026
Open
AndreasKaratzas and others added 3 commits October 17, 2026 04:43
@AndreasKaratzas @claude
[ROCm] Keep the old AMD layout byte-compatible; trust main only; retr…
af61f6f 
…y infra failures

- test-template-amd.j2: restore main's native/DinD env, depends_on, agent tags
  and k8s plugin for checkouts without build.sh; new-layout build steps retry
  on stack_error / agent_stop / agent_refused plus the step's own rules.
- Trusted repo (rocm/vllm-ci) only for non-PR builds of main (bootstrap and
  generator), so release branches and forks write to rocm/vllm-ci-pr.
- README: AITER nightly auto-selection, trust rule.
- Tests for the trust rule by branch, retry rules and GPU job env per layout.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas
Merge remote-tracking branch 'origin/main' into akaratza_revamp_ci_de…
83950c8 
…ployment

Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas @claude
[ci-selector] Model the generator's manual-only devices in the replica
69ddee9 
#826 made `_step_should_run` skip steps on MANUAL_ONLY_DEVICES (L4) before
anything else. The contract test execs that function's source in a namespace
without the name, so it failed with a NameError once a PR touching the
generator ran the ci_selector tests. Take MANUAL_ONLY_DEVICES from the
generator into handwritten.py, check it first in the replica, and add the
manual-only devices to the contract test's input space.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas

AndreasKaratzas commented Oct 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Full CI build:

AndreasKaratzas and others added 3 commits October 18, 2026 07:02
@AndreasKaratzas @claude
Merge origin/main into akaratza_revamp_ci_deployment
4462175 
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas @claude
Revert "[ci-selector] Model the generator's manual-only devices in th…
db87919 
…e replica"

This reverts commit 69ddee9. It modelled #826's MANUAL_ONLY_DEVICES, which
main reverted in #828: the generator no longer defines the name, so
importing it would break ci_selector.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas @claude
Merge origin/main into akaratza_revamp_ci_deployment
7aee02a 
Conflict in buildkite/test-template-amd.j2 with #833 (one AMD job title
pattern ":amd: (<pool tag>) <Title>"):
- build steps: keep this PR's build.sh layout (steps rendered from vLLM's
  hardware_tests/amd.yaml); the old-layout branch takes #833's label
  ":docker: Refresh ROCm base".
- GPU jobs: take #833's label "{{ step.label }}" (amd.py now builds the full
  title) with this PR's condition `step.dind == false and not amd_build_steps`.

tests/test_amd_template.py: the build.sh-layout tests find the GPU job by its
#833 label, and by its amd_mi* agent queue instead of a "mi..." label prefix.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas AndreasKaratzas mentioned this pull request Oct 19, 2026
Closed
@AndreasKaratzas
AndreasKaratzas marked this pull request as ready for review October 19, 2026 05:37
@AndreasKaratzas @claude
[ROCm] Write PR images to rocm/vllm-dev
327790e 
Untrusted ROCm builds (PRs, forks, release branches, the AITER nightly) now
push to rocm/vllm-dev instead of rocm/vllm-ci-pr, which is private and could
not be made public. rocm/vllm-dev is public, already receives PR images
(ci_base-build-*), and its tags (ci_base*, nightly*) cannot collide with
ours (build-*, pr<N>-*, cache-pr*). The trust rule is unchanged: trusted
main builds write rocm/vllm-ci and never read rocm/vllm-dev.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
AndreasKaratzas added a commit to AndreasKaratzas/vllm that referenced this pull request Oct 19, 2026
@AndreasKaratzas @claude
[ROCm] Name rocm/vllm-dev as the PR image repo
6ed1847 
ci-infra now points untrusted ROCm builds at rocm/vllm-dev instead of
rocm/vllm-ci-pr (vllm-project/ci-infra#820). Only comments change here.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas AndreasKaratzas mentioned this pull request Oct 19, 2026
Open
AndreasKaratzas and others added 5 commits October 21, 2026 01:10
@AndreasKaratzas @claude
[ROCm] ROCm release builds in vLLM's release pipeline; release refs m…
4c9d835 
…ay be shared

- README: vLLM's release pipeline now builds the ROCm release on its own
  cpu_queue_release agents (vllm#61939); there is no amd-release pipeline
  to set up.
- ci_selector: the ROCm release runs CI's build.sh, so a release ref can
  also be a live step's script. Replace the test that release refs never
  overlap live steps with the property it guards: such a shared file keeps
  its live steps' claim and is never zeroed as release-only.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas @claude
[ci-selector] Accept a build.sh checkout without ensure-ci-base-amd
cc7b380 
AMD_ALWAYS_RUN_STEP_KEYS lists the base step of both ROCm build layouts.
A vLLM checkout on the build.sh layout (vllm#61939) has no
ensure-ci-base-amd, so test_auto_run_semantics would fail once VLLM_PIN
moves past it. Expect that key only on the old layout.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas @claude
[ROCm][CI] Address the final review of the AMD pipeline for the build…
f4616be 
….sh layout

- Generator: on the build.sh layout every AMD GPU job is a native pod, as the
  template renders it (a mirror without dind: false no longer renders a
  DinD job), and pods no longer get the artifact/workspace env.
- Template: a change to the AMD CI pytest plugins
  (.buildkite/scripts/hardware_ci/amd_ci_*) unblocks native jobs like
  run-amd-test.sh; multi-node steps fail clearly on the build.sh layout;
  no workspace env on its pods.
- Selector: tools/vllm-rocm/ is AMD-exclusive, so a therock_wheels.py change
  runs the AMD jobs, not every pipeline; the release-ci claim carries the
  generator's run_all escalation.
- Tests: layout-aware base step in test_rust_reaches_hardware_image_consumers,
  a neutral synthetic gated build step instead of a ROCm 7.2 one, the plugin
  prefix in the runtime-dependency existence check.
- README: pin the aiter-nightly-overlay.sh link to a pre-revamp commit; the
  ROCm release builds its own stack and runtime.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas @claude
Merge origin/main into akaratza_revamp_ci_deployment
979a596 
Keeps #737's MI250 queue consolidation in the template next to the
build.sh layout's native agent_tags rule, and updates #685's ROCm kernel
map doc for the build.sh layout (map written in the csrc stage, exported
by the kernel-symbol-map target, uploaded by build.sh ci).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@AndreasKaratzas @claude
[CI][ROCm] Test that build.sh's registry-error retry reaches the base…
fac62ed 
… step

- test_amd_template.py: the base step fixture carries vLLM amd.yaml's
  exit_status 75 rule (build.sh registry error), and the rendered step
  must keep it after the infra rules.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
@khluu
khluu merged commit 5fa07cf into main Oct 27, 2026
3 checks passed
@AndreasKaratzas
AndreasKaratzas deleted the akaratza_revamp_ci_deployment branch October 27, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AndreasKaratzas @khluu