[AndreasKaratzas]

Companion PR: vllm-project/ci-infra#820 (dual-mode pipeline generator — must merge together or with ci-infra first)

Currently on CI and deployment, we have:

• Too many image tiers. The old pipeline had six distinct image layers (base, ci_base, CI image, wheels, test, smoke), each driven by a separate script with its own cache keys and metadata handoffs. Adding or debugging any step required understanding all six.
• CI and release were disconnected. CI built images via ci-bake-rocm.sh; release rebuilt from scratch using a separate pipeline. The bytes shipped to users were never the bytes tested on GPUs.
• Thousands of lines of bash nobody could operate. ci-bake-rocm.sh alone was 2,784 lines. The ROCm build/deploy surface totalled ~12,000 lines of shell with no human able to read, modify or debug it under pressure.

In this PR, the purpose is to have:

Two image tiers, one Dockerfile, one HCL, three readable scripts.

1. Stack (rocm/vllm-ci:stack-rocm-<Ks>) — base OS + ROCm + PyTorch. Built only when docker/Dockerfile.rocm_base or tools/vllm-rocm/therock_wheels.py changes (content key Ks). Pushed once; reused by every subsequent build.
2. Per-commit (rocm/vllm-ci:build-<build-id>) — vLLM compiled on top of the stack. The runtime key Kr (hash of Ks + Dockerfile above the marker + pip requirement files) gates whether the runtime layer is rebuilt. A Python-only commit reuses the runtime; a requirements bump rebuilds it; a csrc-only change rebuilds neither.

One shared docker/Dockerfile.rocm. A named BuildKit context (stack) is injected by the bake HCL, so the same Dockerfile builds on any stack without duplication.

One static docker/docker-bake-rocm.hcl replaces the dynamically generated docker/ci-rocm.hcl. All parameterisation is through HCL variable blocks from environment variables. Targets cover all combinations: stack, runtime, CI test, csrc, wheels, release.

build.sh (240 lines) replaces ~5,000 lines of shell scripts. Six modes: keys, base, ci, check, csrc, release. Each mode is a self-contained named section, readable end-to-end.

Release promotes the CI digest — no rebuild. build.sh release reads the exact digest of the CI-tested image and reuses it. A non-blocking rel-csrc step compiles csrc for all 9 GPU arches (gfx90a/942/950/1100/1101/1200/1201/1150/1151); because csrc inputs rarely change (~8% of commits), this is almost always a BuildKit cache hit.

rocm-release.yaml (53 lines) + publish.sh (200 lines) handle release publishing. A trigger-only amd-release Buildkite pipeline uploads rocm-release.yaml once per main commit. publish.sh copies digests via imagetools create to ECR, Docker Hub, and S3 — no Docker build, no layer transfer.

ROCm 7.2 (apt) deprecated. All rocm_72 stack plumbing removed: Dockerfile.rocm_72_base, stack-rocm_72 bake target, rocm72-check-amd CI step, ROCm 7.2 release pipeline group. TheRock 10 (rocm) is the only supported stack.

ci-infra is dual-mode. The pipeline generator checks for .buildkite/scripts/rocm/build.sh in the vLLM checkout. Present → new pipeline renders. Absent (old checkouts, forks) → existing pipeline unchanged.

Image pre-warming. Runtime tags are content-keyed (runtime-rocm-<Kr>). A cluster cron job can pre-pull by checking tag presence before pulling, avoiding the 200-node simultaneous pull on rare runtime rebuilds.

What is preserved

• Step key image-build-amd — unchanged. All 229 GPU test steps continue to depend on it.
• SETUP_STEP_KEYS = {"refresh-rocm-base-amd", "pre-commit"}
• Smoke tests — openai-smoke and test-smoke stages in Dockerfile.rocm; rel-gpu GPU smoke on amd_mi250_1 in rocm-release.yaml.
• Process lifecycle — run-amd-test.sh handles: CONTAINER_TIMEOUT_S wall-clock timeout (timeout --foreground), structured [amd-ci-teardown] lifecycle log events, exit-code propagation, DPX GPU partition lock, HF mount check, NFS xet redirect, GPU preflight, diagnostics bundle on failure.
• Nightly tags (vllm/vllm-openai-rocm:nightly*, base-nightly*), S3 wheels per main commit, perf-eval ECR image, credential boundaries.
• AITER nightly — ci-infra old-layout path unchanged; new-layout renders the AITER overlay step via AITER_SPEC.

Net change

39 files, +1,894 / −8,922 vs origin/main.

File	Before	After
docker/Dockerfile.rocm	1,016	483
docker/Dockerfile.rocm_base	476	404
docker/docker-bake-rocm.hcl	143	118
.buildkite/scripts/rocm/build.sh	—	240 (new)
.buildkite/scripts/rocm/publish.sh	—	200 (new)
.buildkite/rocm-release.yaml	—	53 (new)
tests/tools/test_rocm_build.py	—	49 (new)
docker/context/base-wheels/.keep	—	new (gfx1250 COPY fix)
.buildkite/scripts/hardware_ci/run-amd-test.sh	1,813	182
.buildkite/hardware_tests/amd.yaml	88	47
.buildkite/release-pipeline.yaml	1,055	~820
docker/Dockerfile.rocm_72_base	388	deleted
docker/ci-rocm.hcl	411	deleted
.buildkite/scripts/ci-bake-rocm.sh	2,784	deleted
ROCm rocm/*.sh scripts (8 files)	~1,600	deleted
ROCm release scripts (3 files)	~300	deleted
requirements/build/rocm.txt	19	deleted

ROCm build/deploy surface after this PR: 1 Dockerfile (483 lines), 1 HCL (118 lines), 3 bash scripts (build 240, publish 200, run-amd-test 182 = 622 lines total).

Follow-up infra tasks

• amd-release Buildkite pipeline — create trigger-only pipeline in vllm-project's org with ROCM_STACK env var.
• rocm/vllm-ci-pr Docker Hub repo — create with push access for the amd-cpu agent token.
• amd-cpu-pr queue split — run PR builds on separate hosts with their own Hub token. One comment in build.sh marks the boundary.
• Cluster pre-warm cron — pre-pull rocm/vllm-ci:runtime-rocm-<Kr> on CI nodes with a tag-presence check.
• Hub tag retention — content-addressed tags; define an expiry window.
• gfx1250 Dockerfiles (Dockerfile.rocm_gfx1250, Dockerfile.rocm_base_gfx1250) — kept as-is, not wired into CI or release; needs owner sign-off before removal.

---

AI assistance (Claude Code) was used to draft this change. Every line was reviewed by the submitter.

[AndreasKaratzas]

Closing to reopen with a clean branch — merge conflict resolution and DCO handling were done incorrectly.