Skip to content

release: publish ahead of a merge, promote after it - #73

Merged
paf0186 merged 2 commits into
masterfrom
keyed-releases
Nov 16, 2026
Merged

paf0186 merged 2 commits into
masterfrom
keyed-releases

Conversation

@paf0186

@paf0186 paf0186 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Rebuilt target releases can now be uploaded from a pull request branch before it merges, and made public in seconds after it does. Until now a rebuild had to wait for the merge, and the uploads take hours: #60's 25 releases took about three, and anyone fetching on master meanwhile got artifacts their checkout called stale.

# on the PR branch, after `ltvm build release`
ltvm target publish rocky9 --kernel <kernel_dir> --arch x86_64    # uploads a draft nobody can fetch
# after the merge, on master
ltvm target promote --all --dry-run
ltvm target promote --all                                          # drafts matching master go public

How it works

  • Tags carry the inputs. A release is tagged rel-<target>-<arch>-<kver>[-<variant>]-<key>. The key digests what the publishing checkout would build from: the container, kernel and image input hashes, the declared ZFS, the release schema, and BUILD_GENERATION, a string replaced (never incremented) when ltvm's build code changes what artifacts contain. Two PRs bumping it conflict in git.
  • Branches publish drafts. From a commit not on the release repository's default branch, publish leaves the release a draft. Fetch never sees drafts.
  • Promote after the merge. ltvm target promote on a clean default-branch checkout makes public each complete draft whose key matches what that checkout computes, newest build per kernel only. A PR whose inputs no longer match master after the merge promotes nothing, safely.
  • Fetch. Takes the published release whose key matches the checkout; otherwise the newest published one, with a note that it was built from other inputs, and an update prompt when it came from a newer ltvm.
  • Pruning. On a publish or promote at the default branch's exact tip, published rel- releases of the same kernel and variant 14 or more days old are deleted, keeping the newest two. Drafts and old unprefixed releases are never pruned.
  • Older ltvm is unaffected. It filters tags by <target>-, never sees rel- releases, and keeps fetching the existing releases, which are never written again.

Publish checks more than before

Publish now refuses a release unless every component it packages is current by the builds' own staleness rules (container, kernel, image against the Lustre baked into it, Lustre against the kernel's Module.symvers, ZFS, MOFED kmods), the checkout is clean and every file the inputs read is committed, and each kernel and image records this build generation and the container it was built in. It exports the build container by the image id it recorded, keeps a release a draft until every asset and its notes (with a completion mark written last) are uploaded, and refuses a second manifest in one release.

That fixes several things that were wrong before this change:

  • publish uploaded stale kernels, images, ZFS or MOFED builds without complaint;
  • the build container was exported by its shared podman tag, so another checkout's rebuild could ship;
  • a republish that failed midway left a public release with mixed assets;
  • fetch assumed the GitHub API lists releases newest first, which it does not;
  • a variant build could rebuild the shared kernel in the variant's container and overwrite the base kernel's metadata (kernels now always build in the base container);
  • build code changes never invalidated cached kernels or the MOFED and ZFS modules built against them;
  • --no-lustre still shipped Lustre baked into the image;
  • --tag could put two kernels' manifests in one release, or let a bootable publish overwrite a target release (target releases no longer take --tag).

Also: tests/test_release_build.py's damage test reused one ExitStack across loop passes and could reach the live GitHub API (separate commit).

Cost

Input hashes and their goldens are unchanged, so nobody's existing builds go stale. Artifacts built before this change cannot be published until rebuilt, because they record no build generation or container id. That is a one-off for the next release rebuild.

docs/RELEASING.md has the workflow and every check.

Tested

  • Unit suite: 4724 passed, 19 skipped (186 new in tests/test_keyed_releases.py, including real-git tests of the default-branch check and the committed-inputs check). ruff, mypy, shell tests pass.
  • Against the live release list, read-only: fetch dry runs resolve; the committed-inputs check passes on this checkout for rocky9 base, rocky9 mofed-24 and ubuntu2404.
  • No release was published, promoted or deleted with this code; GitHub writes are covered by mocked tests only. The remote-podman paths (build --iidfile, save -o, load -i) were checked against podman's client code, not run on a Mac.

Review

Eighteen rounds of independent review (Codex gpt-6.1-sol). The first three found that a fallback based on git ancestry and version ordering could not be made safe across squash merges, forks and installed copies; the design was replaced by drafts plus promote. Later rounds found progressively narrower problems in publish's provenance checks, ending with publish reusing the builds' own staleness functions. Round 17 found nothing; round 18's two findings are fixed here.

Declined, with reasons:

  • A kernel or image records the container ltvm built for it, not the image podman actually ran under the shared tag. Another checkout of the same user rebuilding that container mid-build would go undetected. Closing it means running every build container by image id, a separate refactor; RELEASING.md says not to do that during a release build.
  • Fetch's local release-tag marker is keyed by kernel version, so mainline's stable and a declared series resolving to the same version collide. Mainline releases are not published, and the marker predates this change.

🤖 Generated with Claude Code

Patrick Farrell and others added 2 commits November 16, 2026 15:14
test_damage_since_an_older_build_is_forced built its patches once and
entered them on each pass of its loop.  The ExitStack the first pass
exited no longer patched anything, so the second pass reached the live
GitHub API and failed whenever that answer did not parse.  The patches
are now made fresh for each pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Target releases were tagged by target, arch and kernel version
alone, so a rebuild whose inputs changed but whose kernel did not
had to replace the release master fetches, and could only be
uploaded after the change merged. Uploads take hours.

Releases are now tagged rel-<target>-<arch>-<kver>[-<variant>]-<key>,
where the key digests the publishing checkout's container, kernel
and image inputs, the declared ZFS, the release schema and a
BUILD_GENERATION string that is replaced whenever build code changes
what artifacts contain. Notes name each release's target, arch,
declared kernel, variant, ltvm version and commit, and carry a
completion mark written last.

From a branch, publish uploads the release as a draft; fetch never
sees drafts. After the merge, `ltvm target promote` on the default
branch makes public the drafts whose key matches it, in seconds.
From the default branch, publish goes public directly. Fetch takes
the published release whose key matches the checkout, otherwise the
newest published one with a note, and asks for an update when that
came from a newer ltvm. On a publish or promote at the default
branch's tip, published rel- releases of the same kernel and variant
14 or more days old are pruned, keeping the newest two. Old
unprefixed releases are never written or pruned: older ltvm sees
only those.

Publish now checks every component it packages with the builds' own
staleness rules, refuses a dirty checkout or inputs not in the
commit, records the build container by image id and exports it by
that id, keeps a release a draft until every asset and its notes are
written, and refuses a second manifest in a release. Kernels are
always built in the base variant's container. Builds made before
this change stay current for ordinary use; publishing needs a
rebuild.

docs/RELEASING.md describes the workflow and every check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@paf0186
paf0186 merged commit 9950907 into master Nov 16, 2026
1 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant