Repository navigation
build: ltvm build release rebuilds every published target - #72
Merged
Merged
Conversation
Rebuilding the releases by hand took four driver scripts and lost hours to failures that showed up only after long builds: a status check that read the wrong exit code, disks that filled, and a kernel and an image package that no longer built. `ltvm build release --lustre-tree T` takes the set to build from the GitHub releases for the host's arch and runs it in three passes: every target's container and image package layers first (a new `build image --base-only`), then `build all` per target and kernel, breadth first, in parallel slots pinned to NUMA nodes, each with its own Lustre clone; then reads each result back before counting it. It checks disk space for the whole run before starting, logs each build to its own file, ends with a table and summary.json, and keeps a ledger so a rerun skips what finished. docs/RELEASING.md has the procedure, including keeping a shared host's podman storage on a large filesystem between rebuilds. Also: `target list` used an older rule than `target fetch` to tell a base release from a variant's and showed no release for debian13, opensuse156 and ubuntu2604-server; both now share fetch's rule. The low disk warning looks at podman's real storage location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From an independent review of the first version: - A second run sharing the work directory checked its own commit out in the Lustre trees the first was building in; the work directory is now locked for a run. - Ctrl-C stopped only the ltvm processes, so podman builds under them kept running and image export's cleanup was skipped. Each build now runs in its own session; stopping interrupts the whole group, then kills what is left after two minutes, and a build cannot start between the stop and its look at what is running. - Verification could pass a kernel with no boot files or modules, a snapshot with no modules, and an image without this Lustre. It now checks the kernel's outputs, the snapshot's modules, and the image's input hash with the snapshot's Lustre digests folded in, as build_image does; and it checks the kernel directory the build actually used (from its log, kept in the ledger), not the newest one cached. - Kernel, Lustre and ZFS builds sized make -j from os.cpu_count(), so a slot pinned to 32 CPUs ran 128 jobs. They now follow the CPU affinity, which also helps any build run under taskset. - The disk check now counts TMPDIR (image export), sizes slots from the builds left, needs nothing when nothing is left, and leaves podman's storage unchecked on macOS, where it is in the podman machine. Publishing in the docs names the arch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Two runs with different work directories could still build into the same artifacts. A run now also locks the artifacts directory. - A targets.yaml error dropped that target from the plan without a word; it now stops the run. - The summary names each build's exact kernel directory, and the publishing steps use it: the short name picks the newest kernel cached, which need not be the one built. - The preflight runs for every target, finished ones too, so each keeps the build container that publishing exports. - Lustre clones no longer borrow the source checkout's objects, so a later run from another source still works; old borrowing clones are made again. - The disk check counts what podman keeps per target and what the trees keep per build, not only per slot. - SIGTERM stops the builds as Ctrl-C does, before the locks go. - The ledger is not used from an ltvm checkout with uncommitted changes, and `--fresh` ignores it when something outside the two commits changed, such as a newer distro kernel. - A cross build's tool prebuild runs through the podman cleanup wrapper, so an interrupted one does not leave its container running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Which builds were finished was decided before the run took its locks, so another run could replace what was verified in between. The locks now come first; only a dry run plans without them. - The cross build's tool prebuild went through the podman cleanup wrapper with check=True, which lets a cleanup EOF pass as success with nothing checking the tools came out. Any nonzero exit now fails it. - The ledger named ltvm by its baked version, which a branch checkout does not refresh; it now uses the checkout's HEAD. - --fresh now forces each kernel, Lustre and image build, as the operator using it expects; the docs say what it leaves cached. - The disk check counts the preflight's podman needs when no builds are left, and charges each build's artifacts to the filesystem its target's directory is on, cold storage for an archived target. - More jobs than CPUs gave a slot an empty CPU set, and taskset then failed every command in it; that is refused now, as is an empty set given by hand. - Worker threads start inside the interrupt handler, so a Ctrl-C while they start still stops what they launched. - The docs say mainline's moving kernels are not mapped from releases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- On an interrupt, the run waited by joining its worker threads again, which on Python 3.12 can return while a worker is still verifying or writing the ledger. Workers now signal when they are done, and the run waits for that. - release was not among the commands that share new files with the artifacts group, so its locks and work directory came out 0644 and blocked the next group member. - The disk check counted a target as cached when its build container was, though its image package layers may not be. - Parallel cross-build preflights mounted the shared targets/common with a private SELinux label, taking it from each other; it is shared now. - With fewer builds left than slots, the automatic CPU sets were cut for the larger count and the unused CPUs dropped; they are now computed for the slots used. - The docs no longer claim a newer distro kernel is picked up by itself: it comes from targets.yaml and the Lustre tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A build's ledger entry survived a later failed attempt at it, so after a forced rebuild failed partway, leaving old and new kernel outputs mixed, the next plain run counted it done. The entry is now dropped before each attempt and written again only once it verifies. - Verification took the snapshot's recorded module digest on trust; it now recomputes it, so a snapshot that lost a module fails. - The disk check reserves the preflight's working room even when no builds are left. - The docs say how a rebuild picks its kernel, including the cases that move it to a newer one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Two runs with their own artifacts and work directories still retagged each other's podman images. A run now also locks the podman storage it builds into; each directory is locked once even when two of them coincide. - A run launched in the background can inherit SIGINT ignored, and its builds then skipped their cleanup when stopped. It restores a working SIGINT handler for its builds. - The disk check counts the Lustre ccache each new target grows in the work directory, and leaves out filesystems the run needs nothing on. - A lock directory that cannot be written is an error, not a crash. - The docs say a change to ltvm's build code needs --fresh, since the caches key on the build inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Stopping set SIGINT ignored before it shut the launch door, so a build starting at that moment inherited it and could not be interrupted. The runner is now marked stopped under its launch lock first. - A release carrying ZFS would have been rebuilt without it. Such releases are listed and not built; the docs say to use `build all --zfs` for them. - The podman storage lock lived in the storage's parent, which need not be writable and is shared by unrelated stores. It is now a lock file in the user's cache named for the storage. - A cross build's preflight writes its tools into the target's artifacts, which the disk check now counts; with --no-preflight only targets with builds left count for new containers. - The docs say refreshing package layers means removing every ltvm-image image, since targets share them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test run on Sookie wedged for over an hour: fuse-overlayfs spun under two concurrent cached podman builds, and both `build all` processes waited on it with no output and no end. Nothing in the release run noticed. A build whose log has not grown for --stall-minutes (60 by default) is now interrupted like a Ctrl-C, killed after the grace period, and reported as stalled, so the rest of the run goes on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Verification compared the snapshot with its own recorded digests only, so a kernel rebuilt with another ABI under an older snapshot passed. It now checks the live kernel's Module.symvers against the one the snapshot was built with. - A reused Lustre clone kept the old configure state when only the commit changed, so the build could carry the old version. A clone whose commit changes is cleaned first. - Releases are found by their manifest's name rather than their tag, which `target publish --tag` can set to anything. - Automatic slots are capped by the targets left, not the builds: one target's kernels run one at a time. - The podman probes made while planning time out with an error rather than hang behind a wedged build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two wedged test runs on Sookie came down to podman, not the build: rootless podman ignores storage.conf's runroot and keeps every store's locks and mount state for a user in $XDG_RUNTIME_DIR/containers. A release run beside another podman workload of the same user, even on separate storage, wedged fuse-overlayfs both times. The podman lock is now one per user rather than per storage, and the docs explain it. A build stopped by Ctrl-C or SIGTERM is reported as interrupted, not as failed at an unknown step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An error while waiting on a build (a full log disk, say) dropped it from the runner untracked and still running. Any end of the wait now stops and reaps it, and the watchdog's log line is best effort. - Verification recomputed the snapshot's module digest, which hashes nanosecond file times a snapshot copy need not keep, so a good build could fail. The ledger now keeps each verified snapshot's modules by path and size, and a rerun compares against that. - The disk check reserved nothing for a ccache directory that existed but was empty; it now reserves what each may still grow by. - The default work directory is per user, since git refuses a clone another user owns; the artifacts lock stays shared. - The docs say CPU pinning reaches local podman's containers only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The ledger's record of a snapshot's modules used their paths and sizes, so a same-size rebuild from other sources passed as the verified one. It now hashes their contents. - A ZFS asset anywhere in a release kept every kernel in it from being built; ZFS is now judged per manifest, by its kernel. - With a remote podman service the storage path is the server's, so the disk check leaves it unchecked, as on macOS. - The per-user podman lock no longer asks podman where its storage is, so planning queries podman once. - When builds were stopped, the summary says how to find and remove the build containers podman can leave behind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kernel or image copy cut short left files that the completeness check, which only asked whether they existed, still accepted: a later build hit the cache and a release run counted the result as good. Kernel files must now match the sizes meta.json records (every kernel on hand here, fetched or built, does) and an image must not be empty. Also from review: a mainline release candidate matched the numbers of a declared series; release candidates are now reported as unmatched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An image cut short by an interrupted copy passed as long as it was not empty, and a build or release run then kept it. An image now has to hold every block its ext4 superblock declares; that matches exactly on every real image checked, built and fetched. - On macOS the Lustre ccaches are podman volumes in the podman machine, but the release disk check charged a full cache per target to the host's work directory. They are left out, like podman's storage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The ledger's fingerprint covered only the snapshot's Lustre modules, so a later run accepted a snapshot that had lost mount.lustre or lctl, or a kernel whose installed modules were cut short. It now covers every file of the snapshot by contents and the installed kernel modules by path and size. - A kernel published with ZFS could be scheduled again through another manifest of the same kernel, and that build would replace the ZFS one without ZFS. Such kernels are now left out whichever manifest names them. - The docs say an aarch64 builder's podman machine needs 8 GiB: BTF generation for the SLES kernels was killed in a 4 GiB one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The ledger's fingerprint did not cover the image, so an image another build installed for the same kernel passed as the verified one. It now records which versioned image is current, and its size. - A build whose recorded files had changed was run again, but without --force, so its kernel's cache could hand back the damage and the run record it as good. Such builds are now forced. - The base build's fingerprint took in variant snapshots nested in its snapshot, so building a variant made the base look changed. They are left out, as publishing leaves them out of the base release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… crash - Each target's kernel ccache lives beside its artifacts and can grow to ccache's cap; the release disk check now reserves what it may still take there. - On macOS a target whose podman cache volume does not exist yet may fall back to a host ccache, so the host room is reserved for it. - The kernel size check read vmlinux after checking it existed; a rebuild renaming it in between raised instead of reporting the kernel incomplete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A build's ledger entry was deleted when an attempt began, so after an interrupted forced rebuild a plain resume saw ordinary new work, did not force it, and the kernel cache could hand back the half-replaced kernel to be recorded as good. The ledger now marks the attempt instead, and every run forces that build until one verifies. On macOS, a host ccache directory with contents means the build fell back to it, so its room is reserved even when a podman volume exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The fingerprint a rerun checks did not cover the kernel binaries or build tree, so an interrupted kernel rebuild that left them partly replaced, with the same lengths, passed. It now hashes vmlinux and vmlinuz and lists the build tree by path and size. - A file vanishing while the fingerprint was taken raised, ending the whole release run; it now counts as a change, and the build is rebuilt. - An ext4 superblock with an impossible block size made the image check build a huge number; such an image is not whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A Lustre clone moved to the new commit before its old build state was cleaned, so an interruption between the two left a retry seeing the right HEAD and skipping the clean. The clean now comes first. - On macOS the image export's conversion worker runs in the podman machine, beyond a signal to the build's process group, so a stopped release left it running. It goes through the podman cleanup wrapper, and any nonzero exit fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mainline release built from `stable` carries only its version (7.0.8), which mapped to the numbered 7.0 series, so a release run would have rebuilt and republished it under the wrong kernel name. Every release of a target that declares latest, stable or longterm is now listed as unmatched, as the docs promised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a build's commits had changed, its ledger entry no longer matched and its saved fingerprint went unchecked, so a kernel damaged since it verified ran again without --force and its cache could hand the damage back. The last verified fingerprint is now compared whatever its commits, and a build whose files changed is forced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macOS idle-sleeps a Mac left alone even on AC power, and every build in a release run pauses with it: one aarch64 kernel build took four hours of wall clock for 30 minutes of work overnight. A real run now holds an idle sleep assertion with caffeinate for as long as it lasts. A closed lid still sleeps the Mac, so the docs say to leave it open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ltvm build release --lustre-tree <tree>rebuilds every published target for this host's arch in one command, and finds what is going to break early. Rebuilding the releases for #60 by hand took four driver scripts and lost hours to failures that only showed up after long builds: a driver reading the wrong exit status, full disks, a kernel that no longer compiled, and distro packages that no longer installed.Stacked on #70 (the image build fixes), which it needs to build any image today. The base is
image-build-fixes; retarget to master once #70 lands.What it does
--only TARGET[:KERNEL]narrows it. Releases it cannot map are listed, not built: ZFS releases, and every release of a target with moving kernels (mainline).build image --base-only. A broken package fails here in minutes; on its first real run it caught both the Ubuntu 26.04 locales break and the Whamcloud gitiles 404 within 20 minutes.ltvm build allper target and kernel, breadth first (every target's first kernel before any target's second), in parallel slots pinned to whole NUMA nodes, each with its own Lustre clone. One target's kernels never build at once. make's job count now follows CPU affinity everywhere, so a pinned slot no longer runs one job per host CPU.--force.--freshforces everything.--stall-minutes) is stopped and reported. Ctrl-C and SIGTERM stop every build's process group cleanly.caffeinate; overnight, idle sleep had stretched a 30 minute kernel build to four hours.summary.jsonwith each build's exact kernel directory fortarget publish.Also in this branch:
target listused an older rule thantarget fetchto tell a base release from a variant's, and showed no release for debian13, opensuse156 and ubuntu2604-server. They now share fetch's rule.docs/RELEASING.md has the procedure: x86_64 on a large Linux host, aarch64 on a Mac with an 8 GiB podman machine, keeping a shared host's podman storage off a small root filesystem, and publishing from the summary.
Tested
Review
Twenty rounds of independent review (Codex gpt-6.1-sol), with fixes between them, until a round found nothing new and one more confirmed it. The early rounds found real bugs in locking, cancellation and verification; the late ones found edge cases needing interrupted or concurrent manual work on the artifacts.
Declined, with reasons:
🤖 Generated with Claude Code