Skip to content

build: ltvm build release rebuilds every published target - #72

Merged
paf0186 merged 24 commits into
image-build-fixesfrom
release-build
Nov 16, 2026
Merged

paf0186 merged 24 commits into
image-build-fixesfrom
release-build

Conversation

@paf0186

@paf0186 paf0186 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

ltvm build release --lustre-tree <tree> rebuilds every published target for this host's arch in one command, and finds what is going to break early. Rebuilding the releases for #60 by hand took four driver scripts and lost hours to failures that only showed up after long builds: a driver reading the wrong exit status, full disks, a kernel that no longer compiled, and distro packages that no longer installed.

ltvm build release --lustre-tree ~/lustre-release --dry-run   # the plan, CPU sets, disk check
ltvm build release --lustre-tree ~/lustre-release

Stacked on #70 (the image build fixes), which it needs to build any image today. The base is image-build-fixes; retarget to master once #70 lands.

What it does

  • Plan. The set to build is what GitHub publishes for the arch: each base manifest, mapped back to its kernel in targets.yaml (25 releases today, 12 x86_64 and 13 aarch64). --only TARGET[:KERNEL] narrows it. Releases it cannot map are listed, not built: ZFS releases, and every release of a target with moving kernels (mainline).
  • Preflight. Every target's build container and image package layers first, in parallel, through a new build image --base-only. A broken package fails here in minutes; on its first real run it caught both the Ubuntu 26.04 locales break and the Whamcloud gitiles 404 within 20 minutes.
  • Builds. ltvm build all per target and kernel, breadth first (every target's first kernel before any target's second), in parallel slots pinned to whole NUMA nodes, each with its own Lustre clone. One target's kernels never build at once. make's job count now follows CPU affinity everywhere, so a pinned slot no longer runs one job per host CPU.
  • Verify. A build counts only when its kernel (binary sizes, Module.symvers matching the snapshot), Lustre snapshot (from the requested commit) and image (whole ext4, current against its inputs and this Lustre) all check out. An exit status of 0 is not enough.
  • Resume. A ledger records each verified build with the ltvm and Lustre commits and a fingerprint of everything it left to publish. A rerun skips builds still matching; a build whose files changed, or whose last attempt never verified, is rebuilt with --force. --fresh forces everything.
  • Safety. It checks disk space for the whole run first (artifacts, podman storage, TMPDIR, Lustre clones and ccaches, per filesystem). It takes locks so two runs cannot collide; rootless podman keeps one run directory per user whatever storage it uses, so the podman lock is per user. A build silent for an hour (--stall-minutes) is stopped and reported. Ctrl-C and SIGTERM stop every build's process group cleanly.
  • Mac. A run holds the Mac out of idle sleep with caffeinate; overnight, idle sleep had stretched a 30 minute kernel build to four hours.
  • Output. A log per build, a summary table, summary.json with each build's exact kernel directory for target publish.

Also in this branch:

  • target list used an older rule than target fetch to tell a base release from a variant's, and showed no release for debian13, opensuse156 and ubuntu2604-server. They now share fetch's rule.
  • The kernel and image completeness checks used by every build reject a truncated kernel (sizes against meta.json) and a truncated image (ext4 superblock), and a cross build's tool prebuild and the macOS image worker go through the podman cleanup wrapper.

docs/RELEASING.md has the procedure: x86_64 on a large Linux host, aarch64 on a Mac with an 8 GiB podman machine, keeping a shared host's podman storage off a small root filesystem, and publishing from the summary.

Tested

  • Unit suite: 4339 passed, 19 skipped. ruff, mypy, shell tests pass.
  • On Sookie, at the final head and earlier ones:
    • ubuntu2404 and ubuntu2604 built in parallel on NUMA nodes 4-5 and 6-7, verified, 25 minutes, exit 0.
    • At the final head (d8b1d5b), a rerun after the ltvm commit changed rebuilt both from cache in 10 minutes and verified them with every check; the next rerun found both done in under a minute.
    • SIGTERM mid-build stopped both builds and their podman processes within seconds and wrote the summary (exit 130).
    • The podman probe timeout and the store lock were both exercised by a wedged podman.
  • The checks on real artifacts were compared against every kernel, snapshot and image on Sookie and here (Module.symvers digest, kernel sizes, ext4 geometry): all matched.

Review

Twenty rounds of independent review (Codex gpt-6.1-sol), with fixes between them, until a round found nothing new and one more confirmed it. The early rounds found real bugs in locking, cancellation and verification; the late ones found edge cases needing interrupted or concurrent manual work on the artifacts.

Declined, with reasons:

  • A kernel cache with a partly installed modules tree, on a run with no earlier verified build to compare against. This is the kernel cache's own completeness rule, used by every build; the fix is to record the module inventory when a kernel is built, a separate change.
  • A ZFS asset left behind after a ZFS release is republished without ZFS keeps that release out of the plan; telling would mean downloading every manifest. The release is named in a warning, and no ZFS releases exist.
  • On macOS, a podman cache volume that exists but was never writable makes the disk check leave out host ccache room. Reserving it always would refuse runs that fit on a Mac.
  • CPU pinning does not reach containers of a podman service reached through CONTAINER_HOST (documented).

🤖 Generated with Claude Code

Patrick Farrell and others added 24 commits November 15, 2026 22:31
Rebuilding the releases by hand took four driver scripts and lost
hours to failures that showed up only after long builds: a status
check that read the wrong exit code, disks that filled, and a kernel
and an image package that no longer built.

`ltvm build release --lustre-tree T` takes the set to build from the
GitHub releases for the host's arch and runs it in three passes:
every target's container and image package layers first (a new
`build image --base-only`), then `build all` per target and kernel,
breadth first, in parallel slots pinned to NUMA nodes, each with its
own Lustre clone; then reads each result back before counting it.
It checks disk space for the whole run before starting, logs each
build to its own file, ends with a table and summary.json, and keeps
a ledger so a rerun skips what finished.  docs/RELEASING.md has the
procedure, including keeping a shared host's podman storage on a
large filesystem between rebuilds.

Also: `target list` used an older rule than `target fetch` to tell a
base release from a variant's and showed no release for debian13,
opensuse156 and ubuntu2604-server; both now share fetch's rule.  The
low disk warning looks at podman's real storage location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From an independent review of the first version:

- A second run sharing the work directory checked its own commit out
  in the Lustre trees the first was building in; the work directory
  is now locked for a run.
- Ctrl-C stopped only the ltvm processes, so podman builds under them
  kept running and image export's cleanup was skipped.  Each build now
  runs in its own session; stopping interrupts the whole group, then
  kills what is left after two minutes, and a build cannot start
  between the stop and its look at what is running.
- Verification could pass a kernel with no boot files or modules, a
  snapshot with no modules, and an image without this Lustre.  It now
  checks the kernel's outputs, the snapshot's modules, and the image's
  input hash with the snapshot's Lustre digests folded in, as
  build_image does; and it checks the kernel directory the build
  actually used (from its log, kept in the ledger), not the newest one
  cached.
- Kernel, Lustre and ZFS builds sized make -j from os.cpu_count(),
  so a slot pinned to 32 CPUs ran 128 jobs.  They now follow the CPU
  affinity, which also helps any build run under taskset.
- The disk check now counts TMPDIR (image export), sizes slots from
  the builds left, needs nothing when nothing is left, and leaves
  podman's storage unchecked on macOS, where it is in the podman
  machine.  Publishing in the docs names the arch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Two runs with different work directories could still build into
  the same artifacts.  A run now also locks the artifacts directory.
- A targets.yaml error dropped that target from the plan without a
  word; it now stops the run.
- The summary names each build's exact kernel directory, and the
  publishing steps use it: the short name picks the newest kernel
  cached, which need not be the one built.
- The preflight runs for every target, finished ones too, so each
  keeps the build container that publishing exports.
- Lustre clones no longer borrow the source checkout's objects, so a
  later run from another source still works; old borrowing clones are
  made again.
- The disk check counts what podman keeps per target and what the
  trees keep per build, not only per slot.
- SIGTERM stops the builds as Ctrl-C does, before the locks go.
- The ledger is not used from an ltvm checkout with uncommitted
  changes, and `--fresh` ignores it when something outside the two
  commits changed, such as a newer distro kernel.
- A cross build's tool prebuild runs through the podman cleanup
  wrapper, so an interrupted one does not leave its container running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Which builds were finished was decided before the run took its
  locks, so another run could replace what was verified in between.
  The locks now come first; only a dry run plans without them.
- The cross build's tool prebuild went through the podman cleanup
  wrapper with check=True, which lets a cleanup EOF pass as success
  with nothing checking the tools came out.  Any nonzero exit now
  fails it.
- The ledger named ltvm by its baked version, which a branch checkout
  does not refresh; it now uses the checkout's HEAD.
- --fresh now forces each kernel, Lustre and image build, as the
  operator using it expects; the docs say what it leaves cached.
- The disk check counts the preflight's podman needs when no builds
  are left, and charges each build's artifacts to the filesystem its
  target's directory is on, cold storage for an archived target.
- More jobs than CPUs gave a slot an empty CPU set, and taskset then
  failed every command in it; that is refused now, as is an empty set
  given by hand.
- Worker threads start inside the interrupt handler, so a Ctrl-C while
  they start still stops what they launched.
- The docs say mainline's moving kernels are not mapped from releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- On an interrupt, the run waited by joining its worker threads
  again, which on Python 3.12 can return while a worker is still
  verifying or writing the ledger.  Workers now signal when they are
  done, and the run waits for that.
- release was not among the commands that share new files with the
  artifacts group, so its locks and work directory came out 0644 and
  blocked the next group member.
- The disk check counted a target as cached when its build container
  was, though its image package layers may not be.
- Parallel cross-build preflights mounted the shared targets/common
  with a private SELinux label, taking it from each other; it is
  shared now.
- With fewer builds left than slots, the automatic CPU sets were cut
  for the larger count and the unused CPUs dropped; they are now
  computed for the slots used.
- The docs no longer claim a newer distro kernel is picked up by
  itself: it comes from targets.yaml and the Lustre tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A build's ledger entry survived a later failed attempt at it, so
  after a forced rebuild failed partway, leaving old and new kernel
  outputs mixed, the next plain run counted it done.  The entry is now
  dropped before each attempt and written again only once it verifies.
- Verification took the snapshot's recorded module digest on trust;
  it now recomputes it, so a snapshot that lost a module fails.
- The disk check reserves the preflight's working room even when no
  builds are left.
- The docs say how a rebuild picks its kernel, including the cases
  that move it to a newer one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Two runs with their own artifacts and work directories still
  retagged each other's podman images.  A run now also locks the
  podman storage it builds into; each directory is locked once even
  when two of them coincide.
- A run launched in the background can inherit SIGINT ignored, and
  its builds then skipped their cleanup when stopped.  It restores a
  working SIGINT handler for its builds.
- The disk check counts the Lustre ccache each new target grows in
  the work directory, and leaves out filesystems the run needs
  nothing on.
- A lock directory that cannot be written is an error, not a crash.
- The docs say a change to ltvm's build code needs --fresh, since the
  caches key on the build inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Stopping set SIGINT ignored before it shut the launch door, so a
  build starting at that moment inherited it and could not be
  interrupted.  The runner is now marked stopped under its launch lock
  first.
- A release carrying ZFS would have been rebuilt without it.  Such
  releases are listed and not built; the docs say to use
  `build all --zfs` for them.
- The podman storage lock lived in the storage's parent, which need
  not be writable and is shared by unrelated stores.  It is now a lock
  file in the user's cache named for the storage.
- A cross build's preflight writes its tools into the target's
  artifacts, which the disk check now counts; with --no-preflight only
  targets with builds left count for new containers.
- The docs say refreshing package layers means removing every
  ltvm-image image, since targets share them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test run on Sookie wedged for over an hour: fuse-overlayfs spun
under two concurrent cached podman builds, and both `build all`
processes waited on it with no output and no end.  Nothing in the
release run noticed.

A build whose log has not grown for --stall-minutes (60 by default)
is now interrupted like a Ctrl-C, killed after the grace period, and
reported as stalled, so the rest of the run goes on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Verification compared the snapshot with its own recorded digests
  only, so a kernel rebuilt with another ABI under an older snapshot
  passed.  It now checks the live kernel's Module.symvers against the
  one the snapshot was built with.
- A reused Lustre clone kept the old configure state when only the
  commit changed, so the build could carry the old version.  A clone
  whose commit changes is cleaned first.
- Releases are found by their manifest's name rather than their tag,
  which `target publish --tag` can set to anything.
- Automatic slots are capped by the targets left, not the builds:
  one target's kernels run one at a time.
- The podman probes made while planning time out with an error
  rather than hang behind a wedged build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two wedged test runs on Sookie came down to podman, not the build:
rootless podman ignores storage.conf's runroot and keeps every store's
locks and mount state for a user in $XDG_RUNTIME_DIR/containers.  A
release run beside another podman workload of the same user, even on
separate storage, wedged fuse-overlayfs both times.  The podman lock
is now one per user rather than per storage, and the docs explain it.

A build stopped by Ctrl-C or SIGTERM is reported as interrupted, not
as failed at an unknown step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An error while waiting on a build (a full log disk, say) dropped it
  from the runner untracked and still running.  Any end of the wait
  now stops and reaps it, and the watchdog's log line is best effort.
- Verification recomputed the snapshot's module digest, which hashes
  nanosecond file times a snapshot copy need not keep, so a good build
  could fail.  The ledger now keeps each verified snapshot's modules by
  path and size, and a rerun compares against that.
- The disk check reserved nothing for a ccache directory that existed
  but was empty; it now reserves what each may still grow by.
- The default work directory is per user, since git refuses a clone
  another user owns; the artifacts lock stays shared.
- The docs say CPU pinning reaches local podman's containers only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The ledger's record of a snapshot's modules used their paths and
  sizes, so a same-size rebuild from other sources passed as the
  verified one.  It now hashes their contents.
- A ZFS asset anywhere in a release kept every kernel in it from
  being built; ZFS is now judged per manifest, by its kernel.
- With a remote podman service the storage path is the server's, so
  the disk check leaves it unchecked, as on macOS.
- The per-user podman lock no longer asks podman where its storage
  is, so planning queries podman once.
- When builds were stopped, the summary says how to find and remove
  the build containers podman can leave behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kernel or image copy cut short left files that the completeness
check, which only asked whether they existed, still accepted: a later
build hit the cache and a release run counted the result as good.
Kernel files must now match the sizes meta.json records (every kernel
on hand here, fetched or built, does) and an image must not be empty.

Also from review: a mainline release candidate matched the numbers of
a declared series; release candidates are now reported as unmatched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An image cut short by an interrupted copy passed as long as it was
  not empty, and a build or release run then kept it.  An image now
  has to hold every block its ext4 superblock declares; that matches
  exactly on every real image checked, built and fetched.
- On macOS the Lustre ccaches are podman volumes in the podman machine,
  but the release disk check charged a full cache per target to the
  host's work directory.  They are left out, like podman's storage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The ledger's fingerprint covered only the snapshot's Lustre modules,
  so a later run accepted a snapshot that had lost mount.lustre or
  lctl, or a kernel whose installed modules were cut short.  It now
  covers every file of the snapshot by contents and the installed
  kernel modules by path and size.
- A kernel published with ZFS could be scheduled again through another
  manifest of the same kernel, and that build would replace the ZFS one
  without ZFS.  Such kernels are now left out whichever manifest names
  them.
- The docs say an aarch64 builder's podman machine needs 8 GiB: BTF
  generation for the SLES kernels was killed in a 4 GiB one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The ledger's fingerprint did not cover the image, so an image another
  build installed for the same kernel passed as the verified one.  It
  now records which versioned image is current, and its size.
- A build whose recorded files had changed was run again, but without
  --force, so its kernel's cache could hand back the damage and the run
  record it as good.  Such builds are now forced.
- The base build's fingerprint took in variant snapshots nested in its
  snapshot, so building a variant made the base look changed.  They
  are left out, as publishing leaves them out of the base release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… crash

- Each target's kernel ccache lives beside its artifacts and can grow
  to ccache's cap; the release disk check now reserves what it may
  still take there.
- On macOS a target whose podman cache volume does not exist yet may
  fall back to a host ccache, so the host room is reserved for it.
- The kernel size check read vmlinux after checking it existed; a
  rebuild renaming it in between raised instead of reporting the
  kernel incomplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A build's ledger entry was deleted when an attempt began, so after an
interrupted forced rebuild a plain resume saw ordinary new work, did
not force it, and the kernel cache could hand back the half-replaced
kernel to be recorded as good.  The ledger now marks the attempt
instead, and every run forces that build until one verifies.

On macOS, a host ccache directory with contents means the build fell
back to it, so its room is reserved even when a podman volume exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The fingerprint a rerun checks did not cover the kernel binaries or
  build tree, so an interrupted kernel rebuild that left them partly
  replaced, with the same lengths, passed.  It now hashes vmlinux and
  vmlinuz and lists the build tree by path and size.
- A file vanishing while the fingerprint was taken raised, ending the
  whole release run; it now counts as a change, and the build is
  rebuilt.
- An ext4 superblock with an impossible block size made the image check
  build a huge number; such an image is not whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A Lustre clone moved to the new commit before its old build state
  was cleaned, so an interruption between the two left a retry seeing
  the right HEAD and skipping the clean.  The clean now comes first.
- On macOS the image export's conversion worker runs in the podman
  machine, beyond a signal to the build's process group, so a stopped
  release left it running.  It goes through the podman cleanup wrapper,
  and any nonzero exit fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mainline release built from `stable` carries only its version
(7.0.8), which mapped to the numbered 7.0 series, so a release run
would have rebuilt and republished it under the wrong kernel name.
Every release of a target that declares latest, stable or longterm is
now listed as unmatched, as the docs promised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a build's commits had changed, its ledger entry no longer matched
and its saved fingerprint went unchecked, so a kernel damaged since it
verified ran again without --force and its cache could hand the damage
back.  The last verified fingerprint is now compared whatever its
commits, and a build whose files changed is forced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macOS idle-sleeps a Mac left alone even on AC power, and every
build in a release run pauses with it: one aarch64 kernel build
took four hours of wall clock for 30 minutes of work overnight.
A real run now holds an idle sleep assertion with caffeinate for
as long as it lasts. A closed lid still sleeps the Mac, so the
docs say to leave it open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@paf0186
paf0186 merged commit b060ddc into image-build-fixes Nov 16, 2026
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant