Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
d273c9c
build: ltvm build release rebuilds every published target
Nov 16, 2026
a82ac7b
build: release build review fixes
Nov 16, 2026
f676f34
build: release build second review fixes
Nov 16, 2026
771354f
build: release build third review fixes
Nov 16, 2026
c894d73
build: release build fourth review fixes
Nov 16, 2026
f2f6465
build: release build fifth review fixes
Nov 16, 2026
daaa765
build: release build sixth review fixes
Nov 16, 2026
bd67d16
build: release build seventh review fixes
Nov 16, 2026
c35910a
build: release builds stop a stalled build
Nov 16, 2026
e9f0a4c
build: release build eighth review fixes
Nov 16, 2026
d842a1e
build: one release run per user, and interrupted builds say so
Nov 16, 2026
61dab6e
build: release build ninth review fixes
Nov 16, 2026
de90850
build: release build tenth review fixes
Nov 16, 2026
f8d48ee
build: truncated kernels and images are not complete
Nov 16, 2026
9177365
build: a truncated image is not complete; macOS caches stay off the host
Nov 16, 2026
1b6a2c2
build: the ledger records everything a build leaves to publish
Nov 16, 2026
f7644fc
build: the ledger knows the image, a failed check forces the rebuild
Nov 16, 2026
f15e4a8
build: the disk check counts kernel ccaches; a racing rename is not a…
Nov 16, 2026
428c011
build: an interrupted attempt is forced until it verifies
Nov 16, 2026
bced294
build: the ledger fingerprint covers the kernel itself
Nov 16, 2026
4a33126
build: cleanup before HEAD moves; macOS image worker is cleaned up
Nov 16, 2026
7c0357c
build: targets with moving kernels are left to build all
Nov 16, 2026
d8b1d5b
build: damage since an older verified build forces the rebuild
Nov 16, 2026
92a4fa6
build: a release build keeps a Mac awake
Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions docs/RELEASING.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,134 @@
# Releasing

## Rebuilding the target releases

A change that moves kernel or image input hashes (the shared kernel
config, a guest module, a Dockerfile) leaves every published release
stale, so they are all rebuilt and republished together. One command
per arch rebuilds everything published for it, from one Lustre commit:

```bash
ltvm build release --lustre-tree <lustre tree> --dry-run # the plan
ltvm build release --lustre-tree <lustre tree>
```

The set to build comes from the GitHub releases: every base release
for the arch, mapped back to its kernel in targets.yaml. `--only
rocky9` or `--only rocky9:5.14-rhel9.7` narrows it. Releases whose
kernel cannot be mapped back are listed as unmatched and not built.
That includes every release of a target with moving kernels (mainline's
`stable`, `latest`, `longterm`), since a version cannot say which name
or series built it; build those with `ltvm build all`. The Lustre tree
must have no uncommitted changes to tracked files, since the release
records its commit.

It runs in three passes, so what breaks shows early:

1. **Preflight**: each target's build container and image package
layers (`ltvm build image --base-only`), in parallel. A distro
package that no longer installs fails here in minutes. Podman
caches the layers, so the full image builds reuse them.
2. **Builds**: `ltvm build all` for each target and kernel, every
target's first kernel before any target's second, so a kernel
problem common to a target shows on its first build. Builds run
in parallel slots (`--jobs`; one per 32 CPUs, at most four), each
pinned to whole NUMA nodes (`--cpus` sets them by hand; the pinning
reaches local podman's containers, not those of a podman service reached
through `CONTAINER_HOST`) and given
its own clone of the Lustre tree. Two kernels of one target never
build at once.
3. **Verify**: a build counts only when its kernel, its Lustre
snapshot (from the commit asked for) and an image that is current
against its inputs are all there; an exit status of 0 is not enough.

Each build logs to its own file, and the run ends with a table and a
`summary.json` beside the logs. A build whose log stops growing for an
hour (`--stall-minutes`) is stopped and reported as stalled: a wedged
podman or fuse-overlayfs would otherwise hold the run for good. The exit status is 0 only when every
build is good. Before starting it checks that the artifacts and
podman storage, TMPDIR and the work directory have room for the run,
and refuses when they do not. On macOS podman's storage is inside the
podman machine and is not checked.

Finished builds go in a ledger in the work directory
(`<artifacts>/.release-build/<user>`, with the Lustre clones). Rerunning the
same command with the same ltvm and Lustre commit skips them, so after
a failure, fix the cause and run it again. The ledger knows only those
two commits: an ltvm checkout with uncommitted changes skips nothing.

After anything else changes, `--fresh` runs every build with
`--force`, rebuilding each kernel, Lustre and image. That includes a
change to ltvm's own build code that leaves the build inputs alone:
the caches key on the inputs, so without `--fresh` such a change does
not reach artifacts already built. Kernels are chosen by the usual
rules either way: the release targets.yaml and the Lustre tree name,
unless that one can no longer be downloaded, when the newest available
one stands in; mainline's series names take the newest point release.
So a rebuild can move to a newer kernel.

Podman's cached package layers are not rebuilt by either. Targets
with the same image Dockerfile share them, so to refresh them remove
every `ltvm-image-*` image (`podman images`), not just one target's.

The preflight runs for every target, finished ones too, so each still
has its build container for publishing. A release that carries ZFS is
listed and not built: rebuild it with `ltvm build all --zfs`.

One run at a time uses an artifacts directory or a work directory, and
one per user runs at all (see below for why); a second is refused.

### Where to run it

- **x86_64** on a large Linux host. One with 128 threads in 8 NUMA
nodes runs four slots, each on two nodes.
- **aarch64** on an Apple Silicon Mac, one slot. Give its podman
machine 8 GiB of memory: BTF generation for the SLES 6.4 kernels
(pahole) needs about 3.5 GiB, and is killed in a 4 GiB machine.
The run holds the Mac out of idle sleep with `caffeinate` while it
lasts; a closed lid still sleeps it, so leave the lid open.

Neither needs the other: run both at once.

### A shared host whose root filesystem is small

Rootless podman keeps its storage under `~/.local/share/containers`,
and a kernel build needs about 25 GiB there. Put it, the artifacts
and podman's temporary files on a large filesystem, and keep them
there between rebuilds: the build containers, the kernel source
packages and the image package layers are what make the next rebuild
start warm.

```bash
mkdir -p /mnt/data/$USER-ltvm/{containers,tmp,artifacts}
cat > /mnt/data/$USER-ltvm/storage.conf <<EOF2
[storage]
driver = "overlay"
runroot = "/run/user/$(id -u)/$USER-ltvm-containers"
rootless_storage_path = "/mnt/data/$USER-ltvm/containers"
EOF2
export CONTAINERS_STORAGE_CONF=/mnt/data/$USER-ltvm/storage.conf
export TMPDIR=/mnt/data/$USER-ltvm/tmp
export LTVM_ARTIFACTS_DIR=/mnt/data/$USER-ltvm/artifacts
```

`graphroot` does not move rootless storage (podman 4.9 ignores it);
`rootless_storage_path` does. `runroot` is ignored too: every rootless
podman of one user keeps its locks and mount state in
`$XDG_RUNTIME_DIR/containers`, whatever storage it uses. So a second
storage does not keep two sets of builds by one user apart, and
running two at once (two storages, or a release run beside other podman
builds) can wedge fuse-overlayfs. `build release` therefore runs one
at a time per user.

### Publishing

Publish once the change has merged, from the artifacts the build left:
`ltvm target publish <target> --kernel <kernel_dir> --arch <arch>` for
each build in `summary.json`, with the same `LTVM_ARTIFACTS_DIR`. Use
its `kernel_dir`, the full directory name, rather than the short kernel
name: the short name picks the newest kernel cached, which need not be
the one built.

## Rebuilding Pre-built QEMU Binaries

Rocky Linux ships QEMU without microvm support, so we publish
Expand Down
76 changes: 75 additions & 1 deletion ltvm
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,7 @@ try:
cmd_build_kernel,
cmd_build_lustre,
cmd_build_mofed_kmods,
cmd_build_release,
cmd_build_shell,
cmd_build_zfs,
cmd_claim,
Expand Down Expand Up @@ -955,8 +956,81 @@ def build_parser() -> argparse.ArgumentParser:
help="Path to Lustre source tree (default: cwd) -- used to "
"find per-kernel Lustre staging to bake into the image",
)
bip.add_argument(
"--base-only",
action="store_true",
help="Build only the image's package layers (no kernel modules or "
"Lustre), which podman caches for the full image build",
)
bip.set_defaults(func=cmd_build_image)

brp = build_sub.add_parser(
"release",
parents=[common],
help="Rebuild every published target release for this arch "
"(see docs/RELEASING.md)",
)
brp.add_argument(
"--lustre-tree",
help="Lustre tree whose HEAD commit to build (default: cwd)",
)
brp.add_argument(
"--only",
action="append",
metavar="TARGET[:KERNEL]",
help="Build only this target, or one kernel of it (repeatable)",
)
brp.add_argument(
"--jobs",
type=int,
default=None,
help="Builds at once (default: one per 32 CPUs, at most 4; 1 on macOS)",
)
brp.add_argument(
"--cpus",
metavar="SET:SET...",
help="CPU set per job, colon separated, e.g. 0-15,64-79:16-31,80-95 "
"(default: whole NUMA nodes shared out)",
)
brp.add_argument(
"--fresh",
action="store_true",
help="Run every build, forcing its kernel, Lustre and image, not "
"only those not yet finished at these commits (after something "
"outside the ltvm and Lustre commits changed)",
)
brp.add_argument(
"--stall-minutes",
type=int,
default=60,
metavar="N",
help="Stop a build whose log has not grown for N minutes and report "
"it as stalled (default 60; 0 never stops one)",
)
brp.add_argument(
"--no-preflight",
action="store_true",
help="Skip building every container and image package layer first",
)
brp.add_argument(
"--dry-run",
action="store_true",
help="Show the plan, CPU sets and disk check, and build nothing",
)
brp.add_argument(
"--work-dir",
metavar="DIR",
help="Lustre clones and the ledger of finished builds "
"(default: <artifacts>/.release-build)",
)
brp.add_argument(
"--log-dir",
metavar="DIR",
help="Per-build logs and summary.json "
"(default: <work-dir>/logs/<time>)",
)
brp.set_defaults(func=cmd_build_release)

# Note: build lustre's --force has a slightly different meaning
# ("Clean and reconfigure before building") than the shared help text,
# but the flag behaviour is identical.
Expand Down Expand Up @@ -3152,7 +3226,7 @@ def parse_cli(
# not become group-writable.
_ARTIFACT_WRITERS: dict[str, frozenset[str] | None] = {
"build": frozenset(
{"all", "container", "kernel", "image", "zfs", "mofed-kmods"}
{"all", "container", "kernel", "image", "zfs", "mofed-kmods", "release"}
),
"target": frozenset(
{"build", "fetch", "clean", "delete", "archive", "restore"}
Expand Down
9 changes: 8 additions & 1 deletion ltvm_pkg/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
retire_stale_zfs,
targets_unformatted,
)
from ltvm_pkg.image_build import build_image, image_status
from ltvm_pkg.image_build import build_image, build_image_base, image_status
from ltvm_pkg.kernel_build import (
SrpmNotFoundError,
build_kernel,
Expand Down Expand Up @@ -116,10 +116,12 @@
from ltvm_pkg.cli.fetch import ( # noqa: E402
_KVER_PREFIX_RE,
_RHEL_RE,
_fetched_kernel_name,
_find_release_url,
_gh_api,
_gh_next_link,
_gh_release_upload,
_is_base_stem,
_kernel_release_signature,
_list_releases,
_release_matches_kernel,
Expand All @@ -132,6 +134,7 @@
cmd_make_reinstall,
cmd_make_uninstall,
)
from ltvm_pkg.cli.release import cmd_build_release # noqa: E402
from ltvm_pkg.cli.setup import ( # noqa: E402
_current_version,
_git,
Expand Down Expand Up @@ -205,7 +208,9 @@
"_error",
"_find_release_url",
"_gate_lustre_validation",
"_fetched_kernel_name",
"_gh_api",
"_is_base_stem",
"_gh_next_link",
"_gh_release_upload",
"_git",
Expand All @@ -227,9 +232,11 @@
"_vm_call",
"argparse",
"build_image",
"build_image_base",
"build_kernel",
"build_lustre",
"cmd_build_all",
"cmd_build_release",
"cmd_build_container",
"cmd_build_image",
"cmd_build_kernel",
Expand Down
14 changes: 13 additions & 1 deletion ltvm_pkg/cli/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,11 @@ def _warn_low_disk() -> None:
A build that runs out of space fails late, deep in make or
podman, with nothing pointing at the cause.
"""
from ltvm_pkg.release_build import podman_store
from ltvm_pkg.target_config import ARTIFACTS_DIR

roots = [ARTIFACTS_DIR, Path.home() / ".local/share/containers"]
store = podman_store()
roots = [ARTIFACTS_DIR] + ([store] if store else [])
seen: set[int] = set()
for root in roots:
p = root
Expand Down Expand Up @@ -893,6 +895,16 @@ def cmd_build_image(args: argparse.Namespace) -> int:
if err is not None:
return err

if getattr(args, "base_only", False):
with _podman_machine_autostop() as autostop:
try:
tag = _cli_attr("build_image_base")(tc)
except Exception as e:
return _error(f"Image build failed: {e}", use_json)
_output({"target": args.target, "base_tag": tag}, use_json)
autostop.success = True
return EXIT_OK

timer = StepTimer(f"build image {args.target}", quiet=use_json)
with _podman_machine_autostop() as autostop:
kernel = getattr(args, "kernel", None)
Expand Down
43 changes: 24 additions & 19 deletions ltvm_pkg/cli/fetch.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,29 @@ def _gh_next_link(headers: str) -> str | None:
_KERNEL_RELEASE_TAILS = ("_lustre-default", "-lustre")


def _is_base_stem(stem: str, known_variants: list[str]) -> bool:
"""Whether a release asset's name, less its extensions, is a base
release rather than a variant's.

Two complementary rules, because neither alone is enough: an exact
match against the variants the target declares (catches mofed-24,
whose trailing segment "24" the heuristic below reads as a kver),
plus a heuristic for variants we don't know about (a kver's last
segment always carries a digit, a name like "-mofed" does not).
Some kernel releases end in a word: the flavor of a SUSE kernel
(6.4.0-150700.53.66_lustre-default), the CONFIG_LOCALVERSION of a
Debian one (6.12.107-lustre).
"""
if any(stem.endswith(f"-{v}") for v in known_variants):
return False
last_seg = stem.rsplit("-", 1)[-1]
return not (
last_seg
and not any(ch.isdigit() for ch in last_seg)
and not stem.endswith(_KERNEL_RELEASE_TAILS)
)


def _kernel_release_signature(kname: str) -> str | None:
"""Derive a substring found in release tag/asset names for ``kname``.

Expand Down Expand Up @@ -307,25 +330,7 @@ def _asset_stem(name: str) -> str:
continue
stem = _asset_stem(name)
if variant == "base":
# A base lookup must not pick up a variant's asset.
# Two complementary rules, because neither alone is
# enough: an exact match against the variants this
# target declares (catches mofed-24, whose trailing
# segment "24" the heuristic below reads as a kver),
# plus a heuristic for variants we don't know about
# (a kver's last segment always carries a digit, a
# name like "-mofed" does not).
if any(stem.endswith(f"-{v}") for v in known_variants):
continue
# Some kernel releases end in a word: the flavor of a
# SUSE kernel (6.4.0-150700.53.66_lustre-default), the
# CONFIG_LOCALVERSION of a Debian one (6.12.107-lustre).
last_seg = stem.rsplit("-", 1)[-1]
if (
last_seg
and not any(ch.isdigit() for ch in last_seg)
and not stem.endswith(_KERNEL_RELEASE_TAILS)
):
if not _is_base_stem(stem, known_variants):
continue
elif not stem.endswith(f"-{variant}"):
continue
Expand Down
Loading
Loading