Context
Issue #75 suggested adopting nix-oci (which wraps nix2container) as the image-building backend. I've decided not to adopt nix-oci itself for now (single maintainer, no releases yet, large flake-parts surface area, expands the supply chain for a security-focused project). See #75 for the full reasoning.
However, the underlying engine, nix2container, is worth evaluating on its own merits as a possible replacement for the current pkgs.dockerTools.buildLayeredImage call in lib/mkAgentImage.nix.
nix2container is mature: started in 2021, 30+ contributors, a v1.0.0 tag.
Potential benefits
- Does not write image tarballs to the Nix store, which can reduce store usage and build I/O.
- Can skip layers that have already been pushed to a registry, speeding up push/pull cycles.
- Popularity-based / store-path layering for better layer sharing across images that share dependencies (relevant since all agent images share a common base package set).
These are most valuable if/when agent-images moves towards registry-based distribution rather than purely local podman/docker load of tarballs.
Open questions to research
- Does
nix2container's buildImage cleanly support everything mkAgentImage currently relies on?
fakeRootCommands equivalent (we create /etc/passwd, /etc/group, nsswitch.conf, owned directories, chown, nix-ld symlink). nix2container uses perms / copyToRoot style options; confirm parity.
includeNixDB equivalent for the withNix images (registering the Nix DB inside the image).
- Multi-layer / layered-image behaviour comparable to
buildLayeredImage.
config.Env, Entrypoint, User, WorkingDir parity (straightforward).
- Does the string-context trick we rely on for nix-ld libraries (library store paths entering the closure via
config.Env references) still hold with nix2container, or do we need to add them to contents explicitly?
- How do the current
bats tests need to change, if at all? Output is a different artefact (nix2container produces a JSON image spec + a copyTo* app rather than a .tar.gz); the test harness loads a tarball today.
- Impact on consumers:
agent-box and standalone podman/docker users currently load a tarball. nix2container's primary workflow is copyToDockerDaemon / copyToPodman / copyTo. Do we still need a tarball output for the existing UX, and can nix2container produce one?
- Cross-system / macOS remote-builder story (we currently document building Linux images from macOS via a remote builder). Confirm
nix2container works the same way.
- Supply chain: adding
nix2container as a flake input. It is a single upstream maintainer (nlewo) but with a long history and broad community use; assess acceptable vs current zero extra image-build inputs.
Decision criteria
Only worth doing if at least one of:
- We commit to registry-based distribution (then layer-skipping is a real win), or
- The store-usage / build-time reduction is measurable and meaningful for the full image matrix.
If we stay purely on local tarball loading, the benefit is marginal and the status quo (dockerTools, zero extra inputs) likely wins.
Next steps
- Prototype: port one image (e.g. the minimal test image) to
nix2container on a branch and compare build time, closure size, and store usage.
- Verify the nix-ld and
withNix images still behave identically under the existing bats suite (adapting the load step as needed).
- Decide based on the criteria above.
Context
Issue #75 suggested adopting
nix-oci(which wrapsnix2container) as the image-building backend. I've decided not to adoptnix-ociitself for now (single maintainer, no releases yet, largeflake-partssurface area, expands the supply chain for a security-focused project). See #75 for the full reasoning.However, the underlying engine,
nix2container, is worth evaluating on its own merits as a possible replacement for the currentpkgs.dockerTools.buildLayeredImagecall inlib/mkAgentImage.nix.nix2containeris mature: started in 2021, 30+ contributors, av1.0.0tag.Potential benefits
These are most valuable if/when
agent-imagesmoves towards registry-based distribution rather than purely localpodman/docker loadof tarballs.Open questions to research
nix2container'sbuildImagecleanly support everythingmkAgentImagecurrently relies on?fakeRootCommandsequivalent (we create /etc/passwd, /etc/group, nsswitch.conf, owned directories, chown, nix-ld symlink).nix2containerusesperms/copyToRootstyle options; confirm parity.includeNixDBequivalent for thewithNiximages (registering the Nix DB inside the image).buildLayeredImage.config.Env,Entrypoint,User,WorkingDirparity (straightforward).config.Envreferences) still hold withnix2container, or do we need to add them to contents explicitly?batstests need to change, if at all? Output is a different artefact (nix2containerproduces a JSON image spec + acopyTo*app rather than a.tar.gz); the test harness loads a tarball today.agent-boxand standalonepodman/dockerusers currentlyloada tarball.nix2container's primary workflow iscopyToDockerDaemon/copyToPodman/copyTo. Do we still need a tarball output for the existing UX, and cannix2containerproduce one?nix2containerworks the same way.nix2containeras a flake input. It is a single upstream maintainer (nlewo) but with a long history and broad community use; assess acceptable vs current zero extra image-build inputs.Decision criteria
Only worth doing if at least one of:
If we stay purely on local tarball loading, the benefit is marginal and the status quo (
dockerTools, zero extra inputs) likely wins.Next steps
nix2containeron a branch and compare build time, closure size, and store usage.withNiximages still behave identically under the existingbatssuite (adapting the load step as needed).