Skip to content

Evaluate nix2container as the image-building backend #80

Description

@nothingnesses

Context

Issue #75 suggested adopting nix-oci (which wraps nix2container) as the image-building backend. I've decided not to adopt nix-oci itself for now (single maintainer, no releases yet, large flake-parts surface area, expands the supply chain for a security-focused project). See #75 for the full reasoning.

However, the underlying engine, nix2container, is worth evaluating on its own merits as a possible replacement for the current pkgs.dockerTools.buildLayeredImage call in lib/mkAgentImage.nix.

nix2container is mature: started in 2021, 30+ contributors, a v1.0.0 tag.

Potential benefits

  • Does not write image tarballs to the Nix store, which can reduce store usage and build I/O.
  • Can skip layers that have already been pushed to a registry, speeding up push/pull cycles.
  • Popularity-based / store-path layering for better layer sharing across images that share dependencies (relevant since all agent images share a common base package set).

These are most valuable if/when agent-images moves towards registry-based distribution rather than purely local podman/docker load of tarballs.

Open questions to research

  • Does nix2container's buildImage cleanly support everything mkAgentImage currently relies on?
    • fakeRootCommands equivalent (we create /etc/passwd, /etc/group, nsswitch.conf, owned directories, chown, nix-ld symlink). nix2container uses perms / copyToRoot style options; confirm parity.
    • includeNixDB equivalent for the withNix images (registering the Nix DB inside the image).
    • Multi-layer / layered-image behaviour comparable to buildLayeredImage.
    • config.Env, Entrypoint, User, WorkingDir parity (straightforward).
  • Does the string-context trick we rely on for nix-ld libraries (library store paths entering the closure via config.Env references) still hold with nix2container, or do we need to add them to contents explicitly?
  • How do the current bats tests need to change, if at all? Output is a different artefact (nix2container produces a JSON image spec + a copyTo* app rather than a .tar.gz); the test harness loads a tarball today.
  • Impact on consumers: agent-box and standalone podman/docker users currently load a tarball. nix2container's primary workflow is copyToDockerDaemon / copyToPodman / copyTo. Do we still need a tarball output for the existing UX, and can nix2container produce one?
  • Cross-system / macOS remote-builder story (we currently document building Linux images from macOS via a remote builder). Confirm nix2container works the same way.
  • Supply chain: adding nix2container as a flake input. It is a single upstream maintainer (nlewo) but with a long history and broad community use; assess acceptable vs current zero extra image-build inputs.

Decision criteria

Only worth doing if at least one of:

  • We commit to registry-based distribution (then layer-skipping is a real win), or
  • The store-usage / build-time reduction is measurable and meaningful for the full image matrix.

If we stay purely on local tarball loading, the benefit is marginal and the status quo (dockerTools, zero extra inputs) likely wins.

Next steps

  • Prototype: port one image (e.g. the minimal test image) to nix2container on a branch and compare build time, closure size, and store usage.
  • Verify the nix-ld and withNix images still behave identically under the existing bats suite (adapting the load step as needed).
  • Decide based on the criteria above.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requesthelp wantedExtra attention is neededquestionFurther information is requested

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions