Skip to content

Commit b4c9762

Browse files
akramdrew
authored andcommitted
build(docker): split in-cluster build variants for the RFC-0012 three-image layout
RFC-0012 (#2942) split the runtime into three binaries. Match the in-cluster (OpenShift/Buildah) multi-stage variants to the image set the Helm chart and official Dockerfiles expect: - Dockerfile.sandbox.multistage (new): static-musl openshell-sandbox on a scratch base (USER 65532), mirroring Dockerfile.sandbox. - Dockerfile.supervisor.multistage: now builds the dynamically linked GNU openshell-supervisor on a debian base with ca-certificates, mirroring Dockerfile.supervisor (previously it built openshell-sandbox, which predated the RFC-0012 split). - Dockerfile.gateway.multistage: unchanged (openshell-gateway). Part of #3116. Signed-off-by: Akram <akram.benaissi@gmail.com>
1 parent a955ba3 commit b4c9762

2 files changed

Lines changed: 98 additions & 22 deletions

File tree

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# syntax=docker/dockerfile:1.4
2+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
# In-cluster (OpenShift/Buildah) variant of Dockerfile.sandbox.
6+
#
7+
# Upstream CI produces the static musl `openshell-sandbox` binary by building
8+
# inside the project's Nix devShell with the musl cross target, and stages it
9+
# under deploy/docker/.build/prebuilt-binaries. This multi-stage Dockerfile
10+
# reproduces that exact binary in a builder stage so clusters without the Nix
11+
# CI pipeline can build the sandbox image directly from source. The final stage
12+
# is identical to Dockerfile.sandbox.
13+
14+
# ---- Builder: glibc base + Nix, reproduce upstream's cross build -------------
15+
#
16+
# The build must run on a glibc/FHS base (like the CI runners), NOT on the
17+
# minimal nixos/nix image: cross-compiling emits host build-scripts linked for
18+
# x86_64-unknown-linux-gnu whose ELF interpreter is /lib64/ld-linux-x86-64.so.2,
19+
# which the nixos/nix (Alpine, store-only) image does not provide.
20+
FROM debian:bookworm-slim AS builder
21+
22+
RUN apt-get update && apt-get install -y --no-install-recommends \
23+
curl xz-utils ca-certificates git && rm -rf /var/lib/apt/lists/*
24+
25+
# Single-user Nix (no daemon) as root. NIX_CONFIG disables per-build users
26+
# during the install itself (the installer reads config before our nix.conf
27+
# exists); the written nix.conf carries that to later `nix develop` runs, plus
28+
# flakes and the flake's cachix substituter for prebuilt toolchain/z3/aws-lc.
29+
RUN export NIX_CONFIG="build-users-group =" && \
30+
mkdir -m 0755 /nix && \
31+
curl -L https://nixos.org/nix/install -o /tmp/nix-install.sh && \
32+
sh /tmp/nix-install.sh --no-daemon && \
33+
mkdir -p /etc/nix && \
34+
printf 'experimental-features = nix-command flakes\naccept-flake-config = true\nbuild-users-group =\nsandbox = false\n' > /etc/nix/nix.conf
35+
36+
# Put single-user Nix on PATH directly (Buildah RUN is not a login shell, so the
37+
# profile script is unreliable) and point it at Debian's CA bundle for HTTPS
38+
# substituter fetches.
39+
ENV HOME=/root \
40+
PATH=/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
41+
NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
42+
43+
WORKDIR /build
44+
COPY . .
45+
46+
# Build the sandbox binary exactly like CI: same devShell, same cargo command
47+
# and target triple. Everything runs in a single layer: build, stage the static
48+
# binary at /, then delete the Nix store and Cargo target. The binary is static
49+
# musl (it needs nothing from /nix at runtime), so the committed builder layer
50+
# shrinks to the binary alone, keeping the intermediate commit fast and within
51+
# the node's ephemeral-storage budget.
52+
RUN nix develop -c bash -euo pipefail -c '\
53+
GIT_DIR=/nonexistent cargo auditable build --release \
54+
--target x86_64-unknown-linux-musl \
55+
--package openshell-sandbox --bin openshell-sandbox' && \
56+
cp target/x86_64-unknown-linux-musl/release/openshell-sandbox /openshell-sandbox && \
57+
cd / && rm -rf /nix /build /root/.cache /tmp/*
58+
59+
# ---- Runtime: identical to deploy/docker/Dockerfile.sandbox ------------------
60+
#
61+
# Runtime-specific bootstrap tools belong to their compute drivers rather than
62+
# this portable image, so the sandbox runtime image is a bare scratch layer.
63+
FROM scratch AS sandbox
64+
65+
ARG TARGETARCH
66+
67+
# Keep the binary root-owned for image-volume mounts and executable by the
68+
# sandbox runtime's non-root UID.
69+
COPY --from=builder --chmod=0555 /openshell-sandbox /openshell-sandbox
70+
71+
# Drivers may override this identity to match the admitted workload. Keep the
72+
# standalone image non-root by default.
73+
USER 65532:65532
74+
75+
ENTRYPOINT ["/openshell-sandbox"]

‎deploy/docker/Dockerfile.supervisor.multistage‎

Lines changed: 23 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,13 @@
44

55
# In-cluster (OpenShift/Buildah) variant of Dockerfile.supervisor.
66
#
7-
# Upstream CI produces the static musl `openshell-sandbox` binary by building
8-
# inside the project's Nix devShell with the musl cross target, and stages it
9-
# under deploy/docker/.build/prebuilt-binaries. This multi-stage Dockerfile
10-
# reproduces that exact binary in a builder stage so clusters without the Nix
11-
# CI pipeline can build the supervisor image directly from source. The final
12-
# stage is identical to Dockerfile.supervisor.
7+
# Upstream CI builds the dynamically linked GNU `openshell-supervisor` binary
8+
# inside the project's Nix devShell with the gnu cross target (z3 and aws-lc
9+
# statically embedded, standard ELF interpreter), and stages it under
10+
# deploy/docker/.build/prebuilt-binaries. This multi-stage Dockerfile reproduces
11+
# that exact artifact in a builder stage so clusters without the Nix CI pipeline
12+
# can build the supervisor image directly from source. The final stage is
13+
# identical to Dockerfile.supervisor.
1314

1415
# ---- Builder: glibc base + Nix, reproduce upstream's cross build -------------
1516
#
@@ -43,30 +44,30 @@ ENV HOME=/root \
4344
WORKDIR /build
4445
COPY . .
4546

46-
# Build the sandbox binary exactly like CI: same devShell, same cargo command
47-
# and target triple. Everything runs in a single layer: build, stage the static
48-
# binary at /, then delete the Nix store and Cargo target. The binary is static
49-
# musl (it needs nothing from /nix at runtime), so the committed builder layer
50-
# shrinks to the binary alone, keeping the intermediate commit fast and within
51-
# the node's ephemeral-storage budget.
47+
# Build the supervisor exactly like CI: same devShell, same cargo command and
48+
# target triple. The gnu cross toolchain emits a standard ELF interpreter, so
49+
# the binary runs on a plain Debian base without post-processing. Everything
50+
# runs in a single layer: build, stage the binary at /, then delete the Nix
51+
# store and Cargo target so the committed builder layer stays small.
5252
RUN nix develop -c bash -euo pipefail -c '\
5353
GIT_DIR=/nonexistent cargo auditable build --release \
54-
--target x86_64-unknown-linux-musl \
55-
--package openshell-sandbox --bin openshell-sandbox' && \
56-
cp target/x86_64-unknown-linux-musl/release/openshell-sandbox /openshell-sandbox && \
54+
--target x86_64-unknown-linux-gnu \
55+
--package openshell-supervisor --bin openshell-supervisor' && \
56+
cp target/x86_64-unknown-linux-gnu/release/openshell-supervisor /openshell-supervisor && \
5757
cd / && rm -rf /nix /build /root/.cache /tmp/*
5858

5959
# ---- Runtime: identical to deploy/docker/Dockerfile.supervisor ---------------
6060
#
61-
# Alpine supplies nftables and iptables for pod-namespace egress enforcement.
62-
FROM alpine:3.22 AS supervisor
61+
# The dynamically linked GNU supervisor binary needs a glibc runtime plus a CA
62+
# bundle for its outbound TLS connections.
63+
FROM debian:bookworm-slim AS supervisor
6364

6465
ARG TARGETARCH
6566

66-
RUN apk add --no-cache nftables iptables iptables-legacy
67+
RUN apt-get update \
68+
&& apt-get install -y --no-install-recommends ca-certificates \
69+
&& rm -rf /var/lib/apt/lists/*
6770

68-
# Keep the binary root-owned for Podman image-volume mounts and executable by
69-
# the Kubernetes network sidecar's non-root proxy UID.
70-
COPY --from=builder --chmod=0555 /openshell-sandbox /openshell-sandbox
71+
COPY --from=builder --chmod=0555 /openshell-supervisor /openshell-supervisor
7172

72-
ENTRYPOINT ["/openshell-sandbox"]
73+
ENTRYPOINT ["/openshell-supervisor"]

0 commit comments

Comments
 (0)