Skip to content

chore: make dev container setup work without npm registry access - #97

Open
ayeshurun wants to merge 5 commits into
mainfrom
dev/alonyeshurun/remove-devcontainer-node-feature
Open

chore: make dev container setup work without npm registry access#97
ayeshurun wants to merge 5 commits into
mainfrom
dev/alonyeshurun/remove-devcontainer-node-feature

Conversation

@ayeshurun

@ayeshurun ayeshurun commented Aug 26, 2026

Copy link
Copy Markdown
Owner

Problem

Dev container creation fails on networks that block registry.npmjs.org. The ghcr.io/devcontainers/features/node:2 feature aborts while installing pnpm:

npm error code ECONNRESET
npm error network request to https://registry.npmjs.org/pnpm failed,
  reason: Client network socket disconnected before secure TLS connection was established
ERROR: Feature "Node.js (via nvm), yarn and pnpm." failed to install!

This happens during the image build, so the container never starts.

Investigating that failure surfaced a second, independent defect: even with Node removed, apt-get update fails in this base image, and the setup script was swallowing that failure and exiting 0 with none of the required build packages installed.

Root causes

1. npm registry blocked by network policy. The request fails in ~20ms, and nodejs.org downloads fine in the same layer, so general egress is healthy. The 73s in the log is npm's retry backoff. Node.js was in the container for exactly one reason: npm install -g changie. There is no package.json anywhere in the repo.

2. Stale Yarn apt source in the base image. mcr.microsoft.com/devcontainers/python:1-3.12-bullseye ships /etc/apt/sources.list.d/yarn.list plus an RSA keyring whose signing subkeys expired 2026-01-23. Yarn now signs InRelease with EdDSA key 62D54FD4003F6525, so apt reports NO_PUBKEY and apt-get update exits 100. This is not network-specific and is not caused by the npm block — it fails for every user of this image. TLS inspection was ruled out: the certificate chain is genuine Google Trust Services for CN=yarnpkg.com.

3. The failure was silent. The script ran apt-get update && apt-get install .... POSIX ignores errexit for any command in an AND-OR list other than the last, so the failing update did not trip set -e — it simply short-circuited the install and the script returned 0. Result: cmake, pkg-config, libcairo2-dev and python3-dev were never installed, with no error surfaced.

Note: an earlier revision of this description claimed the non-root apt-get update "aborted the script under set -e". That was wrong for the same && reason, and is corrected above.

Changes

Remove the Node.js feature. changie is a standalone Go binary, so it is installed directly from its upstream GitHub release and verified against the published checksums.txt. The version is pinned (overridable via CHANGIE_VERSION). This drops an entire JavaScript toolchain from a Python-only container.

Fix the apt failure. apt-get update and apt-get install are now separate statements, and the stale Yarn source is removed before update runs. Without both changes the split alone would only make the existing failure visible, not fix it. Scope note: apt-get update exits non-zero for signature failures like this one, so it now aborts under set -e, but it still exits 0 when a source is merely unreachable. install remains the real gate there — an unavailable package fails loudly rather than being silently skipped, which was the actual bug. APT::Update::Error-Mode=any would make update strict, but was deliberately not adopted: it would turn a transient mirror blip into a hard container-build failure on exactly the restricted networks this PR targets.

Use sudo when not running as root. postCreateCommand runs as the remote user, where bare apt-get cannot write to /var/lib/apt. The script detects its own uid, preflights that sudo exists and works non-interactively, and behaves identically as either user. Proxy variables are forwarded explicitly via --preserve-env, since sudo resets the environment by default. pip3 is deliberately not run under sudo: Debian's pip falls back to --user and sudo would strip PIP_INDEX_URL.

Install the verified binary without materialising archive entries. The changie member is extracted to stdout (tar -xzOf) and redirected into a fresh regular file inside a 0700 temp dir, which is then non-empty-checked before install. Extracting normally and installing the resulting path would follow a link entry: an archive whose changie member is a symlink to /etc/shadow caused sudo install -m 0755 to read it as root and write a world-readable copy to /usr/local/bin/changie. tar -O emits the member's contents, and link entries carry none, so both symlinks and hard links yield 0 bytes and are rejected. This is defence in depth, not a fix for the underlying trust model — see below.

Support an optional git-ignored .devcontainer/local.env. For environments needing an internal package mirror. It is parsed as KEY=value data against a fixed allowlist, never sourced, so it cannot execute commands or clobber script variables. Worth noting: pypi.org can be reachable while files.pythonhosted.org is blocked, so pip resolves dependencies and only then fails on download. .devcontainer/local.env.example documents the pattern; the real file is git-ignored.

Verification

Run in mcr.microsoft.com/devcontainers/python:1-3.12-bullseye. Assertions check installed outcomes, not just exit codes — an exit-code-only check is what let root cause 3 hide.

Case Result
Full script, non-root user, linux/amd64 exit 0; cmake/pkg-config/libcairo2-dev/python3-dev present; deps importable; changie version v1.26.0
Full script, non-root user, linux/arm64 same
Full script as root (sudo prefix empty) exit 0, binary at /usr/local/bin/changie, mode 0755
Yarn source left in place apt-get update exits 100 and the script now aborts instead of skipping installs
Checksum: empty file / no match / duplicate entries / name spoofed into field 1 / tampered archive rejected in all five cases
Archive whose changie member is a symlink to /etc/shadow or /etc/passwd 0 bytes extracted, install refused, nothing written (pre-fix: leaked /etc/shadow to a 0755 file)
Archive whose changie member is a hard link, or an empty regular file 0 bytes extracted, install refused
Archive whose changie member is a directory with a child file child's contents streamed (21 bytes), install proceeds — archive-internal bytes only, see note below
Archive members named ../evil, /changie, or ../changie tar exits 2 (Not found in archive), script aborts before install; nothing written anywhere on disk
Bad CHANGIE_VERSION curl: (22) 404, exit 22
local.env containing shell metacharacters, unlisted keys, CRLF not executed; sudo_cmd and PATH unclobbered; unlisted keys ignored
sudo env handling plain sudo preserves 0/4 vars; --preserve-env preserves 3/3 proxy vars and does not leak PIP_INDEX_URL

Notes and trade-offs

  • Checksum scope — read this narrowly. checksums.txt is fetched from the same release as the archive, so an actor who can replace the asset can replace its checksum entry. It detects corruption, truncation and CDN inconsistency; it is not source authentication and must not be read as an end-to-end supply-chain boundary. After the link-entry fix, a substituted release yields user-level executable compromise — the same exposure already inherent in this container's unpinned requirements-dev.txt and in CI's npm i -g changie. Pinning per-architecture digests in this repo would narrow that, but only for this one path, and it conflicts with the documented CHANGIE_VERSION override unless a version→digest map is introduced. Deliberately deferred to a repo-wide supply-chain decision, not an oversight. "Pinned version" here should not be read as "reproducible build": the base image tag, feature major tag and most of requirements-dev.txt remain mutable.
  • .github/workflows/changelog-existence.yml still uses npm i -g changie. GitHub-hosted runners reach npmjs fine, so it is left alone. The dev container pins 1.26.0 while CI floats to latest; latest is currently 1.26.0, so this is future drift risk rather than a present mismatch.
  • No CI covers these paths — deferred to No CI coverage for .devcontainer/** and scripts/** — silent provisioning failures can ship #98. fab-build.yml path-filters to src/**, tests/**, pyproject.toml, tox.toml and requirements*.txt, so .devcontainer/**, scripts/** and .gitignore receive zero jobs. Confirmed on this PR: gh pr checks 97 returns only the changelog and title jobs; fab-build does not run at all. That is plausibly why root cause 3 went unnoticed, and two independent reviewers flagged it. Tracked separately so this network bugfix does not grow a CI surface.
  • Archive members that are neither a link nor a plain regular file. Two shapes stream archive-internal bytes past the non-empty check. GNU tar concatenates same-named members, so tar -O may emit several payloads joined together; this does not reliably yield a broken binary, since ELF tolerates trailing bytes (verified: /bin/true plus a trailing member produced a 39,698-byte file that executed successfully). A directory member named changie likewise streams its children's contents (verified: a changie/ directory holding a 21-byte child emitted exactly those 21 bytes, exit 0). Neither is a new exposure — both require the same substituted-release trust failure that could simply ship a malicious single member — and neither can cause host-file disclosure or a privileged link-follow, because the bytes always originate inside the checksum-gated archive rather than from a host path. No additional guard added.
  • CONTRIBUTING.md says changie is "pre-installed in the development container", which remains accurate.
  • Not verified: whether the npm changie package and the GitHub release are byte-identical builds.
  • Pre-existing and left alone: black and mypy are referenced in contributor docs but absent from requirements-dev.txt, and pip console scripts land in ~/.local/bin, which is not on PATH for the non-root user.

Alon Yeshurun and others added 2 commits August 26, 2026 11:24
The Node.js dev container feature failed to install on networks that block
registry.npmjs.org, breaking container creation before setup could start.

Node was only present to run `npm install -g changie`. changie ships as a
standalone Go binary, so install it directly from its upstream GitHub release
with checksum verification and drop the Node feature entirely. This removes a
full JavaScript toolchain from an otherwise Python-only container.

Also fix two latent issues in the setup script:

- apt-get and the binary install now use sudo when not running as root.
  postCreateCommand runs as the remote user, where the previous bare apt-get
  exited 100 and aborted the script under `set -e`.
- Support an optional git-ignored .devcontainer/local.env for environments that
  need an internal package mirror. pypi.org can be reachable while
  files.pythonhosted.org is blocked, which makes pip resolve and then fail on
  download.

Verified end to end in the base image on amd64 and arm64, as both root and the
non-root remote user.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0c37de44-e0a0-4090-8f7e-299889108578
- Split apt-get update/install into separate statements. As the left operand
  of &&, a failing update did not trip set -e, so package installation was
  skipped silently and cmake was never installed.
- Remove the base image's stale Yarn apt source. Its bundled RSA keyring
  predates Yarn's switch to an EdDSA signing key, so apt-get update fails
  verification and exits 100 on any network.
- Parse .devcontainer/local.env as KEY=value data against an allowlist instead
  of sourcing it, so the file cannot run commands or clobber script variables.
- Forward proxy variables explicitly through sudo, which resets the
  environment by default and previously discarded them.
- Require exactly one matching checksum entry before verification.
- Harden curl: https-only redirects, bounded retries and timeouts.
- Resolve requirements files from the repo root rather than the caller's cwd.
- Preflight sudo availability and non-interactive use.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0c37de44-e0a0-4090-8f7e-299889108578
@ayeshurun
ayeshurun force-pushed the dev/alonyeshurun/remove-devcontainer-node-feature branch from 6cfe12b to c345bbd Compare August 26, 2026 08:25
Alon Yeshurun and others added 2 commits August 26, 2026 11:56
Silence a ShellCheck SC2054 false positive by quoting the sudo
--preserve-env argument, whose commas belong to sudo's option list
rather than to the bash array.

Use `env` as a no-op prefix on the root path instead of an empty array.
Expanding an empty array under `set -u` is an error on bash < 4.4.

Document that apt honours only lowercase proxy variables, so setting
HTTP_PROXY alone leaves apt without a proxy while pip and curl work.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0c37de44-e0a0-4090-8f7e-299889108578
… entry

A tar member named `changie` could be a symlink or hard link. The previous
`tar -xzf` materialised that entry, and the following `install` under sudo
followed it, copying an arbitrary root-readable file into world-readable
/usr/local/bin. Verified in mcr.microsoft.com/devcontainers/python:1-3.12-bullseye:
a `changie -> /etc/shadow` member produced a 0755 root-owned copy of
/etc/shadow readable by the unprivileged remote user.

Extract the member's contents with `tar -O` instead. Link entries carry no
content, so they yield zero bytes and the added non-empty check rejects them;
the shell redirect always creates a regular file inside the 0700 mktemp
directory, removing link semantics from the privileged step entirely.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0c37de44-e0a0-4090-8f7e-299889108578
The comment claimed the install path "fails closed". That is only true for
link entries. Verified in the target image that a directory member named
`changie` streams its child's contents through `tar -O` (21 bytes, exit 0),
passing the non-empty check; duplicate members concatenate similarly.

Neither is exploitable: the bytes originate inside the checksum-gated
archive, never from a host path, so the guard still prevents the privileged
install from following a link into the filesystem. Reword to state that
bound precisely rather than overclaiming.

Comment-only change; bash -n and shellcheck --severity=style pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0c37de44-e0a0-4090-8f7e-299889108578
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant