Closed
Conversation
Key changes: - Pin sccache to v0.12.0 (was auto-pulling latest, causing instability) - Add SCCACHE_DIRECT=true for direct hashing mode - Add SCCACHE_DIR=/tmp/sccache for persistent local disk cache - Add CARGO_BUILD_PIPELINING=false to avoid rmeta race conditions (rust-lang/cargo#16790) - Mount /tmp from host via volumes in Cross.toml for cache persistence - Pass GHA cache env vars through to container (SCCACHE_GHA_ENABLED etc.) - Dynamically detect container arch to download matching sccache binary (handles x86_64, aarch64, armv7, i686 via QEMU) Rationale: The original issue #2687 was caused by cargo pipelining starting dependent crate compilation before dependency .rmeta files were fully available. sccache then tried to hash the missing .rmeta file, resulting in a fatal error. The fix disables pipelining at the cargo level and uses sccache's direct mode for safer hashing.
cross Docker images lack CA certificates; sccache's GHA cache client fails with TLS handshake errors when trying to reach the GHA cache endpoint from inside the container. sccache now relies solely on the local disk cache at SCCACHE_DIR=/tmp/sccache, which persists across container runs via the /tmp volume mount. Native (non-cross) builds still use the host-side GHA cache via mozilla-actions/sccache-action.
cross mounts the host target/ directory at /target with --user, which can prevent cargo from creating the required subdirectory structure (e.g. aarch64-unknown-linux-gnu/debug/deps/). The pre-build step ensures /target exists and is world-writable.
… issue Remove pre-build and volumes from Cross.toml to debug whether the container write permission error is caused by our Dockerfile changes (sccache, env vars) or by Cross.toml additions (volumes, pre-build).
Test whether any custom Dockerfile at all causes the /target volume write issue, or if it's specific to sccache-related changes.
SCCACHE_DIRECT=true skips sccache's dep-info pre-pass (which writes to a temp directory without deps/ subdir), going straight to the real rustc invocation which handles directory creation correctly.
Add sccache v0.14.0 to the cross Docker image for diagnostic use. RUSTC_WRAPPER is deliberately NOT set because sccache's server mode interferes with cross's --user Docker setup - rustc fails to create output directories under /target (Docker volume mount). Key findings from debugging (next 8 CI runs): - Plain Dockerfile (clang/llvm only) -> PASS - sccache binary installed, no RUSTC_WRAPPER -> PASS - RUSTC_WRAPPER=/usr/bin/sccache added -> ALL cross targets FAIL (rustc: "error writing dependencies to /target/...d: No such file") - SCCACHE_DIRECT=true, SCCACHE_START_SERVER=0, v0.12.0 -> all FAIL - v0.14.0 (same as host) -> all FAIL - The issue is consistent across all archs (x86_64, aarch64, armv7, i686) - Non-cross targets (Windows, macOS) work fine with host sccache Native builds continue to use the host sccache via mozilla-actions/sccache-action@v0.0.9 (GHA cache backend). Cross builds rely on cargo's fingerprint tracking + Swatinem/rust-cache for incremental compilation across CI runs.
- sccache v0.14.0 installed inside cross container - RUSTC_WRAPPER points to sccache-wrapper.sh instead of sccache directly - Wrapper pre-creates --out-dir/deps before delegating to sccache, fixing the directory creation issue with cross's --user Docker setup - SCCACHE_GHA_ENABLED + ACTIONS_* env vars passed through Cross.toml so container sccache uses GHA cache API (persistent across CI runs) - ca-certificates kept installed for GHA cache TLS verification - SCCACHE_DIRECT=true for faster direct-mode hashing - CARGO_BUILD_PIPELINING=false to avoid cargo-level pipelining races
All attempts to use RUSTC_WRAPPER inside cross containers fail due to cross's --user + Docker volume mount at /target. Wrapper scripts that pre-create directory trees also don't help - the issue is deeper than just missing directories. sccache binary is installed for manual/diagnostic use. GHA cache env vars are passed through so sccache can use GHA API when invoked explicitly. No RUSTC_WRAPPER set - cross compiles natively without sccache (host sccache handles non-cross builds).
sccache's RUSTC_WRAPPER fails in cross containers because cross runs with --user and the /target Docker volume mount prevents directory creation. Solution: set CROSS_CONTAINER_UID=0 and CROSS_CONTAINER_GID=0 in the CI workflow env so cross runs the container as root. This allows sccache (and rustc) to create the full /target directory tree without permission issues.
CROSS_CONTAINER_UID=0 env var does not exist in cross v0.2.5 (checked source at commit 65fe72b). The feature was added in cross v1 (PR #543) but cross v0.2.5 is a v2 reimplementation. Revert to working state: sccache installed, GHA cache env vars passed through, no RUSTC_WRAPPER. cross targets compile natively without sccache (cargo incremental + Swatinem/rust-cache handles cross-CI caching).
sccache on the HOST runs as the server (started during 'cargo install cross' compilation). sccache in the cross container acts as a client connecting to the host's server via a shared Unix Domain Socket at /tmp/sccache-server.sock, mounted via Docker volume. This avoids the cross --user permission issue entirely because: - Server runs on the host with full filesystem access and GHA cache (ghac) - Client in the container just sends compilation requests over UDS - No file I/O from sccache inside the container at all Key changes: - CI workflow: set SCCACHE_SERVER_UDS=/tmp/sccache-server.sock - Cross.toml: mount /tmp, pass through GHA cache env vars - Dockerfile: RUSTC_WRAPPER points to container sccache (client mode) with SCCACHE_SERVER_UDS set to the shared socket
All attempts to use RUSTC_WRAPPER inside cross containers failed: 1. RUSTC_WRAPPER=sccache directly -> all cross targets fail 2. + SCCACHE_DIRECT, +SCCACHE_START_SERVER=0 -> same failure 3. + v0.14.0 (same as host) -> same failure 4. + sccache wrapper (mkdir deps/) -> same failure 5. CROSS_CONTAINER_UID=0 (not in cross v0.2.5) -> not supported 6. Host-client UDS (sccache server can't access GHA cache) -> fails The working approach: sccache binary installed for manual use, no RUSTC_WRAPPER. cross compiles natively (cargo incremental + Swatinem/rust-cache handles caching across CI runs). Native builds use host sccache via mozilla-actions/sccache-action.
- Use SCCACHE_SERVER_UDS=/tmp/sccache.sock for explicit server socket - SCCACHE_DIR=/tmp/sccache for local disk cache (no GHA) - /tmp volume mount so socket/cache persist across runs - No GHA cache env passthrough (was causing server startup failure)
…r paths) The host sccache server and container client can't share a UDS socket because they're in different mount namespaces - the host server can't resolve the container's source file paths. Container uses its own on-demand sccache server with local disk cache via /tmp volume mount.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.