build(profile): stop building vendored C/C++ and DWARF nobody reads - #512
Open
justin13888 wants to merge 1 commit into
Open
build(profile): stop building vendored C/C++ and DWARF nobody reads#512justin13888 wants to merge 1 commit into
justin13888 wants to merge 1 commit into
Conversation
An unset [profile.dev] means opt-level 0 with full debug info, and two costs follow from it that together are most of a worktree's target/. cmake-rs and cc both derive their build type from the OPT_LEVEL and DEBUG that cargo hands the build script, so the three packages compiling vendored C/C++ were building it at -O0 -g: libjxl.a at 181.7 MiB against 8.1 MiB for the same archive at Release. That is paid again in every binary statically linking it, and again in every superseded build-script output directory, which cargo never reclaims — build, test and clippy leave three complete libjxl trees behind (894 + 887 + 772 MiB). The oracles under tooling/ already pass -DCMAKE_BUILD_TYPE=Release; these three delegate to jpegxl_src::build() and to cc::Build, neither of which accepts that define, so the profile is the only lever they share. Packed DWARF is then copied into each of the 172 test binaries an --all-features build links. A sampled one was 0.78 MiB of .text under 64.8 MiB of debug sections. split-debuginfo = "unpacked" emits it once per codegen unit as .dwo files the binaries reference, so no debug information is lost; what changes is that a binary is no longer self-contained, which is why the comment says to debug and profile them where they were built. Measured over a full build + test + clippy + edit cycle from cargo clean: 14.15 GiB to 8.48 GiB, with the cold `cargo test --no-run` falling from 296s to 100s — the JXL differential tests had been running against an -O0 reference encoder. Lowering [profile.dev] debug to "line-tables-only" on top of this measured byte-identical, so the debug level is left alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
[profile.dev]was never set, so it ran at cargo's defaultopt-level = 0, debug = 2. Two costs follow from that, and between them they are most of what a worktree'starget/holds. This adds 31 lines toCargo.tomland changes nothing else.Why
1. Three packages were compiling vendored C/C++ at
-O0 -gcmake-rsandccboth derive their build type from theOPT_LEVEL/DEBUGenv cargo hands the build script. Under an unset dev profile that is aDebugbuild of code that is not ours and is never stepped through from Rust:libjxl.aOUT_DIRThat cost is paid again in every binary that statically links the archive, and again in every superseded build-script output directory — which cargo never reclaims. A fresh tree that has run only
build,testandclippyalready holds three complete libjxl build trees (894 + 887 + 772 MiB) plus two Adobe DNG SDK trees. Runningclippyaftertestadded 1.61 GiB tobuild/by itself. That is the mechanism by which a worktree drifts past 18 GiB.The hand-written oracles under
tooling/already pass-DCMAKE_BUILD_TYPE=Releasefor exactly this reason. These three cannot: they delegate tojpegxl_src::build()and tocc::Build, neither of which accepts that define, so the cargo profile is the only lever they share.This does not stop the accumulation — it makes each copy cost 54 MiB instead of 887 MiB, turning ~2.5 GiB of dead duplication into ~160 MiB.
2. Packed DWARF was copied into all 172 test binaries
An
--all-featurestest build links 172 binaries and each got its own copy of its dependencies' debug info. A sampled 67 MiB test binary was 0.78 MiB of.textunder 64.8 MiB of debug sections — 98%. Across a sample of 13, debug sections were 86.5% of bytes.split-debuginfo = "unpacked"emits the DWARF once per codegen unit as.dwofiles the binaries reference (verified:DW_AT_GNU_dwo_nameresolves, and the 6138.dwofiles total 92 MiB). No debug information is lost. What changes is that a binary is no longer self-contained — one copied out oftarget/loses its debug info — hence the note in the comment to debug and profile them where they were built.Measurements
Cold
cargo test --workspace --all-features --no-run, each fromcargo clean. Both levers were also measured alone, anddebug = "line-tables-only"was measured on top of the chosen pair:debug = "line-tables-only"alonesplit-debuginfoalone-O2aloneline-tables-onlyThe last row is why the dev profile's debug level is left alone: lowering it buys nothing once the DWARF is split, so there is no reason to accept the debuggability tradeoff.
Full
build→test→clippy→ edit cycle, same seven stages before and after:cargo buildcargo test --no-runcargo clippy−40% per worktree, and the cold test build is ~3× faster — the JXL differential tests had been running against an
-O0reference encoder, so this is a correctness-neutral speedup as well as a disk saving.Validation
Run in a clean worktree at
masterwith all submodules initialised:mise run fmt-check— passmise run lint— passmise run test— pass, 3767 tests across 202 suites, 0 failuresAlso verified: per-package overrides do reach the build-script env (
OPT_LEVEL=2 DEBUG=false) and carry into thetestprofile; noprofile package spec did not match any packageswarning under plaincargo build, where thetooling/oracles are absent from the compiled graph; andlibjxl.ain this branch's own build directory is 8.1 MiB.Notes for review
[profile.mutants]inheritstestatdebug = 0, so split debuginfo is a no-op there and the mutation gate is unaffected.[profile.bench]and[profile.fast-debug]inheritrelease, so their native builds were alreadyRelease/RelWithDebInfo.incremental/is untouched by this change and is now the largest single component at 3.37 GiB. That is separate headroom, deliberately left out of scope.