queue-runner: fix dynamic derivation scheduling and resolution - #1836
Open
amaanq wants to merge 3 commits into
Open
queue-runner: fix dynamic derivation scheduling and resolution#1836amaanq wants to merge 3 commits into
amaanq wants to merge 3 commits into
Conversation
amaanq
force-pushed
the
dyn-drv-queue-fixes
branch
from
July 23, 2026 18:09
c295496 to
8a60790
Compare
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/dynamic-derivations-next-steps/79409/1 |
…teps When a step resolves to a different drv path, its rdeps get migrated to the resolved step with `make_rdep`, which records an empty `OutputNameChain`. Dynamic rdeps lose their relation that way, so `pop_dynamic_rdeps` finds nothing when the resolved step finishes and the generated derivation never gets scheduled; anything depending on it fails resolution forever.
`flatten_chain` pushed the outermost output name onto the end of the chain, which is where `pop` reads from, so a depth-2 input like `drv^out^hello` resolved outermost-first and asked the outer derivation for the inner one's output. The outermost name now goes at the front, matching the stack order `OutputNameChain` documents. I've also added a debug trace for each resolution hop.
amaanq
force-pushed
the
dyn-drv-queue-fixes
branch
from
August 20, 2026 22:27
8a60790 to
4cfe787
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When trying to run nix-ninja jobs with dynamic derivation through the queue runner, I hit three bugs in the two-phase CA dance. Regular CA builds only ever produce depth-1 chains, so none of these paths had really been exercised before it seems.
make_rdep, which records an emptyOutputNameChain. As such, dynamic rdeps lost their relation, sopop_dynamic_rdepsfound nothing when the resolved step finished and the generated derivation was never scheduled.flatten_chainpushed the outermost output name onto the pop end of the chain, so an input likedrv^out^helloresolved outermost-first and asked the outer derivation for the inner one's output.ResolvedStepCreationFailedand retried forever instead of finishing the build as cached. This is the early cutoff case the included test located atearly-cutoff-sequential.tcovers.With these three bugs fixed, nix-ninja's example jobs build green end to end for me locally. The outer derivation runs nix-ninja inside the sandbox, the generated derivations get scheduled as real steps, and dependents resolve through the two-level chain!