fix(deps): keep packages the running process loaded requireable across an install that re-keys them - #10595
Conversation
…s an install that re-keys them
pnpm keys a virtual-store directory by the package's peer-resolution hash, so
an install that changes the dependency set gives the same name@version a NEW
directory and deletes the one this process loaded its modules from. Node keeps
the loaded module objects, but not the files - so any require the loaded code
deferred past load time resolves against the deleted directory and throws
MODULE_NOT_FOUND.
Every package the process loaded out of the workspace's own virtual store is
exposed to this, and an env is the worst case: @teambit/aspect defers
require('./babel/babel-config') until getCompiler() is called - which the
install flow itself does right after the package-manager run, when it compiles
components and reloads envs. The install then dies with "Cannot find module
'./babel/babel-config'", and since the env never loads, bit create surfaces it
as the misleading `template "react" was not found`.
Replacing the in-memory instances instead does not work - verified by trying:
every reload path (reloadMovedEnvs, loading components as aspects) has to
consult the registered env to do its work, and consulting it is exactly what
throws. reloadMovedEnvs is additionally a no-op for these envs: it filters on
env.__path/env.id, which only plugin-loaded envs carry, never aspect-registered
ones.
So the fix follows the rule an OS applies to a running binary's deleted files:
what the process has loaded stays available for the process's lifetime. Before
the package-manager run, snapshot which virtual-store directories back entries
in require.cache; afterwards, restore any that vanished from their re-keyed
twin - same name@version, different peer hash - whose package content is
identical (same tarball; the peer set only affects the sibling dependency
symlinks, which are relative and stay valid). pnpmPruneModules learns to skip
directories backing require.cache entries so it does not re-delete a restored
one; a later command's process, which has nothing loaded from it, prunes it.
Found while investigating teambit#10465, where the envs that used to be core aspects
become ordinary packages resolved out of the workspace's virtual store and made
this failure fatal for 8 of 14 e2e shards. Reproduced and verified there with
deps-in-capsules.e2e.ts: the second install re-keys a dozen loaded
@teambit/*@1.0.1042 slots; without the fix the suite fails on the babel-config
require, with it all tests pass and the debug log shows each removed slot
restored.
(cherry picked from commit e792b9f)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…modules The preservation added in e792b9f keeps virtual-store directories the running process loaded modules from requireable when an install re-keys them, by scanning require.cache - which only CJS modules appear in. ESM modules live in node's ESM module map, which has no enumeration API, so an ESM env relocated by an install was still exposed to the same MODULE_NOT_FOUND on any import it deferred past load time. aspect-loader now records every file it loads through dynamic import() - the env-plugin loader and loadEsm(), the only two such call sites - in a Symbol.for-keyed global Set, and the preservation scans that set alongside require.cache. A global symbol rather than a shared import because the reader lives in the pnpm package manager, which aspect-loader must not depend on (the dependency runs the other way), and Symbol.for resolves to the same key even when a module is duplicated in node_modules; both sides document the contract and point at each other. Only ESM entry files are recorded, not their transitive static imports: those are fully loaded into memory and never re-read, while the entry's own package directory - where deferred imports and config-file reads point - is restored wholly. The realpath is recorded alongside the given spelling so a load reached through a node_modules symlink is attributed to the .pnpm directory that owns it. Verified: 55 specs across both components (4 new for the recorder, 2 new for the reader), deps-in-capsules.e2e.ts still green, npm run lint green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 06fa5ec)
The restores ran under an unbounded Promise.all, one recursive fs.copy per removed directory, right after the engine has just saturated the disk - review flagged the burst as a hazard in constrained CI/container environments. The common case is zero removed directories and the checks stay cheap; when there are any, serial restore bounds the I/O with no meaningful cost (the deps-in-capsules repro restores 26 dirs and stays green). Also logs an aggregate removed/restored/duration line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 1bd71e7)
PR Summary by QodoPreserve loaded pnpm virtual-store packages across peer-hash rekeys
AI Description
Diagram
High-Level Assessment
Files changed (8)
|
Code Review by Qodo
1.
|
…pace root The scanning compared loaded module filenames against a prefix built with path.resolve(rootDir), but node resolves a module's filename through its realpath, so require.cache is keyed by the real spelling even when the install was handed the symlinked one. A workspace reached through a symlink therefore matched nothing: the snapshot came back empty and the whole preservation silently turned into a no-op - including on macOS, where a temp dir under /var is really under /private/var, so every e2e workspace is such a case. Match against both spellings of the virtual store (the given one and its realpath, the latter kept for --preserve-symlinks), and record each slot the way the file that revealed it was spelled, so the existence check and the restore that follow address the directory the module was actually loaded from. Both scanners now share one walker. Raised by review on teambit#10595. The two new specs fail on the previous code with an empty result and pass now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Fixed in 63ccfc1 — the finding was correct.
Both scanners now share one walker that matches against the given spelling and its realpath (the given one kept for Two regression specs added: they fail on the previous code with an empty result and pass now. I skipped the suggested Windows drive-letter case normalization — both sides of the comparison come from the same |
|
Code review by qodo was updated up to the latest commit 63ccfc1 |
…y patched twin findDonorDirName matched on escaped name and version alone, but pnpm encodes a patch in the same suffix as the peer set - depPathToDirName turns foo@1.0.0(patch_hash=abc) into foo@1.0.0_patch_hash=abc - so an unpatched slot could be restored from a patched one and vice versa. A patch changes the package's own files, unlike a peer set, which only changes the sibling symlinks, so the restored directory would no longer match the modules the process already loaded from it. Require the patch segment to be equal; when it is not, no donor is found and the restore is skipped, which is where this started. Two more from the same review: - the reader of the global ESM record spread whatever occupied the well-known symbol, so a value that is not iterable would throw out of every install and prune. It is a global held by convention, not by types: treat anything that is not a set of paths as absent (CJS preservation still works), and have the recorder replace a wrong-typed value rather than lose every later add(). - pnpmPruneModules scanned the loaded modules before knowing whether it had anything to remove. Compute the extraneous set first and return early - the common case does no scanning at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
All three addressed in 18254b4. 1. Donor content not verified — real, and I could pin down exactly when. I did not take the pre-install fingerprint route from the suggestion. It would add a file read per loaded slot to every install to cover a case the dir name already answers for free, and the snapshot is deliberately pure in-memory so it costs nothing when there is nothing to preserve. 2. Unchecked ESM global value — agreed, and the reader was the dangerous half: it spread the value unguarded, so a non-iterable there would throw out of every install and prune. It now treats anything that isn't a set of paths as absent (CJS preservation still works) and filters non-string entries. The recorder replaces a wrong-typed value instead of losing every later 3. Prune scans cache always — fair. Six new specs cover the three changes. |
|
Code review by qodo was updated up to the latest commit 18254b4 |
A slot holds its dependencies too, as symlinks under the same node_modules, so a loaded path that kept that spelling instead of being realpathed (--preserve-symlinks, or an ESM load recorded by the name it was given) names the dependency rather than the slot's owner. The snapshot bound each slot to whichever package the first such path named, and a slot attributed to the wrong one finds no donor - findDonorDirName requires the dir name to start with the escaped package name - so it is never restored and the deferred requires it was meant to protect still throw. Prefer whichever loaded path names the owner, whatever order the paths arrive in. A non-owner attribution is kept only as a fallback, for a slot named after something other than <pkg>@<version> (a tarball or git dependency), where no path can match and a restore was never possible anyway. The prefix test both sites need is now one helper. Raised by review on teambit#10595. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Fixed in 34c4137 — correct, and the invariant was already implied, just never enforced where it was decided. A slot holds its dependencies as symlinks under the same On the second half of the suggestion — I kept a non-owner attribution as a fallback rather than dropping the entry. A slot named after something other than Two specs: the first fails on the previous code with |
|
Code review by qodo was updated up to the latest commit 34c4137 |
Both call sites recorded the path before awaiting the dynamic import, so a rejected import left it in the process-global set. A load that failed leaves nothing in memory whose files have to stay around, and the record makes pnpmPruneModules treat that virtual-store slot as in use - keeping a directory nothing is using for the rest of the process's life. Record after the import resolves, and state the rule in the recorder's contract. Raised by review on teambit#10595. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Fixed in 696c727. Both call sites now record after the The consequence is bounded — a slot nothing is using survives one extra prune — but the record is supposed to mean "this process has something loaded from here", and a rejected import leaves nothing in memory whose files need to stay around. The recorder's contract now says so, since it is the only thing keeping the two writers honest. |
|
Code review by qodo was updated up to the latest commit 696c727 |
…from-manifest The loaded-package preservation work this branch carried was upstreamed as #10595, so master and the branch both hold it and every conflict is between the branch's original and the reviewed version that landed. Master's version wins throughout: it is the same work with the realpath spellings, the patch-hash donor check, the slot-owner attribution and the prune early-return added on top. plugins.ts and aspect-loader's loadEsm merged cleanly into a double recordLoadedEsmFile - the branch recorded before the load, master after it, on the grounds that only a load that succeeded leaves something in memory worth keeping files for. Kept master's single post-load call in both. The branch-only aspect-loader changes (the versionless loaded-aspect lookup, the core-aspect manifest guard, the requested-id-preserving def dedup) are untouched - master never had them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s nested copy
An install that decides a hoisted copy satisfies what was nested deletes the
nested directory - and if this process loaded modules out of it, node keeps the
module objects but not the files, so any require the loaded code deferred past
load time throws MODULE_NOT_FOUND. `@teambit/aspect` defers
`require('./babel/babel-config')` until `getCompiler()`, which the install flow
itself calls right after the package-manager run, so the install dies with
`Cannot find module './babel/babel-config'`.
This is the same hazard #10595 fixed for the virtual store, reached by a
different route. That snapshot only scans `node_modules/.pnpm`, so a package
loaded from a root component's own node_modules was invisible to it and the
preservation was inert. root-components' hoisted-linker suite hit exactly this:
17 loaded directories under `.bit_roots/teambit.harmony_aspect/node_modules`
were removed by the second install.
`reloadMovedEnvs` does not cover it either - it skips any env without a
`__path`, which is only set for plugin-registered envs, and `@teambit/aspect`'s
AspectEnv is registered by its aspect provider. Reloading is no substitute
anyway: consulting the registered env is what triggers the deferred require.
So preserve these the same way: snapshot the package directories under the
workspace's node_modules that back loaded modules, and afterwards restore any
that vanished from a same-version copy found by walking the node_modules chain
up from where it used to be - what node itself would resolve now - bounded at
the workspace root. A donor must match on version, since the point is to keep
serving the files belonging to the modules already in memory. The loaded-files
plumbing shared with the virtual-store module moves to loaded-module-files.ts.
Verified with root-components.e2e.ts's hoisted-linker suite: 0 passing/1 failing
before (the before-all hook died on the babel-config require), 12 passing after,
with the debug log showing 17 of 17 removed directories restored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extracted from #10465 so the fix can land on its own.
Problem
pnpm keys a virtual-store directory by the package's peer-resolution hash, so an install that changes the dependency set gives the same
name@versiona new directory and deletes the one this process loaded its modules from. Node keeps the loaded module objects, but not the files — so anyrequire/importthe loaded code defers past load time resolves against the deleted directory and throwsMODULE_NOT_FOUND.Every package the process loaded out of the workspace's own virtual store is exposed to this, and an env is the worst case:
@teambit/aspectdefersrequire('./babel/babel-config')untilgetCompiler()is called — which the install flow itself does right after the package-manager run, when it compiles components and reloads envs. The install then dies withCannot find module './babel/babel-config'.Replacing the in-memory instances instead does not work (verified by trying): every reload path —
reloadMovedEnvs, loading components as aspects — has to consult the registered env to do its work, and consulting it is exactly what throws.reloadMovedEnvsis additionally a no-op for these envs: it filters onenv.__path/env.id, which only plugin-loaded envs carry.Fix
Follow the rule an OS applies to a running binary's deleted files: what the process has loaded stays available for the process's lifetime.
name@version, different peer hash — whose package content is identical (same tarball; the peer set only affects the sibling dependency symlinks, which are relative and stay valid from the restored location).pnpmPruneModulesskips directories backing loaded modules so it does not re-delete a restored one. A later command's process, which has nothing loaded from it, prunes it.require.cache. ESM modules live in node's ESM module map, which has no enumeration API, soaspect-loaderrecords every file it loads through dynamicimport()(the env-plugin loader andloadEsm(), the only two such call sites) in aSymbol.for-keyed global set that the pnpm side reads. A global symbol rather than a shared import because the reader lives in the pnpm package manager, whichaspect-loadermust not depend on — the dependency runs the other way — andSymbol.forresolves to the same key even when a module is duplicated innode_modules. Both sides document the contract and point at each other.Commits
Cherry-picked from #10465 (
-xtrailers reference the originals):fix(deps): keep packages the running process loaded requireable across an install that re-keys themfix(deps): extend loaded-package preservation across installs to ESM modulesfix(deps): restore removed loaded virtual-store dirs sequentially#10465 also carries
fix(generator): stop reporting an env-load failure as "template not found", which is not included here: it fixesgetTemplateWithIdOrEnvFallback, a method that only exists on that branch, so there is nothing on master for it to apply to. It stays with #10465.Verification
bit test teambit.harmony/aspect-loader teambit.dependencies/pnpm— 55/55 passing (4 new specs for the ESM recorder, 6 new for the snapshot/donor/prune logic).npm run lint(tsc + oxlint) green.deps-in-capsules.e2e.ts: the second install re-keys a dozen loaded@teambit/*slots; without the fix the suite fails on the babel-config require, with it all tests pass and the debug log shows each removed slot restored.🤖 Generated with Claude Code