perf(lazy_turtle): targeted ancestor+subtree map in materialize_submodule - #2989
Merged
Conversation
…dule Avoid scanning the entire shell model with target_model.named_modules() on every materialize_submodule call. Instead build only the target submodule's ancestor chain plus its descendant subtree when recurse=True. This removes an O(total_modules * modules_per_layer) overhead that dominated per-module load time for large MoE models. - materialize_submodule now accepts optional module_path, recurse, tie_weights, and show_progress kwargs and passes a targeted modules_by_name map into _copy_checkpoint_tensors_into_submodule. - _copy_checkpoint_tensors_into_submodule accepts an optional modules_by_name map and falls back to a full named_modules() scan only when none is provided, preserving backwards compatibility. - Add regression tests ensuring the targeted map is used and transpose hints are still resolved for ancestors and descendants. - Add benchmark and lifecycle profiler scripts for measuring the map construction and turtle -> shell -> disk overhead.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LazyTurtle.materialize_submodulewas buildingdict(target_model.named_modules())on every call, which isO(total_modules × modules_per_layer)and dominated per-module load time for large MoE models. This change replaces that full-model scan with a targeted map that contains only the target submodule, its ancestors, and (whenrecurse=True) its descendants, which is exactly what_resolve_prefer_transposed_hintneeds.materialize_submodulenow accepts optionalmodule_path,recurse,tie_weights, andshow_progresskwargs, builds the targeted map, and passes it to_copy_checkpoint_tensors_into_submodule._copy_checkpoint_tensors_into_submodulenow accepts an optionalmodules_by_namemap and falls back to a fullnamed_modules()scan only when none is provided.tests/test_lazy_turtle_materialize_map.pywith regression tests that monkeypatchnn.Module.named_modulesto raise if a full model scan is used, while still verifyingis_transposedhints on ancestors and descendants and numerical parity of the loaded weights.scripts/benchmark_lazy_turtle_materialize_map.pyandscripts/profile_lazy_turtle_lifecycle.pyfor micro-benchmarking map construction and measuring the full turtle -> shell -> disk lifecycle on a synthetic MoE model.Validation:
ruff check ...cleanpytest -q tests/test_structure.py tests/test_lazy_turtle_materialize_map.py— 5 passedpython scripts/benchmark_lazy_turtle_materialize_map.pyandLAZY_LIFECYCLE_LAYERS=2 python scripts/profile_lazy_turtle_lifecycle.pyrun successfully on CPU