SG-45189 Bypass Windows MAX_PATH in tk-core filesystem copy/delete - #1132
stevelittlefish wants to merge 3 commits into
Conversation
Move _to_extended_path from zip.py to filesystem.py (add a `force` flag) and apply the \?\ extended-length prefix across the deep-traversal helpers: copy_folder, move_folder, safe_delete_folder, compute_folder_size, copy_file, touch_file and ensure_folder_exists. Directory creation is force-extended because Windows caps new directories at MAX_PATH-12 (248). Extends the SG-44086 unzip fix to the localize/clone_cache copy path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1132 +/- ##
==========================================
+ Coverage 80.09% 80.11% +0.01%
==========================================
Files 203 203
Lines 19537 19543 +6
==========================================
+ Hits 15649 15657 +8
+ Misses 3888 3886 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The extended-length prefix disables the automatic '.'/'..'/'/' normalization Windows performs on normal paths, so a path containing os.pardir (e.g. the os.path.join(dir, os.pardir) handed to safe_delete_folder) became an invalid "...\config_install_backup\.." extended path that no longer resolved to its parent - the folder was never deleted. This broke bootstrap_tests.test_backups on Windows. _to_extended_path now normpath-normalizes internally before prefixing (without resolving relative paths against the cwd), and skips normalization for already-extended paths. Adds a regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ruff flagged the "\?\ " backslash-space as a deprecated escape sequence in the new regression test's docstring; double it so the string is valid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
The long-path handling is applied consistently at the relevant filesystem call sites and is backed by targeted Windows-only regression/integration tests for the reported failure mode.
Pull request overview
This PR extends tk-core’s Windows long-path support (\?\ prefix) beyond unzip/download to filesystem copy/delete operations, addressing MAX_PATH failures during deep bundle localization (SG-45189).
Changes:
- Moved
_to_extended_path()intotank.util.filesystemand added aforceoption for APIs that walk internally (e.g.,os.walk,shutil.rmtree). - Applied extended-path handling across deep-traversal filesystem helpers (
copy_folder,move_folder,safe_delete_folder,compute_folder_size, plus targeted updates tocopy_file,touch_file,ensure_folder_exists). - Relocated and expanded tests into
tests/util_tests/test_filesystem.py, including a Windows-only integration test that reproduces the >260-char leaf scenario.
File summaries
| File | Description |
|---|---|
| tests/util_tests/test_zip.py | Removes now-relocated _to_extended_path unit tests from zip tests. |
| tests/util_tests/test_filesystem.py | Adds _to_extended_path(force=...) coverage and a Windows-only long-path copy/delete integration test. |
| python/tank/util/zip.py | Switches unzip path prefixing to use filesystem._to_extended_path (no behavior change intended). |
| python/tank/util/filesystem.py | Introduces _to_extended_path(force=...) and applies it to copy/delete/deep traversal helpers to bypass MAX_PATH. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
carlos-villavicencio-adsk
left a comment
There was a problem hiding this comment.
Thank you for this one!
Problem
On Windows, the advanced project setup wizard fails part-way through with
OSError/FileNotFoundErrorwhen copying bundles whose files are nested deeper than the 260-characterMAX_PATHlimit (e.g.tk-framework-desktopserver's bundled per-platform Python, andtk-framework-alias's Sphinx docs). The failing call chain iscore_localize.do_localize→descriptor.clone_cache→filesystem.copy_folder→shutil.copy.This is the follow-up flagged in SG-44086 (PR #1118), which fixed the same limit for the unzip/download path only. As Joel noted there, "other tk-core filesystem calls (
os.path.exists, copies, git descriptors) can still hit MAX_PATH elsewhere." This PR extends the same fix to the copy/delete path.Fix
The
_to_extended_path()helper (previously private toutil/zip.py) is moved toutil/filesystem.pyand given aforceflag. On Windows it prepends the\?\extended-length prefix (\?\UNC\for network paths); off Windows and for ineligible/short paths it is a no-op. This requires noLongPathsEnabledregistry key or app manifest — it works on any Windows machine.It is applied across the deep-traversal helpers:
copy_folder– extends each path at the point of use. Directory creation (os.mkdir) is force-extended because Windows caps a new directory atMAX_PATH - 12(248), not 260.safe_delete_folder/compute_folder_size– force-extend the root soshutil.rmtree/os.walkinternal descent inherits the prefix.move_folder– extends each deep file beforeos.stat/os.chmod/os.remove.copy_file,touch_file,ensure_folder_exists– extend their file/dir paths (force for the dir-creatingmakedirs).copy_folder's returned file list is unchanged (plain paths); callers re-extend as needed.util/zip.pynow imports the helper fromfilesystem(no behaviour change).Tests
TestToExtendedPathintotests/util_tests/test_filesystem.pyand addedforcecoverage.TestCopyFolderLongPaths— a Windows-only integration test that builds a short-rooted tree with a leaf > 260 chars and verifiescopy_folder+safe_delete_foldersucceed (the exact SG-45189 scenario).\?\handling disabled, the new long-path tests fail (integration test errors, prefix-assertion tests fail) while all short-path tests still pass — confirming the tests genuinely exercise the fix.util_testspackage: 223 passed / 6 skipped on Windows (Python 3).🤖 Generated with Claude Code