Repository navigation
Avoid port collisions in three CPU unit tests - #9123
ArchanaChetan07 wants to merge 1 commit into
Conversation
TestZeroGradClip hardcoded MASTER_PORT=29500, the first port that get_master_port() hands to xdist worker 0. The re-init in test_default_destroy_releases_cached_groups went through env:// and bound a TCP port. The affine-transfer test put its TCPStore on 29000 + getpid() % 1000, which overlaps worker 0's range. Under pytest -n these can fail with EADDRINUSE. Use get_master_port() in the first, a file:// store (as the test above it already does) in the second, and a FileStore with the previous 60 s timeout in the third. Fixes deepspeedai#9103 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
LGTM, Thank you for fix! |
Fixes #9103.
Three CPU unit tests can grab a TCP port that the test harness also hands out, so they fail with
EADDRINUSEwhen tests run in parallel underpytest -n.get_master_port()intests/unit/common.pygives each xdist worker its own 1000-port range starting at 29500, so these hardcoded or derived ports land inside worker ranges.Changes (test files only):
tests/unit/runtime/zero/test_zero_grad_clip.py:TestZeroGradClipsetMASTER_PORT='29500', which is the first port in xdist worker 0's range. It now usesget_master_port().tests/unit/utils/test_groups.py:test_default_destroy_releases_cached_groupsre-initialized with a bareinit_distributed(dist_backend="gloo"), which goes throughenv://and binds a TCP port. It now re-initializes from afile://store undertmpdir, the same waytest_world_destroy_releases_sequence_parallel_groupsjust above it already does.tests/unit/checkpoint/test_affine_transfer_executor_external_group.py: the weight-update group used aTCPStoreon29000 + os.getpid() % 1000, which overlaps worker 0's range. It now uses aFileStorein the test's temp directory, so it needs no port. The store keeps the previous 60 s timeout (FileStoredefaults to 300 s). The group is still built outside the default world with_new_process_group_helper, which is what the test checks.Duplicate check: no open or closed PR references #9103, and searches for
MASTER_PORT, "port collision" andEADDRINUSEturn up no PR touching these tests. The issue has no comments or linked PRs.Testing: CPU only, WSL2 Ubuntu, 24 vCPUs, no GPU, torch 2.15.0+cpu, Python 3.12.3, based on
b9896d054.-n 4andLOCAL_SIZE=4: 21 passed, 2 skipped each time. The 2 skips are the fp16 cases ofTestZeroGradClip, which skip with "Could not initialize deepspeed" on CPU;masterskips them the same way.-n 4runs withoutLOCAL_SIZE(multi-rank tests skip there): 17 passed, 6 skipped each time.masterand on this branch: same pass and skip results.pre-commit run --files <changed files>: all hooks pass.Compatibility matrix: not applicable. The change touches no library code, only how three tests set up their process-group stores. GPU and other accelerator runs were not done. The affine-transfer test is CPU-only (
skipifon non-CPU accelerators), and the other two changes follow patterns that already exist in the same files.AI assistance: this change was made with AI assistance (Claude Code). I reviewed every changed line and ran the tests above.
🤖 Generated with Claude Code