Skip to content

Avoid port collisions in three CPU unit tests - #9123

Open
ArchanaChetan07 wants to merge 1 commit into
deepspeedai:masterfrom
ArchanaChetan07:fix/9103-test-port-collisions
Open

ArchanaChetan07 wants to merge 1 commit into
deepspeedai:masterfrom
ArchanaChetan07:fix/9103-test-port-collisions

Conversation

@ArchanaChetan07

Copy link
Copy Markdown

Fixes #9103.

Three CPU unit tests can grab a TCP port that the test harness also hands out, so they fail with EADDRINUSE when tests run in parallel under pytest -n. get_master_port() in tests/unit/common.py gives each xdist worker its own 1000-port range starting at 29500, so these hardcoded or derived ports land inside worker ranges.

Changes (test files only):

  • tests/unit/runtime/zero/test_zero_grad_clip.py: TestZeroGradClip set MASTER_PORT='29500', which is the first port in xdist worker 0's range. It now uses get_master_port().
  • tests/unit/utils/test_groups.py: test_default_destroy_releases_cached_groups re-initialized with a bare init_distributed(dist_backend="gloo"), which goes through env:// and binds a TCP port. It now re-initializes from a file:// store under tmpdir, the same way test_world_destroy_releases_sequence_parallel_groups just above it already does.
  • tests/unit/checkpoint/test_affine_transfer_executor_external_group.py: the weight-update group used a TCPStore on 29000 + os.getpid() % 1000, which overlaps worker 0's range. It now uses a FileStore in the test's temp directory, so it needs no port. The store keeps the previous 60 s timeout (FileStore defaults to 300 s). The group is still built outside the default world with _new_process_group_helper, which is what the test checks.

Duplicate check: no open or closed PR references #9103, and searches for MASTER_PORT, "port collision" and EADDRINUSE turn up no PR touching these tests. The issue has no comments or linked PRs.

Testing: CPU only, WSL2 Ubuntu, 24 vCPUs, no GPU, torch 2.15.0+cpu, Python 3.12.3, based on b9896d054.

cd tests
LOCAL_SIZE=4 python -m pytest -n 4 \
  unit/runtime/zero/test_zero_grad_clip.py \
  unit/utils/test_groups.py \
  unit/checkpoint/test_affine_transfer_executor_external_group.py
  • 3 runs with -n 4 and LOCAL_SIZE=4: 21 passed, 2 skipped each time. The 2 skips are the fp16 cases of TestZeroGradClip, which skip with "Could not initialize deepspeed" on CPU; master skips them the same way.
  • 3 more -n 4 runs without LOCAL_SIZE (multi-rank tests skip there): 17 passed, 6 skipped each time.
  • One serial run on master and on this branch: same pass and skip results.
  • pre-commit run --files <changed files>: all hooks pass.

Compatibility matrix: not applicable. The change touches no library code, only how three tests set up their process-group stores. GPU and other accelerator runs were not done. The affine-transfer test is CPU-only (skipif on non-CPU accelerators), and the other two changes follow patterns that already exist in the same files.

AI assistance: this change was made with AI assistance (Claude Code). I reviewed every changed line and ran the tests above.

🤖 Generated with Claude Code

TestZeroGradClip hardcoded MASTER_PORT=29500, the first port that
get_master_port() hands to xdist worker 0. The re-init in
test_default_destroy_releases_cached_groups went through env:// and
bound a TCP port. The affine-transfer test put its TCPStore on
29000 + getpid() % 1000, which overlaps worker 0's range. Under
pytest -n these can fail with EADDRINUSE.

Use get_master_port() in the first, a file:// store (as the test
above it already does) in the second, and a FileStore with the
previous 60 s timeout in the third.

Fixes deepspeedai#9103

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T03:51:40.260473Z 721aecc PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 19, 2026
@jinyouzhi

Copy link
Copy Markdown
Contributor

LGTM, Thank you for fix!
One non-blocking question (the TOCTOU window in get_master_port() —
probe-bind → close → re-bind, not introduced by this PR): is a complete fix
feasible? DistributedTest already launches via file:// init_method, so the
master_port plumbing only survives as a fallback for env:// consumers
(test_zero_grad_clip.py, runtime/zero/utils.py, test_fp32_master_only.py).
Converting those to file:///FileStore would retire get_master_port()
entirely — no ports, no window. Only the explicit tcp:// test in test_dist.py
would still need one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Flaky CPU unit tests: port 29500 EADDRINUSE from hardcoded MASTER_PORT colliding with xdist port ranges

2 participants