Skip to content

[ops] Warn when DeepSpeedCPUAdam is limited to one OpenMP thread - #8938

Merged
delock merged 3 commits into
deepspeedai:masterfrom
0z5a:cpu-adam-omp-warning
Nov 17, 2026
Merged

delock merged 3 commits into
deepspeedai:masterfrom
0z5a:cpu-adam-omp-warning

Conversation

@0z5a

@0z5a 0z5a commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

torchrun sets OMP_NUM_THREADS=1 for every process when --nproc_per_node is larger than 1, unless the variable is already set. DeepSpeedCPUAdam then steps on a single OpenMP thread, and nothing in the log says so. The deepspeed launcher sets the thread count only with --bind_cores_to_rank, which is the path the ZeRO-Offload tutorial documents. Users who launch with torchrun or accelerate get no hint.

Measured effect

Setup: ZeRO-1 with offload_optimizer: cpu, Qwen3-4B in bf16, fp32 gradient accumulation. The time is for engine.step() (CPU Adam plus the copies around it), median over the measured steps.

Hardware Launch OMP_NUM_THREADS=1 threads set per rank Speed-up
one GH200, 72-core host 1 process, variable set to 1 by hand to match the torchrun default 5.6 s 0.8 s (36 threads) 7×
B300 torchrun --nproc_per_node 2, real default 3.92 s 1.76 s (32 threads, half a NUMA node per rank) 2.2×

On B300, each rank steps half of the optimizer state, so the single-thread cost is already split over two processes. In a 2-GPU torchrun job on L40S at the default, the same step took 6.2 s.

The B300 run with threads set also changed the attention path for packed micro-batches. That doesn't touch engine.step(), so the optimizer column only reflects the thread count.

Change

  • single_thread_warning() in deepspeed/ops/adam/cpu_adam.py returns a message when OMP_NUM_THREADS is "1" and the rank has at least two cores (physical cores from psutil, capped by the CPU affinity mask, divided by LOCAL_WORLD_SIZE).
  • DeepSpeedCPUAdam.__init__ logs it once per process, for the first optimizer instance.
  • The CPU Adam tuning section of the ZeRO-Offload tutorial gets a note for torchrun users.

It only warns and does not change the thread count:

  • an explicit OMP_NUM_THREADS=1 may be intentional;
  • the OpenMP runtime may already have read the variable by the time the optimizer is built.

Tests

tests/unit/ops/adam/test_cpu_adam_threads.py is CPU only and does not build the op. It checks these cases:

  • warns at the torchrun default when the rank has spare cores;
  • silent when the variable is unset or larger than 1;
  • silent when the rank has a single core;
  • a single process counts all its cores;
  • hyperthreads are not counted, the affinity mask caps the count, and a None from psutil falls back to os.cpu_count().

yapf, flake8, codespell and the license check pass.

torchrun exports OMP_NUM_THREADS=1 for every process when
--nproc_per_node > 1, unless the variable is already set. The CPU
optimizer's OpenMP loop then runs on a single thread, and nothing says
so. The deepspeed launcher only sets the thread count with
--bind_cores_to_rank.

DeepSpeedCPUAdam now logs one warning per process when
OMP_NUM_THREADS is 1 and the rank has at least two cores (CPU affinity
divided by LOCAL_WORLD_SIZE). The ZeRO-Offload tutorial gets a note for
torchrun users.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@0z5a
0z5a marked this pull request as ready for review November 5, 2026 09:53
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-05T09:59:47.494704Z e01a6ba Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e01a6ba67f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread deepspeed/ops/adam/cpu_adam.py Outdated
if os.environ.get("OMP_NUM_THREADS") != "1":
return None
try:
cores = len(os.sched_getaffinity(0))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Count physical cores before recommending OpenMP threads

On an SMT-enabled host, os.sched_getaffinity() counts logical CPU IDs rather than physical cores. The tutorial and DeepSpeed's get_numactl_cmd() path assign physical cores (psutil.cpu_count(logical=False)), so, for example, two SMT siblings for one physical core make this helper warn that the rank can use two cores and recommend two threads even though OMP_NUM_THREADS=1 is already the intended allocation; larger jobs can be advised to oversubscribe CPUAdam by the SMT factor. Derive physical cores within the affinity mask, or do not present this count as the recommended thread count.

Useful? React with 👍 / 👎.

Comment on lines +29 to +31
return (f"OMP_NUM_THREADS=1, so DeepSpeedCPUAdam steps on one thread although this rank can use {cores_per_rank} "
"cores. torchrun sets OMP_NUM_THREADS=1 when --nproc_per_node > 1; set OMP_NUM_THREADS to the number of "
"cores per rank, or launch with `deepspeed --bind_cores_to_rank`.")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid OMP advice when the selected builder lacks OpenMP

With MLU, SDAA, HPU, or XPU selected, their CPUAdamBuilder/base builder compiles the shared CPU Adam source without an OpenMP flag; MPS does the same whenever Homebrew libomp is unavailable. Under torchrun this code still emits the warning, but changing OMP_NUM_THREADS or core binding cannot make those builds multi-threaded, so the remediation is ineffective and masks the actual build limitation. Gate the warning on OpenMP support or provide a builder-specific remedy.

AGENTS.md reference: AGENTS.md:L19-L19

Useful? React with 👍 / 👎.

self.opt_id = DeepSpeedCPUAdam.optimizer_id
DeepSpeedCPUAdam.optimizer_id = DeepSpeedCPUAdam.optimizer_id + 1
if self.opt_id == 0:
message = single_thread_warning()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip the warning for overlapped ZenFlow CPU Adam

When ZenFlow is enabled with overlap_step=True under torchrun's default environment, this base constructor emits the warning even though ZenFlowCPUAdam deliberately leaves this object's step unused: start_optimizer_process() drives updates through ZenFlowAdam's affinity-sized PinnedThreadPool, whose slices invoke the kernel with parallel=false. Thus OMP_NUM_THREADS=1 does not limit the actual optimizer, and the warning can prompt an unnecessary global thread-count change; suppress it for the overlap implementation or issue diagnostics from the implementation that actually steps.

AGENTS.md reference: AGENTS.md:L19-L19

Useful? React with 👍 / 👎.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>

# Conflicts:
#	deepspeed/ops/adam/cpu_adam.py
@delock
delock self-requested a review November 10, 2026 09:02
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026
@@ -0,0 +1,52 @@
# Copyright (c) Microsoft Corporation.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not use Microsoft copyright header for new files.

cores = len(os.sched_getaffinity(0))
except AttributeError: # not available on macOS
cores = os.cpu_count() or 1
cores_per_rank = cores // max(int(os.environ.get("LOCAL_WORLD_SIZE", "1")), 1)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.cpu_count() count virtual cores, which is physical cores * (hyperthreadings/core). Dense usage should count physical cores. Can you use the following example? Thanks!
total_cores = psutil.cpu_count(logical=False)

The optimizer loop is dense, so hyperthreads do not add throughput.
Count physical cores with psutil, fall back to os.cpu_count() when
psutil cannot tell, and keep the CPU affinity mask as an upper bound.

Use the DeepSpeed Team copyright line in the new test file.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 17, 2026
@0z5a

0z5a commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @delock, both done in 9ea4059:

  • The new test file now starts with # Copyright (c) DeepSpeed Team., like the other recently added files.
  • The warning counts physical cores with psutil.cpu_count(logical=False). If psutil returns None it falls back to os.cpu_count(). When the process is pinned to fewer CPUs (taskset, cpuset), the affinity mask still caps the count, so the message never names more cores than the rank can use. Tests cover a host with two hardware threads per core, a narrower affinity mask, and the None case.

@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 17, 2026
@delock
delock added this pull request to the merge queue Nov 17, 2026
Merged via the queue into deepspeedai:master with commit 7454d78 Nov 17, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants