Repository navigation
[ops] Warn when DeepSpeedCPUAdam is limited to one OpenMP thread - #8938
Conversation
torchrun exports OMP_NUM_THREADS=1 for every process when --nproc_per_node > 1, unless the variable is already set. The CPU optimizer's OpenMP loop then runs on a single thread, and nothing says so. The deepspeed launcher only sets the thread count with --bind_cores_to_rank. DeepSpeedCPUAdam now logs one warning per process when OMP_NUM_THREADS is 1 and the rank has at least two cores (CPU affinity divided by LOCAL_WORLD_SIZE). The ZeRO-Offload tutorial gets a note for torchrun users. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e01a6ba67f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if os.environ.get("OMP_NUM_THREADS") != "1": | ||
| return None | ||
| try: | ||
| cores = len(os.sched_getaffinity(0)) |
There was a problem hiding this comment.
Count physical cores before recommending OpenMP threads
On an SMT-enabled host, os.sched_getaffinity() counts logical CPU IDs rather than physical cores. The tutorial and DeepSpeed's get_numactl_cmd() path assign physical cores (psutil.cpu_count(logical=False)), so, for example, two SMT siblings for one physical core make this helper warn that the rank can use two cores and recommend two threads even though OMP_NUM_THREADS=1 is already the intended allocation; larger jobs can be advised to oversubscribe CPUAdam by the SMT factor. Derive physical cores within the affinity mask, or do not present this count as the recommended thread count.
Useful? React with 👍 / 👎.
| return (f"OMP_NUM_THREADS=1, so DeepSpeedCPUAdam steps on one thread although this rank can use {cores_per_rank} " | ||
| "cores. torchrun sets OMP_NUM_THREADS=1 when --nproc_per_node > 1; set OMP_NUM_THREADS to the number of " | ||
| "cores per rank, or launch with `deepspeed --bind_cores_to_rank`.") |
There was a problem hiding this comment.
Avoid OMP advice when the selected builder lacks OpenMP
With MLU, SDAA, HPU, or XPU selected, their CPUAdamBuilder/base builder compiles the shared CPU Adam source without an OpenMP flag; MPS does the same whenever Homebrew libomp is unavailable. Under torchrun this code still emits the warning, but changing OMP_NUM_THREADS or core binding cannot make those builds multi-threaded, so the remediation is ineffective and masks the actual build limitation. Gate the warning on OpenMP support or provide a builder-specific remedy.
AGENTS.md reference: AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
| self.opt_id = DeepSpeedCPUAdam.optimizer_id | ||
| DeepSpeedCPUAdam.optimizer_id = DeepSpeedCPUAdam.optimizer_id + 1 | ||
| if self.opt_id == 0: | ||
| message = single_thread_warning() |
There was a problem hiding this comment.
Skip the warning for overlapped ZenFlow CPU Adam
When ZenFlow is enabled with overlap_step=True under torchrun's default environment, this base constructor emits the warning even though ZenFlowCPUAdam deliberately leaves this object's step unused: start_optimizer_process() drives updates through ZenFlowAdam's affinity-sized PinnedThreadPool, whose slices invoke the kernel with parallel=false. Thus OMP_NUM_THREADS=1 does not limit the actual optimizer, and the warning can prompt an unnecessary global thread-count change; suppress it for the overlap implementation or issue diagnostics from the implementation that actually steps.
AGENTS.md reference: AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de> # Conflicts: # deepspeed/ops/adam/cpu_adam.py
| @@ -0,0 +1,52 @@ | |||
| # Copyright (c) Microsoft Corporation. | |||
There was a problem hiding this comment.
Should not use Microsoft copyright header for new files.
| cores = len(os.sched_getaffinity(0)) | ||
| except AttributeError: # not available on macOS | ||
| cores = os.cpu_count() or 1 | ||
| cores_per_rank = cores // max(int(os.environ.get("LOCAL_WORLD_SIZE", "1")), 1) |
There was a problem hiding this comment.
os.cpu_count() count virtual cores, which is physical cores * (hyperthreadings/core). Dense usage should count physical cores. Can you use the following example? Thanks!
total_cores = psutil.cpu_count(logical=False)
The optimizer loop is dense, so hyperthreads do not add throughput. Count physical cores with psutil, fall back to os.cpu_count() when psutil cannot tell, and keep the CPU affinity mask as an upper bound. Use the DeepSpeed Team copyright line in the new test file. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
|
Thanks @delock, both done in 9ea4059:
|
torchrun sets
OMP_NUM_THREADS=1for every process when--nproc_per_nodeis larger than 1, unless the variable is already set.DeepSpeedCPUAdamthen steps on a single OpenMP thread, and nothing in the log says so. The deepspeed launcher sets the thread count only with--bind_cores_to_rank, which is the path the ZeRO-Offload tutorial documents. Users who launch with torchrun or accelerate get no hint.Measured effect
Setup: ZeRO-1 with
offload_optimizer: cpu, Qwen3-4B in bf16, fp32 gradient accumulation. The time is forengine.step()(CPU Adam plus the copies around it), median over the measured steps.OMP_NUM_THREADS=1torchrun --nproc_per_node 2, real defaultOn B300, each rank steps half of the optimizer state, so the single-thread cost is already split over two processes. In a 2-GPU torchrun job on L40S at the default, the same step took 6.2 s.
The B300 run with threads set also changed the attention path for packed micro-batches. That doesn't touch
engine.step(), so the optimizer column only reflects the thread count.Change
single_thread_warning()indeepspeed/ops/adam/cpu_adam.pyreturns a message whenOMP_NUM_THREADSis"1"and the rank has at least two cores (physical cores frompsutil, capped by the CPU affinity mask, divided byLOCAL_WORLD_SIZE).DeepSpeedCPUAdam.__init__logs it once per process, for the first optimizer instance.It only warns and does not change the thread count:
OMP_NUM_THREADS=1may be intentional;Tests
tests/unit/ops/adam/test_cpu_adam_threads.pyis CPU only and does not build the op. It checks these cases:Nonefrompsutilfalls back toos.cpu_count().yapf, flake8, codespell and the license check pass.