Skip to content

Offload optimizer states - #9128

Closed
sfc-gh-jaelee wants to merge 4 commits into
deepspeedai:masterfrom
sfc-gh-jaelee:jaelee/v3/zero-memory
Closed

sfc-gh-jaelee wants to merge 4 commits into
deepspeedai:masterfrom
sfc-gh-jaelee:jaelee/v3/zero-memory

Conversation

@sfc-gh-jaelee

Copy link
Copy Markdown

No description provided.

…tream has read it

With overlap_comm, average_tensor reduces each bucket on the reduction stream. A parameter larger than
reduce_bucket_size is reduced straight from its param.grad, which autograd allocated on the compute stream, and on
the ranks that own none of it ZeRO-2 drops that gradient on the host right after queuing the reduction
(reduce_ipg_grads -> clear_grad_attribute). The multi-rank reduction records the reduction stream only on the slices
it copies back to the local rank, so the caching allocator can hand the gradient's memory to the compute stream
before the reduction has read it. The reduction then sums whatever the next kernels wrote there: with a small
reduce_bucket_size every step after the first was skipped as an overflow. The CPU (gloo) path is correct, since it
synchronizes.

average_tensor now records the reduction stream on its whole input first, as the other overlap_comm paths record the
tensors they hand to that stream, so the allocator waits for the reduction.

Tests: under overlap_comm, average_tensor records its device input on the reduction stream before the reduction.

Signed-off-by: Jaeseong Lee <jaeseong.lee@snowflake.com>
…in pinned host memory outside the step

ZeRO-1/2 keeps its FP32 master partitions and the optimizer's state for them (Adam's moments) on every GPU, 12 bytes
per owned parameter, although only the optimizer step reads them. A long-sequence job needs that memory for
activations during the forward and backward; offload_optimizer frees it but moves the step itself to the CPU.

zero_optimization.optimizer_state_swap = {"fraction": f} keeps the step on the GPU and only moves its
state: the tensors live on the GPU from the end of the last backward before a step to the end of that step.

- At the accumulation boundary the end-of-backward epilogue starts copying them back on a side stream; step() waits
  for those copies, runs unchanged, then starts the copies out and frees the GPU storage. Storage is resized to 0 and
  back, so the _hp_mapping views into the masters stay valid; the copy stream is recorded on each tensor, so the
  caching allocator reuses the memory only after the copy has read it. Host copies are pinned and reused.
- fraction in (0, 1]: only the first whole tensors (masters, then moments, in the optimizer's order) holding that
  share of the bytes move, for jobs whose peak has room for the rest. The masters are evicted when the optimizer is
  built, so the first forward already runs without them; the set is chosen again when the moments appear.
- Muon's momentum stays on the GPU: ZeRO folds each gradient into it in the end-of-backward epilogue and commits it
  at the start of step(), before anything is fetched. Its groups' masters still move.
- Checkpoints do not bring the state back: DeepSpeedEngine.save_checkpoint / load_checkpoint, and the optimizer's
  load_state_dict, refresh_fp32_params, update_lp_params and _restore_from_bit16_weights, run with each evicted
  master's .data and each evicted state entry set to its pinned host copy, so a save writes the host copies and a
  load writes into them. A state tensor a load replaces keeps its GPU tensor (the _hp_mapping views point into the
  masters and moments) with the new values in its host copy, so old and new moments are never on the GPU together; a
  load into an engine that has not stepped gets evicted tensors for its new state, one at a time, and the moved set
  is chosen again. The masters are not relinked while their data is on the host; the optimizer-state fragments are
  linked again at the next step. optimizer_state_resident() (nesting) holds the state on the GPU for code that needs
  it there, and a universal checkpoint load uses it. state_dict() outside a hold raises instead of returning freed
  storage; offload_states is refused.

Adam is elementwise and the step runs on the same values, so training is bit-identical to the same job without the
swap. The option needs ZeRO stage 1 or 2 with DeepSpeedZeroOptimizer; it is refused with offload_optimizer, ZenFlow,
and ZeRO-1's BF16_Optimizer path (BF16 parameters with FP32 gradient accumulation).

Tests: the step evicts the state and frees its storage, and the next backward epilogue fetches it back unchanged
(CPU, with a fake ZeRO optimizer); through deepspeed.initialize on one rank, a checkpoint saves and restores the
swapped state (fraction 0.5), training on bit-identically to the engine that never stopped.

Signed-off-by: Jaeseong Lee <jaeseong.lee@snowflake.com>
…by param group

With expert data-parallel 1 a GPU can own so many parameters that its FP32 masters, moments and the step's FP32
gradients do not fit at once (9.4B parameters per H200: 113 GB of master and moments, 150 GB with the gradient), so
the whole-state swap cannot bring the state back for the step.

optimizer_state_swap = {"fraction": f, "stream": true, "group_gib": g} streams the step instead:

- the optimizer's param groups are split into consecutive slices of at most group_gib (default 2) GiB of FP32 master
  before ZeRO flattens them, so each slice is flattened, partitioned and stepped on its own; every slice keeps its
  group's other keys (lr, weight decay, MoE names);
- _optimizer_step evicts the previous slice (whose master the step has already copied into its BF16 partition),
  fetches this slice, starts fetching the next one and waits for this one; copies in and out run on two side
  streams, so both PCIe directions are used at once; the fraction keeps the slices past that share resident;
- before each fetch the host waits until the copies out of the slice two back have finished. Nothing else in the
  step blocks the host, so without this bound it queued every slice's fetch ahead of the GPU while the blocks of the
  slices still being copied out could not be reused yet: the caching allocator grew to the whole GPU, a cudaMalloc
  failed, and the retry freed and unmapped its cache, which the next forward and backward mapped again (seconds per
  step). With the bound, three slices are on the GPU at once and each fetch reuses the blocks of the slice two back.

- The slices are sized and cut over trainable parameters (as ZeRO decides, is_optimized_parameter); a frozen one goes
  with the slice being built, so every slice has something to step.
- The split records each slice's source group. A client LR scheduler built on the unsplit groups is remapped when it
  is passed to deepspeed.initialize: its per-group lists (base_lrs, _last_lr, LambdaLR's lr_lambdas, CyclicLR's
  max_lrs and momenta, ReduceLROnPlateau's min_lrs; SequentialLR and ChainedScheduler through their children) get one
  entry per slice, its group's. Other scheduler classes are refused with an error that says to pass the scheduler as
  a callable or in the DeepSpeed config, so it is built on the slices.
- Each group's optimizer-state fragments are linked once it has state, while that state is on the device:
  _lazy_init_hp_params_optimizer_state used to link all groups right after group 0's first step, so the later
  groups' fragments stayed empty and safe_get_full_optimizer_state raised outside group 0 (with any optimizer that has
  more than one param group, too).
- Checkpoints serve the evicted slices from their host copies, as the whole-state swap does.

Adam is elementwise, so the update equals that of the same split with the state resident (bit-identical in the
tests). group_gib requires stream; changing stream or group_gib changes the param-group count, so a checkpoint does
not load across that change.

Tests: the streamed step holds at most the stepping group, the prefetched next one and the kept ones, and matches the
resident step (CPU, with a fake ZeRO optimizer); through deepspeed.initialize on one rank, every slice's optimizer
state fragments are linked: safe_get_full_optimizer_state and safe_get_full_fp32_param equal the job without the swap.

Signed-off-by: Jaeseong Lee <jaeseong.lee@snowflake.com>
…FP32 zero gradients

initialize_optimizer_states gives every param group an FP32 zero gradient at once and drops them again unless the step
runs on the CPU; only Adagrad builds its state from them (Adam creates its state at the first step). Next to the FP32
masters that doubles the FP32 footprint while the ZeRO optimizer is built, before the optimizer-state swap can evict
anything: the streamed step is for state that does not fit, and GLM-5.3 (BF16) at expert parallel 64 ran out of
memory there (135.5 GiB on an H200) although it trains with the streamed step.

With optimizer_state_swap.stream the optimizer is built without them, except for Adagrad and CPU offload, where they
are used. The whole-state swap keeps DeepSpeed's initialization.

Tests: which configurations skip the zero gradients (the streamed step with Adam only).

Signed-off-by: Jaeseong Lee <jaeseong.lee@snowflake.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T18:15:30.175347Z 61c11b3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61c11b39d0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

for tensor in tensors:
host = self.host.get(id(tensor))
if host is None or host.numel() != tensor.numel():
host = torch.empty(tensor.shape, dtype=tensor.dtype, pin_memory=self._streams.pin_memory)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Allocate swap buffers through the active accelerator

When optimizer_state_swap is enabled on XPU or HPU, this raw torch.empty(..., pin_memory=True) bypasses the device-specific pinning path. Both XPU_Accelerator._torch_empty_pinned and HPU_Accelerator._torch_empty_pinned explicitly allocate and then pin because the allocation API cannot express the required device, so the first eviction can fail or produce pageable buffers and lose the asynchronous-copy behavior; allocate every host copy through get_accelerator().pin_empty_like (or an equivalent accelerator method).

AGENTS.md reference: AGENTS.md:L40-L40

Useful? React with 👍 / 👎.

Comment on lines +313 to +315
if not (group_no in evicted_masters and torch.is_tensor(value) and value.is_floating_point()
and value.numel() > 1):
continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore capturable step tensors to the accelerator

When a legacy ZeRO checkpoint is loaded for an evicted group using torch Adam/AdamW with capturable=True or fused=True, begin_host_hold temporarily makes the master a CPU tensor, so Optimizer.load_state_dict places the scalar step on CPU. This condition only rehomes floating tensors with more than one element, leaving step on CPU after the master is restored to the accelerator; place_restored_optimizer_step documents that these groups require step on the parameter device, and the next optimizer step then fails its device assertion.

Useful? React with 👍 / 👎.

raise ValueError("zero_optimization.optimizer_state_swap does not support ZenFlow")
# The streamed step fetches and steps one param group at a time, so ZeRO has to flatten each slice on
# its own: split the groups before anything below reads them.
if optimizer_state_swap.stream and not init_optimizer.state:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Split Adagrad groups before streaming

With stream=true and torch.optim.Adagrad, this condition is always false because Adagrad eagerly populates optimizer.state in its constructor. initialize_optimizer_states later rebuilds Adagrad on the flattened masters anyway, but the original groups are never sliced, so group_gib is silently ignored and a conventional single large group must be fetched in full during the step, causing the same OOM that streamed mode is intended to avoid.

Useful? React with 👍 / 👎.

self.groups = len(optimizer.single_partition_of_fp32_groups)
self.resident = set(range(self.groups))
sizes = [t.numel() for t in optimizer.single_partition_of_fp32_groups]
budget, total, self.kept = fraction * sum(sizes), 0, set()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Base streamed fractions on all movable state bytes

When fraction < 1 and groups have different state footprints, such as mixed Muon/aux-Adam groups or Adam groups with different amsgrad settings, this budget counts only master elements and fixes kept before moments appear. A state-light early group can therefore exhaust the master-only budget while a later state-heavy group remains resident, making the actual host share far below the configured fraction and potentially exceeding the expected GPU peak; selection needs to be recomputed from the bytes in each group's movable state once that state exists.

Useful? React with 👍 / 👎.

Comment on lines +729 to +730
"stream": false,
"group_gib": 2

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make the documented swap configuration valid

A user copying this example gets a validation error because modes_valid rejects any explicit group_gib unless stream is true, while the example sets stream to false and group_gib to 2. Either omit group_gib from this non-streaming example or enable streaming so the documented configuration can be loaded.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant