Repository navigation
Offload optimizer states - #9128
sfc-gh-jaelee wants to merge 4 commits into
Conversation
…tream has read it With overlap_comm, average_tensor reduces each bucket on the reduction stream. A parameter larger than reduce_bucket_size is reduced straight from its param.grad, which autograd allocated on the compute stream, and on the ranks that own none of it ZeRO-2 drops that gradient on the host right after queuing the reduction (reduce_ipg_grads -> clear_grad_attribute). The multi-rank reduction records the reduction stream only on the slices it copies back to the local rank, so the caching allocator can hand the gradient's memory to the compute stream before the reduction has read it. The reduction then sums whatever the next kernels wrote there: with a small reduce_bucket_size every step after the first was skipped as an overflow. The CPU (gloo) path is correct, since it synchronizes. average_tensor now records the reduction stream on its whole input first, as the other overlap_comm paths record the tensors they hand to that stream, so the allocator waits for the reduction. Tests: under overlap_comm, average_tensor records its device input on the reduction stream before the reduction. Signed-off-by: Jaeseong Lee <jaeseong.lee@snowflake.com>
…in pinned host memory outside the step
ZeRO-1/2 keeps its FP32 master partitions and the optimizer's state for them (Adam's moments) on every GPU, 12 bytes
per owned parameter, although only the optimizer step reads them. A long-sequence job needs that memory for
activations during the forward and backward; offload_optimizer frees it but moves the step itself to the CPU.
zero_optimization.optimizer_state_swap = {"fraction": f} keeps the step on the GPU and only moves its
state: the tensors live on the GPU from the end of the last backward before a step to the end of that step.
- At the accumulation boundary the end-of-backward epilogue starts copying them back on a side stream; step() waits
for those copies, runs unchanged, then starts the copies out and frees the GPU storage. Storage is resized to 0 and
back, so the _hp_mapping views into the masters stay valid; the copy stream is recorded on each tensor, so the
caching allocator reuses the memory only after the copy has read it. Host copies are pinned and reused.
- fraction in (0, 1]: only the first whole tensors (masters, then moments, in the optimizer's order) holding that
share of the bytes move, for jobs whose peak has room for the rest. The masters are evicted when the optimizer is
built, so the first forward already runs without them; the set is chosen again when the moments appear.
- Muon's momentum stays on the GPU: ZeRO folds each gradient into it in the end-of-backward epilogue and commits it
at the start of step(), before anything is fetched. Its groups' masters still move.
- Checkpoints do not bring the state back: DeepSpeedEngine.save_checkpoint / load_checkpoint, and the optimizer's
load_state_dict, refresh_fp32_params, update_lp_params and _restore_from_bit16_weights, run with each evicted
master's .data and each evicted state entry set to its pinned host copy, so a save writes the host copies and a
load writes into them. A state tensor a load replaces keeps its GPU tensor (the _hp_mapping views point into the
masters and moments) with the new values in its host copy, so old and new moments are never on the GPU together; a
load into an engine that has not stepped gets evicted tensors for its new state, one at a time, and the moved set
is chosen again. The masters are not relinked while their data is on the host; the optimizer-state fragments are
linked again at the next step. optimizer_state_resident() (nesting) holds the state on the GPU for code that needs
it there, and a universal checkpoint load uses it. state_dict() outside a hold raises instead of returning freed
storage; offload_states is refused.
Adam is elementwise and the step runs on the same values, so training is bit-identical to the same job without the
swap. The option needs ZeRO stage 1 or 2 with DeepSpeedZeroOptimizer; it is refused with offload_optimizer, ZenFlow,
and ZeRO-1's BF16_Optimizer path (BF16 parameters with FP32 gradient accumulation).
Tests: the step evicts the state and frees its storage, and the next backward epilogue fetches it back unchanged
(CPU, with a fake ZeRO optimizer); through deepspeed.initialize on one rank, a checkpoint saves and restores the
swapped state (fraction 0.5), training on bit-identically to the engine that never stopped.
Signed-off-by: Jaeseong Lee <jaeseong.lee@snowflake.com>
…by param group
With expert data-parallel 1 a GPU can own so many parameters that its FP32 masters, moments and the step's FP32
gradients do not fit at once (9.4B parameters per H200: 113 GB of master and moments, 150 GB with the gradient), so
the whole-state swap cannot bring the state back for the step.
optimizer_state_swap = {"fraction": f, "stream": true, "group_gib": g} streams the step instead:
- the optimizer's param groups are split into consecutive slices of at most group_gib (default 2) GiB of FP32 master
before ZeRO flattens them, so each slice is flattened, partitioned and stepped on its own; every slice keeps its
group's other keys (lr, weight decay, MoE names);
- _optimizer_step evicts the previous slice (whose master the step has already copied into its BF16 partition),
fetches this slice, starts fetching the next one and waits for this one; copies in and out run on two side
streams, so both PCIe directions are used at once; the fraction keeps the slices past that share resident;
- before each fetch the host waits until the copies out of the slice two back have finished. Nothing else in the
step blocks the host, so without this bound it queued every slice's fetch ahead of the GPU while the blocks of the
slices still being copied out could not be reused yet: the caching allocator grew to the whole GPU, a cudaMalloc
failed, and the retry freed and unmapped its cache, which the next forward and backward mapped again (seconds per
step). With the bound, three slices are on the GPU at once and each fetch reuses the blocks of the slice two back.
- The slices are sized and cut over trainable parameters (as ZeRO decides, is_optimized_parameter); a frozen one goes
with the slice being built, so every slice has something to step.
- The split records each slice's source group. A client LR scheduler built on the unsplit groups is remapped when it
is passed to deepspeed.initialize: its per-group lists (base_lrs, _last_lr, LambdaLR's lr_lambdas, CyclicLR's
max_lrs and momenta, ReduceLROnPlateau's min_lrs; SequentialLR and ChainedScheduler through their children) get one
entry per slice, its group's. Other scheduler classes are refused with an error that says to pass the scheduler as
a callable or in the DeepSpeed config, so it is built on the slices.
- Each group's optimizer-state fragments are linked once it has state, while that state is on the device:
_lazy_init_hp_params_optimizer_state used to link all groups right after group 0's first step, so the later
groups' fragments stayed empty and safe_get_full_optimizer_state raised outside group 0 (with any optimizer that has
more than one param group, too).
- Checkpoints serve the evicted slices from their host copies, as the whole-state swap does.
Adam is elementwise, so the update equals that of the same split with the state resident (bit-identical in the
tests). group_gib requires stream; changing stream or group_gib changes the param-group count, so a checkpoint does
not load across that change.
Tests: the streamed step holds at most the stepping group, the prefetched next one and the kept ones, and matches the
resident step (CPU, with a fake ZeRO optimizer); through deepspeed.initialize on one rank, every slice's optimizer
state fragments are linked: safe_get_full_optimizer_state and safe_get_full_fp32_param equal the job without the swap.
Signed-off-by: Jaeseong Lee <jaeseong.lee@snowflake.com>
…FP32 zero gradients initialize_optimizer_states gives every param group an FP32 zero gradient at once and drops them again unless the step runs on the CPU; only Adagrad builds its state from them (Adam creates its state at the first step). Next to the FP32 masters that doubles the FP32 footprint while the ZeRO optimizer is built, before the optimizer-state swap can evict anything: the streamed step is for state that does not fit, and GLM-5.3 (BF16) at expert parallel 64 ran out of memory there (135.5 GiB on an H200) although it trains with the streamed step. With optimizer_state_swap.stream the optimizer is built without them, except for Adagrad and CPU offload, where they are used. The whole-state swap keeps DeepSpeed's initialization. Tests: which configurations skip the zero gradients (the streamed step with Adam only). Signed-off-by: Jaeseong Lee <jaeseong.lee@snowflake.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61c11b39d0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| for tensor in tensors: | ||
| host = self.host.get(id(tensor)) | ||
| if host is None or host.numel() != tensor.numel(): | ||
| host = torch.empty(tensor.shape, dtype=tensor.dtype, pin_memory=self._streams.pin_memory) |
There was a problem hiding this comment.
Allocate swap buffers through the active accelerator
When optimizer_state_swap is enabled on XPU or HPU, this raw torch.empty(..., pin_memory=True) bypasses the device-specific pinning path. Both XPU_Accelerator._torch_empty_pinned and HPU_Accelerator._torch_empty_pinned explicitly allocate and then pin because the allocation API cannot express the required device, so the first eviction can fail or produce pageable buffers and lose the asynchronous-copy behavior; allocate every host copy through get_accelerator().pin_empty_like (or an equivalent accelerator method).
AGENTS.md reference: AGENTS.md:L40-L40
Useful? React with 👍 / 👎.
| if not (group_no in evicted_masters and torch.is_tensor(value) and value.is_floating_point() | ||
| and value.numel() > 1): | ||
| continue |
There was a problem hiding this comment.
Restore capturable step tensors to the accelerator
When a legacy ZeRO checkpoint is loaded for an evicted group using torch Adam/AdamW with capturable=True or fused=True, begin_host_hold temporarily makes the master a CPU tensor, so Optimizer.load_state_dict places the scalar step on CPU. This condition only rehomes floating tensors with more than one element, leaving step on CPU after the master is restored to the accelerator; place_restored_optimizer_step documents that these groups require step on the parameter device, and the next optimizer step then fails its device assertion.
Useful? React with 👍 / 👎.
| raise ValueError("zero_optimization.optimizer_state_swap does not support ZenFlow") | ||
| # The streamed step fetches and steps one param group at a time, so ZeRO has to flatten each slice on | ||
| # its own: split the groups before anything below reads them. | ||
| if optimizer_state_swap.stream and not init_optimizer.state: |
There was a problem hiding this comment.
Split Adagrad groups before streaming
With stream=true and torch.optim.Adagrad, this condition is always false because Adagrad eagerly populates optimizer.state in its constructor. initialize_optimizer_states later rebuilds Adagrad on the flattened masters anyway, but the original groups are never sliced, so group_gib is silently ignored and a conventional single large group must be fetched in full during the step, causing the same OOM that streamed mode is intended to avoid.
Useful? React with 👍 / 👎.
| self.groups = len(optimizer.single_partition_of_fp32_groups) | ||
| self.resident = set(range(self.groups)) | ||
| sizes = [t.numel() for t in optimizer.single_partition_of_fp32_groups] | ||
| budget, total, self.kept = fraction * sum(sizes), 0, set() |
There was a problem hiding this comment.
Base streamed fractions on all movable state bytes
When fraction < 1 and groups have different state footprints, such as mixed Muon/aux-Adam groups or Adam groups with different amsgrad settings, this budget counts only master elements and fixes kept before moments appear. A state-light early group can therefore exhaust the master-only budget while a later state-heavy group remains resident, making the actual host share far below the configured fraction and potentially exceeding the expected GPU peak; selection needs to be recomputed from the bytes in each group's movable state once that state exists.
Useful? React with 👍 / 👎.
| "stream": false, | ||
| "group_gib": 2 |
There was a problem hiding this comment.
Make the documented swap configuration valid
A user copying this example gets a validation error because modes_valid rejects any explicit group_gib unless stream is true, while the example sets stream to false and group_gib to 2. Either omit group_gib from this non-streaming example or enable streaming so the documented configuration can be loaded.
Useful? React with 👍 / 👎.
No description provided.