Skip to content

[checkpoint] Fused copies, symmetric-memory window and staging release for prepared affine transfers - #9118

Draft
jinyouzhi wants to merge 5 commits into
deepspeedai:masterfrom
jinyouzhi:affine-fused-window
Draft

jinyouzhi wants to merge 5 commits into
deepspeedai:masterfrom
jinyouzhi:affine-fused-window

Conversation

@jinyouzhi

Copy link
Copy Markdown
Contributor

Stacked on #9094 (RFC #9026, step after "prepare once, run per step"). Until #9094 merges, only the last three commits belong to this PR:

  • [checkpoint] Fused region copies for prepared affine transfers
  • [comm] Expose symmetric-memory rendezvous and peer tensors
  • [checkpoint] Copy window shards in place and release staging between runs

As agreed with @0z5a on #9026, the fused copy and the window transport share one PR and one copy-table format. copy_ stays the default everywhere.

What changes

Fused copies (opt-in copier=)

  • prepare_transfers(..., copier=fused_copies) replaces each round's pack, unpack and local-copy lists with a single Triton kernel launch (copy_tiles). The kernel reads a table of addresses, strides and extents for each region.
  • sequential_copies (one copy_ per view) is the default and keeps current behaviour.
  • fused_copies falls back to copy_ for views it cannot tile, for host tensors, and when Triton is unavailable.
  • The kernel uses only generic Triton ops. The same code runs on CUDA and on XPU (Triton XPU backend). Triton is imported lazily, so CPU workers never touch a device driver.
  • The copier is built inside _agree. If building it fails, every process refuses the batch.

Symmetric-memory window (opt-in window=)

  • deepspeed.comm gains symm_mem_rendezvous and symm_mem_peer_tensor, so torch.distributed stays behind deepspeed/comm.
  • Tensors placed in a TransferWindow are copied directly into or out of the peer's window, pull or push, and skip the staged rounds.
  • Window copies use the same copier as the staged lists.
  • TransferStats.mapped_bytes reports how many bytes went through the window.
  • The window layout is part of the agreed fingerprint, so mismatched placements refuse the batch on every process.
  • Available on execute_transfer, execute_transfers and prepare_transfers.

Staging lifecycle

  • PreparedTransfer.release_staging() frees the staging arenas, round views and copier tables between runs, for example during training steps.
  • The next run() re-stages them. With check=True, a failure while re-staging refuses on every process.

Why this is not a duplicate

#9094 adds the prepared handle with copy_ only. This PR is the follow-up proposed in #9026 §5: the fused copy, the window transport and release_staging(). No other open PR touches affine_transfer_executor.py beyond #9094.

Tests

pytest tests/unit/checkpoint/test_affine_transfer_executor.py tests/unit/checkpoint/test_affine_transfer.py
Platform Result
CPU, 4 processes, gloo (DS_ACCELERATOR=cpu) 77 passed, 11 skipped (window and Triton tests need a GPU)
4× NVIDIA L20, torch 2.14.1+cu130 88 passed
4× Intel Arc Pro B70, xe driver (kernel 6.19.10), torch 2.14.0.dev20260804+xpu, triton 3.8.0 (xpu), xccl 88 passed

New tests:

  • every copy list goes through the copier;
  • fused copies match copy_ for every region shape (strided, transposed, misaligned, 0-d, empty);
  • window transfers for each placement × copier, prepared and one-shot;
  • schedules leave window copies out of the rounds;
  • a released handle stages again on its next run (2 and 4 processes).

Performance

Workload: Qwen3-4B, 398 bf16 tensors. 2-D weights move from a row split to a column split over N processes, and 1-D weights move between shifted even splits. Each cell is the median of 5 warm PreparedTransfer.run() calls in ms (lower is better); copy_ vs fused_copies.

8× Arc Pro B70 (cards 8–15, xccl):

N staging copy_ fused Δ
2 8 MiB 162.6 171.6 +5.5%
2 64 MiB 123.7 124.2 +0.5%
4 8 MiB 164.9 165.0 0.0%
4 64 MiB 137.6 136.7 −0.7%
8 8 MiB 103.2 99.6 −3.6%
8 64 MiB 109.0 104.7 −4.0%

4× L20:

N staging copy_ fused Δ
2 64 MiB 103.5 102.1 −1.4%
4 8 MiB 84.4 81.5 −3.4%
4 64 MiB 86.1 80.4 −6.6%
  • The fused copy helps more as N grows, because each round has more regions. At N=2 the runs are dominated by communication, and on B70 the 8 MiB case is slightly slower.
  • prepare_transfers with fused_copies costs an extra 50–75 ms once, to build the tables and compile the kernel. This is why copy_ stays the default.
  • Window transport, run() median at 64 MiB staging:
Platform N staged window pull window push
L20 4 83.6 83.8 79.2
B70 2 123.7 90.5 98.3

Compatibility

  • Only the affine checkpoint transfer path changes, and every new feature is opt-in. ZeRO stages, offload and the engine are untouched.
  • Tested with bf16 only. Multi-node runs and the window across nodes were not tested. The window needs torch.distributed._symmetric_memory.

AI assistance was used for this change.

0z5a and others added 2 commits November 16, 2026 11:23
A weight sync moves the same batch every step, and every
execute_transfers call repeats validation, the alias check, scheduling,
the digest agreement, arena allocation and region views before the
first byte moves.

prepare_transfers does that work once and returns a PreparedTransfer
whose run only moves data. The handle keeps the arenas that deepspeedai#9027's
schedule reports, and for each round one list of (destination, source)
views: the packs before a send and the unpacks after a receive, plus
one list of local copies. execute_transfers is now prepare_transfers
followed by one run, with the same collectives as before. Python's
cyclic collector is paused while the views are built: tens of
thousands stay alive together, and the collections their allocation
triggered made a one-shot call at 8 processes on CPU about a third
slower than master.

A buffer that moves after prepare, as reload_states moves offloaded
ZeRO-3 partitions, would leave those views writing into the old storage. So
run checks by default: each process compares its buffers (the held
tensors, or requests rebuilt from the live state) with what it
prepared, and one small all-gather refuses on every process, before
data moves, if a buffer moved on any of them or the processes are not
running the same handle for the same time.

The shape of prepare_transfers and the buffer layout check follow
#8 (a8c34e3); this is its staged half, without the
window transport.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-authored-by: Jin, Youzhi <youzhi.jin@intel.com>
…they run

Two handles prepared for the same batch over different buffers had the
same schedule digest, so run's check passed when each process ran a
different one, and the peers exchanged payloads of different batches.
Each process now counts its prepare_transfers calls and adds the count
to the prepare all-gather. A handle's identity is a hash of the digest
and the gathered counts, the same on every process, and the check
compares that instead of the digest. The scheme is jinyouzhi's, from
the review.

execute_transfers no longer goes through a handle. It runs the same
preflight and agreement, then builds each round's views when the move
reaches that round, as it did before prepare_transfers, so building
them overlaps the previous round's transfer. On L20 GPUs, building them
all first made a one-shot call 10-16 ms slower than master.

The docs and the handle's docstring now say that run() checks only the
tensors the handle holds: a view such as zero_param_storage_shard
returns keeps the old storage when reload_states re-points the
partition, so such handles need run(requests) rebuilt from the live
state. They also say that a handle keeps its buffers allocated until it
is dropped.

The prepared-run test no longer reads the handle's private rounds, and
the garbage-collector test refuses through a too-small budget instead
of patching a private helper.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
prepare_transfers takes a copier that turns each round's pack/unpack list
and the local copy list into one callable. sequential_copies (default)
keeps one copy_ per region; fused_copies builds a per-alignment tile
table at prepare time and moves the list with one Triton launch per
class on CUDA/XPU, falling back to copy_ elsewhere.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
symm_mem_rendezvous allocates a buffer every member of a group can address
and exchanges its handles; symm_mem_peer_tensor maps one peer's buffer into
this process. On backends whose peer views carry no device index (XPU),
the view is rebuilt from the mapped address.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…runs

TransferWindow is a symmetric-memory pool every process of the group can address. An operation between two processes whose source or target lies in a window is copied in place by one of them (the target pulls or the source pushes) with no message, arena or budget; the rest of the batch keeps the staged protocol. Ranks agree on which shards lie in a window as part of the schedule digest, and a run with window copies ends with a group barrier.

Window copies are (destination, source) pairs like a round's packs and unpacks, so the same copier builds them: fused_copies covers copies whose source or destination lies in a peer's mapped window.

PreparedTransfer.release_staging() frees the arenas between runs; the next run allocates them again, and a process that cannot refuses on every process when run checks.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants