Repository navigation
[checkpoint] Fused copies, symmetric-memory window and staging release for prepared affine transfers - #9118
Draft
jinyouzhi wants to merge 5 commits into
Draft
[checkpoint] Fused copies, symmetric-memory window and staging release for prepared affine transfers#9118jinyouzhi wants to merge 5 commits into
jinyouzhi wants to merge 5 commits into
Conversation
A weight sync moves the same batch every step, and every execute_transfers call repeats validation, the alias check, scheduling, the digest agreement, arena allocation and region views before the first byte moves. prepare_transfers does that work once and returns a PreparedTransfer whose run only moves data. The handle keeps the arenas that deepspeedai#9027's schedule reports, and for each round one list of (destination, source) views: the packs before a send and the unpacks after a receive, plus one list of local copies. execute_transfers is now prepare_transfers followed by one run, with the same collectives as before. Python's cyclic collector is paused while the views are built: tens of thousands stay alive together, and the collections their allocation triggered made a one-shot call at 8 processes on CPU about a third slower than master. A buffer that moves after prepare, as reload_states moves offloaded ZeRO-3 partitions, would leave those views writing into the old storage. So run checks by default: each process compares its buffers (the held tensors, or requests rebuilt from the live state) with what it prepared, and one small all-gather refuses on every process, before data moves, if a buffer moved on any of them or the processes are not running the same handle for the same time. The shape of prepare_transfers and the buffer layout check follow #8 (a8c34e3); this is its staged half, without the window transport. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de> Co-authored-by: Jin, Youzhi <youzhi.jin@intel.com>
…they run Two handles prepared for the same batch over different buffers had the same schedule digest, so run's check passed when each process ran a different one, and the peers exchanged payloads of different batches. Each process now counts its prepare_transfers calls and adds the count to the prepare all-gather. A handle's identity is a hash of the digest and the gathered counts, the same on every process, and the check compares that instead of the digest. The scheme is jinyouzhi's, from the review. execute_transfers no longer goes through a handle. It runs the same preflight and agreement, then builds each round's views when the move reaches that round, as it did before prepare_transfers, so building them overlaps the previous round's transfer. On L20 GPUs, building them all first made a one-shot call 10-16 ms slower than master. The docs and the handle's docstring now say that run() checks only the tensors the handle holds: a view such as zero_param_storage_shard returns keeps the old storage when reload_states re-points the partition, so such handles need run(requests) rebuilt from the live state. They also say that a handle keeps its buffers allocated until it is dropped. The prepared-run test no longer reads the handle's private rounds, and the garbage-collector test refuses through a too-small budget instead of patching a private helper. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
prepare_transfers takes a copier that turns each round's pack/unpack list and the local copy list into one callable. sequential_copies (default) keeps one copy_ per region; fused_copies builds a per-alignment tile table at prepare time and moves the list with one Triton launch per class on CUDA/XPU, falling back to copy_ elsewhere. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
symm_mem_rendezvous allocates a buffer every member of a group can address and exchanges its handles; symm_mem_peer_tensor maps one peer's buffer into this process. On backends whose peer views carry no device index (XPU), the view is rebuilt from the mapped address. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
…runs TransferWindow is a symmetric-memory pool every process of the group can address. An operation between two processes whose source or target lies in a window is copied in place by one of them (the target pulls or the source pushes) with no message, arena or budget; the rest of the batch keeps the staged protocol. Ranks agree on which shards lie in a window as part of the schedule digest, and a run with window copies ends with a group barrier. Window copies are (destination, source) pairs like a round's packs and unpacks, so the same copier builds them: fused_copies covers copies whose source or destination lies in a peer's mapped window. PreparedTransfer.release_staging() frees the arenas between runs; the next run allocates them again, and a process that cannot refuses on every process when run checks. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #9094 (RFC #9026, step after "prepare once, run per step"). Until #9094 merges, only the last three commits belong to this PR:
[checkpoint] Fused region copies for prepared affine transfers[comm] Expose symmetric-memory rendezvous and peer tensors[checkpoint] Copy window shards in place and release staging between runsAs agreed with @0z5a on #9026, the fused copy and the window transport share one PR and one copy-table format.
copy_stays the default everywhere.What changes
Fused copies (opt-in
copier=)prepare_transfers(..., copier=fused_copies)replaces each round's pack, unpack and local-copy lists with a single Triton kernel launch (copy_tiles). The kernel reads a table of addresses, strides and extents for each region.sequential_copies(onecopy_per view) is the default and keeps current behaviour.fused_copiesfalls back tocopy_for views it cannot tile, for host tensors, and when Triton is unavailable._agree. If building it fails, every process refuses the batch.Symmetric-memory window (opt-in
window=)deepspeed.commgainssymm_mem_rendezvousandsymm_mem_peer_tensor, sotorch.distributedstays behinddeepspeed/comm.TransferWindoware copied directly into or out of the peer's window,pullorpush, and skip the staged rounds.TransferStats.mapped_bytesreports how many bytes went through the window.execute_transfer,execute_transfersandprepare_transfers.Staging lifecycle
PreparedTransfer.release_staging()frees the staging arenas, round views and copier tables between runs, for example during training steps.run()re-stages them. Withcheck=True, a failure while re-staging refuses on every process.Why this is not a duplicate
#9094 adds the prepared handle with
copy_only. This PR is the follow-up proposed in #9026 §5: the fused copy, the window transport andrelease_staging(). No other open PR touchesaffine_transfer_executor.pybeyond #9094.Tests
DS_ACCELERATOR=cpu)New tests:
copy_for every region shape (strided, transposed, misaligned, 0-d, empty);Performance
Workload: Qwen3-4B, 398 bf16 tensors. 2-D weights move from a row split to a column split over N processes, and 1-D weights move between shifted even splits. Each cell is the median of 5 warm
PreparedTransfer.run()calls in ms (lower is better);copy_vsfused_copies.8× Arc Pro B70 (cards 8–15, xccl):
copy_4× L20:
copy_prepare_transferswithfused_copiescosts an extra 50–75 ms once, to build the tables and compile the kernel. This is whycopy_stays the default.run()median at 64 MiB staging:release_staging()frees all staging buffers between runs. Without it, memory use is the same as [checkpoint] Prepare an affine transfer once and run it per step #9094.Compatibility
torch.distributed._symmetric_memory.AI assistance was used for this change.