Repository navigation
Description
Refs #8252, #8935, #8936, #9027. Builds on #8885 (ZeRO storage maps), #8519 (AutoTP layer maps), #8623 (plan_transfer), #8809 (executor, merged), #8966 / #8967 (calibration and plan analysis).
Summary
When weights move from trainers on one node to rollout engines on another, the cheapest plan is often not a direct one. Two examples:
- Sending every byte across the network once, into the target node, and then spreading it over NVLink beats every direct plan.
- On 2 × 4 GH200 over Slingshot, the pipelined two-stage plan moves 8 GiB to four engines in about 93 ms. Direct plans take 374–502 ms, and relaying through a single GPU takes 366 ms.
This RFC proposes four things:
- Express the relay as two ordinary
plan_transferplans through an intermediate relay map. No new IR concept is needed, and exactness follows from the existing argument, stage by stage. - Choose between direct and relay plans with a calibrated cost model:
t ≈ max over resources r (bytes through r / capacity of r) + rounds · α_round + segments · α_seg.- The capacities come from
calibrate_links/fit_links([AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) #8966). - The per-resource bytes come from
analyze_transfers([AffineIR] Analyze affine transfer batches before execution (Plan analysis & calibration Part-1) #8967). - One-time setup is priced separately, as a prepare cost spread over the syncs a plan serves.
- The capacities come from
- State the memory, coalescing and delta options the executor needs to make relay practical.
- Keep fabric differences out of the planner (§6):
- the plan says what moves;
- the cost model chooses among plans;
- each transport declares what it can do.
The model was tested against measurements on four machines (GH200 + Slingshot, B300 + NVSwitch, B70 + PCIe, H100 + InfiniBand). It predicted the pipelined relay at about 93 ms before it was run, and the run measured 92.7 / 94.3 ms.
1. Relay as two plans
Let M_s be the source maps (for example the trainers' #8885 storage maps) and M_t the engines' target maps.
The relay map. M_r is any map onto relay GPUs on the target node that has one holder per element.
- Rename (special case). When the source already has one holder per element and the relay locations line up, as with ZeRO-3 partitions and one relay per trainer rank,
M_risM_swith each source rankirenamed torelay(i): the same pieces, the same shard shapes, new locations. - Split (general case). Otherwise a rename either multiplies the traffic or is refused, so
M_rsplits the parameter along axis 0 instead. This isrelay_map(source_map, relay_locations, min_split_bytes=...)in @0z5a's design (§5).
@0z5a's CPU check: one 4,096-element parameter, four relays, four engines that each hold it whole. Both stages were run by applying the plans' segments directly, and every accepted case was exact.
| Source | Relay map | Stage 1 bytes / model | Largest relay shard / model | Sent from |
|---|---|---|---|---|
| ZeRO-3 partition | rename | 1.00 | 0.25 | all four, evenly |
| ZeRO-1/2 replicas | rename | 4.00 | 1.00 | rank 0 only |
| ZeRO-1/2 replicas | split, default selector | 1.00 | 0.25 | rank 0 only |
| ZeRO-1/2 replicas | split, selector picks the matching rank | 1.00 | 0.25 | all four, evenly |
| 8 ZeRO-3 trainers, 4 relays | rename i % 4 |
refused (InvalidMapError) |
||
| 8 ZeRO-3 trainers, 4 relays | split | 1.00 | 0.25 | all eight, evenly |
The two stages:
- Stage 1 =
plan_transfer(M_r, M_s, replica_selector=...). With a one-holderM_rspread evenly over the relays, the model crosses the network exactly once, and with rail-aligned NICs each rail carries1/Wof it. Which source replica feeds each relay piece is the [checkpoint] Batch replica selection for affine transfer plans #8936 policy's choice. The table shows that this choice decides whether the load is spread. - Stage 2 =
plan_transfer(M_t, M_r). An intra-node plan from the relay buffers into the engines' target maps, over NVLink. When every engine needs the whole model, this is an all-gather.
Both are ordinary plans, cached per layout like any other. Nothing in the planner changes for the basic version.
Exactness. Exactness does not depend on the rename: any M_r that passes the map checks works. A rename preserves every condition a map must satisfy: well-formedness, the destination partition, coverage, homogeneity and exact scaling. A split relay map is checked like any other map.
- The correctness note's plan theorem ("any consistent plan is exact, for every replica choice, as long as the replicas agree") applies to each stage.
- Stage 1 therefore writes exactly
extract_r(F). Its replicas agree because they are copies of agreeing source replicas. - So stage 2 writes exactly
extract_t(F). This is the parallelism-invariance corollary applied toM_s → M_r → M_t.
Where each condition is checked:
| Condition | Checked by | When |
|---|---|---|
| Map conditions (tiling of every shard by address, full coverage, consistent holders, scale 1) | plan_transfer's metadata checks, in O(pieces²), never per element |
at plan time, once per layout |
| Every element written exactly once, for any stride | ParamAffineMap.validate_coverage() (O(numel) in Python) |
in tests only |
| The data itself: replicas agree, and the target matches the source | a runtime digest of the moved tensors, compared after stage 2 | after every sync |
No check on a map can catch a map that is self-consistent but describes the wrong memory. That is the job of the digest.
The digest also catches the transport. On @LiRunGuo's two H100 nodes (torch 2.13, NCCL 2.29.7, InfiniBand with GPUDirect), a bare all_to_all_single, with no DeepSpeed code, misplaced one 128 KiB block about once per 60 TB when calls of 0.4–1.7 GiB per peer were in the mix (comment; reported as NVIDIA/nccl#2608). Each wrong block held bytes from 512 KiB or 1 MiB away in the same message. The plans replayed on CPU were exact. So the digest stays on at run time, and a weight whose digest fails is sent again and checked again. Its cost is part of the price (§4).
2. Relay memory
A separate relay buffer costs model / W per relay GPU: 2 GiB per GPU for the 8 GiB example above, and about 17.5 GB per GPU for a 70B bf16 model over 8. Two ways to avoid it:
- Borrow idle memory (near term). The contract is that during a sync the engine lends one idle, contiguous region at a fixed address, at least
model / Win size, and does not read it until stage 2 has finished. The KV cache between rollout rounds is the natural candidate.- One instance, measured by @0z5a: a vLLM engine paused at sleep level 0 lent 44 of its 45 GiB KV allocation. After a prefix-cache reset and resume, greedy outputs were identical token for token, and pause, lend and resume took 0.09 s.
- At sleep level 1 the memory is released, and re-creating it took about 9 s on GH200. So only an engine paused at level 0 should lend.
- That test covers the lender only. In the same experiment the borrower was a trainer allocating from the lent region, and that was not safe.
- Step-0 gradient norms had relative errors of 1.0e-3 and 4.5e-3 in two runs, and 2.3% by step 1 in the first, against 1.4e-4 run-to-run noise.
- The cause is not known (a reuse race is one guess), and that integration was dropped.
- A relay only copies bytes and doesn't go through autograd, so it may not hit this. But writing relay data into a lent region has not been tested.
- Write stage 1 straight into the engines' weights (long term). Each block lands in one engine that needs it, and stage 2 becomes an in-place all-gather among engines.
- This needs no extra memory, but it needs
plan_transferto accept a target map whose pieces fill only part of each shard. - Today
_shards_tilerefuses such a target, by design.
- This needs no extra memory, but it needs
3. Cost model
t ≈ max over resources r ( bytes through r / capacity of r ) + rounds · α_round + segments · α_seg
Resources are everything a byte occupies on its way:
- each GPU's network send and receive;
- each NVLink link;
- any shared link, such as a PCIe root port or a cross-socket hop.
They are separate resources: on GH200, a GPU receiving over NVLink and over the network at once is bound by whichever is slower, not by their sum (see the mixed incast rows below).
Pipelined stages cost the slowest resource's stage 1 plus the last stage-2 round, not t1 + t2.
Collectives such as NCCL's ring all-gather get their own calibrated cost per participant count, because ring efficiency varies with the GPU count (below).
Prepare and run. The formula is the cost of one run. A plan also has a one-time prepare cost:
- validation;
- the schedule digest;
- staging allocation;
- the transport's own setup, such as communicators or registered windows.
Today #8809 repeats the first three on every call. On a prepared handle, the staging arenas stay resident by default, so their allocation stays in t_prepare. A staged-only transport may release them explicitly, and its next run then pays allocation and view rebuilding again, so the policy prices that run with them added (@jinyouzhi).
A plan lives until either side's placement, parallel sizes or ZeRO partitioning changes (spec §10.5). So for a plan that serves n syncs, the policy compares t_prepare / n + t_run. A transport with a cheaper run and a costlier prepare wins only once that cost is spread over enough syncs.
Both costs are visible in the measurements:
- On B300, a first call's setup was about 150–250 ms per ordered GPU pair (below).
- [checkpoint] Execute affine transfer plans with bounded peer staging #8809 also repeats its validation, schedule build, digest agreement and arena allocation on every warm call, so part of each warm call is prepare work as well (below and §5).
- @jinyouzhi's symmetric-memory window paid about 1 ms of control plane per call until the transfer was prepared once (Prepare window transfers once and run them per step jinyouzhi/DeepSpeed#8).
- Measured prepare and run (@jinyouzhi; fit in this comment). Qwen3-4B rebuild, row split to column split over N = 2 / 4 / 8 processes, staged prepared transfers (Prepare window transfers once and run them per step jinyouzhi/DeepSpeed#8):
- Prepare grows with total segments: about 7.8 µs per segment on 8× B70 and 11.9 µs on 8× L20 from N = 4 to 8.
- What prepare spends (breakdown), at N = 8 on B70 / L20:
build_operations42 / 76 ms andplan_digest15 / 24 ms per plan,_validate_buffers23 / 38 ms per request,_validate_aliases7.5 / 11.6 ms, and the agreement all-gather 1.5–2.5 / 0.6–1 ms. So the fixed part is host-side Python, not a collective, andt_prepareis a per-segment term plus a per-tensor term.- Garbage collection adds a full collection to about every other prepare (each creates about 18k operation objects).
- The 398 tensors have 8 distinct shapes, so a caller that shares plans across same-shape tensors removes most of the per-plan part.
- Run on L20, refit with each process's regions (received, sent and local: about 1.6k / 2.2k / 4.2k at N = 2 / 4 / 8; fit): bytes per process plus a per-region term fits all three N within 4%, at 20.4 GB/s + 4.7 µs per region with 8 MiB staging and 23.4 GB/s + 9.3 µs with 64 MiB. A fourth N pinned it down (@jinyouzhi): fitted on N = 2, 4 and 8 at 8 MiB, the same model predicts N = 3, 5, 6 and 7 within 5.5%, at 4.8 µs per region.
- At 64 MiB the larger constant is not a per-region cost: from N = 3 on, both budgets touch the same number of regions within 2%. The N = 8 / 64 MiB outlier is the host rebuilding about 2k views per round with only two rounds to hide it behind; caching the views brings it from 78 to 64 ms.
- The per-region term is host time, the
copy_launches in_run, and it overlaps the bytes rather than adding to them: the final wait is 59–85 ms of the run at N = 2, and the device has already finished when the host reaches it at N = 8. - Fused copies (a prototype Triton kernel that copies each round's pack, each round's unpack and the local copies as one table of tiles per launch, with the tables built at prepare): the run is 15–18 ms plus the bytes at about 23 GB/s per process, within 5.5% at both budgets, and 1.5% faster than one
copy_per region at N = 2 and 17–32% at N = 8. Building the tables takes 70–230 ms inprepare. With fused copies, the regions term leavest_runand joinst_prepare.
- Run on B70 stays at 121–138 ms (64 MiB) while each process's bytes fall from 2.0 to 0.9 GB, which points to a shared link rather than each process's own.
- Releasing the arenas, re-allocating them and running took 0.6–6.8 s on B70 at 8 × 64 MiB, against about 120 ms for a resident run.
- A whole training-state move (@jinyouzhi), Qwen3-0.6B, ZeRO-1 DP4 → ZeRO-3 DP4 on 4× L20, 1.2 GB of bf16 weights plus 7.2 GB of fp32 master weights and Adam moments: maps 0.06 + plans 0.25 + prepare 0.08 + first run 0.23 = 0.62 s in the running processes, and 0.12 s for a repeated run once prepared. Through a universal checkpoint the same move took 27.4 s, plus a 10.1 s relaunch.
Inputs:
-
fit_links([AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) #8966) gives the policy:- per-pair fits, one way and both ways at once;
- relay fits keyed by (receive path, send path), for a process that receives while it sends;
- fan completion rates keyed by peer set.
The policy derives per-resource capacities from them. Colocated resharding and the all-gather stage use the both-ways fits; the relay GPUs of a pipelined relay use the relay fit.
-
analyze_transfers([AffineIR] Analyze affine transfer batches before execution (Plan analysis & calibration Part-1) #8967) gives each process's sent and received bytes, per-path bytes, and rounds. -
The topology: each process's node, or [AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) #8966's link classes. A [checkpoint] Batch replica selection for affine transfer plans #8936
ReplicaRequestcarries processes and bytes but not where a process sits, and a policy can be any callable object, so the cost-aware policy takes the topology when it is built. The planner needs nothing new. -
Segments come from the plan.
Evidence
Fits use the medians from @0z5a's raw timings. Bold cases were not used to fit any parameter.
GH200, 2 nodes × 4 GPUs, Slingshot (one NIC per GPU), 8 GiB bf16 to 4 engines.
Capacities: 23.4 GB/s receive per rail, 52.1 GB/s per NVLink pair, 68.4 GB/s network send per GPU (fitted from the direct case).
| Case | Model (ms) | Measured (ms) |
|---|---|---|
| Direct: each engine pulls everything from one trainer GPU | 502 (fit) | 502 |
| Relay through one target GPU, 64 MiB pieces | 367 | 366 |
| Sharded direct: each trainer rank sends its shard to every engine | 367 | 374 |
| Two crossed flows (A0→B0, A0→B1, A1→B0) | 91.7 | 91.7 |
| Mixed incast, 50% NVLink + 50% network | 45.8 | 46.0 |
| Mixed incast, 80% NVLink + 20% network | 33.0 | 33.4 |
| Two-stage relay (sharded into node B, then all-gather), serial | 91.7 + 19.9 = 111.6 | 112.4 / 113.5 |
| Same, pipelined with 64 MiB buckets | ≈ 93 (predicted before the run) | 92.7 / 94.3 |
- Incast is flat: 1, 2 and 4 senders into one GPU reach 23.4 / 22.7 / 23.4 GB/s, so on this fabric the policy should not cap inflow.
- Rounds cost almost nothing on the network stage (1 vs 32 rounds: 93.4 vs 92.9 ms) and about 34 µs each on the NVLink all-gather.
- Across the 16, 64 and 256 MiB bucket sizes, in both orders, "stage 1 + the last all-gather round" is within 3.3% of every pipelined run.
H100, 2 nodes × 4 GPUs, InfiniBand (one NDR200 NIC per GPU), @LiRunGuo's 54 transfers of Qwen3-8B and 32B: syncs from a ZeRO-3 trainer on node 0 into replicas on node 1, and replicas growing onto node 0, shrinking back, or changing TP size, each as flat, relay + pipe, and relay + pipe + spread (fit).
- Capacities: 24.2 GB/s per NIC plus 20.4 ms, from the three 32B flat syncs (within 0.4%; 97% of NDR200's 25 GB/s), and 82 GB/s per GPU inside a node, from six 8B moves on node 1. The GPUs are H100 SXM, so this is low for NVLink, and which intra-node transport NCCL uses is still to be checked.
- Held out (33 transfers): network-only moves within −7% to +8% for 8B, and relay + pipe + spread within −1.3% to +4.0% at two replicas per node.
- Not priced yet:
- relay + pipe and relay + pipe + spread move the same bytes per rank, yet spread is 1.11–1.58× faster. The difference sits in the all-to-alls that carry
embed_tokensandlm_head(per-step data), so the pipelined term needs each rank's inter- and intra-node bytes per all-to-all. It then becomes a sum over the all-to-alls of the slowest rank's step, which reduces to stage 1 plus the last stage-2 round when the buckets are even; - the 32B reconfigures, 16–18% over the network term. All of it is in the first all-to-all, 156–166 ms longer than its bytes take at the later steps' rate. Building the new layout without a barrier before the move probably counts the ranks' allocation wait as move time; without that wait they sit 4–7% under the network term;
- a replica grown onto the trainer's GPUs, whose first all-to-all carries the same wait. These rows are to be rerun with a barrier, with per-rank local bytes and regions.
- relay + pipe and relay + pipe + spread move the same bytes per rank, yet spread is 1.11–1.58× faster. The difference sits in the all-to-alls that carry
- The choice: with two or more replicas per node, flat is 1.9–3.3× slower than the best relay, and the model picks a relay every time. Where plans move the same bytes the model ties; breaking ties toward spread, it picks the fastest plan or one within 1.3% of it in 13 of the 14 cases (the exception is +14.7%).
- The source matters as much as the plan: reading from both the old replica and the trainer's partitions, a replica grows onto the trainer's node 4.7–5.3× faster, because nothing crosses the network. That choice needs holders in two layouts (Describe a parameter held in several layouts as one affine map #9052, and [AffineIR] Plan flat shards that keep the parameter's own strides #9070 for column cuts).
B300, one node, 8 GPUs on NVSwitch.
- Capacities: 699 GB/s receive and 716 GB/s send per GPU, flat from 1 to 7 peers.
- NCCL ring all-gather of 8 GiB: 9.7 / 10.8 / 11.2 / 12.4 ms on 2 / 3 / 4 / 8 GPUs. That is 1.58× to 1.15× the bytes/capacity bound, so collectives are calibrated separately.
- NVLS was slower than ring at 2–4 GPUs, and NCCL's default chose ring at every size tried.
- The [checkpoint] Execute affine transfer plans with bounded peer staging #8809 executor is linear in rounds at 8 GPUs: 35.2 / 67.0 / 371.0 ms at 4 / 16 / 128 rounds, i.e. 2.71 ms per round with 64 segments. The warm Qwen3-4B calls below split that into a round cost and a chunk cost. At 2 GPUs the cost per round grows from 0.16 to 0.30 ms as the staging shrinks.
Segments dominate a real model. Rebuilding Qwen3-4B (398 bf16 tensors, 8.0 GB) on B300 with the #8809 executor (64 MiB staging) makes one plan per tensor. @0z5a timed a first call and a second call in the same process, with the same plan and communicators:
| GPUs | Segments | Rounds | First call (ms) | Second call (ms) |
|---|---|---|---|---|
| 2 | 1,592 | 78 | 471 / 394 | 66.9 / 66.7 |
| 4 | 6,368 | 32 | 1,939 | 132.8 |
| 8 | 25,472 | 16 | 10,476 | 611.4 |
(Two launches at 2 GPUs. Earlier one-shot first calls on other days took 449 ms at 2 GPUs and 14.5 s at 8.)
- Most of a first call is one-time setup: communicators, allocator growth and first-use paths, not broken down. The second call is 6–7× faster at 2 GPUs and 17× faster at 8.
- The setup grows roughly with the number of ordered GPU pairs, about 150–250 ms per pair. The per-launch values are 151, 164, 176 and 202 ms here, and 191 and 248 ms for the earlier one-shot calls.
- So it belongs in a prepare cost, paid once per plan and process, not in the cost of every sync.
- Even warm, 8 GPUs take 611 ms against about 10 ms of bytes/capacity. Much of that may be prepare work that [checkpoint] Execute affine transfer plans with bounded peer staging #8809 repeats on every call.
- @0z5a's CPU prototype (gloo, the same segment counts, Qwen3-4B shapes divided by 16) caches that work once. At 8 ranks, a warm call took 701 ms on that machine, 546 ms of it before the first byte moved. Caching the prepare work brought it to 144 ms, and caching the region views as well to 72 ms.
- That is a laptop CPU with no GPU launches, so the numbers don't transfer to B300 directly. The prediction for B300 at 8 GPUs is ≤ 350 ms with prepare cached. If it lands above 450 ms, prepare isn't the main term there, and coalescing goes first.
- So the order is: cache the prepare work first, then measure what coalescing still buys.
- Round and chunk costs. These fits combine the warm calls with the staging sweep above. A chunk is a segment, or a staging-sized piece of one.
- At 8 GPUs:
24.0 ms + 1.29 ms · rounds + 0.178 ms · chunks per processfits all four points (errors −0.9, +0.5, 0 and 0%). That's three parameters on four points, so one more staging size would be a useful check.- The 24 ms term is equivalent to the bytes at an effective ~310 GB/s, well below the 699 GB/s the links carry.
- With bytes/capacity at 699 GB/s (10.75 ms) instead, about 13 ms per call is unexplained.
- At 2 GPUs: about 0.06 ms per chunk per process, with the round cost bending as staging shrinks (above).
- Across GPU counts: one set of constants for all GPU counts misses some points by more than 2×. Before calibrating per GPU count, try @0z5a's split of the chunk term:
α_schedule · total chunksfor prepare, which grows with the GPU count;α_copy · regions this process touchesfor run (5,970 at 8 GPUs: 2,786 packs, 2,786 unpacks, 398 local copies).
- At 8 GPUs:
B70, one node, 8 GPUs on PCIe (from @jinyouzhi on #8966).
- Pair bandwidth ranges from 3.6 to 54 GB/s depending on sender and direction.
- At width 2, fan-in completion ranges from 12.6 to 55.2 GB/s depending on which senders feed the receiver.
- This is why resources include shared links and why
fit_linkskeys fans by peer set: each of the three cases discussed there is bound by a different resource (a sender's link, the receiver, the cross-socket link).
4. Delta transfer
Between GRPO steps, only about 1–2% of bf16 weight elements change. An element changes only when its fp32 master moves past half a bf16 ulp. A lossless sparse delta (indices and values) through the affine path sends just those.
Measured end to end (@0z5a):
- Setup: a trainer on one B300 GPU behind a 10 Gb/s NIC, and rollout on two GH200 nodes, each receiving over a 1 GbE link.
- Transport: plain TCP over ssh (an int32 index and a bf16 value per change), not [checkpoint] Execute affine transfer plans with bounded peer staging #8809 or NCCL.
- Over 7 GRPO steps of Qwen3-4B, 0.90–1.37% of elements changed per step, i.e. 0.43–0.66 GB per step to the two receivers combined.
- The delta sync took 3.4–6.3 s per step (median 4.1 s), not counting the per-sync digest check. A full 8 GB push would take about 70 s per receiver at that link rate.
- Digests over all 796 regions matched, at step 0 and at the end.
- Pricing each receiver's delta bytes against its link gave 2–3.5 s, below the measured time. The gap is the sender's copy, not the network.
- The sender copies each target's payload GPU → host → bytes, one target after the other, before any send starts.
- That takes about 0.53 s per 332 MB target, 23–25% of the sender's communication time.
- Without it, each receiver got 0.092–0.106 GB/s (median 0.098), its single-pair rate.
- The shared 10 Gb/s sender port was not the limit at two receivers (about 0.2 GB/s, ~16% of the port). Each receiver's 1 GbE link was.
- A single-receiver decomposition confirms the cap. The real sender and wire format reach 0.102 GB/s, the same as a raw ssh pipe (0.103). Two flows into one receiver total 0.104–0.111 GB/s.
- So the model prices this sync as delta bytes / 0.10 GB/s per 1 GbE receiver, plus a sender-side copy term.
- The copy term is about 0.6 GB/s per target, serial in the current code. A chunked double buffer would overlap it.
- Two runs are still to do: a direct fan run (both receivers at once) and a 1-vs-2-sender split test. Both need a GH200 node that is down, so there is no date yet. Until then, the fan factor of 1.0 is unmeasured.
- Over NVLink, delta does not pay: 1.19 s against 0.39 s for the full transfer.
- About 0.8 s of the 1.19 s isn't in any timer yet.
- The same run's break-even bandwidth, about 32 GB/s, is the better number. It is an overestimate if part of that 0.8 s is specific to delta.
- That run changed 1.6–2.2% of elements.
The policy prices delta and full the same way: encode/decode and copy time, plus bytes / capacity, plus the digest (below).
Correctness condition: both sides must start from the same previous version, bit for bit. So the digest has to run after every delta, and any mismatch falls back to a full sync.
- This matters in practice. In a dry run that dropped one index on one receiver, a digest checked only at the end passed, because a later delta happened to rewrite that element.
- A digest after every sync caught it at the first sync.
The digest needs its own term in the price. The sync times above exclude it.
- As coded (sha256, serial, after resume), one check took 29.3 s at step 0 and 14.7 s at the end. Checking every step would add about 15 s to a 3.4–6.3 s sync.
- That cost isn't inherent. A GPU-side region checksum should be far cheaper (@0z5a has a prototype tested on CPU).
- The check could also run during rollout, at the price of finding a bad delta one step later.
5. Planner and executor
(@0z5a)
- Planning: two
plan_transfercalls, cached per layout. The policy evaluates the model for direct, relay, sharded and sharded + gather, and keeps the cheapest. - Execution: [checkpoint] Execute affine transfer plans with bounded peer staging #8809's bounded peer staging runs both stages.
- Stage 2 starts on bucket k as soon as stage 1 has landed it.
- 64 MiB buckets were best on Slingshot. On NVSwitch, larger staging keeps paying (8 / 64 / 256 MiB: 371 / 67 / 35 ms).
- Coalescing: in the schedule, not in a map. Per-tensor plans keep the exactness argument per plan, and merging only changes how a round's regions are addressed.
The full design is in @0z5a's comment: #9026 (comment)
- prepared transfers, with schedule state on a handle;
relay_map;- coalescing and adjacency-based direct rounds;
- pipelining through a gate table computed at prepare.
The proposed order of PRs, each useful alone:
- staged prepare/run: [checkpoint] Prepare an affine transfer once and run it per step #9094 by @0z5a, on [checkpoint] Expose the affine transfer schedule as schedule_transfers #9027 (merged), with the structure of Prepare window transfers once and run them per step jinyouzhi/DeepSpeed#8;
relay_mapand the serial relay;- flat relay and adjacency-based direct rounds;
- pipelining;
- small tensors, and the receive side, as the numbers from step 1 direct.
Training-state moves (#9074, @LiRunGuo). The split proposed there for upstream, each piece after this RFC:
- ZeRO-3's rank-agreement checks and barriers over the partition group ([zero] Run ZeRO-3's rank-agreement checks and barriers over the partition group #9076);
- the training-state move on
plan_transfers+execute_transfers, with DP4 → DP2 → DP4 as its test. @jinyouzhi suggests it cover the wider matrix of DeepSpeedExamples#1021 (ZeRO stage, AutoTP and AutoEP size changes); - the relay as two
plan_transferscalls through a relay map (open question 1); - the pipelined execution, after [checkpoint] Expose the affine transfer schedule as schedule_transfers #9027.
The rectangle planner and NativeRollout stay out. A move across TP or EP sizes inside the same processes also needs each engine to hold its own parallel groups (#9081).
6. Layers, and where the fabric enters
Fabric differences touch three layers differently. Examples: GPUDirect or not, host staging, one-sided windows. (Prompted by @delock's question; the current-state details are from @0z5a.)
| Layer | Decides | Where it lives today | Sees the fabric? |
|---|---|---|---|
| Plan | which source region feeds each target region, from which replica; merging adjacent segments | plan_transfer (#8623): replica_selector, _coalesce; batch policy plan_transfers (#8936) |
No. Exactness lives here, so the fabric changes how fast a plan runs, never what it writes |
| Plan choice | direct vs relay vs sharded, bucket size, transport | proposed in this RFC. #8966 and #8967 measure and report only; #9027's schedule_transfers exposes the executor's rounds, directness and arena sizes per process, which can feed the round term without mirroring executor logic. Nothing chooses today |
Only through calibrated numbers |
| Transport | how bytes move | the #8809 executor | Yes |
The executor today (#8809):
- Schedule agreement: before anything moves, all ranks agree on a digest of the schedule. If any rank rejects it, nothing moves.
- Ordering: each round posts all receives before any sends, in one
batch_isend_irecv. - Messages: one packed message per peer per round.
- A round goes in place only when it has exactly one region and that region is contiguous; each side decides separately.
- Otherwise the round is packed through a per-peer staging arena bounded by
StagingBudget(default 8 MiB per peer, per direction). - Large regions are split into budget-sized rounds.
- Buffers: they must be on the current accelerator device when distributed is initialized (CPU otherwise), contiguous, and all of one dtype.
- Rank numbering: source and target ranks are mapped separately (
TransferEndpoints). Copies within one process skip the transport. - Limits: the call is synchronous (no CUDA graphs, no overlap with compute), and only
scale == 1is planned.
Transport capabilities. Fabric-specific code stays in the transport. It declares what it can do, detected by capability rather than by device name, so the planner never branches on the fabric. A declaration states three properties for a pair of processes (on the same host, or across hosts), not for a device (@jinyouzhi):
- What one transfer can address on each side: packed, one contiguous run per peer per message (which is why staging exists); regions, a list of (offset, length) or strided regions in registered memory, as with RDMA scatter/gather; or mapped, peer memory that this process can load from and store to directly.
- Who moves the bytes: two-sided, where both ends post and the receives are armed first; or one-sided, where one end moves the bytes and signals the other (pull, push or put). Completion is either stream-ordered signals or a group fence.
- Which memory and registration it needs: device or pinned host memory (host staging is the device → host case), and whether the buffers must lie in a registered window, which is collective, has fixed addresses and outlives the prepared handle.
In these terms, #8809's staged path is packed, two-sided, device memory, no registration; the window path of jinyouzhi#7 is mapped and one-sided within a host, registered; the put path of jinyouzhi#9 is packed on both sides and one-sided across hosts, registered. The executor picks each operation's path per pair from these properties, as jinyouzhi#9 does in _mapped_direction, and finds the capabilities by probing. NCCL M2N sits outside this per-operation scheme, as a whole-request path (open question 6).
Each declared path is calibrated like any other resource, and the cost model chooses among paths, prepare and run included (§3). Candidates:
- host staging, for accelerators without a device-to-device path;
- GPUDirect RDMA scatter/gather. The [checkpoint] Execute affine transfer plans with bounded peer staging #8809 docstring notes that it changes the per-peer message contract.
- @jinyouzhi's prototypes: symmetric-memory windows within a node and NCCL put/signal across hosts (Copy window shards in place with symmetric memory jinyouzhi/DeepSpeed#7, Put window transfers across hosts with NCCL one-sided puts jinyouzhi/DeepSpeed#9).
PXN is not the relay. NCCL may route an inter-node send through an NVLink peer nearer the NIC (PXN). That is NCCL's own choice, made with or without staging, per message and invisible to the planner. The relay in §1 is planned explicitly and priced by the model.
PXN matters for calibration: a "direct" network sample may already include a PXN hop. So calibration should record the NCCL PXN settings (NCCL_PXN_DISABLE, NCCL_P2P_PXN_LEVEL) and the transport lines from NCCL_DEBUG=INFO with each fit (@0z5a).
Related work: NCCL M2N
NCCL M2N (NVIDIA, arXiv:2610.07516; nccl_m2n in NVIDIA/nccl-extensions) reshards a tensor between two disjoint process groups, each described by a DTensor-style mesh with Shard/Replicate placements.
What it shares with this RFC. Its hierarchical schedule is the relay idea: each unique source contribution crosses the network once per destination NVLink domain, and leaders forward it to replicas over NVLink. Its speed-of-light bound, max(T_network, T_NVLink), has the same max-over-resources form as the cost model above.
The differences that matter for DeepSpeed:
- Layouts. M2N's local tiles are contiguous and equal-sized per rank. DeepSpeed's often are not: uneven head-aligned AutoTP splits, fused and interleaved QKV, replicated blocks inside a sharded tensor, ZeRO flat partitions. The affine maps describe all of these.
- Groups. M2N requires disjoint source and destination groups. Colocated resharding on the same GPUs is in scope here.
- Exactness. Here, the correctness argument's conditions are checked at plan time, and the data is digested after each sync. The M2N paper validates its output bitwise in its benchmarks.
- Choosing plans. M2N derives its schedule from the two layouts. Here, a calibrated model chooses among direct, relay and sharded plans per fabric, which matters on non-uniform fabrics such as the B70 PCIe tree above.
Where it helps. M2N is a native NCCL transport, and per-call and per-segment overhead is the main overhead this RFC measures. Using M2N, or its approach, as an execution backend for the affine plans it can describe is worth evaluating. So far (open question 6), it fits as a whole-request path rather than as a backend for a plan's segments.
Open questions
- Partial targets in
plan_transfer(for in-place relay): an explicit flag, or a separate entry point, so the default keeps refusing a target that does not tile its shards?- @LiRunGuo's alternative (comment): each relay shard is the band of its holder's own destination, so it is a view of the engine's weights and a whole shard, and no partial target is needed. Stage 2 then copies each band onto itself, which is skipped, but [checkpoint] Execute affine transfer plans with bounded peer staging #8809's
_validate_aliasesrefuses the stage because a relay rank's source lies inside its own target. The executor would have to allow that when no segment reads what another one writes. - [rollout] In-memory relayout between sharded layouts (two-stage relay, pipelined execution) and ZeRO-3 training-state move #9074 implements this formulation (
affine_plan.py). @LiRunGuo's proposal for the executor (comment): drop a local segment whose source and target are the same memory with the same strides, and check overlap only on what the remaining segments read and write. - Answered by @0z5a (comment): [checkpoint] Check aliases on the regions a transfer moves #9093 makes that change. The whole-buffer check stays the fast path; when buffers overlap, the executor checks the regions each process reads and writes, and drops a local copy whose source and target are the same memory at the same strides, counting its region as read.
relay_maptakes the bands as its definition, so no partial-target flag is needed, andmin_split_bytesonly decides which weights are relayed.
- @LiRunGuo's alternative (comment): each relay shard is the band of its holder's own destination, so it is a view of the engine's weights and a whole shard, and no partial target is needed. Stage 2 then copies each band onto itself, which is skipped, but [checkpoint] Execute affine transfer plans with bounded peer staging #8809's
- Coalescing across tensors: @0z5a proposes doing it in the schedule rather than in a map. On the receive side, a tensor's rows come from different senders, so what helps there: unpacking with cached views, or gathering into a sender-ordered buffer and then assembling?
- @jinyouzhi's answer on 8× L20 (comment): gather into the sender-ordered arena and assemble with one copy kernel per round, prepared once. Cached views change the run by less than 1% up to N = 6 at 8 MiB; the fused kernel is the fastest at every N and budget (§3). Open: a kernel that also covers XPU (the Triton XPU backend or a SYCL kernel), with
copy_kept for views it cannot describe as rows of contiguous columns. - Order, from @0z5a (comment): the fused copy comes after the prepared transfers, as its own PR, with
copy_as the default path. [checkpoint] Prepare an affine transfer once and run it per step #9094 keeps each round's pack and unpack copies in one list on the handle, so a per-round table can replace that list without changingprepareorrun.
- @jinyouzhi's answer on 8× L20 (comment): gather into the sender-ordered arena and assemble with one copy kernel per round, prepared once. Cached views change the run by less than 1% up to N = 6 at 8 MiB; the fused kernel is the fastest at every N and budget (§3). Open: a kernel that also covers XPU (the Triton XPU backend or a SYCL kernel), with
- Candidate plans: is enumerating direct / relay / sharded / sharded + gather enough at first, or should the policy search?
- Round and chunk costs: fitted on B300 at 2 and 8 GPUs, from the staging sweep plus warm calls. Does the prepare/run split (
α_schedule · total chunks+α_copy · regions touched) explain the dependence on GPU count? Prepared-transfer timings on B70 and L20, with a breakdown ofprepareand per-process region counts, now exist (above). On L20, a fourth N and fused copies pinned the run term (§3): about 5 µs of host time per region, overlapping the bytes, and nothing per region once the copies are fused. Still needed: rows on B300 and GH200. - Other fabrics: multi-node InfiniBand, measured by @LiRunGuo (comment): 2 nodes × 4 H100 with 4 × NDR200 per node, a ZeRO-3 trainer on node 0 syncing into rollout replicas on node 1, every sync digest-checked.
- A "flat" sync, where every replica pulls its own copy, takes 1.04–1.08× of its inter-node bytes over the node's four NICs at 25 GB/s, for Qwen3-8B and 32B at 1, 2 and 4 replicas. The network term alone prices it.
- With 2 and 4 replicas, relay + pipelining + spreading large weights over the ranks takes 1.11–1.30× of one model over the NICs, and the remainder grows with the replica count.
- His 54 transfers are fitted in §3. Open: per-step bytes for the stage-2 term, a rerun of the reconfigures and grows with a barrier before the move, and the intra-node topology and NCCL transport.
- NCCL M2N as a backend: for plans whose layouts M2N can describe, how does it compare with [checkpoint] Execute affine transfer plans with bounded peer staging #8809 on the same hardware, and can a plan's segments be handed to it?
- One data point (@0z5a, B300, 2→2 with Qwen3-4B shapes, all bit-exact, median of five warm calls), to TP2 (8.05 GB) / to two TP1 (16.09 GB):
nccl_m2nPACK, one reshard per tensor: 24.8 / 36.0 ms;- [checkpoint] Execute affine transfer plans with bounded peer staging #8809 at 128 MiB staging: 39.3 / 43.6 ms;
- [checkpoint] Execute affine transfer plans with bounded peer staging #8809 at the default 8 MiB: 66.6 / 101.5 ms;
- NCCL, one message per GPU pair: 6.0 / 11.6 ms.
- M2N's Group API and DIRECT mode hung at the first call in that setup.
- The [checkpoint] Execute affine transfer plans with bounded peer staging #8809 rows include the prepare work it repeats on every call, so this is to be repeated with prepared transfers before reading anything into the gap.
- With prepared transfers (@jinyouzhi, comment), on 8× L20 (PCIe, no NVLink), 4 → 4 over disjoint groups, every Qwen3-4B weight (8.04 GB), all bit-exact, medians of five warm runs: M2N PACK, one reshard per tensor, 259.1 ms; prepared [checkpoint] Execute affine transfer plans with bounded peer staging #8809 113.9 / 119.8 ms at 8 / 64 MiB, and 112.4 / 115.7 ms with fused copies. The gap is bandwidth, not per-call overhead: on the 778 MB embedding alone M2N reaches 31 GB/s. DIRECT and PIPE need GPU-initiated networking and did not run there, the Group API failed in
ncclDevCommCreate, and M2N's first call costs about 850 ms. - On handing it a plan's segments: not as segments. M2N takes a whole tensor per call, described by a mesh and Shard/Replicate placements with equal tiles. So it fits as a whole-request path: the policy chooses it when both maps can be written as M2N placements and the groups are disjoint, and calibrates it like any other path.
- On @LiRunGuo's InfiniBand setup (Qwen3-8B, 2 nodes): his relay plans were 3–13% faster than M2N's default PACK transport, and both were exact. M2N's beta PIPE transport returned wrong values in 3 of 4 layouts. That comparison uses his bucketed
all_to_all_single, not [checkpoint] Execute affine transfer plans with bounded peer staging #8809, and is to be repeated with prepared [checkpoint] Execute affine transfer plans with bounded peer staging #8809 transfers.
- One data point (@0z5a, B300, 2→2 with Qwen3-4B shapes, all bit-exact, median of five warm calls), to TP2 (8.05 GB) / to two TP1 (16.09 GB):
- The transport declaration: what is the smallest set of capabilities that covers host staging, RDMA scatter/gather and one-sided windows? For prepared state, @0z5a's answer is that schedule state lives on the prepared handle, and transport state such as windows hangs off it.
- For the capabilities, @jinyouzhi's answer (comment): three properties per pair of processes, now in §6: what one transfer addresses on each side, who moves the bytes, and which memory and registration it needs.
- For staging, @jinyouzhi's answer (comment):
runchecks the handle by default, on all three paths (staged, direct and window): the buffers a caller passes are views of the old storage, so even staged copies don't follow a moved buffer (correction). Prepare window transfers once and run them per step jinyouzhi/DeepSpeed#8 (a8c34e3) compares each buffer'sdata_ptr, size, dtype and device with whatpreparerecorded and combines the result with a one-element all-reduce, so if a buffer moved on any rank, for example afterreload_states(), every rank raises before data moves;- arenas stay resident on the handle, sized from what the schedule needs ([checkpoint] Expose the affine transfer schedule as schedule_transfers #9027 reports it per process) rather than budget × peers;
preparemay take caller-provided storage, such as the lent region of §2, under its fixed-address contract;- a rank-uniform
release_staging()frees them for staged-only transports, while window transports keep them pinned.
- One weight held in two layouts (from @LiRunGuo: a surviving TP shard and the trainer's ZeRO-3 partitions).
plan_transferrefuses it, since two source regions must be equal or disjoint. Cutting each layout's pieces at the other's boundaries, with the union of holders on each piece, makes it plannable without a planner change.- On CPU, over 300 random small cases (uneven TP sizes, any subset of surviving ranks, 1–6 ZeRO-3 partitions), every row-split case was accepted and exact (137 of 137).
- For a column split, a cut ZeRO-3 box that spans several rows is no longer a dense run of the flat partition, and the planner refuses it (84 of 163). One piece per row is accepted, but exceeds
max_pieces_per_mapfor large weights. - Agreed answer: the constructor is Describe a parameter held in several layouts as one affine map #9052, row case first. For the column case, @0z5a's planner change, [AffineIR] Plan flat shards that keep the parameter's own strides #9070, accepts a flat shard whose pieces sit at the parameter's own row-major strides, with one shift for the whole shard. With Describe a parameter held in several layouts as one affine map #9052 on top of [AffineIR] Plan flat shards that keep the parameter's own strides #9070, every random two-layout case on CPU plans exactly into both a row-split and a column-split target, including the 82 column cuts per target that Describe a parameter held in several layouts as one affine map #9052 alone refuses (comment). The correctness note gets the matching tiling lemma.
Who does what
| Part | Owner |
|---|---|
| Exactness argument, cost model (including prepare/run and the choice of transport), and the cost-aware policy on #8936 | @Achyuthan-S |
Two-stage planning (relay_map), executor staging, prepared transfers (#9094) and pipelining, the region alias check (#9093), coalescing in the schedule, schedule_transfers (#9027) |
@0z5a |
| Calibration and plan analysis (#8966, #8967) | @jinyouzhi |
| Transport prototypes as transport state on the prepared handle: symmetric-memory windows and NCCL put/signal (jinyouzhi#7, jinyouzhi#9); prepared-versus-staged numbers on B70 | @jinyouzhi |
InfiniBand measurements; the rollout's shard-to-shard path on plan_transfer; ZeRO-3 over a subgroup (#9076) and the training-state move (#9074); a layout-independent digest kernel (offered) |
@LiRunGuo |
Measurements: GH200 and B300 by @0z5a; B70 by @jinyouzhi; InfiniBand (H100, NDR200) by @LiRunGuo. Feedback welcome, especially from @delock.
Activity
0z5a commentedon Nov 7, 2026
Checked this against my raw timings and current master. The GH200 table, the B300 collectives, the rebuild table and the executor description all match. A few things to change, most important first.
§1: the relay map is a rename only when each element has one holder. On CPU, for one 4,096-element parameter with four relays and four engines each holding it whole, I ran both stages by applying the plans' segments directly:
i % 4InvalidMapError)All the accepted cases are exact. So "crosses the network exactly once" and "
model / Wper relay" need a source with one holder per element, or a relay map that isn't a rename. The exactness argument doesn't depend on the rename, since anyM_rthat passes the map checks works. I'd defineM_ras any map with one holder per element, with the rename as the special case, and let the #8936 policy choose the source replica.§2: the lending test covers the lender only. What I showed is that vLLM survives: KV overwritten, prefix cache reset, greedy output identical on resume. In the same experiment the borrower was a trainer allocating from that region, and it was not safe: step-0 gradient norms had relative errors of 1.0e-3 and 4.5e-3 in two runs, and 2.3% by step 1 in the first, against 1.4e-4 run-to-run noise. I didn't find the cause (a reuse race is my guess) and dropped that integration. A relay only copies bytes and doesn't go through autograd, so it may not hit this, but writing relay data into a lent region hasn't been tested.
§3: the 8-GPU fit needs a constant. I can only reproduce "within 1%" as
24.0 ms + 1.29 ms · rounds + 0.178 ms · chunks per process(errors −0.9, +0.5, 0, 0%). Without the constant, 1.3 and 0.18 give 11 ms against 35.2 at 256 MiB, and 44 against 67 at 64 MiB. Using bytes/capacity (10.75 ms) as the constant, as the §3 formula does, gives 21.7 and 54.6, so about 13 ms is unexplained. That's three parameters on four points, so one more staging size would be a useful check.§3: setup per ordered pair. The Nov 6 runs give 151, 164, 176 and 202 ms, and the earlier cold calls give 191 and 248. I'd write "about 150–250 ms".
§4: the fan and split runs aren't in progress. One of the GH200 machines is down, so neither has started, and I don't have a date. The single-receiver inference stands, but the fan factor of 1.0 is still unmeasured.
§4: the digest needs a term in the price. §4 requires a digest after every delta, but the sync times exclude it. As coded (sha256, serial, after resume) one check took 29.3 s at step 0 and 14.7 s at the end, so checking every step would add about 15 s against a 3.4–6.3 s sync. That cost isn't inherent. A GPU-side region checksum should be far cheaper (I have a prototype tested on CPU, not yet on GPU), and the check could also run during rollout, at the price of finding a bad delta one step late.
§4: over NVLink. About 0.8 s of the 1.19 s delta time isn't in any timer, and I haven't split it. The conclusion holds, but the same run's break-even bandwidth, about 32 GB/s, is the better number, and it's an overestimate if part of that 0.8 s is specific to delta. That run also changed 1.6–2.2% of elements, so "1–2%" fits better than "about 1%".
§6: #8936 is no longer a draft. #9027 now exposes the executor's schedule (
schedule_transfers: rounds, directness and arena sizes per process), which can feed the round term without mirroring executor logic.Open question 6 (M2N vs #8809). One data point on B300: 2→2 with Qwen3-4B tensor shapes, all bit-exact, two warm-ups then the median of five calls.
nccl_m2n, one reshard per tensor, PACKM2N's Group API and DIRECT mode hung at the first call in my setup (NCCL 2.30.7), and I don't know why, so per-tensor PACK is the only M2N row. M2N and #8809 at 128 MiB are both 3–7× above the one-message floor. The #8809 rows are warm calls, but
execute_transfersrepeats its validation and agreement on every call, so they include that work. I'll come back to this with §5.§5 is still on me.
0z5a commentedon Nov 7, 2026
§5 details, which the TODO asked for. Nothing here is built except a throwaway CPU prototype of step 1, and none of it has run on GPU.
1. Prepared transfers come first.
A warm
execute_transferscall still isn't a run. Before the first byte moves it repeats_validate_buffers,build_operationsand a sort over the whole batch's chunks rather than this rank's,plan_digest,_validate_aliases, arena allocation and the digest all-gather. So the second-call column in §3 mixes a term in total chunks with a term in the regions one process touches.I rebuilt the same layouts on CPU over gloo, with Qwen3-4B shapes divided by 16: the same segment counts as your table, but 31 MB and one round. A throwaway prototype on master caches the prepare work per batch, then the region views as well. Times are rank 0 over 11 warm calls, median (min), in ms. Every call passed the demo's full check.
This is a laptop CPU with other load on it, with no GPU launches and almost no payload, so the CPU and B300 columns can't be compared directly, and I can't give the split on B300. The prototype also skips the agreement, which a real version shouldn't do. If B300 behaves the same way, two things change:
α_schedule · total chunks(prepare) plusα_copy · regions this process touches(run) before calibrating per GPU count. That second count is a different axis: 5,970 at 8 GPUs (2,786 packs, 2,786 unpacks, 398 local copies).@jinyouzhi's
prepare_transfersin jinyouzhi#8 already has this shape: validation, agreement, arenas and views once,run()only moves data, andexecute_transfersis prepare plus one run. Its prepared staging column gains without windows too. I'd land the staged half on master first, separately from the windows in #7, and on top of #9027 so that_lanesstays the one schedule computation. @jinyouzhi, would you rather split that out, or should I? Two questions on top of that:runcheck the handle? If one process re-prepares and another doesn't, a run can hang or move bytes on a stale schedule. One small digest all-gather would catch that before anything moves. add allreduce test #8 deliberately removed host collectives per run, which makes sense for its small messages, while a sync happens once per step. So should the check be on or off by default?That answers the prepared-state half of open question 7: schedule state lives on the handle, and transport state such as windows hangs off it.
2. Relay planning.
relay_map(source_map, relay_locations, min_split_bytes=...)returns anM_rwith one holder per element. It renames when the source already has one holder per element and the locations line up, and otherwise splits on axis 0 (see the table in my earlier comment). Stage 1 isplan_transfer(M_r, M_s, replica_selector=...)and stage 2 isplan_transfer(M_t, M_r), or #8936'splan_transfersfor a batch. A serial relay is then two ordinary executor calls, with no executor change, so this can land early.Small tensors get one holder instead of being split. At 8 GPUs, the 145 1-D tensors of Qwen3-4B are 0.005% of the bytes, and one holder each takes the chunk count from 25,536 to 17,416 (schedule only).
3. Coalescing (open question 2): in the schedule, not in a map.
ParamAffineMapdescribes one logical parameter, and the exactness argument applies per plan, so I'd keep per-tensor plans and no batch-level map. Merging only changes how a round's regions are addressed, not what gets written. Whether regions can merge depends on the storage layout. For the 8-GPU rebuild at 64 MiB, one process, 112 messages each way:(The 2,793 includes 7 regions already sent in place.)
So a flat relay removes packing, but a flat target ordered by tensor doesn't help the receive side, because a tensor's rows come from different senders. For targets that need separate tensors, I see two options:
torch.cat(pieces, out=tensor): 398 calls at 8 GPUs instead of 3,184 region copies. From the torch source this is one batched kernel per call for contiguous inputs, but I haven't timed it on CUDA. It needs one extra model per GPU, so it only applies when lent memory allows. The transfer itself is your 64-segment row: 67 ms at 64 MiB, 35 at 256.If a relay can't be flat,
torch.cat(regions, out=arena)batches the packing, about 25 regions per message at 8 GPUs (on CPU, 400 regions: 0.09 ms against 1.33). Torch has no batched unpack.The executor change is small.
_round_is_directaccepts only a one-entry round today. It would accept entries adjacent in one buffer, and a direct round needn't be chunked by the staging budget, since it uses no staging. A follow-up branch of mine, not pushed yet, already sends every contiguous region in place, as one message of up to 1 GiB. On the one-buffer all-gather of 8 GiB, at 8 GPUs on B300, that took 21.3 ms in one round, against 371 / 67 / 35 ms at 8 / 64 / 256 MiB and 12.4 ms for NCCL's ring in the same run. At 2 GPUs it took 10.9 ms, against 158.7 / 23.9 / 16.1 ms and 11.7 ms for the ring.4. Pipelining.
Stage 2 round
kneeds stage 1 up to roundg(k). I'd compute that table at prepare from the two schedules and include it in the digest. With real Qwen3-4B shapes and four relays, equal budgets and request order giveg(k) = kover 32 rounds at 64 MiB. Unequal budgets don't, so it's computed rather than assumed.Execution:
g(k).Stage 1 never waits on stage 2, and stage 2 keeps the arm-then-send rule within each round. Gates only point from stage 1 to stage 2, so the graph is acyclic. That argument is host-level. Two NCCL communicators active on one GPU also need a consistent issue order across ranks unless they run on separate streams, which I'd verify in torch. The overlap bench ran
isend/irecvnext to an all-gather on a second group without hanging.This is the shape of my GH200 overlap bench, with one difference: stage 2 there was an NCCL all-gather, not executor rounds. The stage 1 bucket interval was 2.81–2.93 ms with and without the overlap, and the total was the slowest rail plus about 0.67 ms for the last round. Executor rounds in stage 2 are untested.
Open question 1 doesn't block this: a separate relay buffer needs no partial targets. In-place relay stays open.
5. Order and predictions.
Each PR is useful alone:
relay_mapand the serial relay.Predictions, none run yet:
The first row is today's warm time minus 80% of the fastest CPU before-first-byte time, rounded up. The second is about two thirds of the CPU saving. Both assume the B300 host spends at least that share of this laptop's time on the same work. That's the weak link: a much faster host would miss for that reason alone, and the decision rule would still apply.
For open question 6, I'd repeat the M2N table from my earlier comment with #8809 prepared, as promised there, before reading anything into the gap.
Achyuthan-S commentedon Nov 8, 2026
Thanks @0z5a, this is a careful review, and the CPU relay-map table makes §1 much sharper. I've updated the RFC body:
M_ris now any map with one holder per element, with the rename as a special case, plus your table, and the source replica chosen by the [checkpoint] Batch replica selection for affine transfer plans #8936 policy.On the cost model, I prefer your prepare/run split to per-GPU-count constants:
t_prepare ≈ α_schedule · total chunks + setup per ordered pairt_run ≈ bytes / cap + α_round · rounds + α_copy · regions this process touchesRounds and regions would be read from #9027's schedule. Once the staged prepare/run lands, warm runs at 2 / 4 / 8 GPUs plus a fourth staging size would fit both terms, and show whether the ~13 ms per call at 8 GPUs belongs to prepare or to run. I'll set that fit up as soon as those timings exist.
On checking the handle in
run, I'd make it on by default. A sync happens once per step, so one small all-gather is cheap next to it, and a stale schedule is exactly the kind of silent failure the rest of the path is built to refuse. Paths with many small messages could opt out.Your PR order works for me.
LiRunGuo commentedon Nov 8, 2026
Thanks @Achyuthan-S and @0z5a. Following up on a call with Achyuthan: some data for open questions 5 and 6, and a first step of moving our shard-to-shard path onto the affine IR.
Open question 5, multi-node InfiniBand.
Open question 6, NCCL M2N, on the same 8B transfers, 2 nodes:
all_to_all_single, not [checkpoint] Execute affine transfer plans with bounded peer staging #8809. I'll repeat it with [checkpoint] Execute affine transfer plans with bounded peer staging #8809 once prepared transfers land, as @0z5a suggests.Our path on
plan_transfer(in our fork):zero_param_storage_affine_map, and the training-state move reads the fp32 master and Adam moments throughzero_optimizer_storage_affine_maps/zero_optimizer_storage_shard.plan_transferplans them. Our preference order (same rank, same node, caller order, least loaded) is thereplica_selector. Execution is unchanged.plan_transfercalls through a relay map with one holder per element, as @0z5a defined it._validate_aliaseswould refuse that stage anyway, because a relay rank's source (its band) lies inside its own target. The executor would have to allow a source inside the target when no segment reads what another one writes.plan_transferdoesn't take today, which we plan with our own planner for those weights:relay_mapand the selector for [checkpoint] Batch replica selection for affine transfer plans #8936 when they land, then compare our all-to-all with [checkpoint] Execute affine transfer plans with bounded peer staging #8809 (prepared) on InfiniBand, as agreed.Digest (the runtime check in §1's table, and §4's check after every delta):
One caveat: with 32B across two nodes, our path still shows an intermittent single-weight mismatch: 4 of 142 checks, only in long mixed sequences of moves. I'm tracking it down, and the digest is what caught it.
Achyuthan-S commentedon Nov 10, 2026
Thanks @LiRunGuo, this is really helpful. The InfiniBand table answers open question 5, and it's great to see your path running on
plan_transferwith the same inter-node bytes as your own planner.The CSV. Yes please, attached here would be best, so the fit can be reproduced. If it has inter-node bytes per rank as well as per node, the model can take the per-NIC maximum, which is what should separate "relay + pipe" from "+ spread". I'll fit it and post predicted against measured here.
A first read of the table: in all six rows, "flat" takes 1.04–1.08× of its inter-node bytes over the node's four NICs at 25 GB/s, so the network term alone prices it. With 2 and 4 replicas, "relay + pipe + spread" takes 1.11–1.30× of one model over the NICs, and the remainder grows with the replica count. I'd expect that to be the second stage, which the intra-node bytes in the CSV will let the model price.
One weight in two layouts.
plan_transferrefuses this because any two source regions must be equal or disjoint. I tried a map-side fix on master: cut each layout's pieces at the other layout's boundaries, and give every cut piece the union of its holders. I ran 300 random cases of a surviving TP shard plus ZeRO-3 partitions on small 2-D weights (uneven TP sizes, any subset of surviving ranks, 1–6 partitions), planned each into a two-rank target, and applied the plans' segments directly on CPU:max_pieces_per_mapof 4,096.Each (process, layout) gets its own location, which
TransferEndpointsalready allows, so the planner needs no change for the row case. I haven't run these plans through #8809 yet. I'm happy to write this constructor. @0z5a, for the column case, would you rather haveplan_transferaccept a strided sub-box of a flat run, or keep one piece per row with a larger budget?The digest. It would be very welcome; it's the runtime check §1 and §4 depend on. One question: with several rollout replicas, is each replica's digest compared with the trainer's separately? That would also check that the replicas agree (P2). @0z5a has a GPU-side region checksum prototype tested on CPU, so it may be worth settling on one kernel together.
The 32B mismatch. If it helps narrow it down, replaying a failing sequence's plans on CPU with index tensors (each element holding its logical position) shows quickly whether a plan or the execution is wrong. Happy to help with that.
I've added the InfiniBand results, your in-place relay (open question 1), the M2N comparison (open question 6) and the two-layout case (new open question 8) to the RFC body, with credit.
0z5a commentedon Nov 10, 2026
@Achyuthan-S on the column case: I'd rather have
plan_transferaccept it, narrowly — a flat shard whose pieces sit at the parameter's own row-major strides, one shift for the whole shard, which is how a ZeRO partition stores a box cut out of its rows. I tried that on top of #9011 (bea2e09): about 50 lines inaffine_transfer.py.ParamAffineMap.validate().max_pieces_per_map; planning takes 0.4–5.8 s against about 1 ms; and into a column-split target the rows don't coalesce, so 8,192 segments (TP2) or 16,384 (TP4) and a 0.6–1.3 MB plan, against 16 or 32 segments. Into a row-split target both end at 8 segments.So please go ahead with the constructor as a separate PR, as you offered on #9045, with the row case first. I'll open the planner change on top of #9011 once that lands, so the column case works without a larger budget.
Achyuthan-S commentedon Nov 11, 2026
Thanks @0z5a, accepting the column case in the planner is cleaner than a bigger budget, and 600 exact shards through #8809, including at a 256 B staging budget, make the case well.
Agreed on the split. I'll open the constructor as a separate PR with the row case first, built on #9010's
affine_geometryso the box arithmetic stays in one place. Its cut pieces keep their strides and their shard's offset, so once your change lands on top of #9011, the column case should need nothing more from the constructor.I'll add the single-shift case to the correctness note: with one shift per flat shard, a piece's address is the row-major index plus that shift, which is injective, so disjoint logical boxes, the bounds and the volume sum give condition D directly.
LiRunGuo commentedon Nov 11, 2026
@delock, and @tohtana / @tjruwase as owners of
deepspeed/runtime/: could you say whether this piece is accepted as part of this RFC? It is open as draft #9045 so the design can be discussed on code, and it stays a draft until you decide.The piece. It is the first step of the bottom-up plan @Achyuthan-S and I agreed on. #8936's policy and other transports come later.
plan_transferplans each parameter once.execute_transfersbatch into a fixed-address arena, so CUDA graphs captured on it stay valid.deepspeed/runtime/rollout/affine_weight_sync.py, with tests and a docs section. No existing source file changes.Numbers. 4× RTX 4090 (PCIe), ZeRO-3, bf16, median of 5 syncs. The baseline gathers each parameter whole and copies the local slice, which is #9023's approach, here on bf16 rather than fp32.
@Achyuthan-S has reviewed it and run its tests on CPU merged with current master. If it is accepted, I'll mark it ready and follow with a tutorial and a DeepSpeedExamples example.
jinyouzhi commentedon Nov 11, 2026
Sorry @0z5a for the slow reply on this — that's on me.
On who splits the staged prepare/run: I think it belongs with you. The prepared handle is mostly schedule state (validation, agreement, arenas, views), it needs to sit on #9027 so _lanes stays the one schedule computation, and it overlaps your unpushed in-place/direct-round branch. Please feel free to lift whatever is useful from jinyouzhi#8 ( prepare_transfers , the staged-path tests); I'll read and wire it with more transport choices.
On my side I'll keep refining the transport prototypes (symmetric-memory windows, NCCL put/signal) so they hang off your handle as transport state, which also gives a concrete answer to open question 7, i.e. the smallest capability declaration that covers them. Once your handle lands I'll rebase #7/#9 onto it and post prepared vs. staged numbers on B70.
On your two questions:
• Handle check in run : agree with @Achyuthan-S, on by default. The small-message window path is the one that wants to opt out.
• Releasing arenas between runs: I'd keep them resident on the handle by default, and not release them implicitly after each run.
• Releasing doesn't free much on its own. Dropping the tensors returns the memory to the caching allocator, not to the engine. Making it usable elsewhere needs empty_cache and a fresh cudaMalloc on the next run, which moves allocation, one of the prepare costs, back into run.
• It breaks anything bound to the address. The cached views, CUDA graphs, and any registered window (symmetric memory, ncclCommWindowRegister ) all depend on it. So release is a transport capability: window transports must pin the arena, and the staged path may release it.
• Two opt-ins instead:
• prepare accepts caller-provided storage, e.g. the lent KV-cache region from §2, so the caller owns its lifetime under the "fixed address during a sync" contract;
• an explicit, rank-uniform handle.release_staging() , with re-allocation and view rebuild on the next run, for staged-only transports.
• Size the arenas from what the schedule needs (#9027 reports arena sizes per process), not from budget × peers. Long term, flat relay and direct rounds need no arena at all, so it becomes the fallback for non-contiguous layouts.
• NCCL M2N makes the same choice. Its staging is a library-owned, bounded pool, allocated with ncclMemAlloc and registered once per (comm, buffer) as a symmetric window. It stays resident for the runtime epoch and is freed only at ncclM2nFinalize . By default that's 2 GiB × 4 slots for PACK ( NCCL_RESHARD_PACK_BUFFSIZES ). The source notes that registration is collective, so runtime eviction is avoided to keep ranks in lockstep. ncclReshardWithWindow is its caller-provided-memory path.
Achyuthan-S commentedon Nov 11, 2026
Thanks @jinyouzhi, that settles both questions, and the M2N comparison is a useful reference point. I've recorded them in the RFC body (§3, the PR order in §5, open question 7, and the ownership table):
runchecks the handle by default, and the small-message window path can opt out.preparetaking caller-provided storage (the lent region of §2) and a rank-uniformrelease_staging()for staged-only transports. Window transports keep theirs pinned.For the cost model, this keeps staging allocation in
t_prepare. Afterrelease_staging(), the next run pays allocation and view rebuilding again, so the policy prices that run with them added. Your prepared-versus-staged numbers on B70 will be the first data on how much that is.jinyouzhi commentedon Nov 11, 2026
On column support in the planner: I agree it's worth having for these scenarios. Today a flat shard can only hold a dense run, either a plain line or a box stored as a line. This change would add a third form: a box kept at the parameter's own row-major strides, with one shift for the whole shard. That lets the two-layout case and other column splits plan without a larger budget.
@0z5a, about your ~50-line planner change: would you rather open it as a PR stacked on #9011, or push it straight into #9011? I think you have access. Either works for me. I will push #9010 to merge first, Thanks.
jinyouzhi commentedon Nov 11, 2026
Prepared vs. staged numbers on B70, as promised.
Setup: one host, 8× Arc Pro B70 (PCIe), torch 2.14.0.dev20260804+xpu, oneCCL built from source. Qwen3-4B rebuild: 398 bf16 tensors (8.05 GB), one plan per tensor. 2-D weights go from a row split to a column split over the same N processes; 1-D ones to an even split shifted by half a shard. That gives 1447 / 5063 / 18367 segments at 2 / 4 / 8, fewer than your 398·N² because the 1-D tensors don't fan out fully. Staging only, no window.
executeis master8a6b53fe.prepare/runis jinyouzhi#8c0ed827b, whoseexecutematches master within noise. Times are the max over ranks, median of 5 after 2 warm-ups. Every mode is checked bitwise against the source.empty_cache(), re-allocating, then running takes 0.6–6.8 s at 8×64 MiB across sessions (vs. ~120 ms resident). That supports resident-by-default and keepingrelease_staging()opt-in.executeat 8×64 MiB ranges 190–784 ms (it prepares and allocates its staging buffers on every call, so it inherits the allocation noise). Preparedrunstays at 119–163 ms.Same benchmark on 8× L20 (PCIe, torch 2.14.1+cu130, NCCL 2.30.7), jinyouzhi#8
8e20b87c:Re-allocation after a release (median). jinyouzhi#8
8e20b87cnow allocates each direction's staging arenas once and gives each peer a slice, instead of one allocation per peer;runis unchanged on both:¹ Noisy across sessions; interleaved A/B runs in the same session gave 2.5 / 4.0 s before and 3.0 / 2.4 s after, so no change at this size.
Script: https://gist.github.com/jinyouzhi/65a65d56f875ad2faef80f946e78c876. AI assistance (GitHub Copilot) was used to write the benchmark and this summary; I reviewed the numbers.
Achyuthan-S commentedon Nov 11, 2026
Thanks @jinyouzhi, this is the data the prepare/run split needed. I fitted it against the model in §3; every number below comes from your two tables.
Prepare. A cost per total segment fits the step from N=4 to N=8 with almost no intercept: about 7.8 µs per segment on B70 and 11.9 µs on L20, the same order as @0z5a's 12–18 µs per chunk on CPU. It doesn't fit N=2: there, prepare is about 16 ms (B70) and 19 ms (L20) above that line on both machines. So
t_preparehas a fixed part that a per-segment term misses. With only three values of N I can't tell whether that part is constant or depends on N. Would a breakdown ofprepareinto validation, schedule build, digest agreement and arena allocation be easy to log? That would separate the terms better than more N.Run on L20. Bytes over a per-process link plus a small round term fits N=2 and N=4 within 4%: about 18.9 GB/s per process, with rounds costing almost nothing (the 8 and 64 MiB runs differ by 1–4 ms). It predicts 46 ms at N=8 against 64–78 ms measured, so N=8 has another 17–32 ms. My guess is the per-region copy term, since each process touches more regions as N grows, but I'd want the per-process region counts before claiming it.
Run on B70. In this rebuild each process sends (N−1)/N² of the model: 2.0, 1.5 and 0.9 GB at N = 2, 4 and 8. So per-process bytes fall 2.3× while
runstays at 121–138 ms (64 MiB). That suggests each process's own link is not the bottleneck here. The total crossing processes grows from 4.0 to 7.0 GB, which would fit a shared link such as the cross-socket hop in your #8966 B70 data. Pricing it would need which pairs share that link, which the fan ratesfit_linkskeys by peer set are meant to capture.Release. 0.6–6.8 s to release, re-allocate and run at 8 × 64 MiB on B70, against about 120 ms resident, makes the case for resident-by-default and an opt-in
release_staging()very concretely.I'll add these to §3 as the first prepare/run measurements, and refit once there are per-process region counts (from #9027's
schedule_transfers) and a prepare breakdown.Achyuthan-S commentedon Nov 12, 2026
One more input for the prepared handle, from @jinyouzhi's ZeRO-3 runs with #8842 (DeepSpeedExamples#1021, comment 6733267750).
offload_states()/reload_states()move every ZeRO-3 weight partition to new storage:reload_statesassignslp_param_buffer.dataa new tensor and re-points each partition, and each aliased master, into it.A plan survives that, since it depends only on the layout (spec §10.5). But a handle that cached region views before the reload would keep copying into the old storage, and nothing would fail. The run check agreed above compares schedules across ranks, so it wouldn't see a buffer that moved on every rank.
One option:
runcompares each bound buffer'sdata_ptrand size with whatpreparerecorded, and folds a mismatch into the same all-gather, so every rank refuses together and the caller re-prepares. It needs nothing from the engine. @0z5a, would that fit the handle?jinyouzhi commentedon Nov 12, 2026
@Achyuthan-S thanks, your fits match ours. I profiled
prepare_transfersstage by stage (same Qwen3-4B rebuild, 398 tensors; per-rank wall times, median of 7, GC off — see the note below).Prepare breakdown, N=8 (ms)
_validate_buffers(per request)build_operations(per plan)plan_digest(per plan)_validate_aliasesall_gather+ syncAt N=2 the same four Python stages are 13–16 ms on B70 and 28–35 ms on L20, and the agreement is 0.2–0.3 ms. So the intercept/residual you saw is host-side Python, not a collective. Most of it scales with segments;
_validate_aliasesand the per-request part of_validate_buffersscale with tensor count and stay flat as N changes.Two things inflate the prepare numbers I posted earlier:
build_operations/plan_digestper plan object, and the benchmark builds a fresh plan per tensor. The 398 tensors have only 8 distinct shapes, so a caller that shares plans across same-shape tensors would cut ~57 ms (B70) / ~100 ms (L20) at N=8 to a few ms.Regions per rank (budget-independent; first two ranks shown):
runissues roughly one pack/unpack copy per region, so ~2.2k copies per rank at N=4 and ~4.2k at N=8. At typical launch costs that is tens of ms at N=8, consistent with your per-region term. Fusing the packs into one gather kernel per round would be the lever there.Stale buffers after
offload_states/reload_states. Agreed this is real, but only on two paths:A check that only one rank does would hang the others, so whatever we do has to be collective. For direct messages, the cheapest fix is to rebuild the view at run time; there are only a few, so this is cheap and needs no collective. Window copies depend on offsets every rank has published, so they need the
data_ptr/size check folded into an agreement round inrun(~1 ms per run at N=8 here). Otherwiserunshould refuse and require a re-prepare.1 remaining item
Achyuthan-S commentedon Nov 13, 2026
Thanks @jinyouzhi, the breakdown answers it: the fixed part of
t_prepareis host-side Python, not a collective. So the model becomes a per-segment term (build_operations,plan_digest) plus a per-tensor term (_validate_buffers,_validate_aliases), with the agreement a millisecond or two. The GC effect and plan reuse are worth recording too: with 8 distinct shapes among 398 tensors, a caller that shares plans removes most of the per-plan part.Run, refit with your region counts (L20). With each process sending (N−1)/N² of the model, bytes alone gave 47 ms at N=8 against 64–78 ms measured. Adding a per-region term fits all three N within 4% at both budgets:
Regions per process are your max over the ranks shown (recv + send + local: about 1.6k, 2.2k and 4.2k at N = 2, 4, 8). The per-region constant isn't pinned down: it is two parameters on three points per budget, and the constant doubles between budgets. But the term accounts for the N=8 excess that bytes alone missed, and fusing the packs into one gather kernel per round would shrink exactly that term. A run with fused packs, or a fourth N, would separate the two.
Stale buffers. a8c34e3 is the check I had in mind (data_ptr, size, dtype and device against prepare, one all-reduce, every rank refuses together), and your correction means all three paths need it. #9045 resolves its shards at every sync since 48a3575, so it follows a reload on its side too.
I'll add the prepare breakdown, the run refit and the handle check to §3 and open question 7 in the body.
jinyouzhi commentedon Nov 13, 2026
One more data point for the in-memory vs. checkpoint comparison, measured independently of #9074: LiRunGuo opened #9074 about an hour and a half before I ran this, and I only saw it afterwards. It agrees with #9074 and adds three things #9074 does not cover: both paths on one machine, a change across ZeRO stages, and a repeated run of an already prepared transfer.
Setup: Qwen3-0.6B, bf16, AdamW, 4× L20, ZeRO-1 DP4 → ZeRO-3 DP4,
OMP_NUM_THREADS=8, checkpoints on local NVMe. Each phase is the max over ranks; 3 runs agree within ~1.5 s and the table shows the median. The live path plans each tensor withplan_transfer, then uses the executor's prepare/run path from my B70 numbers above.ds_to_universal17.5 + load 5.3The live move carries 1.2 GB of bf16 weights plus 7.2 GB of fp32 master weights,
exp_avgandexp_avg_sq(1240 requests, 4969 segments). I checked it with per-state sums against the source, which agree within 3e-14 relative. That is weaker than #9074's bitwise digest, and I have not run a bitwise check on this path.What this adds:
Hardware for the numbers above: 4× NVIDIA L20, torch 2.14.1+cu130. AI assistance was used to write the benchmark script and draft this comment; I reviewed the numbers and the text.
LiRunGuo commentedon Nov 14, 2026
Thanks @Achyuthan-S.
The CSV. Both files are in this gist: https://gist.github.com/LiRunGuo/4036b5cfec93c1975e02dafac5d20435
two_node_h100_transfers.csv: the 54 measured transfers (Qwen3-8B and Qwen3-32B: syncs into TP4 ×1, TP2 ×2 and TP1 ×4 on node 1, and a TP4 replica growing onto node 0 and shrinking back, from the old replica or from both the replica and the trainer's partitions; flat, relay + pipe, relay + pipe + spread). Medians, min and max of 4–6 runs, stages, transfer count, inter-node bytes in total and into the busiest node, intra-node bytes.two_node_h100_per_rank_bytes.csv: for the same 54 plans, each rank's inter- and intra-node bytes sent and received (bf16), so the model can take the per-NIC maximum. These come from rebuilding each plan on CPU with the same planner; the planned inter-node totals equal the measured runs' in all 54 rows.Digest with several replicas. Per replica now. Each replica's digests are compared with the trainer's on their own: one all-reduce with one slot per replica, so it costs the same as the old sum over all rollout ranks, and a failure names the replica as well as the weight. That checks P2 directly. The planner comparison I mentioned on #9074 ran with it (Qwen3-8B on two nodes, 270 checks, none failed). Happy to settle on one kernel with @0z5a's region checksum.
Buffers that move after
offload_states()/reload_states(). As you noted above, #9045 resolves its shards at everysyncsince 48a3575 and has a test for it (offload, reload, a check that the partitions really moved, then a sync that must match exactly). With a prepared handle it would pass the rebuilt requests torun(requests)as in @jinyouzhi's a8c34e3, so thedata_ptrcheck covers it.The 32B mismatch. Thanks for the suggestion; I did exactly that. I rebuilt every plan of the failing sequence (190 moves, 29 distinct plans, Qwen3-32B's 707 weights, the real shapes on 8 ranks over two nodes) and replayed them on CPU with index tensors, executing each move as our executor does (local copies, then all sends packed, then all receives written, stage by stage, a reused replica reading and writing the same buffer). Every target element held its own index after every move, for all 707 weights, so the plans are right and the cause is in the execution (or the hardware). A self-test that corrupts one element of one transfer is caught, on the right rank.
I then reran the sequence on GPU with two more checks. The old replica was checked on its own right before every timed move, and each trainer partition's digest was compared with its value at load after every move. One of 48 moves failed again. In it, the old replica had been right just before the move and the trainer never changed, so the grow itself wrote the wrong values, into the new replica.
A longer rerun ran the same sequence three times and dumped the bad shard element by element on a failure. Two of 162 checks failed. Neither fits the pattern of the earlier failures (a row-parallel weight in the flat grow from both holders): one was a grow from the old replica alone with relay+pipe+spread, the other a plain sync from the trainer. The dump of the grow failure shows what went wrong. One transfer wrote the whole shard: 20 MiB, contiguous on both sides, from the old replica on the other node, inside one
all_to_all_single. It has two wrong blocks, 116 KiB and 128 KiB, on the same 128 KiB grid. Each holds exactly the bytes of another block of the same message, one from 512 KiB later and one from 1 MiB earlier. There are no zeros and nothing from outside the message, and the bytes are still wrong on a recheck. Both failures are in transfers that cross the two nodes. That pointed below the plan and our pack/unpack, at the network transport (torch 2.13, NCCL 2.29.7, IB with GPUDirect).So I ran
all_to_all_singlealone between two nodes: no DeepSpeed, no model, every word encoding its iteration, its source rank and its offset, buffers allocated and freed as our executor does, ten minutes per setting.NCCL_IB_QPS_PER_CONNECTION=4,NCCL_NCHANNELS_PER_NET_PEER=8), none in 40 TB with NCCL's defaults.So the mismatch is below DeepSpeed and below our executor:
torch.distributed.all_to_all_singleon torch 2.13.0 / NCCL 2.29.7 over IB with GPUDirect, on this cluster. Not settled: whether the two settings are needed (0 in 40 TB with the defaults is not significant against 3 in 84 TB), and whether it is tied to these machines or this NCCL version. What caught it is the digest after every move, which is one more reason to have that check at run time. The 8B runs passed every check (39 configurations, and 270 per-replica checks in the planner comparison). Next I'm separating the two settings, and I'll report it to NCCL with the standalone test.LiRunGuo commentedon Nov 14, 2026
A follow-up on the 32B mismatch, because my last comment left the two NCCL settings open. I ran the bare
all_to_all_singletest again at the same message sizes, 20 minutes per setting, on another pair of nodes:NCCL_NCHANNELS_PER_NET_PEER=8aloneNCCL_IB_QPS_PER_CONNECTION=4aloneAchyuthan-S commentedon Nov 14, 2026
Thanks @LiRunGuo, here is the fit of your 54 transfers to §3's model,
t ≈ c0 + max over resources (bytes / capacity). The resources are each rank's NIC (inter-node bytes sent or received) and each GPU's intra-node traffic, both taken per rank fromtwo_node_h100_per_rank_bytes.csv. The 12 shrinks that move nothing (0.7–5 ms) are left out.Fitted (9 rows)
Held out (33 rows), measured against the model
4-7:1>0-7:1from both sourcesWhat it says:
The choice, which is what the policy needs: in each of the 14 groups (model, operation, layout, source) with bytes to move, does the cheapest predicted plan match the fastest measured one?
One question on the hardware: 72–82 GB/s per GPU for an all-to-all inside a node is low for H100s on NVLink. Are the four GPUs of a node on NVLink or on PCIe? That decides whether the intra-node resource is per GPU or per pair.
I'll add the fit to §3 and open question 5 in the body.
Achyuthan-S commentedon Nov 14, 2026
I've updated the body with the last few days of this thread:
all_to_all_singlemisplacement on InfiniBand, so the digest stays on at run time and a weight that fails it is sent again._validate_aliasesproposal. @0z5a, both are open for you.LiRunGuo commentedon Nov 15, 2026
Thanks @Achyuthan-S, this is a useful fit. Before the questions, one thing it found in my measurements rather than in the model.
Per-step data. Every row also has one profiled move, which I hadn't put in the CSV: for each
all_to_all_single, the slowest rank's pack, exchange and unpack times and the largest bytes any rank sent and received. It is now in the gist astwo_node_h100_steps.csv(493 steps for the 54 rows). Two caveats. It is one move per row, with a device synchronize around each step, so the steps add up to more than the median (8B into TP1 × 4, relay + pipe: 0.525 s against 0.338 s). And the sent and received maxima can come from different ranks.Buckets are 64 weights in parameter order, by count rather than size, and all-to-all
kcarries stage 1 of bucketkand stage 2 of bucketk − 1.1. The last stage-2 round. The gap between relay + pipe and relay + pipe + spread sits in the steps that carry the two largest weights:
embed_tokens(bucket 0) andlm_head(the last bucket). For 8B,lm_headis 151936 × 4096 = 1.16 GiB; for 32B, 151936 × 5120 = 1.45 GiB.Without spread, one rank of the node fetches each of them whole and then sends it to the node's other ranks in stage 2.
Per step, 8B sync into TP1 × 4 (ms):
The middle steps are the same. The first two differ in the same way as the last two, through
embed_tokens: without spread, one NIC per node carries all of it in stage 1, and one GPU fans it out in stage 2.So the largest block alone would miss half of it. What the bound needs is each rank's bytes per step, inter- and intra-node. I can rebuild those from the plans on CPU, as I did for the per-rank CSV, with the largest block per step and the local bytes and regions you ask for in 3. Untested: ordering the buckets so that
lm_headis not last would hide its stage 2 behind the next bucket's stage 1.2. The 32B reconfigures (+16–18%). As far as the profile shows, it is not the one-peer pattern.
4-7:4>0-7:4from the old replica moves the same 61.03 GiB across nodes as the sync into4-7:4, one peer per NIC against four.I think this is how I measure. A reconfigure first builds the new layout on every rank, which allocates the new replica's arena, and then starts the move without a barrier. The move time I report is the slowest rank's. So a rank that finishes its allocation first waits inside the first all-to-all for the others, and that wait is counted as move time. Syncs have no layout step and show no such first step. I haven't verified it; the next run will time the move after a barrier and record when each rank enters it.
On the digest: that run recorded only pass or fail. The failures I dumped later on the same moves were all the misplaced 128 KiB block from my earlier comment, which reproduces with the bare collective, with no timing involved; it's now NVIDIA/nccl#2608. So I don't connect it to the time.
3. Growing onto the trainer's node. The profile points the same way, more strongly:
So most of the +53% to +179% is probably the same wait, not the local copies. I'll add per-rank local bytes and regions with the rebuild in 1, so that the per-region term can be fitted once the first step is cleaned up.
4. NVLink or PCIe. The GPUs are H100 SXM parts (94 GB), so NVLink by form factor. But I haven't checked
nvidia-smi topo -mor which transport NCCL picks for the intra-node connections; the NCCL log I have covers only the inter-node ones. In the six TP4 → TP1 × 4 rows on node 1, steps 3–7 run at 80–82 GB/s per GPU in each direction, matching your 82 GB/s. Steps 1–2 run at about 50 GB/s. 80 GB/s is about 27 GB/s per GPU pair, which is low for NVLink. So I'll check the topology and the transport, and measure an intra-node all-to-all by itself, before the intra-node term is read as the link's capacity.To sum up for §3: the network term and the spread rows stand. The 32B reconfigure and grow rows should be read with the first-step wait in mind until I rerun them, and the stage-2 term needs per-step bytes, which I'll add.
Achyuthan-S commentedon Nov 15, 2026
Thanks @LiRunGuo, the per-step file settles most of it. Your numbers match
two_node_h100_steps.csv: the 8B TP1 × 4 steps, the last step's 3.85 GiB sent against 1.51 GiB received without spread (1.14 GiB both ways with it), and the 230 ms against 75 ms first all-to-all.The first-step wait. For the three 32B reconfigures from the old replica, the first all-to-all is 156–166 ms longer than its bytes take at the later steps' rate. Without that, they sit 4–7% under the network term, the same range as the 8B syncs, so your explanation fits them. For the grows from both sources this file can't confirm it: it has one profiled move per row, and for relay + pipe that move's first-step excess (161 ms) is larger than the whole gap between the median and the model (101 ms). So those need the rerun with the barrier.
Stage 2. Agreed that the largest block alone misses it. With each rank's inter- and intra-node bytes per all-to-all, the pipelined term becomes a sum over the all-to-alls of the slowest rank's step, each priced as the larger of its network and intra-node parts. With even buckets, where stage 1 is the slower part of each step, that reduces to §3's stage 1 plus the last stage-2 round, and
embed_tokensandlm_headare exactly where the buckets aren't even. The same sum would also price your bucket-ordering idea before it is run. I'll fit it to the medians once the rebuild is in, rather than to the profiled step times, which the synchronizes inflate.I've updated §3 and open question 5 to read the reconfigure and grow rows with the first-step wait, and linked NVIDIA/nccl#2608 from §1.
jinyouzhi commentedon Nov 15, 2026
@Achyuthan-S @0z5a, results for open questions 2, 4, 6 and 7, all on one 8× L20 node (PCIe), with the code of jinyouzhi#8 (a8c34e3). The workload is the same Qwen3-4B rebuild as before (2-D weights split by rows → split by columns), using prepared, warm runs.
Question 4: a fourth N
I reran N = 2 to 8. N = 2, 4 and 8 came back within 1–2 ms of the earlier numbers.
How to read the table:
run()to its final stream synchronize. "Wait" is that synchronize. Both are taken on the slowest rank.At 8 MiB, the per-region constant holds. I fitted your bytes + regions model on N = 2, 4 and 8 only. It predicts the new N = 3, 5, 6 and 7 within −5.4% to +2.7%. The per-region constant is 4.8 µs (5.2 µs when fitted on all seven points); you had 4.7 µs.
At 64 MiB, the larger constant is not a per-region cost.
The run is closer to the larger of host time and device time than to their sum.
copy_launches in_run.So in the model the per-region term is a host launch cost that overlaps the bytes term, not one added to it. The fused packs below confirm this.
On B70 the same sweep fails at N = 3 inside oneCCL's allreduce ("Non-uniform work-groups are not supported by the target device"), so these rows are L20 only.
Question 2: cached views or a sender-ordered buffer
I tried both on top of the same prepared handle. The exchange is the same in all three variants (same rounds, same
batch_isend_irecv); only the copies change:_runas in add allreduce test #8, which rebuilds each region's view and issues onecopy_per region.copy_per region.Every variant was checked bit-exact against the source. Medians of 5 warm runs, in ms:
What this shows:
So my answer to question 2: gather into the sender-ordered arena and assemble with one copy kernel per round, prepared once. In the model, the regions term then leaves
t_runand joinst_prepare.Caveats on the kernel:
copy_. There were none in this workload.Question 6: NCCL M2N against prepared transfers
M2N needs disjoint groups, so this is 4 → 4: ranks 0–3 hold the row split and ranks 4–7 receive the column split.
nccl-extensions0.2.0 (nccl.m2n), the same NCCL 2.30.7 that torch uses, and one persistentHandle.m2n.reshardper tensor. The prepared runs useTransferEndpoints(source={i: i}, target={i: 4 + i}).Observations:
ncclDevCommCreate("User requested GIN but not all ranks in the communicator support GIN"), so these modes need GPU-initiated networking.m2n.group_start()/group_end()fails with the samencclDevCommCreateerror: with PACK at entry 394 of 398, and with DIRECT or PIPE at entry 0.prepareinstead.These numbers are for PCIe without NVLink. They do not contradict @0z5a's B300 row, where M2N PACK beat #8809 at 128 MiB. That #8809 row was not prepared, and the fabric differs.
On whether a plan's segments can be handed to M2N: not as segments. M2N takes a whole tensor per call, described by a mesh and Shard/Replicate placements, with equal tiles. So it fits as a whole-request path: the policy chooses it when both maps can be written as M2N placements and the groups are disjoint, and calibrates it like any other path.
Question 7: the transport declaration
From the three paths that exist today (staged in #8809, symmetric-memory windows in jinyouzhi#7, NCCL put/signal in jinyouzhi#9), I think three properties per path are enough. Each is declared for a pair of processes (same host, or across hosts), not for a device.
What one transfer can address on each side:
Packed is why staging exists. Mapped needs no message at all.
Who moves the bytes:
Completion is either stream-ordered signals or a group fence. Put window transfers across hosts with NCCL one-sided puts jinyouzhi/DeepSpeed#9 uses signals on CUDA and XPU and barriers elsewhere.
Which memory and registration it needs:
The three existing paths in these terms:
The executor would pick each operation's path per pair from these properties. jinyouzhi#9 already does this in
_mapped_direction: an operation between hosts is a put only if both of its regions are packed and in windows, and otherwise it is staged. The planner sees only the calibrated cost of each path, withprepareandrunpriced separately. Capabilities are found by probing: whether a symmetric-memory rendezvous succeeds, whether peers can access each other's memory, and whether the backend supports put/signal.Hardware and software: 8× NVIDIA L20 (PCIe, no NVLink), torch 2.14.1+cu130, NCCL 2.30.7, Triton 3.8.0, nccl-extensions 0.2.0. Medians of 5 warm runs; 7 for the host and wait split. AI assistance was used to write the benchmark and kernel scripts and to draft this comment.
Achyuthan-S commentedon Nov 15, 2026
Thanks @jinyouzhi, this settles most of four open questions at once. I refit your tables and get the same picture: bytes plus regions, fitted on N = 2, 4 and 8 at 8 MiB, gives 4.8 µs per region and predicts N = 3, 5, 6 and 7 within 5.5%. The fused runs fit 15–18 ms plus about 23 GB/s per process within 5.2% at each budget.
I've updated the body:
copy_for views it can't tile.For the policy, this makes
t_prepare / n + t_runconcrete. At N = 8, fused saves 11–25 ms per run, so even a 230 ms table build is paid back within 22 runs. At N = 2 it saves 1.6 ms, so it takes at least 44 runs. A weight sync that reuses one prepared handle every step clears both.@0z5a, @jinyouzhi: should the fused copy go in with the prepared transfers (jinyouzhi#8), or after them?
0z5a commentedon Nov 16, 2026
Open question 1. @LiRunGuo, agreed on
_validate_aliases: it's #9093. The whole-buffer check stays as the fast path. Only when buffers overlap does the executor check the regions each process actually reads and writes. A local copy whose source and target are the same memory at the same strides is dropped and counted as read, so another write into it is still refused. On CPU at 2 and 4 processes, stage two of a relay, with each rank's band a view of its own full copy, now completes exactly with no local copy; master refuses it. A band viewed at a peer's rows is still refused on every rank before data moves.On
relay_map, I'd take your formulation as the definition. Each relay shard is the band of its holder's own destination, so stage two never needs a partial target, and the flag this question asks about isn't needed.min_split_bytesthen only decides which weights are relayed at all.Fused copy. @Achyuthan-S, I'd land it after the prepared transfers, as its own PR. Three reasons:
copy_anyway. Withcopy_as the default path and the kernel as an opt-in replacement for it, the default stays testable on CPU CI.To make that cheap later, PR 1 of §5 will keep each round's pack and unpack regions in one place on the prepared handle, so that a per-round table can replace them without changing
prepare/run. @jinyouzhi, if you'd rather carry the kernel, it would sit on top of that.jinyouzhi commentedon Nov 17, 2026
@0z5a thanks, agreed on the order. I'll carry the fused copy as its own PR on top of #9094.
copy_stays the default; the kernel is an opt-in replacement for a round's copies, Triton on CUDA first, with XPU (Triton XPU backend, or SYCL where needed) from our side.One question on the window transports (jinyouzhi#7 symmetric-memory window, jinyouzhi#9 put/signal): should the fused copy be done together with them?
release_staging()).My proposal is one table format designed for both, with the fused copy PR first and the window transport next on the same handle. Does that fit your plan for §5, or would you rather see them in one PR?
I'm reviewing #9094 now and will post there with L20 and B70 numbers.
0z5a commentedon Nov 17, 2026
PR 1 of §5 is open: #9094, staged prepare/run on #9027, with the structure of jinyouzhi#8 and @jinyouzhi as co-author. As said in my earlier comment, the handle keeps each round's packs, unpacks and local copies as lists of (destination, source) views, so a fused copy can replace a list without changing
prepare_transfersorrun.release_staging()and fused copies are not in it.0z5a commentedon Nov 17, 2026
@jinyouzhi, together works for §5: one PR on top of #9094 with the fused copy and the window transport, and one table format for both from the start, so a round's table can point its destinations into either the staging arena or a mapped peer window.
copy_stays the default, as you said, and #9094's per-round view lists are what the fused copy replaces.jinyouzhi commentedon Nov 18, 2026
@0z5a I opened #9118 as a draft on top of #9094, as we agreed. Only its last three commits are new:
prepare_transfers(..., copier=fused_copies)replaces a round's pack, unpack and local copies with one Triton launch. The same kernel runs on CUDA and XPU.copy_stays the default.window=TransferWindow(...), shards are copied straight into or out of the peer's symmetric-memory window. These copies use the same copier and table format as the staged lists. Thesymm_memcalls are wrapped indeepspeed.comm.PreparedTransfer.release_staging(). It frees the arenas and copy tables between runs, and the nextrun()stages them again. Withcheck=True, a re-staging failure refuses on every process.B70 numbers: Qwen3-4B, row split to column split, warm
run()median in ms:copy_)copy_as the default.Tests pass on CPU, 4× L20 and 4× B70. Details are in the PR description. I'll mark it ready once #9094 lands, and will rebase if #9094 changes in the meantime.