Skip to content

[RFC] Two-stage relay for cross-node weight sync, priced by a calibrated cost model #9026

Open

Description

@Achyuthan-S

Refs #8252, #8935, #8936, #9027. Builds on #8885 (ZeRO storage maps), #8519 (AutoTP layer maps), #8623 (plan_transfer), #8809 (executor, merged), #8966 / #8967 (calibration and plan analysis).

Summary

When weights move from trainers on one node to rollout engines on another, the cheapest plan is often not a direct one. Two examples:

  • Sending every byte across the network once, into the target node, and then spreading it over NVLink beats every direct plan.
  • On 2 × 4 GH200 over Slingshot, the pipelined two-stage plan moves 8 GiB to four engines in about 93 ms. Direct plans take 374–502 ms, and relaying through a single GPU takes 366 ms.

This RFC proposes four things:

  1. Express the relay as two ordinary plan_transfer plans through an intermediate relay map. No new IR concept is needed, and exactness follows from the existing argument, stage by stage.
  2. Choose between direct and relay plans with a calibrated cost model:
    t ≈ max over resources r (bytes through r / capacity of r) + rounds · α_round + segments · α_seg.
  3. State the memory, coalescing and delta options the executor needs to make relay practical.
  4. Keep fabric differences out of the planner (§6):
    • the plan says what moves;
    • the cost model chooses among plans;
    • each transport declares what it can do.

The model was tested against measurements on four machines (GH200 + Slingshot, B300 + NVSwitch, B70 + PCIe, H100 + InfiniBand). It predicted the pipelined relay at about 93 ms before it was run, and the run measured 92.7 / 94.3 ms.

1. Relay as two plans

Let M_s be the source maps (for example the trainers' #8885 storage maps) and M_t the engines' target maps.

The relay map. M_r is any map onto relay GPUs on the target node that has one holder per element.

  • Rename (special case). When the source already has one holder per element and the relay locations line up, as with ZeRO-3 partitions and one relay per trainer rank, M_r is M_s with each source rank i renamed to relay(i): the same pieces, the same shard shapes, new locations.
  • Split (general case). Otherwise a rename either multiplies the traffic or is refused, so M_r splits the parameter along axis 0 instead. This is relay_map(source_map, relay_locations, min_split_bytes=...) in @0z5a's design (§5).

@0z5a's CPU check: one 4,096-element parameter, four relays, four engines that each hold it whole. Both stages were run by applying the plans' segments directly, and every accepted case was exact.

Source Relay map Stage 1 bytes / model Largest relay shard / model Sent from
ZeRO-3 partition rename 1.00 0.25 all four, evenly
ZeRO-1/2 replicas rename 4.00 1.00 rank 0 only
ZeRO-1/2 replicas split, default selector 1.00 0.25 rank 0 only
ZeRO-1/2 replicas split, selector picks the matching rank 1.00 0.25 all four, evenly
8 ZeRO-3 trainers, 4 relays rename i % 4 refused (InvalidMapError)
8 ZeRO-3 trainers, 4 relays split 1.00 0.25 all eight, evenly

The two stages:

  • Stage 1 = plan_transfer(M_r, M_s, replica_selector=...). With a one-holder M_r spread evenly over the relays, the model crosses the network exactly once, and with rail-aligned NICs each rail carries 1/W of it. Which source replica feeds each relay piece is the [checkpoint] Batch replica selection for affine transfer plans #8936 policy's choice. The table shows that this choice decides whether the load is spread.
  • Stage 2 = plan_transfer(M_t, M_r). An intra-node plan from the relay buffers into the engines' target maps, over NVLink. When every engine needs the whole model, this is an all-gather.

Both are ordinary plans, cached per layout like any other. Nothing in the planner changes for the basic version.

Exactness. Exactness does not depend on the rename: any M_r that passes the map checks works. A rename preserves every condition a map must satisfy: well-formedness, the destination partition, coverage, homogeneity and exact scaling. A split relay map is checked like any other map.

  • The correctness note's plan theorem ("any consistent plan is exact, for every replica choice, as long as the replicas agree") applies to each stage.
  • Stage 1 therefore writes exactly extract_r(F). Its replicas agree because they are copies of agreeing source replicas.
  • So stage 2 writes exactly extract_t(F). This is the parallelism-invariance corollary applied to M_s → M_r → M_t.

Where each condition is checked:

Condition Checked by When
Map conditions (tiling of every shard by address, full coverage, consistent holders, scale 1) plan_transfer's metadata checks, in O(pieces²), never per element at plan time, once per layout
Every element written exactly once, for any stride ParamAffineMap.validate_coverage() (O(numel) in Python) in tests only
The data itself: replicas agree, and the target matches the source a runtime digest of the moved tensors, compared after stage 2 after every sync

No check on a map can catch a map that is self-consistent but describes the wrong memory. That is the job of the digest.

The digest also catches the transport. On @LiRunGuo's two H100 nodes (torch 2.13, NCCL 2.29.7, InfiniBand with GPUDirect), a bare all_to_all_single, with no DeepSpeed code, misplaced one 128 KiB block about once per 60 TB when calls of 0.4–1.7 GiB per peer were in the mix (comment; reported as NVIDIA/nccl#2608). Each wrong block held bytes from 512 KiB or 1 MiB away in the same message. The plans replayed on CPU were exact. So the digest stays on at run time, and a weight whose digest fails is sent again and checked again. Its cost is part of the price (§4).

2. Relay memory

A separate relay buffer costs model / W per relay GPU: 2 GiB per GPU for the 8 GiB example above, and about 17.5 GB per GPU for a 70B bf16 model over 8. Two ways to avoid it:

  • Borrow idle memory (near term). The contract is that during a sync the engine lends one idle, contiguous region at a fixed address, at least model / W in size, and does not read it until stage 2 has finished. The KV cache between rollout rounds is the natural candidate.
    • One instance, measured by @0z5a: a vLLM engine paused at sleep level 0 lent 44 of its 45 GiB KV allocation. After a prefix-cache reset and resume, greedy outputs were identical token for token, and pause, lend and resume took 0.09 s.
    • At sleep level 1 the memory is released, and re-creating it took about 9 s on GH200. So only an engine paused at level 0 should lend.
    • That test covers the lender only. In the same experiment the borrower was a trainer allocating from the lent region, and that was not safe.
      • Step-0 gradient norms had relative errors of 1.0e-3 and 4.5e-3 in two runs, and 2.3% by step 1 in the first, against 1.4e-4 run-to-run noise.
      • The cause is not known (a reuse race is one guess), and that integration was dropped.
      • A relay only copies bytes and doesn't go through autograd, so it may not hit this. But writing relay data into a lent region has not been tested.
  • Write stage 1 straight into the engines' weights (long term). Each block lands in one engine that needs it, and stage 2 becomes an in-place all-gather among engines.
    • This needs no extra memory, but it needs plan_transfer to accept a target map whose pieces fill only part of each shard.
    • Today _shards_tile refuses such a target, by design.

3. Cost model

t ≈ max over resources r ( bytes through r / capacity of r )  +  rounds · α_round  +  segments · α_seg

Resources are everything a byte occupies on its way:

  • each GPU's network send and receive;
  • each NVLink link;
  • any shared link, such as a PCIe root port or a cross-socket hop.

They are separate resources: on GH200, a GPU receiving over NVLink and over the network at once is bound by whichever is slower, not by their sum (see the mixed incast rows below).

Pipelined stages cost the slowest resource's stage 1 plus the last stage-2 round, not t1 + t2.

Collectives such as NCCL's ring all-gather get their own calibrated cost per participant count, because ring efficiency varies with the GPU count (below).

Prepare and run. The formula is the cost of one run. A plan also has a one-time prepare cost:

  • validation;
  • the schedule digest;
  • staging allocation;
  • the transport's own setup, such as communicators or registered windows.

Today #8809 repeats the first three on every call. On a prepared handle, the staging arenas stay resident by default, so their allocation stays in t_prepare. A staged-only transport may release them explicitly, and its next run then pays allocation and view rebuilding again, so the policy prices that run with them added (@jinyouzhi).

A plan lives until either side's placement, parallel sizes or ZeRO partitioning changes (spec §10.5). So for a plan that serves n syncs, the policy compares t_prepare / n + t_run. A transport with a cheaper run and a costlier prepare wins only once that cost is spread over enough syncs.

Both costs are visible in the measurements:

  • On B300, a first call's setup was about 150–250 ms per ordered GPU pair (below).
  • [checkpoint] Execute affine transfer plans with bounded peer staging #8809 also repeats its validation, schedule build, digest agreement and arena allocation on every warm call, so part of each warm call is prepare work as well (below and §5).
  • @jinyouzhi's symmetric-memory window paid about 1 ms of control plane per call until the transfer was prepared once (Prepare window transfers once and run them per step jinyouzhi/DeepSpeed#8).
  • Measured prepare and run (@jinyouzhi; fit in this comment). Qwen3-4B rebuild, row split to column split over N = 2 / 4 / 8 processes, staged prepared transfers (Prepare window transfers once and run them per step jinyouzhi/DeepSpeed#8):
    • Prepare grows with total segments: about 7.8 µs per segment on 8× B70 and 11.9 µs on 8× L20 from N = 4 to 8.
    • What prepare spends (breakdown), at N = 8 on B70 / L20: build_operations 42 / 76 ms and plan_digest 15 / 24 ms per plan, _validate_buffers 23 / 38 ms per request, _validate_aliases 7.5 / 11.6 ms, and the agreement all-gather 1.5–2.5 / 0.6–1 ms. So the fixed part is host-side Python, not a collective, and t_prepare is a per-segment term plus a per-tensor term.
      • Garbage collection adds a full collection to about every other prepare (each creates about 18k operation objects).
      • The 398 tensors have 8 distinct shapes, so a caller that shares plans across same-shape tensors removes most of the per-plan part.
    • Run on L20, refit with each process's regions (received, sent and local: about 1.6k / 2.2k / 4.2k at N = 2 / 4 / 8; fit): bytes per process plus a per-region term fits all three N within 4%, at 20.4 GB/s + 4.7 µs per region with 8 MiB staging and 23.4 GB/s + 9.3 µs with 64 MiB. A fourth N pinned it down (@jinyouzhi): fitted on N = 2, 4 and 8 at 8 MiB, the same model predicts N = 3, 5, 6 and 7 within 5.5%, at 4.8 µs per region.
      • At 64 MiB the larger constant is not a per-region cost: from N = 3 on, both budgets touch the same number of regions within 2%. The N = 8 / 64 MiB outlier is the host rebuilding about 2k views per round with only two rounds to hide it behind; caching the views brings it from 78 to 64 ms.
      • The per-region term is host time, the copy_ launches in _run, and it overlaps the bytes rather than adding to them: the final wait is 59–85 ms of the run at N = 2, and the device has already finished when the host reaches it at N = 8.
      • Fused copies (a prototype Triton kernel that copies each round's pack, each round's unpack and the local copies as one table of tiles per launch, with the tables built at prepare): the run is 15–18 ms plus the bytes at about 23 GB/s per process, within 5.5% at both budgets, and 1.5% faster than one copy_ per region at N = 2 and 17–32% at N = 8. Building the tables takes 70–230 ms in prepare. With fused copies, the regions term leaves t_run and joins t_prepare.
    • Run on B70 stays at 121–138 ms (64 MiB) while each process's bytes fall from 2.0 to 0.9 GB, which points to a shared link rather than each process's own.
    • Releasing the arenas, re-allocating them and running took 0.6–6.8 s on B70 at 8 × 64 MiB, against about 120 ms for a resident run.
  • A whole training-state move (@jinyouzhi), Qwen3-0.6B, ZeRO-1 DP4 → ZeRO-3 DP4 on 4× L20, 1.2 GB of bf16 weights plus 7.2 GB of fp32 master weights and Adam moments: maps 0.06 + plans 0.25 + prepare 0.08 + first run 0.23 = 0.62 s in the running processes, and 0.12 s for a repeated run once prepared. Through a universal checkpoint the same move took 27.4 s, plus a 10.1 s relaunch.

Inputs:

Evidence

Fits use the medians from @0z5a's raw timings. Bold cases were not used to fit any parameter.

GH200, 2 nodes × 4 GPUs, Slingshot (one NIC per GPU), 8 GiB bf16 to 4 engines.
Capacities: 23.4 GB/s receive per rail, 52.1 GB/s per NVLink pair, 68.4 GB/s network send per GPU (fitted from the direct case).

Case Model (ms) Measured (ms)
Direct: each engine pulls everything from one trainer GPU 502 (fit) 502
Relay through one target GPU, 64 MiB pieces 367 366
Sharded direct: each trainer rank sends its shard to every engine 367 374
Two crossed flows (A0→B0, A0→B1, A1→B0) 91.7 91.7
Mixed incast, 50% NVLink + 50% network 45.8 46.0
Mixed incast, 80% NVLink + 20% network 33.0 33.4
Two-stage relay (sharded into node B, then all-gather), serial 91.7 + 19.9 = 111.6 112.4 / 113.5
Same, pipelined with 64 MiB buckets ≈ 93 (predicted before the run) 92.7 / 94.3
  • Incast is flat: 1, 2 and 4 senders into one GPU reach 23.4 / 22.7 / 23.4 GB/s, so on this fabric the policy should not cap inflow.
  • Rounds cost almost nothing on the network stage (1 vs 32 rounds: 93.4 vs 92.9 ms) and about 34 µs each on the NVLink all-gather.
  • Across the 16, 64 and 256 MiB bucket sizes, in both orders, "stage 1 + the last all-gather round" is within 3.3% of every pipelined run.

H100, 2 nodes × 4 GPUs, InfiniBand (one NDR200 NIC per GPU), @LiRunGuo's 54 transfers of Qwen3-8B and 32B: syncs from a ZeRO-3 trainer on node 0 into replicas on node 1, and replicas growing onto node 0, shrinking back, or changing TP size, each as flat, relay + pipe, and relay + pipe + spread (fit).

  • Capacities: 24.2 GB/s per NIC plus 20.4 ms, from the three 32B flat syncs (within 0.4%; 97% of NDR200's 25 GB/s), and 82 GB/s per GPU inside a node, from six 8B moves on node 1. The GPUs are H100 SXM, so this is low for NVLink, and which intra-node transport NCCL uses is still to be checked.
  • Held out (33 transfers): network-only moves within −7% to +8% for 8B, and relay + pipe + spread within −1.3% to +4.0% at two replicas per node.
  • Not priced yet:
    • relay + pipe and relay + pipe + spread move the same bytes per rank, yet spread is 1.11–1.58× faster. The difference sits in the all-to-alls that carry embed_tokens and lm_head (per-step data), so the pipelined term needs each rank's inter- and intra-node bytes per all-to-all. It then becomes a sum over the all-to-alls of the slowest rank's step, which reduces to stage 1 plus the last stage-2 round when the buckets are even;
    • the 32B reconfigures, 16–18% over the network term. All of it is in the first all-to-all, 156–166 ms longer than its bytes take at the later steps' rate. Building the new layout without a barrier before the move probably counts the ranks' allocation wait as move time; without that wait they sit 4–7% under the network term;
    • a replica grown onto the trainer's GPUs, whose first all-to-all carries the same wait. These rows are to be rerun with a barrier, with per-rank local bytes and regions.
  • The choice: with two or more replicas per node, flat is 1.9–3.3× slower than the best relay, and the model picks a relay every time. Where plans move the same bytes the model ties; breaking ties toward spread, it picks the fastest plan or one within 1.3% of it in 13 of the 14 cases (the exception is +14.7%).
  • The source matters as much as the plan: reading from both the old replica and the trainer's partitions, a replica grows onto the trainer's node 4.7–5.3× faster, because nothing crosses the network. That choice needs holders in two layouts (Describe a parameter held in several layouts as one affine map #9052, and [AffineIR] Plan flat shards that keep the parameter's own strides #9070 for column cuts).

B300, one node, 8 GPUs on NVSwitch.

  • Capacities: 699 GB/s receive and 716 GB/s send per GPU, flat from 1 to 7 peers.
  • NCCL ring all-gather of 8 GiB: 9.7 / 10.8 / 11.2 / 12.4 ms on 2 / 3 / 4 / 8 GPUs. That is 1.58× to 1.15× the bytes/capacity bound, so collectives are calibrated separately.
  • NVLS was slower than ring at 2–4 GPUs, and NCCL's default chose ring at every size tried.
  • The [checkpoint] Execute affine transfer plans with bounded peer staging #8809 executor is linear in rounds at 8 GPUs: 35.2 / 67.0 / 371.0 ms at 4 / 16 / 128 rounds, i.e. 2.71 ms per round with 64 segments. The warm Qwen3-4B calls below split that into a round cost and a chunk cost. At 2 GPUs the cost per round grows from 0.16 to 0.30 ms as the staging shrinks.

Segments dominate a real model. Rebuilding Qwen3-4B (398 bf16 tensors, 8.0 GB) on B300 with the #8809 executor (64 MiB staging) makes one plan per tensor. @0z5a timed a first call and a second call in the same process, with the same plan and communicators:

GPUs Segments Rounds First call (ms) Second call (ms)
2 1,592 78 471 / 394 66.9 / 66.7
4 6,368 32 1,939 132.8
8 25,472 16 10,476 611.4

(Two launches at 2 GPUs. Earlier one-shot first calls on other days took 449 ms at 2 GPUs and 14.5 s at 8.)

  • Most of a first call is one-time setup: communicators, allocator growth and first-use paths, not broken down. The second call is 6–7× faster at 2 GPUs and 17× faster at 8.
    • The setup grows roughly with the number of ordered GPU pairs, about 150–250 ms per pair. The per-launch values are 151, 164, 176 and 202 ms here, and 191 and 248 ms for the earlier one-shot calls.
    • So it belongs in a prepare cost, paid once per plan and process, not in the cost of every sync.
  • Even warm, 8 GPUs take 611 ms against about 10 ms of bytes/capacity. Much of that may be prepare work that [checkpoint] Execute affine transfer plans with bounded peer staging #8809 repeats on every call.
    • @0z5a's CPU prototype (gloo, the same segment counts, Qwen3-4B shapes divided by 16) caches that work once. At 8 ranks, a warm call took 701 ms on that machine, 546 ms of it before the first byte moved. Caching the prepare work brought it to 144 ms, and caching the region views as well to 72 ms.
    • That is a laptop CPU with no GPU launches, so the numbers don't transfer to B300 directly. The prediction for B300 at 8 GPUs is ≤ 350 ms with prepare cached. If it lands above 450 ms, prepare isn't the main term there, and coalescing goes first.
    • So the order is: cache the prepare work first, then measure what coalescing still buys.
  • Round and chunk costs. These fits combine the warm calls with the staging sweep above. A chunk is a segment, or a staging-sized piece of one.
    • At 8 GPUs: 24.0 ms + 1.29 ms · rounds + 0.178 ms · chunks per process fits all four points (errors −0.9, +0.5, 0 and 0%). That's three parameters on four points, so one more staging size would be a useful check.
      • The 24 ms term is equivalent to the bytes at an effective ~310 GB/s, well below the 699 GB/s the links carry.
      • With bytes/capacity at 699 GB/s (10.75 ms) instead, about 13 ms per call is unexplained.
    • At 2 GPUs: about 0.06 ms per chunk per process, with the round cost bending as staging shrinks (above).
    • Across GPU counts: one set of constants for all GPU counts misses some points by more than 2×. Before calibrating per GPU count, try @0z5a's split of the chunk term:
      • α_schedule · total chunks for prepare, which grows with the GPU count;
      • α_copy · regions this process touches for run (5,970 at 8 GPUs: 2,786 packs, 2,786 unpacks, 398 local copies).

B70, one node, 8 GPUs on PCIe (from @jinyouzhi on #8966).

  • Pair bandwidth ranges from 3.6 to 54 GB/s depending on sender and direction.
  • At width 2, fan-in completion ranges from 12.6 to 55.2 GB/s depending on which senders feed the receiver.
  • This is why resources include shared links and why fit_links keys fans by peer set: each of the three cases discussed there is bound by a different resource (a sender's link, the receiver, the cross-socket link).

4. Delta transfer

Between GRPO steps, only about 1–2% of bf16 weight elements change. An element changes only when its fp32 master moves past half a bf16 ulp. A lossless sparse delta (indices and values) through the affine path sends just those.

Measured end to end (@0z5a):

  • Setup: a trainer on one B300 GPU behind a 10 Gb/s NIC, and rollout on two GH200 nodes, each receiving over a 1 GbE link.
  • Transport: plain TCP over ssh (an int32 index and a bf16 value per change), not [checkpoint] Execute affine transfer plans with bounded peer staging #8809 or NCCL.
  • Over 7 GRPO steps of Qwen3-4B, 0.90–1.37% of elements changed per step, i.e. 0.43–0.66 GB per step to the two receivers combined.
  • The delta sync took 3.4–6.3 s per step (median 4.1 s), not counting the per-sync digest check. A full 8 GB push would take about 70 s per receiver at that link rate.
  • Digests over all 796 regions matched, at step 0 and at the end.
  • Pricing each receiver's delta bytes against its link gave 2–3.5 s, below the measured time. The gap is the sender's copy, not the network.
    • The sender copies each target's payload GPU → host → bytes, one target after the other, before any send starts.
    • That takes about 0.53 s per 332 MB target, 23–25% of the sender's communication time.
    • Without it, each receiver got 0.092–0.106 GB/s (median 0.098), its single-pair rate.
  • The shared 10 Gb/s sender port was not the limit at two receivers (about 0.2 GB/s, ~16% of the port). Each receiver's 1 GbE link was.
    • A single-receiver decomposition confirms the cap. The real sender and wire format reach 0.102 GB/s, the same as a raw ssh pipe (0.103). Two flows into one receiver total 0.104–0.111 GB/s.
  • So the model prices this sync as delta bytes / 0.10 GB/s per 1 GbE receiver, plus a sender-side copy term.
    • The copy term is about 0.6 GB/s per target, serial in the current code. A chunked double buffer would overlap it.
    • Two runs are still to do: a direct fan run (both receivers at once) and a 1-vs-2-sender split test. Both need a GH200 node that is down, so there is no date yet. Until then, the fan factor of 1.0 is unmeasured.
  • Over NVLink, delta does not pay: 1.19 s against 0.39 s for the full transfer.
    • About 0.8 s of the 1.19 s isn't in any timer yet.
    • The same run's break-even bandwidth, about 32 GB/s, is the better number. It is an overestimate if part of that 0.8 s is specific to delta.
    • That run changed 1.6–2.2% of elements.

The policy prices delta and full the same way: encode/decode and copy time, plus bytes / capacity, plus the digest (below).

Correctness condition: both sides must start from the same previous version, bit for bit. So the digest has to run after every delta, and any mismatch falls back to a full sync.

  • This matters in practice. In a dry run that dropped one index on one receiver, a digest checked only at the end passed, because a later delta happened to rewrite that element.
  • A digest after every sync caught it at the first sync.

The digest needs its own term in the price. The sync times above exclude it.

  • As coded (sha256, serial, after resume), one check took 29.3 s at step 0 and 14.7 s at the end. Checking every step would add about 15 s to a 3.4–6.3 s sync.
  • That cost isn't inherent. A GPU-side region checksum should be far cheaper (@0z5a has a prototype tested on CPU).
  • The check could also run during rollout, at the price of finding a bad delta one step later.

5. Planner and executor

(@0z5a)

  • Planning: two plan_transfer calls, cached per layout. The policy evaluates the model for direct, relay, sharded and sharded + gather, and keeps the cheapest.
  • Execution: [checkpoint] Execute affine transfer plans with bounded peer staging #8809's bounded peer staging runs both stages.
    • Stage 2 starts on bucket k as soon as stage 1 has landed it.
    • 64 MiB buckets were best on Slingshot. On NVSwitch, larger staging keeps paying (8 / 64 / 256 MiB: 371 / 67 / 35 ms).
  • Coalescing: in the schedule, not in a map. Per-tensor plans keep the exactness argument per plan, and merging only changes how a round's regions are addressed.

The full design is in @0z5a's comment: #9026 (comment)

  • prepared transfers, with schedule state on a handle;
  • relay_map;
  • coalescing and adjacency-based direct rounds;
  • pipelining through a gate table computed at prepare.

The proposed order of PRs, each useful alone:

  1. staged prepare/run: [checkpoint] Prepare an affine transfer once and run it per step #9094 by @0z5a, on [checkpoint] Expose the affine transfer schedule as schedule_transfers #9027 (merged), with the structure of Prepare window transfers once and run them per step jinyouzhi/DeepSpeed#8;
  2. relay_map and the serial relay;
  3. flat relay and adjacency-based direct rounds;
  4. pipelining;
  5. small tensors, and the receive side, as the numbers from step 1 direct.

Training-state moves (#9074, @LiRunGuo). The split proposed there for upstream, each piece after this RFC:

  1. ZeRO-3's rank-agreement checks and barriers over the partition group ([zero] Run ZeRO-3's rank-agreement checks and barriers over the partition group #9076);
  2. the training-state move on plan_transfers + execute_transfers, with DP4 → DP2 → DP4 as its test. @jinyouzhi suggests it cover the wider matrix of DeepSpeedExamples#1021 (ZeRO stage, AutoTP and AutoEP size changes);
  3. the relay as two plan_transfers calls through a relay map (open question 1);
  4. the pipelined execution, after [checkpoint] Expose the affine transfer schedule as schedule_transfers #9027.

The rectangle planner and NativeRollout stay out. A move across TP or EP sizes inside the same processes also needs each engine to hold its own parallel groups (#9081).

6. Layers, and where the fabric enters

Fabric differences touch three layers differently. Examples: GPUDirect or not, host staging, one-sided windows. (Prompted by @delock's question; the current-state details are from @0z5a.)

Layer Decides Where it lives today Sees the fabric?
Plan which source region feeds each target region, from which replica; merging adjacent segments plan_transfer (#8623): replica_selector, _coalesce; batch policy plan_transfers (#8936) No. Exactness lives here, so the fabric changes how fast a plan runs, never what it writes
Plan choice direct vs relay vs sharded, bucket size, transport proposed in this RFC. #8966 and #8967 measure and report only; #9027's schedule_transfers exposes the executor's rounds, directness and arena sizes per process, which can feed the round term without mirroring executor logic. Nothing chooses today Only through calibrated numbers
Transport how bytes move the #8809 executor Yes

The executor today (#8809):

  • Schedule agreement: before anything moves, all ranks agree on a digest of the schedule. If any rank rejects it, nothing moves.
  • Ordering: each round posts all receives before any sends, in one batch_isend_irecv.
  • Messages: one packed message per peer per round.
    • A round goes in place only when it has exactly one region and that region is contiguous; each side decides separately.
    • Otherwise the round is packed through a per-peer staging arena bounded by StagingBudget (default 8 MiB per peer, per direction).
    • Large regions are split into budget-sized rounds.
  • Buffers: they must be on the current accelerator device when distributed is initialized (CPU otherwise), contiguous, and all of one dtype.
  • Rank numbering: source and target ranks are mapped separately (TransferEndpoints). Copies within one process skip the transport.
  • Limits: the call is synchronous (no CUDA graphs, no overlap with compute), and only scale == 1 is planned.

Transport capabilities. Fabric-specific code stays in the transport. It declares what it can do, detected by capability rather than by device name, so the planner never branches on the fabric. A declaration states three properties for a pair of processes (on the same host, or across hosts), not for a device (@jinyouzhi):

  • What one transfer can address on each side: packed, one contiguous run per peer per message (which is why staging exists); regions, a list of (offset, length) or strided regions in registered memory, as with RDMA scatter/gather; or mapped, peer memory that this process can load from and store to directly.
  • Who moves the bytes: two-sided, where both ends post and the receives are armed first; or one-sided, where one end moves the bytes and signals the other (pull, push or put). Completion is either stream-ordered signals or a group fence.
  • Which memory and registration it needs: device or pinned host memory (host staging is the device → host case), and whether the buffers must lie in a registered window, which is collective, has fixed addresses and outlives the prepared handle.

In these terms, #8809's staged path is packed, two-sided, device memory, no registration; the window path of jinyouzhi#7 is mapped and one-sided within a host, registered; the put path of jinyouzhi#9 is packed on both sides and one-sided across hosts, registered. The executor picks each operation's path per pair from these properties, as jinyouzhi#9 does in _mapped_direction, and finds the capabilities by probing. NCCL M2N sits outside this per-operation scheme, as a whole-request path (open question 6).

Each declared path is calibrated like any other resource, and the cost model chooses among paths, prepare and run included (§3). Candidates:

PXN is not the relay. NCCL may route an inter-node send through an NVLink peer nearer the NIC (PXN). That is NCCL's own choice, made with or without staging, per message and invisible to the planner. The relay in §1 is planned explicitly and priced by the model.

PXN matters for calibration: a "direct" network sample may already include a PXN hop. So calibration should record the NCCL PXN settings (NCCL_PXN_DISABLE, NCCL_P2P_PXN_LEVEL) and the transport lines from NCCL_DEBUG=INFO with each fit (@0z5a).

Related work: NCCL M2N

NCCL M2N (NVIDIA, arXiv:2610.07516; nccl_m2n in NVIDIA/nccl-extensions) reshards a tensor between two disjoint process groups, each described by a DTensor-style mesh with Shard/Replicate placements.

What it shares with this RFC. Its hierarchical schedule is the relay idea: each unique source contribution crosses the network once per destination NVLink domain, and leaders forward it to replicas over NVLink. Its speed-of-light bound, max(T_network, T_NVLink), has the same max-over-resources form as the cost model above.

The differences that matter for DeepSpeed:

  • Layouts. M2N's local tiles are contiguous and equal-sized per rank. DeepSpeed's often are not: uneven head-aligned AutoTP splits, fused and interleaved QKV, replicated blocks inside a sharded tensor, ZeRO flat partitions. The affine maps describe all of these.
  • Groups. M2N requires disjoint source and destination groups. Colocated resharding on the same GPUs is in scope here.
  • Exactness. Here, the correctness argument's conditions are checked at plan time, and the data is digested after each sync. The M2N paper validates its output bitwise in its benchmarks.
  • Choosing plans. M2N derives its schedule from the two layouts. Here, a calibrated model chooses among direct, relay and sharded plans per fabric, which matters on non-uniform fabrics such as the B70 PCIe tree above.

Where it helps. M2N is a native NCCL transport, and per-call and per-segment overhead is the main overhead this RFC measures. Using M2N, or its approach, as an execution backend for the affine plans it can describe is worth evaluating. So far (open question 6), it fits as a whole-request path rather than as a backend for a plan's segments.

Open questions

  1. Partial targets in plan_transfer (for in-place relay): an explicit flag, or a separate entry point, so the default keeps refusing a target that does not tile its shards?
  2. Coalescing across tensors: @0z5a proposes doing it in the schedule rather than in a map. On the receive side, a tensor's rows come from different senders, so what helps there: unpacking with cached views, or gathering into a sender-ordered buffer and then assembling?
    • @jinyouzhi's answer on 8× L20 (comment): gather into the sender-ordered arena and assemble with one copy kernel per round, prepared once. Cached views change the run by less than 1% up to N = 6 at 8 MiB; the fused kernel is the fastest at every N and budget (§3). Open: a kernel that also covers XPU (the Triton XPU backend or a SYCL kernel), with copy_ kept for views it cannot describe as rows of contiguous columns.
    • Order, from @0z5a (comment): the fused copy comes after the prepared transfers, as its own PR, with copy_ as the default path. [checkpoint] Prepare an affine transfer once and run it per step #9094 keeps each round's pack and unpack copies in one list on the handle, so a per-round table can replace that list without changing prepare or run.
  3. Candidate plans: is enumerating direct / relay / sharded / sharded + gather enough at first, or should the policy search?
  4. Round and chunk costs: fitted on B300 at 2 and 8 GPUs, from the staging sweep plus warm calls. Does the prepare/run split (α_schedule · total chunks + α_copy · regions touched) explain the dependence on GPU count? Prepared-transfer timings on B70 and L20, with a breakdown of prepare and per-process region counts, now exist (above). On L20, a fourth N and fused copies pinned the run term (§3): about 5 µs of host time per region, overlapping the bytes, and nothing per region once the copies are fused. Still needed: rows on B300 and GH200.
  5. Other fabrics: multi-node InfiniBand, measured by @LiRunGuo (comment): 2 nodes × 4 H100 with 4 × NDR200 per node, a ZeRO-3 trainer on node 0 syncing into rollout replicas on node 1, every sync digest-checked.
    • A "flat" sync, where every replica pulls its own copy, takes 1.04–1.08× of its inter-node bytes over the node's four NICs at 25 GB/s, for Qwen3-8B and 32B at 1, 2 and 4 replicas. The network term alone prices it.
    • With 2 and 4 replicas, relay + pipelining + spreading large weights over the ranks takes 1.11–1.30× of one model over the NICs, and the remainder grows with the replica count.
    • His 54 transfers are fitted in §3. Open: per-step bytes for the stage-2 term, a rerun of the reconfigures and grows with a barrier before the move, and the intra-node topology and NCCL transport.
  6. NCCL M2N as a backend: for plans whose layouts M2N can describe, how does it compare with [checkpoint] Execute affine transfer plans with bounded peer staging #8809 on the same hardware, and can a plan's segments be handed to it?
  7. The transport declaration: what is the smallest set of capabilities that covers host staging, RDMA scatter/gather and one-sided windows? For prepared state, @0z5a's answer is that schedule state lives on the prepared handle, and transport state such as windows hangs off it.
    • For the capabilities, @jinyouzhi's answer (comment): three properties per pair of processes, now in §6: what one transfer addresses on each side, who moves the bytes, and which memory and registration it needs.
    • For staging, @jinyouzhi's answer (comment):
      • run checks the handle by default, on all three paths (staged, direct and window): the buffers a caller passes are views of the old storage, so even staged copies don't follow a moved buffer (correction). Prepare window transfers once and run them per step jinyouzhi/DeepSpeed#8 (a8c34e3) compares each buffer's data_ptr, size, dtype and device with what prepare recorded and combines the result with a one-element all-reduce, so if a buffer moved on any rank, for example after reload_states(), every rank raises before data moves;
      • arenas stay resident on the handle, sized from what the schedule needs ([checkpoint] Expose the affine transfer schedule as schedule_transfers #9027 reports it per process) rather than budget × peers;
      • prepare may take caller-provided storage, such as the lent region of §2, under its fixed-address contract;
      • a rank-uniform release_staging() frees them for staged-only transports, while window transports keep them pinned.
  8. One weight held in two layouts (from @LiRunGuo: a surviving TP shard and the trainer's ZeRO-3 partitions). plan_transfer refuses it, since two source regions must be equal or disjoint. Cutting each layout's pieces at the other's boundaries, with the union of holders on each piece, makes it plannable without a planner change.

Who does what

Part Owner
Exactness argument, cost model (including prepare/run and the choice of transport), and the cost-aware policy on #8936 @Achyuthan-S
Two-stage planning (relay_map), executor staging, prepared transfers (#9094) and pipelining, the region alias check (#9093), coalescing in the schedule, schedule_transfers (#9027) @0z5a
Calibration and plan analysis (#8966, #8967) @jinyouzhi
Transport prototypes as transport state on the prepared handle: symmetric-memory windows and NCCL put/signal (jinyouzhi#7, jinyouzhi#9); prepared-versus-staged numbers on B70 @jinyouzhi
InfiniBand measurements; the rollout's shard-to-shard path on plan_transfer; ZeRO-3 over a subgroup (#9076) and the training-state move (#9074); a layout-independent digest kernel (offered) @LiRunGuo

Measurements: GH200 and B300 by @0z5a; B70 by @jinyouzhi; InfiniBand (H100, NDR200) by @LiRunGuo. Feedback welcome, especially from @delock.

Activity

  1. 0z5a commented on Nov 7, 2026

    @0z5a
    Contributor

    Checked this against my raw timings and current master. The GH200 table, the B300 collectives, the rebuild table and the executor description all match. A few things to change, most important first.

    §1: the relay map is a rename only when each element has one holder. On CPU, for one 4,096-element parameter with four relays and four engines each holding it whole, I ran both stages by applying the plans' segments directly:

    Source Relay map Stage 1 bytes / model Largest relay shard / model Sent from
    ZeRO-3 partition rename 1.00 0.25 all four, evenly
    ZeRO-1/2 replicas rename 4.00 1.00 rank 0 only
    ZeRO-1/2 replicas split, default selector 1.00 0.25 rank 0 only
    ZeRO-1/2 replicas split, selector picks the matching rank 1.00 0.25 all four, evenly
    8 ZeRO-3 trainers, 4 relays rename i % 4 refused (InvalidMapError)
    8 ZeRO-3 trainers, 4 relays split 1.00 0.25 all eight, evenly

    All the accepted cases are exact. So "crosses the network exactly once" and "model / W per relay" need a source with one holder per element, or a relay map that isn't a rename. The exactness argument doesn't depend on the rename, since any M_r that passes the map checks works. I'd define M_r as any map with one holder per element, with the rename as the special case, and let the #8936 policy choose the source replica.

    §2: the lending test covers the lender only. What I showed is that vLLM survives: KV overwritten, prefix cache reset, greedy output identical on resume. In the same experiment the borrower was a trainer allocating from that region, and it was not safe: step-0 gradient norms had relative errors of 1.0e-3 and 4.5e-3 in two runs, and 2.3% by step 1 in the first, against 1.4e-4 run-to-run noise. I didn't find the cause (a reuse race is my guess) and dropped that integration. A relay only copies bytes and doesn't go through autograd, so it may not hit this, but writing relay data into a lent region hasn't been tested.

    §3: the 8-GPU fit needs a constant. I can only reproduce "within 1%" as 24.0 ms + 1.29 ms · rounds + 0.178 ms · chunks per process (errors −0.9, +0.5, 0, 0%). Without the constant, 1.3 and 0.18 give 11 ms against 35.2 at 256 MiB, and 44 against 67 at 64 MiB. Using bytes/capacity (10.75 ms) as the constant, as the §3 formula does, gives 21.7 and 54.6, so about 13 ms is unexplained. That's three parameters on four points, so one more staging size would be a useful check.

    §3: setup per ordered pair. The Nov 6 runs give 151, 164, 176 and 202 ms, and the earlier cold calls give 191 and 248. I'd write "about 150–250 ms".

    §4: the fan and split runs aren't in progress. One of the GH200 machines is down, so neither has started, and I don't have a date. The single-receiver inference stands, but the fan factor of 1.0 is still unmeasured.

    §4: the digest needs a term in the price. §4 requires a digest after every delta, but the sync times exclude it. As coded (sha256, serial, after resume) one check took 29.3 s at step 0 and 14.7 s at the end, so checking every step would add about 15 s against a 3.4–6.3 s sync. That cost isn't inherent. A GPU-side region checksum should be far cheaper (I have a prototype tested on CPU, not yet on GPU), and the check could also run during rollout, at the price of finding a bad delta one step late.

    §4: over NVLink. About 0.8 s of the 1.19 s delta time isn't in any timer, and I haven't split it. The conclusion holds, but the same run's break-even bandwidth, about 32 GB/s, is the better number, and it's an overestimate if part of that 0.8 s is specific to delta. That run also changed 1.6–2.2% of elements, so "1–2%" fits better than "about 1%".

    §6: #8936 is no longer a draft. #9027 now exposes the executor's schedule (schedule_transfers: rounds, directness and arena sizes per process), which can feed the round term without mirroring executor logic.

    Open question 6 (M2N vs #8809). One data point on B300: 2→2 with Qwen3-4B tensor shapes, all bit-exact, two warm-ups then the median of five calls.

    to TP2 (8.05 GB) to two TP1 (16.09 GB)
    nccl_m2n, one reshard per tensor, PACK 24.8 ms 36.0 ms
    #8809, 128 MiB staging 39.3 ms 43.6 ms
    #8809, default 8 MiB 66.6 ms 101.5 ms
    NCCL, one message per GPU pair 6.0 ms 11.6 ms

    M2N's Group API and DIRECT mode hung at the first call in my setup (NCCL 2.30.7), and I don't know why, so per-tensor PACK is the only M2N row. M2N and #8809 at 128 MiB are both 3–7× above the one-message floor. The #8809 rows are warm calls, but execute_transfers repeats its validation and agreement on every call, so they include that work. I'll come back to this with §5.

    §5 is still on me.

  2. 0z5a commented on Nov 7, 2026

    @0z5a
    Contributor

    §5 details, which the TODO asked for. Nothing here is built except a throwaway CPU prototype of step 1, and none of it has run on GPU.

    1. Prepared transfers come first.

    A warm execute_transfers call still isn't a run. Before the first byte moves it repeats _validate_buffers, build_operations and a sort over the whole batch's chunks rather than this rank's, plan_digest, _validate_aliases, arena allocation and the digest all-gather. So the second-call column in §3 mixes a term in total chunks with a term in the regions one process touches.

    I rebuilt the same layouts on CPU over gloo, with Qwen3-4B shapes divided by 16: the same segment counts as your table, but 31 MB and one round. A throwaway prototype on master caches the prepare work per batch, then the region views as well. Times are rank 0 over 11 warm calls, median (min), in ms. Every call passed the demo's full check.

    Ranks Segments B300 warm CPU warm Before the first byte Prepare cached Views cached too
    2 1,592 66.8 53 (44) 32 (28) 19 (17) 10.5 (9.2)
    4 6,368 132.8 176 (137) 108 (80) 57 (47) 36 (31)
    8 25,472 611.4 701 (459) 546 (328) 144 (116) 72 (54)

    This is a laptop CPU with other load on it, with no GPU launches and almost no payload, so the CPU and B300 columns can't be compared directly, and I can't give the split on B300. The prototype also skips the agreement, which a real version shouldn't do. If B300 behaves the same way, two things change:

    • "611 ms against about 10 ms" doesn't yet show that coalescing is a requirement. Caching comes first, then we see what's left to coalesce.
    • The slope in §3 should grow with GPU count anyway. Prepare costs 12–18 µs per total chunk here, and total chunks are N times the chunks per process used in the §3 fit (3,184 at 8 GPUs). So I'd try α_schedule · total chunks (prepare) plus α_copy · regions this process touches (run) before calibrating per GPU count. That second count is a different axis: 5,970 at 8 GPUs (2,786 packs, 2,786 unpacks, 398 local copies).

    @jinyouzhi's prepare_transfers in jinyouzhi#8 already has this shape: validation, agreement, arenas and views once, run() only moves data, and execute_transfers is prepare plus one run. Its prepared staging column gains without windows too. I'd land the staged half on master first, separately from the windows in #7, and on top of #9027 so that _lanes stays the one schedule computation. @jinyouzhi, would you rather split that out, or should I? Two questions on top of that:

    • Should run check the handle? If one process re-prepares and another doesn't, a run can hang or move bytes on a stale schedule. One small digest all-gather would catch that before anything moves. add allreduce test #8 deliberately removed host collectives per run, which makes sense for its small messages, while a sync happens once per step. So should the check be on or off by default?
    • Could arenas be released between runs? They are 884 MiB per GPU at 8 GPUs and 64 MiB, and relay memory may be lent (§2).

    That answers the prepared-state half of open question 7: schedule state lives on the handle, and transport state such as windows hangs off it.

    2. Relay planning.

    relay_map(source_map, relay_locations, min_split_bytes=...) returns an M_r with one holder per element. It renames when the source already has one holder per element and the locations line up, and otherwise splits on axis 0 (see the table in my earlier comment). Stage 1 is plan_transfer(M_r, M_s, replica_selector=...) and stage 2 is plan_transfer(M_t, M_r), or #8936's plan_transfers for a batch. A serial relay is then two ordinary executor calls, with no executor change, so this can land early.

    Small tensors get one holder instead of being split. At 8 GPUs, the 145 1-D tensors of Qwen3-4B are 0.005% of the bytes, and one holder each takes the chunk count from 25,536 to 17,416 (schedule only).

    3. Coalescing (open question 2): in the schedule, not in a map.

    ParamAffineMap describes one logical parameter, and the exactness argument applies per plan, so I'd keep per-tensor plans and no batch-level map. Merging only changes how a round's regions are addressed, not what gets written. Whether regions can merge depends on the storage layout. For the 8-GPU rebuild at 64 MiB, one process, 112 messages each way:

    Layout on that side Send regions Receive regions
    Separate tensors 2,793 2,793
    One buffer, ordered by tensor 112 (a relay holds only its own rows) 2,793
    One buffer, ordered by sender 112 112

    (The 2,793 includes 7 regions already sent in place.)

    So a flat relay removes packing, but a flat target ordered by tensor doesn't help the receive side, because a tensor's rows come from different senders. For targets that need separate tensors, I see two options:

    • Unpack with cached views (2,793 regions, from step 1). A run-table kernel with a per-chunk fallback only if launches still dominate after that.
    • Gather into a sender-ordered buffer, then assemble with torch.cat(pieces, out=tensor): 398 calls at 8 GPUs instead of 3,184 region copies. From the torch source this is one batched kernel per call for contiguous inputs, but I haven't timed it on CUDA. It needs one extra model per GPU, so it only applies when lent memory allows. The transfer itself is your 64-segment row: 67 ms at 64 MiB, 35 at 256.

    If a relay can't be flat, torch.cat(regions, out=arena) batches the packing, about 25 regions per message at 8 GPUs (on CPU, 400 regions: 0.09 ms against 1.33). Torch has no batched unpack.

    The executor change is small. _round_is_direct accepts only a one-entry round today. It would accept entries adjacent in one buffer, and a direct round needn't be chunked by the staging budget, since it uses no staging. A follow-up branch of mine, not pushed yet, already sends every contiguous region in place, as one message of up to 1 GiB. On the one-buffer all-gather of 8 GiB, at 8 GPUs on B300, that took 21.3 ms in one round, against 371 / 67 / 35 ms at 8 / 64 / 256 MiB and 12.4 ms for NCCL's ring in the same run. At 2 GPUs it took 10.9 ms, against 158.7 / 23.9 / 16.1 ms and 11.7 ms for the ring.

    4. Pipelining.

    Stage 2 round k needs stage 1 up to round g(k). I'd compute that table at prepare from the two schedules and include it in the digest. With real Qwen3-4B shapes and four relays, equal budgets and request order give g(k) = k over 32 rounds at 64 MiB. Unequal budgets don't, so it's computed rather than assumed.

    Execution:

    • Stage 1 receives are all posted up front, directly into the flat relay. There's no arena, so this costs no staging memory.
    • Stage 2 runs its rounds in order, in its own group, and each round first waits for the stage 1 receive at g(k).

    Stage 1 never waits on stage 2, and stage 2 keeps the arm-then-send rule within each round. Gates only point from stage 1 to stage 2, so the graph is acyclic. That argument is host-level. Two NCCL communicators active on one GPU also need a consistent issue order across ranks unless they run on separate streams, which I'd verify in torch. The overlap bench ran isend/irecv next to an all-gather on a second group without hanging.

    This is the shape of my GH200 overlap bench, with one difference: stage 2 there was an NCCL all-gather, not executor rounds. The stage 1 bucket interval was 2.81–2.93 ms with and without the overlap, and the total was the slowest rail plus about 0.67 ms for the last round. Executor rounds in stage 2 are untested.

    Open question 1 doesn't block this: a separate relay buffer needs no partial targets. In-place relay stays open.

    5. Order and predictions.

    Each PR is useful alone:

    1. Staged prepare/run, with @jinyouzhi.
    2. relay_map and the serial relay.
    3. Flat relay and adjacency-based direct rounds.
    4. Pipelining.
    5. Small tensors, and the receive side as the numbers from step 1 direct.

    Predictions, none run yet:

    Case Prediction If it misses
    Prepare cached, B300, same layouts, 2 / 4 / 8 GPUs ≤ 50 / 75 / 350 ms (today 66.8 / 132.8 / 611.4) Above 450 ms at 8 GPUs, prepare isn't the main term on B300 and coalescing goes first
    Views cached too, 8 GPUs at least 40 ms less Leave them out of the first PR
    Flat relay, sender-ordered buffer, assembly, 8 GPUs, 64 MiB ≤ 100 ms Measure the assembly alone
    Pipelined relay in executor rounds, GH200, 64 MiB ≤ 102 ms (serial is 112–114) Keep the serial relay until that's understood

    The first row is today's warm time minus 80% of the fastest CPU before-first-byte time, rounded up. The second is about two thirds of the CPU saving. Both assume the B300 host spends at least that share of this laptop's time on the same work. That's the weak link: a much faster host would miss for that reason alone, and the decision rule would still apply.

    For open question 6, I'd repeat the M2N table from my earlier comment with #8809 prepared, as promised there, before reading anything into the gap.

  3. Achyuthan-S commented on Nov 8, 2026

    @Achyuthan-S
    ContributorAuthor

    Thanks @0z5a, this is a careful review, and the CPU relay-map table makes §1 much sharper. I've updated the RFC body:

    • §1: M_r is now any map with one holder per element, with the rename as a special case, plus your table, and the source replica chosen by the [checkpoint] Batch replica selection for affine transfer plans #8936 policy.
    • §2: the lending result is scoped to the lender, with your borrower numbers and the untested relay case.
    • §3: the 8-GPU fit now carries its 24 ms term, and the setup range is 150–250 ms per ordered pair. The coalescing conclusion now waits for prepared transfers, with your B300 prediction and decision rule.
    • §4: the fan and split runs are marked as not started, and the digest has its own term in the price. The NVLink case uses the ~32 GB/s break-even and 1.6–2.2%.
    • §5–§6, Q6, Q7: your design and PR order, [checkpoint] Expose the affine transfer schedule as schedule_transfers #9027, the B300 M2N table, and the prepared-state answer.

    On the cost model, I prefer your prepare/run split to per-GPU-count constants:

    • t_prepare ≈ α_schedule · total chunks + setup per ordered pair
    • t_run ≈ bytes / cap + α_round · rounds + α_copy · regions this process touches

    Rounds and regions would be read from #9027's schedule. Once the staged prepare/run lands, warm runs at 2 / 4 / 8 GPUs plus a fourth staging size would fit both terms, and show whether the ~13 ms per call at 8 GPUs belongs to prepare or to run. I'll set that fit up as soon as those timings exist.

    On checking the handle in run, I'd make it on by default. A sync happens once per step, so one small all-gather is cheap next to it, and a stale schedule is exactly the kind of silent failure the rest of the path is built to refuse. Paths with many small messages could opt out.

    Your PR order works for me.

  4. LiRunGuo commented on Nov 8, 2026

    @LiRunGuo
    Contributor

    Thanks @Achyuthan-S and @0z5a. Following up on a call with Achyuthan: some data for open questions 5 and 6, and a first step of moving our shard-to-shard path onto the affine IR.

    Open question 5, multi-node InfiniBand.

    • Setup: 2 nodes × 4 H100 SXM, 4 × NDR200 per node.
      • The trainer is ZeRO-3 on node 0; the rollout is on node 1.
      • Each sync is ZeRO-3 → the rollout's weights.
      • Times are medians of 4–6 runs, and every sync was digest-checked.
    • The bound: M2N's speed-of-light bound is one model over the node's 4 NICs at 25 GB/s each: 164 ms for 8B and 655 ms for 32B.
    • "flat": every replica pulls its own copy across.
    • "relay + pipe": two stages, with the second stage of bucket k−1 in the same all-to-all as the first stage of bucket k.
    • "+ spread": replicas fetch bands of large weights, one band per rank.
    Model Rollout on node 1 flat relay + pipe relay + pipe + spread Inter-node GiB (flat → relay)
    Qwen3-8B (16.4 GB) TP4 ×1 0.177 s 0.177 s 0.177 s 15.3
    TP2 ×2 0.348 s 0.232 s 0.187 s 30.5 → 15.3
    TP1 ×4 0.686 s 0.338 s 0.213 s 61.0 → 15.3
    Qwen3-32B (65.5 GB) TP4 ×1 0.695 s 0.695 s 0.695 s 61.0
    TP2 ×2 1.378 s 0.806 s 0.726 s 122.1 → 61.0
    TP1 ×4 2.727 s 0.941 s 0.820 s 244.1 → 61.0
    • With the relay, one copy crosses whatever the replica count. The best plans reach 77–93% (8B) and 80–94% (32B) of the bound above.
    • I have a CSV of 54 transfers for the cost model: these syncs, plus a replica growing onto the second node and shrinking back, with inter- and intra-node bytes. I can attach it or send it directly.

    Open question 6, NCCL M2N, on the same 8B transfers, 2 nodes:

    Our path on plan_transfer (in our fork):

    • Sources: the ZeRO-3 holdings now come from zero_param_storage_affine_map, and the training-state move reads the fp32 master and Adam moments through zero_optimizer_storage_affine_maps / zero_optimizer_storage_shard.
    • Plans: each weight's pieces become a source map and a target map of the rollout's fixed-address weights, and plan_transfer plans them. Our preference order (same rank, same node, caller order, least loaded) is the replica_selector. Execution is unchanged.
    • Relay: two plan_transfer calls through a relay map with one holder per element, as @0z5a defined it.
      • Each relay shard is the band of its holder's own destination that it receives, so it is a view of the engine's weights, not a separate buffer.
      • Every relay shard is therefore a whole shard, and the in-place relay of open question 1 needs no partial targets.
      • The second plan contains a copy of each band onto itself, which is the same memory and is skipped.
      • [checkpoint] Execute affine transfer plans with bounded peer staging #8809's _validate_aliases would refuse that stage anyway, because a relay rank's source (its band) lies inside its own target. The executor would have to allow a source inside the target when no segment reads what another one writes.
    • Two things plan_transfer doesn't take today, which we plan with our own planner for those weights:
      • holders of one weight in two layouts, e.g. a surviving TP shard and the trainer's ZeRO-3 partitions (refused as replicas with different piece boundaries);
      • a partial target, such as refilling one lost TP rank.
    • Tests: unit tests pass with both planners, and inter-node bytes are identical to our own planner's on every layout we tried.
      • On one 4-GPU node split into two fake nodes, syncs and reconfigurations of Qwen3-1.7B take the same time with either planner, and all 44 moves passed the digest check.
      • Planning is slower, 0.2–1.0 s per Qwen3-8B layout on 8 ranks against 0.01–0.06 s, but plans are cached per layout pair.
    • Integration plan: I'll swap the relay map for relay_map and the selector for [checkpoint] Batch replica selection for affine transfer plans #8936 when they land, then compare our all-to-all with [checkpoint] Execute affine transfer plans with bounded peer staging #8809 (prepared) on InfiniBand, as agreed.

    Digest (the runtime check in §1's table, and §4's check after every delta):

    • Each element's hash is keyed by its logical position, and the hashes are summed. Trainer partitions and rollout shards can therefore be compared without gathering, in any layout.
    • A Triton kernel digests a 16.4 GB rank in 76 ms on an A800, and a flipped bit is traced to its weight and rank.
    • Happy to contribute it.

    One caveat: with 32B across two nodes, our path still shows an intermittent single-weight mismatch: 4 of 142 checks, only in long mixed sequences of moves. I'm tracking it down, and the digest is what caught it.

  5. Achyuthan-S commented on Nov 10, 2026

    @Achyuthan-S
    ContributorAuthor

    Thanks @LiRunGuo, this is really helpful. The InfiniBand table answers open question 5, and it's great to see your path running on plan_transfer with the same inter-node bytes as your own planner.

    The CSV. Yes please, attached here would be best, so the fit can be reproduced. If it has inter-node bytes per rank as well as per node, the model can take the per-NIC maximum, which is what should separate "relay + pipe" from "+ spread". I'll fit it and post predicted against measured here.

    A first read of the table: in all six rows, "flat" takes 1.04–1.08× of its inter-node bytes over the node's four NICs at 25 GB/s, so the network term alone prices it. With 2 and 4 replicas, "relay + pipe + spread" takes 1.11–1.30× of one model over the NICs, and the remainder grows with the replica count. I'd expect that to be the second stage, which the intra-node bytes in the CSV will let the model price.

    One weight in two layouts. plan_transfer refuses this because any two source regions must be equal or disjoint. I tried a map-side fix on master: cut each layout's pieces at the other layout's boundaries, and give every cut piece the union of its holders. I ran 300 random cases of a surviving TP shard plus ZeRO-3 partitions on small 2-D weights (uneven TP sizes, any subset of surviving ranks, 1–6 partitions), planned each into a two-rank target, and applied the plans' segments directly on CPU:

    • when the TP split is along rows, all 137 cases were accepted and every plan was exact;
    • when it is along columns, 79 were accepted and exact, and 84 were refused. A ZeRO-3 box that spans several rows, cut at a column boundary, is no longer a dense run of the flat partition, and the planner accepts only dense runs there. One piece per row is accepted, but a 4096 × 4096 weight over 8 partitions with a TP2 column split would need over 8,192 pieces, above the default max_pieces_per_map of 4,096.

    Each (process, layout) gets its own location, which TransferEndpoints already allows, so the planner needs no change for the row case. I haven't run these plans through #8809 yet. I'm happy to write this constructor. @0z5a, for the column case, would you rather have plan_transfer accept a strided sub-box of a flat run, or keep one piece per row with a larger budget?

    The digest. It would be very welcome; it's the runtime check §1 and §4 depend on. One question: with several rollout replicas, is each replica's digest compared with the trainer's separately? That would also check that the replicas agree (P2). @0z5a has a GPU-side region checksum prototype tested on CPU, so it may be worth settling on one kernel together.

    The 32B mismatch. If it helps narrow it down, replaying a failing sequence's plans on CPU with index tensors (each element holding its logical position) shows quickly whether a plan or the execution is wrong. Happy to help with that.

    I've added the InfiniBand results, your in-place relay (open question 1), the M2N comparison (open question 6) and the two-layout case (new open question 8) to the RFC body, with credit.

  6. 0z5a commented on Nov 10, 2026

    @0z5a
    Contributor

    @Achyuthan-S on the column case: I'd rather have plan_transfer accept it, narrowly — a flat shard whose pieces sit at the parameter's own row-major strides, one shift for the whole shard, which is how a ZeRO partition stores a box cut out of its rows. I tried that on top of #9011 (bea2e09): about 50 lines in affine_transfer.py.

    • The one premise that changes is shard tiling by address. Two column halves of the same rows interleave, so their address intervals overlap even though no address is shared. With one shift per shard the address is the row-major index plus that shift, which is injective, so disjoint logical boxes, the bounds and the volume sum prove the tiling. The correctness note would need that case.
    • Still refused: two shifts on one shard, overlapping boxes, a piece past the shard's end, a mix with dense runs of another width. The existing affine suites pass (588 on CPU).
    • A random mix like yours (uneven TP, any surviving subset, 1–6 partitions, two-rank target), 300 cases: unchanged planner 134/134 along rows, 99 accepted and 67 refused along columns; with the change all 300 accepted and exact. Run through [checkpoint] Execute affine transfer plans with bounded peer staging #8809 on 4 gloo ranks with index tensors, all 600 target shards are exact, also at a 256 B staging budget that chunks the strided segments. With the roles reversed (the cut map as the target), all 300 are exact too, and every cut map passes ParamAffineMap.validate().
    • One piece per row costs more than the budget. 4096 × 4096, 8 partitions, TP2 column split, both TP shards alive: 16,384 pieces against 32, so 4× the default max_pieces_per_map; planning takes 0.4–5.8 s against about 1 ms; and into a column-split target the rows don't coalesce, so 8,192 segments (TP2) or 16,384 (TP4) and a 0.6–1.3 MB plan, against 16 or 32 segments. Into a row-split target both end at 8 segments.

    So please go ahead with the constructor as a separate PR, as you offered on #9045, with the row case first. I'll open the planner change on top of #9011 once that lands, so the column case works without a larger budget.

  7. Achyuthan-S commented on Nov 11, 2026

    @Achyuthan-S
    ContributorAuthor

    Thanks @0z5a, accepting the column case in the planner is cleaner than a bigger budget, and 600 exact shards through #8809, including at a 256 B staging budget, make the case well.

    Agreed on the split. I'll open the constructor as a separate PR with the row case first, built on #9010's affine_geometry so the box arithmetic stays in one place. Its cut pieces keep their strides and their shard's offset, so once your change lands on top of #9011, the column case should need nothing more from the constructor.

    I'll add the single-shift case to the correctness note: with one shift per flat shard, a piece's address is the row-major index plus that shift, which is injective, so disjoint logical boxes, the bounds and the volume sum give condition D directly.

  8. LiRunGuo commented on Nov 11, 2026

    @LiRunGuo
    Contributor

    @delock, and @tohtana / @tjruwase as owners of deepspeed/runtime/: could you say whether this piece is accepted as part of this RFC? It is open as draft #9045 so the design can be discussed on code, and it stays a draft until you decide.

    The piece. It is the first step of the bottom-up plan @Achyuthan-S and I agreed on. #8936's policy and other transports come later.

    • Sources: a ZeRO model's storage maps.
    • Targets: a tensor-parallel layout held by every inference replica.
    • Planning: plan_transfer plans each parameter once.
    • Execution: each sync is one execute_transfers batch into a fixed-address arena, so CUDA graphs captured on it stay valid.
    • Size: one new module, deepspeed/runtime/rollout/affine_weight_sync.py, with tests and a docs section. No existing source file changes.

    Numbers. 4× RTX 4090 (PCIe), ZeRO-3, bf16, median of 5 syncs. The baseline gathers each parameter whole and copies the local slice, which is #9023's approach, here on bf16 rather than fp32.

    • Qwen3-8B into TP4: 0.096 s against 0.778 s, with 36 MiB of extra peak memory against 1220 MiB.
    • Qwen3-1.7B: 0.04–0.21 s against 0.19 s. There is no gain at TP1, where every rank needs the whole model.
    • Correctness: the two copies were identical every time.

    @Achyuthan-S has reviewed it and run its tests on CPU merged with current master. If it is accepted, I'll mark it ready and follow with a tutorial and a DeepSpeedExamples example.

  9. jinyouzhi commented on Nov 11, 2026

    @jinyouzhi
    Contributor

    Sorry @0z5a for the slow reply on this — that's on me.

    On who splits the staged prepare/run: I think it belongs with you. The prepared handle is mostly schedule state (validation, agreement, arenas, views), it needs to sit on #9027 so  _lanes  stays the one schedule computation, and it overlaps your unpushed in-place/direct-round branch. Please feel free to lift whatever is useful from jinyouzhi#8 ( prepare_transfers , the staged-path tests); I'll read and wire it with more transport choices.

    On my side I'll keep refining the transport prototypes (symmetric-memory windows, NCCL put/signal) so they hang off your handle as transport state, which also gives a concrete answer to open question 7, i.e. the smallest capability declaration that covers them. Once your handle lands I'll rebase #7/#9 onto it and post prepared vs. staged numbers on B70.

    On your two questions:

    • Handle check in  run : agree with @Achyuthan-S, on by default. The small-message window path is the one that wants to opt out.
    • Releasing arenas between runs: I'd keep them resident on the handle by default, and not release them implicitly after each run.
    • Releasing doesn't free much on its own. Dropping the tensors returns the memory to the caching allocator, not to the engine. Making it usable elsewhere needs  empty_cache  and a fresh  cudaMalloc  on the next run, which moves allocation, one of the prepare costs, back into run.
    • It breaks anything bound to the address. The cached views, CUDA graphs, and any registered window (symmetric memory,  ncclCommWindowRegister ) all depend on it. So release is a transport capability: window transports must pin the arena, and the staged path may release it.
    • Two opt-ins instead:
    •  prepare  accepts caller-provided storage, e.g. the lent KV-cache region from §2, so the caller owns its lifetime under the "fixed address during a sync" contract;
    • an explicit, rank-uniform  handle.release_staging() , with re-allocation and view rebuild on the next run, for staged-only transports.
    • Size the arenas from what the schedule needs (#9027 reports arena sizes per process), not from budget × peers. Long term, flat relay and direct rounds need no arena at all, so it becomes the fallback for non-contiguous layouts.
    • NCCL M2N makes the same choice. Its staging is a library-owned, bounded pool, allocated with  ncclMemAlloc  and registered once per (comm, buffer) as a symmetric window. It stays resident for the runtime epoch and is freed only at  ncclM2nFinalize . By default that's 2 GiB × 4 slots for PACK ( NCCL_RESHARD_PACK_BUFFSIZES ). The source notes that registration is collective, so runtime eviction is avoided to keep ranks in lockstep.  ncclReshardWithWindow  is its caller-provided-memory path.

  10. Achyuthan-S commented on Nov 11, 2026

    @Achyuthan-S
    ContributorAuthor

    Thanks @jinyouzhi, that settles both questions, and the M2N comparison is a useful reference point. I've recorded them in the RFC body (§3, the PR order in §5, open question 7, and the ownership table):

    For the cost model, this keeps staging allocation in t_prepare. After release_staging(), the next run pays allocation and view rebuilding again, so the policy prices that run with them added. Your prepared-versus-staged numbers on B70 will be the first data on how much that is.

  11. jinyouzhi commented on Nov 11, 2026

    @jinyouzhi
    Contributor

    On column support in the planner: I agree it's worth having for these scenarios. Today a flat shard can only hold a dense run, either a plain line or a box stored as a line. This change would add a third form: a box kept at the parameter's own row-major strides, with one shift for the whole shard. That lets the two-layout case and other column splits plan without a larger budget.

    @0z5a, about your ~50-line planner change: would you rather open it as a PR stacked on #9011, or push it straight into #9011? I think you have access. Either works for me. I will push #9010 to merge first, Thanks.

  12. jinyouzhi commented on Nov 11, 2026

    @jinyouzhi
    Contributor

    Prepared vs. staged numbers on B70, as promised.

    Setup: one host, 8× Arc Pro B70 (PCIe), torch 2.14.0.dev20260804+xpu, oneCCL built from source. Qwen3-4B rebuild: 398 bf16 tensors (8.05 GB), one plan per tensor. 2-D weights go from a row split to a column split over the same N processes; 1-D ones to an even split shifted by half a shard. That gives 1447 / 5063 / 18367 segments at 2 / 4 / 8, fewer than your 398·N² because the 1-D tensors don't fan out fully. Staging only, no window. execute is master 8a6b53fe. prepare/run is jinyouzhi#8 c0ed827b, whose execute matches master within noise. Times are the max over ranks, median of 5 after 2 warm-ups. Every mode is checked bitwise against the source.

    N budget segments rounds execute (master) prepare run arena / proc
    2 8 MiB 1447 311 187.2 ms 17.8 ms 171.4 ms 16 MiB
    2 64 MiB 1447 32 137.9 ms 13.6 ms 123.2 ms 128 MiB
    4 8 MiB 5063 78 198.8 ms 28.0 ms 171.4 ms 48 MiB
    4 64 MiB 5063 8 164.7 ms 27.3 ms 137.5 ms 380 MiB
    8 8 MiB 18367 16 192.1 ms 131.3 ms 117.7 ms 109 MiB
    8 64 MiB 18367 2 399.3 ms (190–784) 130.8 ms 121.3 ms 889 MiB
    • Prepare cost grows quickly with segment count: about 10% of a call at 2 processes, and at 8 processes 131 ms of a ~190–300 ms call. Caching it brings the 8-process rebuild to ~120 ms, in the same direction as your B300 prediction.
    • B70 is link-bound, not schedule-bound: 2→8 stays roughly flat (123 → 121 ms at 64 MiB) because every process moves about the same bytes over its own PCIe link. That is unlike the B300 curve.
    • Releasing the arenas is expensive on B70: dropping them, calling empty_cache(), re-allocating, then running takes 0.6–6.8 s at 8×64 MiB across sessions (vs. ~120 ms resident). That supports resident-by-default and keeping release_staging() opt-in.
    • Variance: per-call execute at 8×64 MiB ranges 190–784 ms (it prepares and allocates its staging buffers on every call, so it inherits the allocation noise). Prepared run stays at 119–163 ms.

    Same benchmark on 8× L20 (PCIe, torch 2.14.1+cu130, NCCL 2.30.7), jinyouzhi#8 8e20b87c:

    N budget execute prepare run
    2 8 MiB 147.6 ms 39.1 ms 107.2 ms
    2 64 MiB 133.2 ms 30.8 ms 102.8 ms
    4 8 MiB 143.2 ms 59.4 ms 83.1 ms
    4 64 MiB 141.4 ms 58.8 ms 82.1 ms
    8 8 MiB 300.1 ms 218.8 ms 63.6 ms
    8 64 MiB 313.9 ms 215.2 ms 78.1 ms

    Re-allocation after a release (median). jinyouzhi#8 8e20b87c now allocates each direction's staging arenas once and gives each peer a slice, instead of one allocation per peer; run is unchanged on both:

    N budget B70 before B70 after L20 before L20 after
    2 8 MiB 0.1 ms 0.1 ms 0.1 ms 0.1 ms
    2 64 MiB 123.7 ms 4.7 ms 1.0 ms 1.0 ms
    4 8 MiB 125.0 ms 2.8 ms 0.5 ms 0.5 ms
    4 64 MiB 359.9 ms 36.1 ms 3.0 ms 2.9 ms
    8 8 MiB 179.8 ms 103.3 ms 1.7 ms 1.9 ms
    8 64 MiB 0.5–4.0 s ¹ 2.4–3.0 s ¹ 14.4 ms 24.4 ms

    ¹ Noisy across sessions; interleaved A/B runs in the same session gave 2.5 / 4.0 s before and 3.0 / 2.4 s after, so no change at this size.

    Script: https://gist.github.com/jinyouzhi/65a65d56f875ad2faef80f946e78c876. AI assistance (GitHub Copilot) was used to write the benchmark and this summary; I reviewed the numbers.

  13. Achyuthan-S commented on Nov 11, 2026

    @Achyuthan-S
    ContributorAuthor

    Thanks @jinyouzhi, this is the data the prepare/run split needed. I fitted it against the model in §3; every number below comes from your two tables.

    Prepare. A cost per total segment fits the step from N=4 to N=8 with almost no intercept: about 7.8 µs per segment on B70 and 11.9 µs on L20, the same order as @0z5a's 12–18 µs per chunk on CPU. It doesn't fit N=2: there, prepare is about 16 ms (B70) and 19 ms (L20) above that line on both machines. So t_prepare has a fixed part that a per-segment term misses. With only three values of N I can't tell whether that part is constant or depends on N. Would a breakdown of prepare into validation, schedule build, digest agreement and arena allocation be easy to log? That would separate the terms better than more N.

    Run on L20. Bytes over a per-process link plus a small round term fits N=2 and N=4 within 4%: about 18.9 GB/s per process, with rounds costing almost nothing (the 8 and 64 MiB runs differ by 1–4 ms). It predicts 46 ms at N=8 against 64–78 ms measured, so N=8 has another 17–32 ms. My guess is the per-region copy term, since each process touches more regions as N grows, but I'd want the per-process region counts before claiming it.

    Run on B70. In this rebuild each process sends (N−1)/N² of the model: 2.0, 1.5 and 0.9 GB at N = 2, 4 and 8. So per-process bytes fall 2.3× while run stays at 121–138 ms (64 MiB). That suggests each process's own link is not the bottleneck here. The total crossing processes grows from 4.0 to 7.0 GB, which would fit a shared link such as the cross-socket hop in your #8966 B70 data. Pricing it would need which pairs share that link, which the fan rates fit_links keys by peer set are meant to capture.

    Release. 0.6–6.8 s to release, re-allocate and run at 8 × 64 MiB on B70, against about 120 ms resident, makes the case for resident-by-default and an opt-in release_staging() very concretely.

    I'll add these to §3 as the first prepare/run measurements, and refit once there are per-process region counts (from #9027's schedule_transfers) and a prepare breakdown.

  14. Achyuthan-S commented on Nov 12, 2026

    @Achyuthan-S
    ContributorAuthor

    One more input for the prepared handle, from @jinyouzhi's ZeRO-3 runs with #8842 (DeepSpeedExamples#1021, comment 6733267750). offload_states() / reload_states() move every ZeRO-3 weight partition to new storage: reload_states assigns lp_param_buffer.data a new tensor and re-points each partition, and each aliased master, into it.

    A plan survives that, since it depends only on the layout (spec §10.5). But a handle that cached region views before the reload would keep copying into the old storage, and nothing would fail. The run check agreed above compares schedules across ranks, so it wouldn't see a buffer that moved on every rank.

    One option: run compares each bound buffer's data_ptr and size with what prepare recorded, and folds a mismatch into the same all-gather, so every rank refuses together and the caller re-prepares. It needs nothing from the engine. @0z5a, would that fit the handle?

  15. jinyouzhi commented on Nov 12, 2026

    @jinyouzhi
    Contributor

    @Achyuthan-S thanks, your fits match ours. I profiled prepare_transfers stage by stage (same Qwen3-4B rebuild, 398 tensors; per-rank wall times, median of 7, GC off — see the note below).

    Prepare breakdown, N=8 (ms)

    stage B70 L20
    _validate_buffers (per request) 23 38
    build_operations (per plan) 42 76
    plan_digest (per plan) 15 24
    _validate_aliases 7.5 11.6
    rounds / direct payload / arena alloc 1.5 2
    agreement all_gather + sync 1.5–2.5 0.6–1
    total ~100 ~160

    At N=2 the same four Python stages are 13–16 ms on B70 and 28–35 ms on L20, and the agreement is 0.2–0.3 ms. So the intercept/residual you saw is host-side Python, not a collective. Most of it scales with segments; _validate_aliases and the per-request part of _validate_buffers scale with tensor count and stay flat as N changes.

    Two things inflate the prepare numbers I posted earlier:

    • GC. Each prepare creates ~18k operation objects. With GC on, about every other repeat pays a full collection: N=8 alternates 100/215 ms on B70 and 160/220–270 ms on L20. With GC off it is a steady 100 ms on B70 and 160 ms on L20. The earlier medians partly included this.
    • No plan reuse in the benchmark. The executor reuses build_operations / plan_digest per plan object, and the benchmark builds a fresh plan per tensor. The 398 tensors have only 8 distinct shapes, so a caller that shares plans across same-shape tensors would cut ~57 ms (B70) / ~100 ms (L20) at N=8 to a few ms.

    Regions per rank (budget-independent; first two ranks shown):

    N recv send local
    2 529 / 384 384 / 529 529
    4 919 774–919 403
    8 1923 1778–1923 399

    run issues roughly one pack/unpack copy per region, so ~2.2k copies per rank at N=4 and ~4.2k at N=8. At typical launch costs that is tens of ms at N=8, consistent with your per-region term. Fusing the packs into one gather kernel per round would be the lever there.

    Stale buffers after offload_states/reload_states. Agreed this is real, but only on two paths:

    • Staged and local copies re-resolve the request's buffers on every run, so they follow a buffer that moved.
    • Direct (zero-copy) messages and window copies hold views made at prepare time. If storage moves, those keep writing the old storage, silently: the view keeps it alive, so nothing crashes.

    A check that only one rank does would hang the others, so whatever we do has to be collective. For direct messages, the cheapest fix is to rebuild the view at run time; there are only a few, so this is cheap and needs no collective. Window copies depend on offsets every rank has published, so they need the data_ptr/size check folded into an agreement round in run (~1 ms per run at N=8 here). Otherwise run should refuse and require a re-prepare.

  16. 1 remaining item

  17. Achyuthan-S commented on Nov 13, 2026

    @Achyuthan-S
    ContributorAuthor

    Thanks @jinyouzhi, the breakdown answers it: the fixed part of t_prepare is host-side Python, not a collective. So the model becomes a per-segment term (build_operations, plan_digest) plus a per-tensor term (_validate_buffers, _validate_aliases), with the agreement a millisecond or two. The GC effect and plan reuse are worth recording too: with 8 distinct shapes among 398 tensors, a caller that shares plans removes most of the per-plan part.

    Run, refit with your region counts (L20). With each process sending (N−1)/N² of the model, bytes alone gave 47 ms at N=8 against 64–78 ms measured. Adding a per-region term fits all three N within 4% at both budgets:

    budget per process per region N=8 region term
    8 MiB 20.4 GB/s 4.7 µs 20 ms of 63.6
    64 MiB 23.4 GB/s 9.3 µs 40 ms of 78.1

    Regions per process are your max over the ranks shown (recv + send + local: about 1.6k, 2.2k and 4.2k at N = 2, 4, 8). The per-region constant isn't pinned down: it is two parameters on three points per budget, and the constant doubles between budgets. But the term accounts for the N=8 excess that bytes alone missed, and fusing the packs into one gather kernel per round would shrink exactly that term. A run with fused packs, or a fourth N, would separate the two.

    Stale buffers. a8c34e3 is the check I had in mind (data_ptr, size, dtype and device against prepare, one all-reduce, every rank refuses together), and your correction means all three paths need it. #9045 resolves its shards at every sync since 48a3575, so it follows a reload on its side too.

    I'll add the prepare breakdown, the run refit and the handle check to §3 and open question 7 in the body.

  18. jinyouzhi commented on Nov 13, 2026

    @jinyouzhi
    Contributor

    One more data point for the in-memory vs. checkpoint comparison, measured independently of #9074: LiRunGuo opened #9074 about an hour and a half before I ran this, and I only saw it afterwards. It agrees with #9074 and adds three things #9074 does not cover: both paths on one machine, a change across ZeRO stages, and a repeated run of an already prepared transfer.

    Setup: Qwen3-0.6B, bf16, AdamW, 4× L20, ZeRO-1 DP4 → ZeRO-3 DP4, OMP_NUM_THREADS=8, checkpoints on local NVMe. Each phase is the max over ranks; 3 runs agree within ~1.5 s and the table shows the median. The live path plans each tensor with plan_transfer, then uses the executor's prepare/run path from my B70 numbers above.

    phase seconds
    relaunch (launch → engine ready): imports 3.5 (torch 1.3, deepspeed 0.4, transformers 1.8), launcher 2.7, dist init 1.8, model load 0.65, engine init 1.5 10.1
    through a universal checkpoint: save 4.6 + ds_to_universal 17.5 + load 5.3 27.4
    in the running processes: maps 0.06 + plans 0.25 + prepare 0.08 + first run 0.23 0.62
    repeated run (target engine already built and prepared) 0.12
    target engine built in the warm process (model load 0.3 + engine init 1.6 + first step 0.8) 2.8

    The live move carries 1.2 GB of bf16 weights plus 7.2 GB of fp32 master weights, exp_avg and exp_avg_sq (1240 requests, 4969 segments). I checked it with per-state sums against the source, which agree within 3e-14 relative. That is weaker than #9074's bitwise digest, and I have not run a bitwise check on this path.

    What this adds:

    Hardware for the numbers above: 4× NVIDIA L20, torch 2.14.1+cu130. AI assistance was used to write the benchmark script and draft this comment; I reviewed the numbers and the text.

  19. LiRunGuo commented on Nov 14, 2026

    @LiRunGuo
    Contributor

    Thanks @Achyuthan-S.

    The CSV. Both files are in this gist: https://gist.github.com/LiRunGuo/4036b5cfec93c1975e02dafac5d20435

    • two_node_h100_transfers.csv: the 54 measured transfers (Qwen3-8B and Qwen3-32B: syncs into TP4 ×1, TP2 ×2 and TP1 ×4 on node 1, and a TP4 replica growing onto node 0 and shrinking back, from the old replica or from both the replica and the trainer's partitions; flat, relay + pipe, relay + pipe + spread). Medians, min and max of 4–6 runs, stages, transfer count, inter-node bytes in total and into the busiest node, intra-node bytes.
    • two_node_h100_per_rank_bytes.csv: for the same 54 plans, each rank's inter- and intra-node bytes sent and received (bf16), so the model can take the per-NIC maximum. These come from rebuilding each plan on CPU with the same planner; the planned inter-node totals equal the measured runs' in all 54 rows.
    • Ranks 0–3 are node 0 (the trainer), 4–7 node 1. Each GPU has its own NDR200 NIC (25 GB/s).

    Digest with several replicas. Per replica now. Each replica's digests are compared with the trainer's on their own: one all-reduce with one slot per replica, so it costs the same as the old sum over all rollout ranks, and a failure names the replica as well as the weight. That checks P2 directly. The planner comparison I mentioned on #9074 ran with it (Qwen3-8B on two nodes, 270 checks, none failed). Happy to settle on one kernel with @0z5a's region checksum.

    Buffers that move after offload_states() / reload_states(). As you noted above, #9045 resolves its shards at every sync since 48a3575 and has a test for it (offload, reload, a check that the partitions really moved, then a sync that must match exactly). With a prepared handle it would pass the rebuilt requests to run(requests) as in @jinyouzhi's a8c34e3, so the data_ptr check covers it.

    The 32B mismatch. Thanks for the suggestion; I did exactly that. I rebuilt every plan of the failing sequence (190 moves, 29 distinct plans, Qwen3-32B's 707 weights, the real shapes on 8 ranks over two nodes) and replayed them on CPU with index tensors, executing each move as our executor does (local copies, then all sends packed, then all receives written, stage by stage, a reused replica reading and writing the same buffer). Every target element held its own index after every move, for all 707 weights, so the plans are right and the cause is in the execution (or the hardware). A self-test that corrupts one element of one transfer is caught, on the right rank.

    I then reran the sequence on GPU with two more checks. The old replica was checked on its own right before every timed move, and each trainer partition's digest was compared with its value at load after every move. One of 48 moves failed again. In it, the old replica had been right just before the move and the trainer never changed, so the grow itself wrote the wrong values, into the new replica.

    A longer rerun ran the same sequence three times and dumped the bad shard element by element on a failure. Two of 162 checks failed. Neither fits the pattern of the earlier failures (a row-parallel weight in the flat grow from both holders): one was a grow from the old replica alone with relay+pipe+spread, the other a plain sync from the trainer. The dump of the grow failure shows what went wrong. One transfer wrote the whole shard: 20 MiB, contiguous on both sides, from the old replica on the other node, inside one all_to_all_single. It has two wrong blocks, 116 KiB and 128 KiB, on the same 128 KiB grid. Each holds exactly the bytes of another block of the same message, one from 512 KiB later and one from 1 MiB earlier. There are no zeros and nothing from outside the message, and the bytes are still wrong on a recheck. Both failures are in transfers that cross the two nodes. That pointed below the plan and our pack/unpack, at the network transport (torch 2.13, NCCL 2.29.7, IB with GPUDirect).

    So I ran all_to_all_single alone between two nodes: no DeepSpeed, no model, every word encoding its iteration, its source rank and its offset, buffers allocated and freed as our executor does, ten minutes per setting.

    • With every message at or below 256 MiB per pair it was clean: 49,712 exchanges, 159 TB between the nodes, under five NCCL settings.
    • With the sizes of the failing moves in the mix (0.4 to 1.7 GiB to one peer in one call) it reproduces: 3 wrong blocks in 84 TB with our two NCCL settings (NCCL_IB_QPS_PER_CONNECTION=4, NCCL_NCHANNELS_PER_NET_PEER=8), none in 40 TB with NCCL's defaults.
    • Each is one 128 KiB block on the message's 128 KiB grid that holds the bytes belonging 512 KiB or 1 MiB earlier in the same message, still wrong on a recheck: the signature of the dump above. They arrived on both nodes, in messages of 256 MiB and 742 MiB.

    So the mismatch is below DeepSpeed and below our executor: torch.distributed.all_to_all_single on torch 2.13.0 / NCCL 2.29.7 over IB with GPUDirect, on this cluster. Not settled: whether the two settings are needed (0 in 40 TB with the defaults is not significant against 3 in 84 TB), and whether it is tied to these machines or this NCCL version. What caught it is the digest after every move, which is one more reason to have that check at run time. The 8B runs passed every check (39 configurations, and 270 per-replica checks in the planner comparison). Next I'm separating the two settings, and I'll report it to NCCL with the standalone test.

  20. LiRunGuo commented on Nov 14, 2026

    @LiRunGuo
    Contributor

    A follow-up on the 32B mismatch, because my last comment left the two NCCL settings open. I ran the bare all_to_all_single test again at the same message sizes, 20 minutes per setting, on another pair of nodes:

    Settings Sent between the nodes Wrong blocks
    NCCL defaults 80.7 TB 1
    NCCL_NCHANNELS_PER_NET_PEER=8 alone 85.1 TB 2
    NCCL_IB_QPS_PER_CONNECTION=4 alone 79.3 TB 0
    • So the defaults are not free of it. With the earlier run: 1 block in 121 TB at the defaults, 6 in 370 TB over all settings, about one per 60 TB. The differences between the settings are not significant.
    • The signature is the same each time: one 128 KiB block holding the bytes that belong 512 KiB or 1 MiB earlier in the same message, still wrong on a recheck. Three pairs of nodes so far.
    • For the RFC: on such a path the digest after a move is what makes the move safe, and a weight that fails it has to be sent again. I'm adding that to our harness (re-send the weights whose digest fails, then check again), and I'm preparing a report for NCCL with the standalone test.
  21. Achyuthan-S commented on Nov 14, 2026

    @Achyuthan-S
    ContributorAuthor

    Thanks @LiRunGuo, here is the fit of your 54 transfers to §3's model, t ≈ c0 + max over resources (bytes / capacity). The resources are each rank's NIC (inter-node bytes sent or received) and each GPU's intra-node traffic, both taken per rank from two_node_h100_per_rank_bytes.csv. The 12 shrinks that move nothing (0.7–5 ms) are left out.

    Fitted (9 rows)

    • Network: 24.2 GB/s per NIC plus 20.4 ms, from the three Qwen3-32B flat syncs (all within 0.4%). That is 97% of NDR200's 25 GB/s.
    • Intra-node: 82 GB/s per GPU after the same 20 ms (72 GB/s raw), from the six Qwen3-8B TP4 → 4 × TP1 moves on node 1, which move 12.3 GB in and out of each GPU in 0.168–0.172 s.

    Held out (33 rows), measured against the model

    Case Rows Measured vs model
    One stage, network only: Qwen3-8B syncs 5 −6.6% … −1.7%
    One stage, network only: Qwen3-32B syncs into TP4 2 −0.4%
    One stage, network only: Qwen3-8B reconfigures 5 −1.5% … +7.6%
    One stage, network only: Qwen3-32B reconfigures 3 +15.7% … +18.3%
    One stage, intra-node only: Qwen3-8B, 4-7:1>0-7:1 from both sources 2 +3.9% … +4.3%
    Relay + pipe + spread, 2 replicas per node 2 −1.3% … +4.0%
    Relay + pipe + spread, 4 replicas per node 3 +12.5% … +26.3%
    Relay + pipe, 2 replicas per node 2 +15.6% … +22.1%
    Relay + pipe, 4 replicas per node 3 +29.0% … +56.9%
    Growing onto the trainer's node from both sources: Qwen3-8B 3 +0.9% … +12.8%
    Growing onto the trainer's node from both sources: Qwen3-32B 3 +52.8% … +178.5%

    What it says:

    • The network term holds where the network is the bottleneck: flat syncs and the 8B reconfigures within 8%. The 8B syncs come in 2–7% under, so part of the 20 ms probably scales with the number of transfers (§3's segment term) rather than being a constant.
    • Relay + spread at two replicas per node lands on the pipelined bound (−1.3% and +4.0%): stage 2 is hidden behind the network stage.
    • What bytes alone don't see:
      1. The last stage-2 round. Relay + pipe and relay + pipe + spread move the same bytes per rank, or nearly (8B into TP2 × 2: 4.10 GB through each NIC and 4.10 GB intra-node in both), yet spread is 1.11–1.58× faster. §3 prices a pipelined relay as the slowest resource's stage 1 plus the last stage-2 round, and without spread a large weight crosses stage 2 whole. Pricing that needs the largest block per stage-2 bucket. Could the CSV get that column, or the bucket size?
      2. The Qwen3-32B reconfigures are 16–18% over while the 8B ones match, though both send to one peer per NIC (one transfer per weight and rank). Is that the one-peer pattern at large message sizes, as in your NCCL test? The flat one is also the row whose digest failed.
      3. Growing a replica onto the trainer's GPUs. Most of each new shard is copied from the trainer's partition on the same GPU, and the CSV has no local bytes or region counts, so §3's per-region term can't price it yet. Per-rank local bytes and regions would close this.

    The choice, which is what the policy needs: in each of the 14 groups (model, operation, layout, source) with bytes to move, does the cheapest predicted plan match the fastest measured one?

    • Where the plans move different bytes, yes. With two or more replicas per node, flat is 1.9–3.3× slower than the best relay, and the model picks a relay in every such group.
    • Where they move the same bytes, the model ties. Breaking ties toward spread picks the fastest plan, or one within 1.3% of it, in 13 of 14 groups. The exception is the 32B grow onto the trainer's node (relay 0.172 s, spread 0.197 s, +14.7%). Breaking them toward fewer transfers and no spread instead costs up to 24% (8B into TP2 × 2). So until the last round is priced, the policy should spread whenever it relays.
    • The source matters as much as the plan. Reading from both the old replica and the trainer's partitions, growing onto the trainer's node moves nothing across nodes: 0.035 s against 0.187 s from the replica alone for 8B (5.3×), and 0.172 s against 0.807 s for 32B (4.7×). The policy can only take that option when the planner accepts holders in two layouts (Describe a parameter held in several layouts as one affine map #9052, with [AffineIR] Plan flat shards that keep the parameter's own strides #9070 for column cuts).

    One question on the hardware: 72–82 GB/s per GPU for an all-to-all inside a node is low for H100s on NVLink. Are the four GPUs of a node on NVLink or on PCIe? That decides whether the intra-node resource is per GPU or per pair.

    I'll add the fit to §3 and open question 5 in the body.

  22. Achyuthan-S commented on Nov 14, 2026

    @Achyuthan-S
    ContributorAuthor

    I've updated the body with the last few days of this thread:

  23. LiRunGuo commented on Nov 15, 2026

    @LiRunGuo
    Contributor

    Thanks @Achyuthan-S, this is a useful fit. Before the questions, one thing it found in my measurements rather than in the model.

    Per-step data. Every row also has one profiled move, which I hadn't put in the CSV: for each all_to_all_single, the slowest rank's pack, exchange and unpack times and the largest bytes any rank sent and received. It is now in the gist as two_node_h100_steps.csv (493 steps for the 54 rows). Two caveats. It is one move per row, with a device synchronize around each step, so the steps add up to more than the median (8B into TP1 × 4, relay + pipe: 0.525 s against 0.338 s). And the sent and received maxima can come from different ranks.

    Buckets are 64 weights in parameter order, by count rather than size, and all-to-all k carries stage 1 of bucket k and stage 2 of bucket k − 1.

    1. The last stage-2 round. The gap between relay + pipe and relay + pipe + spread sits in the steps that carry the two largest weights: embed_tokens (bucket 0) and lm_head (the last bucket). For 8B, lm_head is 151936 × 4096 = 1.16 GiB; for 32B, 151936 × 5120 = 1.45 GiB.

    • Without spread, one rank of the node fetches each of them whole and then sends it to the node's other ranks in stage 2.

      • In the final step of 8B into TP1 × 4, one GPU sends 3.85 GiB while each receiver gets 1.51 GiB.
      • With spread, every rank sends and receives 1.14 GiB in that step.
    • Per step, 8B sync into TP1 × 4 (ms):

      • relay + pipe: 102, 102, 55, 27, 28, 27, 60, 74
      • relay + pipe + spread: 46, 25, 32, 27, 28, 28, 22, 11

      The middle steps are the same. The first two differ in the same way as the last two, through embed_tokens: without spread, one NIC per node carries all of it in stage 1, and one GPU fans it out in stage 2.

    So the largest block alone would miss half of it. What the bound needs is each rank's bytes per step, inter- and intra-node. I can rebuild those from the plans on CPU, as I did for the per-rank CSV, with the largest block per step and the local bytes and regions you ask for in 3. Untested: ordering the buckets so that lm_head is not last would hide its stage 2 behind the next bucket's stage 1.

    2. The 32B reconfigures (+16–18%). As far as the profile shows, it is not the one-peer pattern.

    • 32B 4-7:4>0-7:4 from the old replica moves the same 61.03 GiB across nodes as the sync into 4-7:4, one peer per NIC against four.
    • Its all-to-alls 2–12 run at 24.0–24.2 GB/s per rank; the sync's run at 23.8–24.5 GB/s.
    • The whole difference is in the first all-to-all: 230 ms for 1.79 GB, against 75 ms in the sync. 8B shows the same, smaller: 46 ms against 37 ms.

    I think this is how I measure. A reconfigure first builds the new layout on every rank, which allocates the new replica's arena, and then starts the move without a barrier. The move time I report is the slowest rank's. So a rank that finishes its allocation first waits inside the first all-to-all for the others, and that wait is counted as move time. Syncs have no layout step and show no such first step. I haven't verified it; the next run will time the move after a barrier and record when each rank enters it.

    On the digest: that run recorded only pass or fail. The failures I dumped later on the same moves were all the misplaced 128 KiB block from my earlier comment, which reproduces with the bare collective, with no timing involved; it's now NVIDIA/nccl#2608. So I don't connect it to the time.

    3. Growing onto the trainer's node. The profile points the same way, more strongly:

    • 32B: the first all-to-all takes 137–167 ms. The other ten take 6–7 ms each with relay or spread (0.37–0.39 GB received per step), and pack, which includes the local copies, takes 2 ms per step.
    • 8B: 15–52 ms for the first, then 4 ms per step.

    So most of the +53% to +179% is probably the same wait, not the local copies. I'll add per-rank local bytes and regions with the rebuild in 1, so that the per-region term can be fitted once the first step is cleaned up.

    4. NVLink or PCIe. The GPUs are H100 SXM parts (94 GB), so NVLink by form factor. But I haven't checked nvidia-smi topo -m or which transport NCCL picks for the intra-node connections; the NCCL log I have covers only the inter-node ones. In the six TP4 → TP1 × 4 rows on node 1, steps 3–7 run at 80–82 GB/s per GPU in each direction, matching your 82 GB/s. Steps 1–2 run at about 50 GB/s. 80 GB/s is about 27 GB/s per GPU pair, which is low for NVLink. So I'll check the topology and the transport, and measure an intra-node all-to-all by itself, before the intra-node term is read as the link's capacity.

    To sum up for §3: the network term and the spread rows stand. The 32B reconfigure and grow rows should be read with the first-step wait in mind until I rerun them, and the stage-2 term needs per-step bytes, which I'll add.

  24. Achyuthan-S commented on Nov 15, 2026

    @Achyuthan-S
    ContributorAuthor

    Thanks @LiRunGuo, the per-step file settles most of it. Your numbers match two_node_h100_steps.csv: the 8B TP1 × 4 steps, the last step's 3.85 GiB sent against 1.51 GiB received without spread (1.14 GiB both ways with it), and the 230 ms against 75 ms first all-to-all.

    The first-step wait. For the three 32B reconfigures from the old replica, the first all-to-all is 156–166 ms longer than its bytes take at the later steps' rate. Without that, they sit 4–7% under the network term, the same range as the 8B syncs, so your explanation fits them. For the grows from both sources this file can't confirm it: it has one profiled move per row, and for relay + pipe that move's first-step excess (161 ms) is larger than the whole gap between the median and the model (101 ms). So those need the rerun with the barrier.

    Stage 2. Agreed that the largest block alone misses it. With each rank's inter- and intra-node bytes per all-to-all, the pipelined term becomes a sum over the all-to-alls of the slowest rank's step, each priced as the larger of its network and intra-node parts. With even buckets, where stage 1 is the slower part of each step, that reduces to §3's stage 1 plus the last stage-2 round, and embed_tokens and lm_head are exactly where the buckets aren't even. The same sum would also price your bucket-ordering idea before it is run. I'll fit it to the medians once the rebuild is in, rather than to the profiled step times, which the synchronizes inflate.

    I've updated §3 and open question 5 to read the reconfigure and grow rows with the first-step wait, and linked NVIDIA/nccl#2608 from §1.

  25. jinyouzhi commented on Nov 15, 2026

    @jinyouzhi
    Contributor

    @Achyuthan-S @0z5a, results for open questions 2, 4, 6 and 7, all on one 8× L20 node (PCIe), with the code of jinyouzhi#8 (a8c34e3). The workload is the same Qwen3-4B rebuild as before (2-D weights split by rows → split by columns), using prepared, warm runs.

    Question 4: a fourth N

    I reran N = 2 to 8. N = 2, 4 and 8 came back within 1–2 ms of the earlier numbers.

    N GB sent per process regions per process, 8 / 64 MiB run at 8 MiB, ms (host + wait) run at 64 MiB, ms (host + wait)
    2 2.01 1442 / 1055 107.0 (48 + 59) 102.9 (18 + 85)
    3 1.79 1750 / 1705 97.9 (56 + 42) 96.2 (60 + 36)
    4 1.51 2241 / 2206 82.8 (55 + 28) 82.3 (54 + 28)
    5 1.29 2739 / 2712 78.2 (60 + 18) 80.4 (67 + 13)
    6 1.12 3240 / 3218 74.4 (62 + 13) 77.1 (71 + 6)
    7 0.99 3737 / 3724 68.3 (57 + 12) 71.3 (66 + 5)
    8 0.88 4245 / 4230 64.2 (64 + 0) 81.8 (82 + 0)

    How to read the table:

    • "Regions" is the maximum over ranks of receive + send + local regions.
    • "Host" runs from the start of run() to its final stream synchronize. "Wait" is that synchronize. Both are taken on the slowest rank.

    At 8 MiB, the per-region constant holds. I fitted your bytes + regions model on N = 2, 4 and 8 only. It predicts the new N = 3, 5, 6 and 7 within −5.4% to +2.7%. The per-region constant is 4.8 µs (5.2 µs when fitted on all seven points); you had 4.7 µs.

    At 64 MiB, the larger constant is not a per-region cost.

    • From N = 3 on, both budgets have the same region counts within 2%. So the 8.6 µs that the same fit gives at 64 MiB cannot come from regions.
    • One fit for both budgets, with one bandwidth and one per-region constant (21.0 GB/s, 6.6 µs), puts 12 of the 14 points within 5%. Both misses are at N = 8: +8.5% at 8 MiB and −10.6% at 64 MiB.

    The run is closer to the larger of host time and device time than to their sum.

    • At N = 2 the device is the limit: 59–85 ms of the run is the final wait.
    • From N = 5 on, the host is the limit: most of the run is the per-region copy_ launches in _run.
    • At N = 8 the device has already finished when the host reaches the final synchronize.

    So in the model the per-region term is a host launch cost that overlaps the bytes term, not one added to it. The fused packs below confirm this.

    On B70 the same sweep fails at N = 3 inside oneCCL's allreduce ("Non-uniform work-groups are not supported by the target device"), so these rows are L20 only.

    Question 2: cached views or a sender-ordered buffer

    I tried both on top of the same prepared handle. The exchange is the same in all three variants (same rounds, same batch_isend_irecv); only the copies change:

    • base: _run as in add allreduce test #8, which rebuilds each region's view and issues one copy_ per region.
    • views: the region views are built once at prepare time; still one copy_ per region.
    • fused: at prepare time, each round's pack, each round's unpack and the local copies each become one table of (address, row stride, rows, columns) tiles. At run time, one Triton kernel copies a whole table. This is the "gather into a sender-ordered buffer, then assemble" option: the arena is already in sender order.

    Every variant was checked bit-exact against the source. Medians of 5 warm runs, in ms:

    N base, 8 MiB views, 8 MiB fused, 8 MiB base, 64 MiB views, 64 MiB fused, 64 MiB
    2 107.2 107.2 105.6 103.2 103.1 101.6
    3 98.0 97.9 95.5 96.6 96.8 93.4
    4 82.9 82.5 78.0 82.4 82.2 78.7
    5 77.9 78.0 72.6 79.9 79.5 73.8
    6 74.4 74.2 66.6 76.7 74.5 66.9
    7 68.4 68.1 60.6 71.1 66.3 58.1
    8 64.2 62.4 53.4 78.1 63.6 53.5

    What this shows:

    • Cached views help little, with one exception. They change the run by less than 1% up to N = 6 at 8 MiB. The exception is N = 8 at 64 MiB, where they bring the run from 78 to 64 ms. So the N = 8 / 64 MiB outlier in question 4 is the Python work of rebuilding about 2k views per round, which the host does with only 2 rounds to hide it behind.
    • Fused is the fastest at every N and budget. It saves 1.5% at N = 2 and 17–32% at N = 8. With fused copies, the run is about 15–18 ms plus the bytes at 23 GB/s per process. That form fits all 14 points within ±5.5%; adding a per-round term makes the fit worse.
    • The per-region cost moves to prepare. Building the tables takes 70–230 ms (Python, not optimized), paid once per prepared handle.

    So my answer to question 2: gather into the sender-ordered arena and assemble with one copy kernel per round, prepared once. In the model, the regions term then leaves t_run and joins t_prepare.

    Caveats on the kernel:

    • It is a prototype outside DeepSpeed (CUDA via Triton).
    • XPU would need the Triton XPU backend or a SYCL kernel.
    • Views it cannot describe as rows of contiguous columns fall back to copy_. There were none in this workload.
    • Long contiguous regions must be cut into many tiles. In my first version, each region was one program, and fused was 30–55% slower than base at N ≤ 4.

    Question 6: NCCL M2N against prepared transfers

    M2N needs disjoint groups, so this is 4 → 4: ranks 0–3 hold the row split and ranks 4–7 receive the column split.

    • Every Qwen3-4B weight is moved, 8.04 GB in total. 1-D weights are split evenly on both sides.
    • Setup: nccl-extensions 0.2.0 (nccl.m2n), the same NCCL 2.30.7 that torch uses, and one persistent Handle.
    • M2N runs as one m2n.reshard per tensor. The prepared runs use TransferEndpoints(source={i: i}, target={i: 4 + i}).
    • All results are bit-exact. Medians of 5 warm runs:
    full model, ms GB/s (model bytes / time) embedding only (778 MB), ms
    M2N PACK, one reshard per tensor 259.1 31.0 25.3
    prepared #8809, 8 MiB 113.9 70.6 11.3
    prepared + fused, 8 MiB 112.4 71.6 11.1
    prepared #8809, 64 MiB 119.8 67.1 12.2
    prepared + fused, 64 MiB 115.7 69.5 10.7

    Observations:

    • The gap is bandwidth, not per-call overhead. On the single 778 MB embedding, M2N still reaches only 31 GB/s.
    • DIRECT and PIPE did not run on this node. Every rank fails in ncclDevCommCreate ("User requested GIN but not all ranks in the communicator support GIN"), so these modes need GPU-initiated networking.
    • The Group API failed. Wrapping the 398 calls in m2n.group_start() / group_end() fails with the same ncclDevCommCreate error: with PACK at entry 394 of 398, and with DIRECT or PIPE at entry 0.
    • M2N's first call costs about 850 ms. Prepared [checkpoint] Execute affine transfer plans with bounded peer staging #8809 pays its setup in prepare instead.

    These numbers are for PCIe without NVLink. They do not contradict @0z5a's B300 row, where M2N PACK beat #8809 at 128 MiB. That #8809 row was not prepared, and the fabric differs.

    On whether a plan's segments can be handed to M2N: not as segments. M2N takes a whole tensor per call, described by a mesh and Shard/Replicate placements, with equal tiles. So it fits as a whole-request path: the policy chooses it when both maps can be written as M2N placements and the groups are disjoint, and calibrates it like any other path.

    Question 7: the transport declaration

    From the three paths that exist today (staged in #8809, symmetric-memory windows in jinyouzhi#7, NCCL put/signal in jinyouzhi#9), I think three properties per path are enough. Each is declared for a pair of processes (same host, or across hosts), not for a device.

    1. What one transfer can address on each side:

      • packed: one contiguous run per peer per message;
      • regions: a list of (offset, length) or strided regions in registered memory, as with RDMA scatter/gather;
      • mapped: peer memory that this process can load from and store to directly.

      Packed is why staging exists. Mapped needs no message at all.

    2. Who moves the bytes:

      • two-sided: both ends post, and the receives are armed first;
      • one-sided: one end moves the bytes and signals the other, as in pull, push or put.

      Completion is either stream-ordered signals or a group fence. Put window transfers across hosts with NCCL one-sided puts jinyouzhi/DeepSpeed#9 uses signals on CUDA and XPU and barriers elsewhere.

    3. Which memory and registration it needs:

      • device or pinned host memory (host staging is the device → host case);
      • whether the buffers must lie in a registered window, which is collective, has fixed addresses and outlives the prepared handle.

    The three existing paths in these terms:

    The executor would pick each operation's path per pair from these properties. jinyouzhi#9 already does this in _mapped_direction: an operation between hosts is a put only if both of its regions are packed and in windows, and otherwise it is staged. The planner sees only the calibrated cost of each path, with prepare and run priced separately. Capabilities are found by probing: whether a symmetric-memory rendezvous succeeds, whether peers can access each other's memory, and whether the backend supports put/signal.

    Hardware and software: 8× NVIDIA L20 (PCIe, no NVLink), torch 2.14.1+cu130, NCCL 2.30.7, Triton 3.8.0, nccl-extensions 0.2.0. Medians of 5 warm runs; 7 for the host and wait split. AI assistance was used to write the benchmark and kernel scripts and to draft this comment.

  26. Achyuthan-S commented on Nov 15, 2026

    @Achyuthan-S
    ContributorAuthor

    Thanks @jinyouzhi, this settles most of four open questions at once. I refit your tables and get the same picture: bytes plus regions, fitted on N = 2, 4 and 8 at 8 MiB, gives 4.8 µs per region and predicts N = 3, 5, 6 and 7 within 5.5%. The fused runs fit 15–18 ms plus about 23 GB/s per process within 5.2% at each budget.

    I've updated the body:

    For the policy, this makes t_prepare / n + t_run concrete. At N = 8, fused saves 11–25 ms per run, so even a 230 ms table build is paid back within 22 runs. At N = 2 it saves 1.6 ms, so it takes at least 44 runs. A weight sync that reuses one prepared handle every step clears both.

    @0z5a, @jinyouzhi: should the fused copy go in with the prepared transfers (jinyouzhi#8), or after them?

  27. 0z5a commented on Nov 16, 2026

    @0z5a
    Contributor

    Open question 1. @LiRunGuo, agreed on _validate_aliases: it's #9093. The whole-buffer check stays as the fast path. Only when buffers overlap does the executor check the regions each process actually reads and writes. A local copy whose source and target are the same memory at the same strides is dropped and counted as read, so another write into it is still refused. On CPU at 2 and 4 processes, stage two of a relay, with each rank's band a view of its own full copy, now completes exactly with no local copy; master refuses it. A band viewed at a peer's rows is still refused on every rank before data moves.

    On relay_map, I'd take your formulation as the definition. Each relay shard is the band of its holder's own destination, so stage two never needs a partial target, and the flag this question asks about isn't needed. min_split_bytes then only decides which weights are relayed at all.

    Fused copy. @Achyuthan-S, I'd land it after the prepared transfers, as its own PR. Three reasons:

    • §5 asks each PR to be useful alone. Prepared staged transfers already move validation, agreement, arenas and views out of the run. On @jinyouzhi's L20 numbers, up to N = 5 at 8 MiB, fused copies save 1.5–6.8% of the prepared run; the kernel's gain is at larger N.
    • Coverage. The kernel is Triton on CUDA today. XPU needs either the Triton XPU backend or a SYCL kernel, and views it can't tile need copy_ anyway. With copy_ as the default path and the kernel as an opt-in replacement for it, the default stays testable on CPU CI.
    • Prepare cost. The table build is 70–230 ms in unoptimized Python, against 11–25 ms saved per run at N = 8. It pays back for a weight sync that reuses one handle, but it is a separate cost to price and optimize.

    To make that cheap later, PR 1 of §5 will keep each round's pack and unpack regions in one place on the prepared handle, so that a per-round table can replace them without changing prepare/run. @jinyouzhi, if you'd rather carry the kernel, it would sit on top of that.

  28. jinyouzhi commented on Nov 17, 2026

    @jinyouzhi
    Contributor

    @0z5a thanks, agreed on the order. I'll carry the fused copy as its own PR on top of #9094. copy_ stays the default; the kernel is an opt-in replacement for a round's copies, Triton on CUDA first, with XPU (Triton XPU backend, or SYCL where needed) from our side.

    One question on the window transports (jinyouzhi#7 symmetric-memory window, jinyouzhi#9 put/signal): should the fused copy be done together with them?

    • For doing them together: with a mapped peer window, a round's table can point its destinations into the peer's window. One kernel then gathers the regions and writes them straight to the peer, so packing and sending become one step. The table format should allow that from the start.
    • For landing them separately: fused copy is useful alone on the staged path (§5). The window needs its own probing, memory registration and a staging lifecycle (release_staging()).

    My proposal is one table format designed for both, with the fused copy PR first and the window transport next on the same handle. Does that fit your plan for §5, or would you rather see them in one PR?

    I'm reviewing #9094 now and will post there with L20 and B70 numbers.

  29. 0z5a commented on Nov 17, 2026

    @0z5a
    Contributor

    PR 1 of §5 is open: #9094, staged prepare/run on #9027, with the structure of jinyouzhi#8 and @jinyouzhi as co-author. As said in my earlier comment, the handle keeps each round's packs, unpacks and local copies as lists of (destination, source) views, so a fused copy can replace a list without changing prepare_transfers or run. release_staging() and fused copies are not in it.

  30. 0z5a commented on Nov 17, 2026

    @0z5a
    Contributor

    @jinyouzhi, together works for §5: one PR on top of #9094 with the fused copy and the window transport, and one table format for both from the start, so a round's table can point its destinations into either the staging arena or a mapped peer window. copy_ stays the default, as you said, and #9094's per-round view lists are what the fused copy replaces.

  31. jinyouzhi commented on Nov 18, 2026

    @jinyouzhi
    Contributor

    @0z5a I opened #9118 as a draft on top of #9094, as we agreed. Only its last three commits are new:

    • Fused copy. prepare_transfers(..., copier=fused_copies) replaces a round's pack, unpack and local copies with one Triton launch. The same kernel runs on CUDA and XPU. copy_ stays the default.
    • Window transport. With window=TransferWindow(...), shards are copied straight into or out of the peer's symmetric-memory window. These copies use the same copier and table format as the staged lists. The symm_mem calls are wrapped in deepspeed.comm.
    • PreparedTransfer.release_staging(). It frees the arenas and copy tables between runs, and the next run() stages them again. With check=True, a re-staging failure refuses on every process.

    B70 numbers: Qwen3-4B, row split to column split, warm run() median in ms:

    N staging execute run (copy_) run (fused)
    2 8 MiB 190 162.6 171.6
    2 64 MiB 137 123.7 124.2
    4 64 MiB 166 137.6 136.7
    8 8 MiB 196 103.2 99.6
    8 64 MiB 274 109.0 104.7
    • Preparing the handle once ([checkpoint] Prepare an affine transfer once and run it per step #9094) is the main gain: 15–60% over one-shot execute.
    • The fused copy gains a few percent, growing with N: about −4% at N=8 on B70, and −3% to −7% at N=4 on L20. At N=2 with small staging, the rounds hold only a few regions and the extra launch costs more than it saves. That fits keeping copy_ as the default.
    • On B70 the window gains more: at N=2 with 64 MiB staging, pull goes from 123.7 to 90.5 ms and push to 98.3 ms. On L20 it changes little.

    Tests pass on CPU, 4× L20 and 4× B70. Details are in the PR description. I'll mark it ready once #9094 lands, and will rebase if #9094 changes in the meantime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions