Repository navigation
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a7b6b345a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| moved = 1 | ||
| if self._world_size > 1: | ||
| # One process raising alone would leave its peers waiting in the transfer. | ||
| local = torch.tensor([self._digest, self._runs, moved], dtype=torch.int64, device=self._device) |
There was a problem hiding this comment.
Include a preparation identity in the run agreement
When ranks hold two separately prepared handles for the same keys, plans, and endpoints but different backing tensors, both handles have the same _digest; if each rank selects a different handle at the same _runs, this all-gather sees identical triples and allows the transfer. The peers then communicate different batches and can write one handle's payload into another handle's targets instead of raising as the documented same-handle check promises, so the agreement needs an identity for the particular prepare_transfers invocation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 57476e8: the handle's identity is now a hash of the schedule digest and each process's count of prepare_transfers calls, gathered at prepare, and run's check compares that instead of the digest. test_handles_prepared_alike_are_told_apart covers it.
| recvs = [staged for receives, _ in prepared._rounds for staged in receives] | ||
| sends = [staged for _, sending in prepared._rounds for staged in sending] |
There was a problem hiding this comment.
Avoid asserting the private prepared-round layout
This test derives arena allocation by walking PreparedTransfer._rounds and _StagedRound internals, so a correct implementation that changes how prepared rounds or arenas are represented will fail even when the public transfer and schedule contracts are unchanged. Assert observable behavior through a public inspection API (or avoid this storage-layout assertion) instead.
AGENTS.md reference: AGENTS.md:L76-L78
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 57476e8: the test no longer reads _rounds; rounds and bytes are checked against TransferStats in the schedule test.
A weight sync moves the same batch every step, and every execute_transfers call repeats validation, the alias check, scheduling, the digest agreement, arena allocation and region views before the first byte moves. prepare_transfers does that work once and returns a PreparedTransfer whose run only moves data. The handle keeps the arenas that deepspeedai#9027's schedule reports, and for each round one list of (destination, source) views: the packs before a send and the unpacks after a receive, plus one list of local copies. execute_transfers is now prepare_transfers followed by one run, with the same collectives as before. Python's cyclic collector is paused while the views are built: tens of thousands stay alive together, and the collections their allocation triggered made a one-shot call at 8 processes on CPU about a third slower than master. A buffer that moves after prepare, as reload_states moves offloaded ZeRO-3 partitions, would leave those views writing into the old storage. So run checks by default: each process compares its buffers (the held tensors, or requests rebuilt from the live state) with what it prepared, and one small all-gather refuses on every process, before data moves, if a buffer moved on any of them or the processes are not running the same handle for the same time. The shape of prepare_transfers and the buffer layout check follow jinyouzhi#8 (a8c34e3); this is its staged half, without the window transport. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de> Co-authored-by: Jin, Youzhi <youzhi.jin@intel.com>
7a7b6b3 to
156c1a8
Compare
Achyuthan-S
left a comment
There was a problem hiding this comment.
Thanks @0z5a. On CPU (macOS, gloo, up to 4 processes, torch 2.8.0), the suite you list gives 766 passed, 4 skipped, against 755 on master (e35282b). run keeps the old order (packs, the batched sends and receives, the bounded drain, unpacks, then the local copies), so execute_transfers as prepare_transfers(...).run(check=False) is the same transfer. With the check removed, test_a_prepared_transfer_writes_where_it_was_prepared fails and the multi-process re-point cases no longer pass.
One case the default check doesn't see, and it's the one the class docstring names. zero_param_storage_shard returns a new view, param.ds_tensor.data.view(-1).narrow(...), and offload_states / reload_states re-point ds_tensor with .data = .... The view the handle holds keeps the old storage, so its data_ptr doesn't change, and run() passes the check and moves the old storage. Your single-process test, with the target held as such a view:
map_ = replicated_map((2, 3), 1)
plan = plan_transfer(map_, map_)
endpoints = TransferEndpoints({0: 0}, {0: 0})
source = torch.arange(6, dtype=torch.float32).reshape(2, 3)
ds_tensor = torch.full((6, ), -1.0)
def shard(): # as zero_param_storage_shard
return ds_tensor.data.view(-1).narrow(0, 0, 6)
prepared = prepare_transfers([TransferRequest('weight', plan, {0: source}, {0: shard()})], endpoints, torch.float32,
StagingBudget(8))
ds_tensor.data = torch.full((6, ), -2.0) # as reload_states
prepared.run() # not refused: writes the old storage, ds_tensor stays -2
prepared.run([TransferRequest('weight', plan, {0: source}, {0: shard()})]) # refusedSo for ZeRO-3 buffers it is run(requests) with fresh views that catches it, which the docs mention as an option. A trainer that offloads its partitions for rollout and reloads them, then calls prepared.run() as in the docs' loop, would keep sending the weights from before the offload. And because the handle holds those views, the old partitions stay allocated while it lives, so offload_states doesn't free them.
Two small changes would cover it, I think:
- In the docs and the class docstring, say that the default check sees only the tensors the handle holds: for views such as
zero_param_storage_shard, passrun(requests)rebuilt from the live state, and drop the handle beforeoffload_statesif that memory should be freed. - The case above as a test, so it stays pinned.
Another option is for prepare_transfers to take a callable that rebuilds the requests, so that run() checks the live state by default. Happy either way.
jinyouzhi
left a comment
There was a problem hiding this comment.
Thanks @0z5a. I went through this against #8 (a8c34e3). The structure is the one we discussed: per-round view lists, a handle check that is on by default, and execute_transfers as prepare + one run. Keeping the per-peer arenas that schedule_transfers reports and leaving the single allocation per direction for later is fine with me. Three points, then device numbers.
1. Codex's P1 is real. I reproduced it with 2 processes on gloo. Two handles prepared for the same key and plan, over different buffers (a filled with 1, b filled with 2). Rank 0 runs a's handle and rank 1 runs b's. Both have the same digest and _runs == 0, so the check passes. Rank 0's a.target then holds [1, 2], half from each batch, and nothing raises. A cheap fix that needs no extra collective:
- Each process counts its own
prepare_transferscalls. - That counter is added as a third column of the prepare all-gather.
- The handle's identity is a hash of the gathered column, which every rank sees identically.
_checkcompares that identity instead of_digest.
Two different prepare calls then never share an identity, even when every plan is the same.
2. One-shot execute_transfers on a device. It now stores a view of every chunk before moving anything. On the L20 that is slower than master, beyond the −1% to +8% you measured on CPU (median ms, Qwen3-4B as in your table, full size, bf16, three launches of each, two of them interleaved; ranges are across launches):
| N | budget | master execute_transfers |
this PR execute_transfers |
prepare |
run() |
run(check=False) |
|---|---|---|---|---|---|---|
| 2 | 8 MiB | 144–146 | 157–158 (one launch 184) | 48 | 109–110 | 108 |
| 2 | 64 MiB | 132–133 | 142–143 | 38 | 105 | 104 |
| 4 | 8 MiB | 143–153 | 161–162 | 77 | 84 | 83 |
| 4 | 64 MiB | 144–152 | 158–159 | 75 | 84 | 82 |
One run of each at N = 2 on one B70 host (oneCCL), median of 7:
| N | budget | master execute_transfers |
this PR execute_transfers |
prepare |
run() |
run(check=False) |
|---|---|---|---|---|---|---|
| 2 | 8 MiB | 186 | 183 | 22 | 161 | 160 |
| 2 | 64 MiB | 137 | 141 | 18 | 124 | 123 |
A warm run() is 24–45% below master's execute_transfers on the L20 and 10–14% below it on the B70. The check costs about 1.5 ms on the L20 and 0.5 ms on the B70. run(requests) costs the same as run(). Each mode's result was compared bitwise with the source and was exact.
The one-shot cost is 10–16 ms on the L20. If that matters for callers that still use execute_transfers once per step, it could keep building each round's views as it goes rather than storing them all first. Otherwise, could the PR body note the device numbers alongside the CPU ones?
3. The handle keeps its buffers allocated. Its views keep every source and target storage alive. After ZeRO-3 offload_states, the device partitions therefore stay allocated as long as a handle prepared on them is alive. The offload then frees nothing, and run refuses after reload_states, as intended. Could the docs say to drop the handle before offloading, rather than only to prepare again after a refusal? This also bears on the release_staging() follow-up, which would free the arenas but not these buffers.
Tests: test_affine_transfer_executor.py passes on CPU (59 passed, 5 skipped). On the B70 host, the multi-process pytest cases hang in the first p2p test with this machine's oneCCL setup, so on XPU I validated only through the bitwise check in the benchmark above, at N = 2.
Hardware and software:
- 8× NVIDIA L20 (PCIe, no NVLink), torch 2.14.1+cu130, NCCL 2.30.7, using 4 of the GPUs.
- Intel Arc Pro B70, torch 2.14.0.dev (XPU), oneCCL, using 2 cards.
AI assistance was used to write the benchmark and reproduction scripts and to draft this review.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 156c1a89ef
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| gc.enable() if enabled else gc.disable() | ||
| prepare_transfers([request], endpoints, torch.float32, StagingBudget(8)) | ||
| assert gc.isenabled() == enabled | ||
| monkeypatch.setattr(executor, '_buffer_layout', refuse) |
There was a problem hiding this comment.
Stop patching the executor's private layout helper
If preparation is refactored to build or validate its views without _buffer_layout, this test fails at monkeypatch.setattr even when prepare_transfers still preserves the caller's GC state on both success and refusal. Inject the failure through a documented collaborator or public input, or restrict the test to observable GC behavior, rather than coupling it to an internal helper.
AGENTS.md reference: AGENTS.md:L80-L80
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 57476e8: the test now refuses through a budget too small for one element, a public input, instead of patching _buffer_layout.
…they run Two handles prepared for the same batch over different buffers had the same schedule digest, so run's check passed when each process ran a different one, and the peers exchanged payloads of different batches. Each process now counts its prepare_transfers calls and adds the count to the prepare all-gather. A handle's identity is a hash of the digest and the gathered counts, the same on every process, and the check compares that instead of the digest. The scheme is jinyouzhi's, from the review. execute_transfers no longer goes through a handle. It runs the same preflight and agreement, then builds each round's views when the move reaches that round, as it did before prepare_transfers, so building them overlaps the previous round's transfer. On L20 GPUs, building them all first made a one-shot call 10-16 ms slower than master. The docs and the handle's docstring now say that run() checks only the tensors the handle holds: a view such as zero_param_storage_shard returns keeps the old storage when reload_states re-points the partition, so such handles need run(requests) rebuilt from the live state. They also say that a handle keeps its buffers allocated until it is dropped. The prepared-run test no longer reads the handle's private rounds, and the garbage-collector test refuses through a too-small budget instead of patching a private helper. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
|
Thanks @Achyuthan-S and @jinyouzhi. I pushed 57476e8 on top of 156c1a8 as its own commit, so only the changes are new. Handles prepared alike (Codex's P1, @jinyouzhi's reproduction). Done as you proposed:
Views from
The handle keeps its buffers allocated (both). The docstring and the docs say to drop the handle before One-shot
Codex's two P2s. The test no longer reads On CPU (gloo, up to 4 processes, torch 2.15), the nine files give 768 passed and 4 skipped (755 on master). yapf and flake8 are clean. |
|
Reviewed 57476e8 in full (executor, tests, docs). The two-step API is a clean design:
Lazy per-round view building ( Non-blocking nits
Merge orderThis conflicts with #9093 (both edit the same preflight region of |
PR 1 of the order in #9026 §5: prepared transfers for the staged path. It builds on the schedule from #9027 (merged) and is two commits on master: the change, and the fixes from review.
A weight sync moves the same batch every step. Every
execute_transferscall repeats the validation, the alias check, scheduling, the digest agreement, arena allocation and region views before the first byte moves.Change.
prepare_transfers(requests, endpoints, dtype, budget, group)does that work once and returns aPreparedTransfer. It is collective, and it refuses a batch exactly asexecute_transfersdoes.schedule_transfersreports ([checkpoint] Expose the affine transfer schedule as schedule_transfers #9027). It also keeps each round's copies in one place: for every peer and round, one list of(destination, source)views, packs before a send and unpacks after a receive, plus one list of local copies.runonly iterates those lists, so a fused copy over a round can later replace a list without changingprepareorrun.execute_transfersruns the same preflight and agreement asprepare_transfers, with the same collectives as before, but keeps nothing: it builds each round's views only when the move reaches that round, as master does, so building them overlaps the previous round's transfer. The existing tests pass unchanged.prepare_transferspauses Python's cyclic garbage collector while it builds the views, and restores its state afterwards, also on a refusal. Tens of thousands of views stay alive together, and the collections that building them triggered made preparing at 8 processes about a third slower. The views hold no reference cycles.The handle check, on by default. The handle holds views into the buffers, and keeps them allocated until it is dropped: drop it before ZeRO-3
offload_statesif that memory should be freed, and prepare again afterreload_states. A buffer that moves afterprepare, as a ZeRO-3 partition does whenreload_statesbrings it back from an offload, would leave those views writing into the old storage without error. Sorunchecks first:data_ptr, shape, stride, dtype and device. It checks either the tensors the handle holds, which catches a buffer re-pointed in place (.data = ...), orrun(requests)rebuilt from the live state, such as freshzero_param_storage_shardviews.zero_param_storage_shardreturns keeps the old storage whenreload_statesre-points the partition, so a handle over such views needsrun(requests)rebuilt from the live state. The docs' loop does that.[identity, run count, moved]then refuses on every process, before any data moves, if a buffer moved on any process, or if the processes are not running the same handle for the same time. The identity is a hash of the schedule digest and each process's count ofprepare_transferscalls, gathered at prepare, so two handles prepared for the same batch are still told apart (@jinyouzhi's scheme).run(check=False)skips that collective, so every process must pass the samecheck.The structure of
prepare_transfers/runand the buffer layout check follow @jinyouzhi's jinyouzhi#8 (a8c34e3). This is its staged half, without the window transport. Two differences:schedule_transfersreports them. add allreduce test #8's single allocation per direction can follow by changing_lanesand the schedule together.Not in this PR:
release_staging()(freeing the arenas between runs) and fused copies, which come after it.Tests (
test_affine_transfer_executor.py, 13 new):_mixed_batchof the schedule tests): three runs of one handle track sources changed in place, exactly, with the same stats asexecute_transfers. The guard elements around every buffer stay untouched.check=False, a re-pointed target keeps receiving into its old storage, which is why the check is the default. Frozen parameters keep their identity and storage over repeated runs.prepare_transfersleaves the garbage collector enabled or disabled as it found it, after a refusal too (here, a budget too small for one element). A target held as azero_param_storage_shard-style view and re-pointed asreload_statesdoes is refused byrun(requests)with fresh views, and preparing again lands in the new storage; checking the held tensors instead fails this case.tests/unit/v1, PR CI runs single-device only, so the multi-process cases run in the nightly multi-process job. The single-process cases run on PRs.On CPU (gloo, up to 4 processes, torch 2.15),
test_affine_*.py,test_autoep_affine*.py,test_zero_affine.pyandv1/moe/test_autoep_checkpoint.pypass: 768 passed, 4 skipped (755 on master). yapf and flake8, at the repo's pinned versions, are clean.Warm calls on CPU. These use Qwen3-4B's 398 tensors with both dimensions divided by 16 (31 MB in bf16), from a row split to a column split, over gloo on one CPU host, at 64 MiB per peer (one round). Each entry is rank 0's median (min) over 11 calls, in ms. The weights are random bf16 bit patterns, and every call was compared bitwise and exact. These numbers show the host work
runno longer repeats; device transfers are not measured here.execute_transfers, masterexecute_transfers, 156c1a8prepare_transfers(once)run()run(check=False)run()is 4–15× faster than a warmexecute_transfers. The check costs 0.3–2.3 ms of that.execute_transfersstopped going through a handle;run()is unchanged by the second commit apart from the identity. At N = 8, over three paired runs, 156c1a8'sexecute_transferswas within −1% to +8% of master's.On devices (@jinyouzhi, on 156c1a8; median ms, Qwen3-4B as above at full size, bf16). 8× L20 (PCIe, no NVLink), using 4, torch 2.14.1+cu130, NCCL 2.30.7; ranges are across three launches:
execute_transfersexecute_transferspreparerun()run(check=False)Intel Arc Pro B70, 2 cards, oneCCL, torch 2.14.0.dev (XPU), median of 7:
execute_transfersexecute_transferspreparerun()run(check=False)run()is 24–45% below master'sexecute_transferson the L20 and 10–14% below it on the B70. The check costs about 1.5 ms on the L20 and 0.5 ms on the B70.Conflicts: #9093 changes the same preflight lines (the alias check and the local copies taken from the schedule). Whichever lands second moves them into
prepare_transfers.