Skip to content

[checkpoint] Prepare an affine transfer once and run it per step - #9094

Open
0z5a wants to merge 2 commits into
deepspeedai:masterfrom
0z5a:affine-prepared-staged
Open

0z5a wants to merge 2 commits into
deepspeedai:masterfrom
0z5a:affine-prepared-staged

Conversation

@0z5a

@0z5a 0z5a commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR 1 of the order in #9026 §5: prepared transfers for the staged path. It builds on the schedule from #9027 (merged) and is two commits on master: the change, and the fixes from review.

A weight sync moves the same batch every step. Every execute_transfers call repeats the validation, the alias check, scheduling, the digest agreement, arena allocation and region views before the first byte moves.

Change.

  • prepare_transfers(requests, endpoints, dtype, budget, group) does that work once and returns a PreparedTransfer. It is collective, and it refuses a batch exactly as execute_transfers does.
  • The handle keeps the per-peer arenas that schedule_transfers reports ([checkpoint] Expose the affine transfer schedule as schedule_transfers #9027). It also keeps each round's copies in one place: for every peer and round, one list of (destination, source) views, packs before a send and unpacks after a receive, plus one list of local copies. run only iterates those lists, so a fused copy over a round can later replace a list without changing prepare or run.
  • execute_transfers runs the same preflight and agreement as prepare_transfers, with the same collectives as before, but keeps nothing: it builds each round's views only when the move reaches that round, as master does, so building them overlaps the previous round's transfer. The existing tests pass unchanged.
  • prepare_transfers pauses Python's cyclic garbage collector while it builds the views, and restores its state afterwards, also on a refusal. Tens of thousands of views stay alive together, and the collections that building them triggered made preparing at 8 processes about a third slower. The views hold no reference cycles.

The handle check, on by default. The handle holds views into the buffers, and keeps them allocated until it is dropped: drop it before ZeRO-3 offload_states if that memory should be freed, and prepare again after reload_states. A buffer that moves after prepare, as a ZeRO-3 partition does when reload_states brings it back from an offload, would leave those views writing into the old storage without error. So run checks first:

  • Each process compares its buffers with what it prepared: key, plan, side, location, data_ptr, shape, stride, dtype and device. It checks either the tensors the handle holds, which catches a buffer re-pointed in place (.data = ...), or run(requests) rebuilt from the live state, such as fresh zero_param_storage_shard views.
  • Only the tensors it is shown: a view such as zero_param_storage_shard returns keeps the old storage when reload_states re-points the partition, so a handle over such views needs run(requests) rebuilt from the live state. The docs' loop does that.
  • One all-gather of [identity, run count, moved] then refuses on every process, before any data moves, if a buffer moved on any process, or if the processes are not running the same handle for the same time. The identity is a hash of the schedule digest and each process's count of prepare_transfers calls, gathered at prepare, so two handles prepared for the same batch are still told apart (@jinyouzhi's scheme).
  • run(check=False) skips that collective, so every process must pass the same check.

The structure of prepare_transfers/run and the buffer layout check follow @jinyouzhi's jinyouzhi#8 (a8c34e3). This is its staged half, without the window transport. Two differences:

Not in this PR: release_staging() (freeing the arenas between runs) and fused copies, which come after it.

Tests (test_affine_transfer_executor.py, 13 new):

  • At 2 and 4 processes, with packed, strided and in-place rounds (the _mixed_batch of the schedule tests): three runs of one handle track sources changed in place, exactly, with the same stats as execute_transfers. The guard elements around every buffer stay untouched.
  • At 2 and 4 processes: a target re-pointed in place on one process makes every process refuse, and no target changes. Requests rebuilt as fresh views of the same storage run. A source copied to new storage on one process is refused, and so is a batch that isn't the prepared one.
  • Two processes running different handles are refused, and both handles still run afterwards. So are two handles prepared alike over different buffers, one run on each process: no target is written, and each handle then moves its own batch. Without the prepare count in the identity, this case no longer passes.
  • In one process: with check=False, a re-pointed target keeps receiving into its old storage, which is why the check is the default. Frozen parameters keep their identity and storage over repeated runs. prepare_transfers leaves the garbage collector enabled or disabled as it found it, after a refusal too (here, a budget too small for one element). A target held as a zero_param_storage_shard-style view and re-pointed as reload_states does is refused by run(requests) with fresh views, and preparing again lands in the new storage; checking the held tensors instead fails this case.
  • With the check removed, the single-process case fails, and the multi-process re-point case no longer passes.
  • Outside tests/unit/v1, PR CI runs single-device only, so the multi-process cases run in the nightly multi-process job. The single-process cases run on PRs.

On CPU (gloo, up to 4 processes, torch 2.15), test_affine_*.py, test_autoep_affine*.py, test_zero_affine.py and v1/moe/test_autoep_checkpoint.py pass: 768 passed, 4 skipped (755 on master). yapf and flake8, at the repo's pinned versions, are clean.

Warm calls on CPU. These use Qwen3-4B's 398 tensors with both dimensions divided by 16 (31 MB in bf16), from a row split to a column split, over gloo on one CPU host, at 64 MiB per peer (one round). Each entry is rank 0's median (min) over 11 calls, in ms. The weights are random bf16 bit patterns, and every call was compared bitwise and exact. These numbers show the host work run no longer repeats; device transfers are not measured here.

N chunks execute_transfers, master execute_transfers, 156c1a8 prepare_transfers (once) run() run(check=False)
2 1,302 19.8 (17.8) 18.8 (18.3) 14.1 4.7 (4.6) 4.4 (4.1)
4 4,628 38.8 (34.7) 37.6 (35.1) 30.7 6.7 (6.0) 5.8 (4.9)
8 17,352 127.8 (120.0) 137.6 (125.7) 127.6 9.2 (8.6) 6.9 (6.2)
  • A warm run() is 4–15× faster than a warm execute_transfers. The check costs 0.3–2.3 ms of that.
  • These were measured on the first commit (156c1a8), before execute_transfers stopped going through a handle; run() is unchanged by the second commit apart from the identity. At N = 8, over three paired runs, 156c1a8's execute_transfers was within −1% to +8% of master's.

On devices (@jinyouzhi, on 156c1a8; median ms, Qwen3-4B as above at full size, bf16). 8× L20 (PCIe, no NVLink), using 4, torch 2.14.1+cu130, NCCL 2.30.7; ranges are across three launches:

N budget master execute_transfers 156c1a8 execute_transfers prepare run() run(check=False)
2 8 MiB 144–146 157–158 (one launch 184) 48 109–110 108
2 64 MiB 132–133 142–143 38 105 104
4 8 MiB 143–153 161–162 77 84 83
4 64 MiB 144–152 158–159 75 84 82

Intel Arc Pro B70, 2 cards, oneCCL, torch 2.14.0.dev (XPU), median of 7:

N budget master execute_transfers 156c1a8 execute_transfers prepare run() run(check=False)
2 8 MiB 186 183 22 161 160
2 64 MiB 137 141 18 124 123
  • A warm run() is 24–45% below master's execute_transfers on the L20 and 10–14% below it on the B70. The check costs about 1.5 ms on the L20 and 0.5 ms on the B70.
  • The one-shot slowdown on the L20 (10–16 ms) came from building every round's views before the first byte; the second commit builds them per round again, as master does. The one-shot column needs a rerun on this head.

Conflicts: #9093 changes the same preflight lines (the alias check and the local copies taken from the schedule). Whichever lands second moves them into prepare_transfers.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 16, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T12:25:02.278976Z 156c1a8 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a7b6b345a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

moved = 1
if self._world_size > 1:
# One process raising alone would leave its peers waiting in the transfer.
local = torch.tensor([self._digest, self._runs, moved], dtype=torch.int64, device=self._device)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Include a preparation identity in the run agreement

When ranks hold two separately prepared handles for the same keys, plans, and endpoints but different backing tensors, both handles have the same _digest; if each rank selects a different handle at the same _runs, this all-gather sees identical triples and allows the transfer. The peers then communicate different batches and can write one handle's payload into another handle's targets instead of raising as the documented same-handle check promises, so the agreement needs an identity for the particular prepare_transfers invocation.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 57476e8: the handle's identity is now a hash of the schedule digest and each process's count of prepare_transfers calls, gathered at prepare, and run's check compares that instead of the digest. test_handles_prepared_alike_are_told_apart covers it.

Comment on lines +301 to +302
recvs = [staged for receives, _ in prepared._rounds for staged in receives]
sends = [staged for _, sending in prepared._rounds for staged in sending]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid asserting the private prepared-round layout

This test derives arena allocation by walking PreparedTransfer._rounds and _StagedRound internals, so a correct implementation that changes how prepared rounds or arenas are represented will fail even when the public transfer and schedule contracts are unchanged. Assert observable behavior through a public inspection API (or avoid this storage-layout assertion) instead.

AGENTS.md reference: AGENTS.md:L76-L78

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 57476e8: the test no longer reads _rounds; rounds and bytes are checked against TransferStats in the schedule test.

A weight sync moves the same batch every step, and every
execute_transfers call repeats validation, the alias check, scheduling,
the digest agreement, arena allocation and region views before the
first byte moves.

prepare_transfers does that work once and returns a PreparedTransfer
whose run only moves data. The handle keeps the arenas that deepspeedai#9027's
schedule reports, and for each round one list of (destination, source)
views: the packs before a send and the unpacks after a receive, plus
one list of local copies. execute_transfers is now prepare_transfers
followed by one run, with the same collectives as before. Python's
cyclic collector is paused while the views are built: tens of
thousands stay alive together, and the collections their allocation
triggered made a one-shot call at 8 processes on CPU about a third
slower than master.

A buffer that moves after prepare, as reload_states moves offloaded
ZeRO-3 partitions, would leave those views writing into the old storage. So
run checks by default: each process compares its buffers (the held
tensors, or requests rebuilt from the live state) with what it
prepared, and one small all-gather refuses on every process, before
data moves, if a buffer moved on any of them or the processes are not
running the same handle for the same time.

The shape of prepare_transfers and the buffer layout check follow
jinyouzhi#8 (a8c34e3); this is its staged half, without the
window transport.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-authored-by: Jin, Youzhi <youzhi.jin@intel.com>
@0z5a
0z5a force-pushed the affine-prepared-staged branch from 7a7b6b3 to 156c1a8 Compare November 16, 2026 10:23
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 16, 2026
@0z5a
0z5a marked this pull request as draft November 16, 2026 10:25

@Achyuthan-S Achyuthan-S left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @0z5a. On CPU (macOS, gloo, up to 4 processes, torch 2.8.0), the suite you list gives 766 passed, 4 skipped, against 755 on master (e35282b). run keeps the old order (packs, the batched sends and receives, the bounded drain, unpacks, then the local copies), so execute_transfers as prepare_transfers(...).run(check=False) is the same transfer. With the check removed, test_a_prepared_transfer_writes_where_it_was_prepared fails and the multi-process re-point cases no longer pass.

One case the default check doesn't see, and it's the one the class docstring names. zero_param_storage_shard returns a new view, param.ds_tensor.data.view(-1).narrow(...), and offload_states / reload_states re-point ds_tensor with .data = .... The view the handle holds keeps the old storage, so its data_ptr doesn't change, and run() passes the check and moves the old storage. Your single-process test, with the target held as such a view:

map_ = replicated_map((2, 3), 1)
plan = plan_transfer(map_, map_)
endpoints = TransferEndpoints({0: 0}, {0: 0})
source = torch.arange(6, dtype=torch.float32).reshape(2, 3)
ds_tensor = torch.full((6, ), -1.0)


def shard():  # as zero_param_storage_shard
    return ds_tensor.data.view(-1).narrow(0, 0, 6)


prepared = prepare_transfers([TransferRequest('weight', plan, {0: source}, {0: shard()})], endpoints, torch.float32,
                             StagingBudget(8))
ds_tensor.data = torch.full((6, ), -2.0)  # as reload_states
prepared.run()  # not refused: writes the old storage, ds_tensor stays -2
prepared.run([TransferRequest('weight', plan, {0: source}, {0: shard()})])  # refused

So for ZeRO-3 buffers it is run(requests) with fresh views that catches it, which the docs mention as an option. A trainer that offloads its partitions for rollout and reloads them, then calls prepared.run() as in the docs' loop, would keep sending the weights from before the offload. And because the handle holds those views, the old partitions stay allocated while it lives, so offload_states doesn't free them.

Two small changes would cover it, I think:

  • In the docs and the class docstring, say that the default check sees only the tensors the handle holds: for views such as zero_param_storage_shard, pass run(requests) rebuilt from the live state, and drop the handle before offload_states if that memory should be freed.
  • The case above as a test, so it stays pinned.

Another option is for prepare_transfers to take a callable that rebuilds the requests, so that run() checks the live state by default. Happy either way.

@jinyouzhi jinyouzhi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @0z5a. I went through this against #8 (a8c34e3). The structure is the one we discussed: per-round view lists, a handle check that is on by default, and execute_transfers as prepare + one run. Keeping the per-peer arenas that schedule_transfers reports and leaving the single allocation per direction for later is fine with me. Three points, then device numbers.

1. Codex's P1 is real. I reproduced it with 2 processes on gloo. Two handles prepared for the same key and plan, over different buffers (a filled with 1, b filled with 2). Rank 0 runs a's handle and rank 1 runs b's. Both have the same digest and _runs == 0, so the check passes. Rank 0's a.target then holds [1, 2], half from each batch, and nothing raises. A cheap fix that needs no extra collective:

  • Each process counts its own prepare_transfers calls.
  • That counter is added as a third column of the prepare all-gather.
  • The handle's identity is a hash of the gathered column, which every rank sees identically.
  • _check compares that identity instead of _digest.

Two different prepare calls then never share an identity, even when every plan is the same.

2. One-shot execute_transfers on a device. It now stores a view of every chunk before moving anything. On the L20 that is slower than master, beyond the −1% to +8% you measured on CPU (median ms, Qwen3-4B as in your table, full size, bf16, three launches of each, two of them interleaved; ranges are across launches):

N budget master execute_transfers this PR execute_transfers prepare run() run(check=False)
2 8 MiB 144–146 157–158 (one launch 184) 48 109–110 108
2 64 MiB 132–133 142–143 38 105 104
4 8 MiB 143–153 161–162 77 84 83
4 64 MiB 144–152 158–159 75 84 82

One run of each at N = 2 on one B70 host (oneCCL), median of 7:

N budget master execute_transfers this PR execute_transfers prepare run() run(check=False)
2 8 MiB 186 183 22 161 160
2 64 MiB 137 141 18 124 123

A warm run() is 24–45% below master's execute_transfers on the L20 and 10–14% below it on the B70. The check costs about 1.5 ms on the L20 and 0.5 ms on the B70. run(requests) costs the same as run(). Each mode's result was compared bitwise with the source and was exact.

The one-shot cost is 10–16 ms on the L20. If that matters for callers that still use execute_transfers once per step, it could keep building each round's views as it goes rather than storing them all first. Otherwise, could the PR body note the device numbers alongside the CPU ones?

3. The handle keeps its buffers allocated. Its views keep every source and target storage alive. After ZeRO-3 offload_states, the device partitions therefore stay allocated as long as a handle prepared on them is alive. The offload then frees nothing, and run refuses after reload_states, as intended. Could the docs say to drop the handle before offloading, rather than only to prepare again after a refusal? This also bears on the release_staging() follow-up, which would free the arenas but not these buffers.

Tests: test_affine_transfer_executor.py passes on CPU (59 passed, 5 skipped). On the B70 host, the multi-process pytest cases hang in the first p2p test with this machine's oneCCL setup, so on XPU I validated only through the bitwise check in the benchmark above, at N = 2.

Hardware and software:

  • 8× NVIDIA L20 (PCIe, no NVLink), torch 2.14.1+cu130, NCCL 2.30.7, using 4 of the GPUs.
  • Intel Arc Pro B70, torch 2.14.0.dev (XPU), oneCCL, using 2 cards.

AI assistance was used to write the benchmark and reproduction scripts and to draft this review.

@0z5a
0z5a marked this pull request as ready for review November 17, 2026 12:19

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 156c1a89ef

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

gc.enable() if enabled else gc.disable()
prepare_transfers([request], endpoints, torch.float32, StagingBudget(8))
assert gc.isenabled() == enabled
monkeypatch.setattr(executor, '_buffer_layout', refuse)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop patching the executor's private layout helper

If preparation is refactored to build or validate its views without _buffer_layout, this test fails at monkeypatch.setattr even when prepare_transfers still preserves the caller's GC state on both success and refusal. Inject the failure through a documented collaborator or public input, or restrict the test to observable GC behavior, rather than coupling it to an internal helper.

AGENTS.md reference: AGENTS.md:L80-L80

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 57476e8: the test now refuses through a budget too small for one element, a public input, instead of patching _buffer_layout.

…they run

Two handles prepared for the same batch over different buffers had the
same schedule digest, so run's check passed when each process ran a
different one, and the peers exchanged payloads of different batches.
Each process now counts its prepare_transfers calls and adds the count
to the prepare all-gather. A handle's identity is a hash of the digest
and the gathered counts, the same on every process, and the check
compares that instead of the digest. The scheme is jinyouzhi's, from
the review.

execute_transfers no longer goes through a handle. It runs the same
preflight and agreement, then builds each round's views when the move
reaches that round, as it did before prepare_transfers, so building
them overlaps the previous round's transfer. On L20 GPUs, building them
all first made a one-shot call 10-16 ms slower than master.

The docs and the handle's docstring now say that run() checks only the
tensors the handle holds: a view such as zero_param_storage_shard
returns keeps the old storage when reload_states re-points the
partition, so such handles need run(requests) rebuilt from the live
state. They also say that a handle keeps its buffers allocated until it
is dropped.

The prepared-run test no longer reads the handle's private rounds, and
the garbage-collector test refuses through a too-small budget instead
of patching a private helper.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@gpu-test-labeler gpu-test-labeler Bot removed the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 17, 2026
@0z5a

0z5a commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @Achyuthan-S and @jinyouzhi. I pushed 57476e8 on top of 156c1a8 as its own commit, so only the changes are new.

Handles prepared alike (Codex's P1, @jinyouzhi's reproduction). Done as you proposed:

  • Each process counts its own prepare_transfers calls, and the count is a third column of the prepare all-gather.
  • The handle's identity is a hash of the schedule digest and that gathered column, so every process computes the same value. run's check compares the identity instead of the digest.
  • Two separate prepare calls therefore never match, even for the same batch.
  • New test, at 2 processes: two handles prepared alike over different buffers, one run on each process, are refused, and no target is written. Both handles still run correctly afterwards. Without the count, the test fails.

Views from zero_param_storage_shard (@Achyuthan-S). I took the docs route:

  • The class docstring and the docs now say that run() checks only the tensors the handle holds. A view taken before reload_states keeps the old storage, so a handle over such views needs run(requests) rebuilt from the live state.
  • The loop in the docs now passes the rebuilt requests.
  • Your case is a test. It asserts that the rebuilt requests are refused after the partition is re-pointed, and that preparing again lands in the new storage. It doesn't assert that run() misses the move, so a callable that rebuilds the requests could still be added later without breaking it. Happy to add that callable here if you'd rather have run() check the live state by default.

The handle keeps its buffers allocated (both). The docstring and the docs say to drop the handle before offload_states if that memory should be freed, and to prepare again after reload_states. release_staging() will free only the arenas, and its PR will say the same.

One-shot execute_transfers (@jinyouzhi).

  • It no longer goes through a handle. It runs the same preflight and agreement, then builds each round's views only when the move reaches that round, as master did. That way, building them overlaps the previous round's transfer instead of preceding the first byte.
  • prepare_transfers still builds them all up front.
  • On CPU, gloo waits for every round, so the overlap can't show there.
  • Could you rerun the one-shot column on the L20? I've added your L20 and B70 tables to the description, with the one-shot column marked as measured on 156c1a8.

Codex's two P2s. The test no longer reads _rounds; the schedule test still checks the rounds and bytes against TransferStats through the public API. The garbage-collector test now refuses through a budget too small for one element, instead of patching _buffer_layout.

On CPU (gloo, up to 4 processes, torch 2.15), the nine files give 768 passed and 4 skipped (755 on master). yapf and flake8 are clean.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 17, 2026
@jinyouzhi

Copy link
Copy Markdown
Contributor

Reviewed 57476e8 in full (executor, tests, docs). The two-step API is a clean design: prepare_transfers does the validation/scheduling/arena work once, run() only moves bytes, and execute_transfers stays as the one-shot path — with both paths sharing _agree, so the docs' closing guarantee that both build their rounds and arenas from the same computation actually holds in the code. All three threads from the earlier review round are addressed and verified in this diff:

  1. Handle identity (the cross-write P1): module-level _preparations = count(), all-gathered as a third column [digest, rejected, preparation] in _agree and folded into _identity via blake2b. Two handles prepared from identical batches on different processes can no longer run each other's data; run() refuses with processes are not running the same prepared transfer for the same time. test_processes_running_different_handles_are_refused covers the cross-run case (refused before any move, sentinels intact) and test_handles_prepared_alike_are_told_apart covers two identical prepares distinguished only by the counter.
  2. Stale views: _buffer_layout captures (key, plan, side, location, data_ptr(), shape, stride, dtype, device) per buffer, so a .data = ... re-pointing is caught by the moved check (a buffer of the prepared transfer moved or changed since prepare_transfers), and run(requests) accepts rebuilt views for the ZeRO-3 offload_states/reload_states lifecycle. The check=False footgun (writes land in the old storage) is honestly documented and tested with the zero_param_storage_shard-style closure.
  3. GC pressure: _collector_paused scopes the disable/restore to view construction only, and the parameterized test verifies the collector state is restored on both success and failure paths (including the StagingBudget failure).

Lazy per-round view building (_round_views) in the one-shot path also resolves the earlier question about pre-materializing views there — the one-shot path no longer pays the up-front view cost.

Non-blocking nits

  1. A check mismatch across ranks (run() on one, run(check=False) on another) hangs in the collective. It is documented ("all processes must pass the same check") and inherent to skipping the collective; a possible follow-up is having check=False still join the identity all-gather while skipping only the moved-check, if the cost is acceptable.
  2. The perf numbers (CPU warmed run() 4-15x faster than warmed execute_transfers; L20 24-45% lower with check ~1.5 ms; B70 10-14% lower) are self-reported — worth an independent run of the CPU multiprocess suite before merge. The B70 multiprocess hang at the first p2p test looks environmental, but a note in the PR would help the next person who hits it.
  3. Reminder: the multiprocess tests under tests/unit/checkpoint/ do not run in PR CI (nightly multiprocess job only), so worth watching the nightly after merge.

Merge order

This conflicts with #9093 (both edit the same preflight region of execute_transfers, same base). Suggest #9093 lands first (after fixing its conj/neg regression — see that PR; on this branch the _agree preflight still runs master's storage-level _validate_aliases, so conj/neg views are still refused here today), then rebase this PR and move #9093's _buffers_alias fast path + region-level _validate_aliases into _agree so prepare_transfers inherits the relay-band fix, and re-run TestB2RelayBandInPlace through the prepared path too. The reverse order would mean porting the whole alias decision into _agree from the other side — a larger diff.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants