Skip to content

[checkpoint] Check aliases on the regions a transfer moves - #9093

Open
0z5a wants to merge 2 commits into
deepspeedai:masterfrom
0z5a:affine-relay-self-alias
Open

0z5a wants to merge 2 commits into
deepspeedai:masterfrom
0z5a:affine-relay-self-alias

Conversation

@0z5a

@0z5a 0z5a commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

For open question 1 in #9026, as @LiRunGuo proposed in #9074 (comment).

In stage two of a relay, a relay rank holds its band as a view of its own destination. It sends the band to its peers and receives the other bands around it, so no element is both read and written. _validate_aliases compares whole buffers, though, so execute_transfers refuses the batch.

Change.

  • The whole-buffer check stays as the fast path. When no target buffer shares storage with a source or another target, nothing changes and nothing extra runs.
  • Otherwise, once the schedule is built, _validate_aliases collects the regions this process reads (sends and local-copy sources) and writes (receives and local-copy targets).
  • A local copy whose source and target are the same memory at the same strides moves nothing. It is dropped, and its region counts as read, so another write into it is still refused.
  • No write may overlap a read or another write. Regions are compared by the bytes they span, so strided regions that interleave without sharing an element are still refused. A band that is a view of its holder's destination is one contiguous run of it, so the relay case doesn't hit that.
  • The check runs before the agreement, as before, so a refusal on any rank stops every rank before data moves.
  • schedule_transfers doesn't see buffers, so it still counts an in-place copy as a local copy, and its docstring says so. TransferStats.local_copy_bytes does not count it.
  • Conjugate and negative views are refused in _flat, on the process that holds one, before the agreement. Otherwise a lazy view of a destination would pass as in place and be dropped (raised by the Codex review, @LiRunGuo and @jinyouzhi). Point-to-point transport does not carry the lazy bit, on master either. With gloo on CPU, a conj shard fails inside the send after the agreement while its peer waits, and a neg-view target is never written. The docs say to pass such views resolved.
  • Region extents use address_range, as _validate_buffers does.

Tests (14 new, in test_affine_transfer_executor.py):

  • TestB2RelayBandInPlace, at 2 and 4 processes: every rank's band is a view of its own full copy. Each rank completes its copy exactly, copies nothing locally, and sends and receives N − 1 bands. With the view moved to a peer's rows, every rank refuses before data moves, and the cause names the overlap.
  • In one process: a band kept in place next to a band copied in. Refusals, each matched to its message, for swapped bands, for a band kept in place that another request writes into, and for two requests writing the same rows.
  • TestB2LazyViewsRefused, at 2 processes: a conj or neg view of the destination, and one on either side of a cross-process segment. Every process refuses, the holder's cause names the view, and nothing is written. On master (gloo, CPU) the destination cases fail on the old message, the cross-process conj cases hang, and the neg-target case completes with nothing written.
  • On master the 3 acceptance cases are refused. If an in-place copy counted as neither read nor written, the kept-band refusal would fail.

On CPU (gloo, up to 4 processes), test_affine_*.py, test_autoep_affine*.py, test_zero_affine.py and v1/moe/test_autoep_checkpoint.py pass: 970 passed, 4 skipped (956 on master daf175b). yapf and flake8 at the pre-commit pins are clean.

How it was found. By reading _validate_aliases against the stage-two shape of Runguo's relay on the affine IR (#9026, Q1): each relay rank's band is a view of its own destination, so the whole-buffer comparison refuses the batch although no element is both read and written. The conj/neg part came out of review, and the cross-process behaviour on master from probing each side and view on CPU.

Production impact (estimate). Today a relay built on execute_transfers cannot run its second stage at all; with this change it runs, and copies nothing for the band it already holds. Batches whose buffers do not share storage see no change and pay nothing extra. Conj/neg views need a complex tensor viewed through .conj() or an explicit neg view, so we expect them to be rare in checkpoints; where they did occur across processes, master hung or left the target unwritten with gloo on CPU, and they are now refused up front.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T09:40:13.964450Z 735b057 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 16, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 735b057f49

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +294 to +295
in_place = read[0] == write[0] and all(size <= 1 or source == target for size, source, target in zip(
operation.chunk_shape, operation.segment.source_strides, operation.segment.target_strides))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Check lazy view bits before dropping in-place copies

When a local source and target view the same storage with identical strides but different lazy-value metadata, this predicate incorrectly treats the copy as a no-op. For example, with a complex base, using {0: base.conj()} as the source and {0: base} as the target passes _flat (same dtype and contiguous) and has equal pointers/strides, so the operation is deleted and base remains unconjugated even though the transfer should produce the conjugated values or reject the alias. Include is_conj()/is_neg() compatibility in the in-place check to avoid reporting success with incorrect checkpoint contents.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 66bad36: _flat refuses a buffer with is_conj() or is_neg() set, so this copy is refused before the alias check instead of being dropped as in place. TestB2LazyViewsRefused covers it, on the destination and across processes.

@LiRunGuo LiRunGuo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @0z5a, this is the check I had in mind, and the fast path keeps the common case free. A few points:

  1. The lazy-bit case the bot raised is real. On CPU, with base a complex64 tensor, execute_transfer(plan, {0: base.conj()}, {0: base}, ...) with a replicated map is refused on master ("a source and a target overlap within the transfer batch") but completes on this head with base unchanged, since the copy is taken as in place. It needs a complex tensor viewed through .conj(), so it is unlikely in practice. But it turns a refusal into a silent wrong result. Refusing buffers with is_conj() or is_neg() in _flat would close it, and so would comparing the two buffers' bits in the in_place predicate. Either way, a one-line test would pin it.

  2. Our relay fits the byte-span rule. A relay holder's band is a range of rows of its own destination shard, so it is one contiguous run of that buffer, whatever the TP layout. Interleaved regions being refused doesn't affect it.

  3. schedule_transfers and in-place copies. The description says schedule_transfers still counts an in-place copy as a local copy. Could the docstring of schedule_transfers (or ProcessSchedule.local_copy_*) say so? A cost model reading it (#9026) would count, in stage two of a relay, one band per rank that never moves.

  4. Ordering with #9094. Both PRs change execute_transfers. Whichever lands second will need the in-place decision made in prepare. It depends on data_ptr, and #9094's buffer check already refuses a run whose buffers moved, so a prepared decision stays valid.

  5. On GPUs. On 2× RTX 3090 with NCCL (torch 2.13.0), test_affine_transfer_executor.py and test_affine_transfer_executor_external_group.py at 735b057 give 57 passed and 5 skipped (the four 4-process cases need 4 GPUs; the external-group case is CPU-only). The same tests on master (e35282b) give 3 failures:

    • TestB2RelayBandInPlace[False-2] and test_a_band_kept_in_place_is_not_copied, as you describe;
    • second_writer, which master refuses with "a source and a target overlap" instead of the expected message.

    I didn't run the 4-process cases on GPUs.

@Achyuthan-S Achyuthan-S left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @0z5a, this reads right to me. On CPU (macOS, gloo, up to 4 processes, torch 2.8.0), the suite you list gives 763 passed, 4 skipped, against 755 on master (e35282b).

Reading _validate_aliases: once the writes are sorted by start, comparing neighbours catches every pair of overlapping writes. With the writes disjoint, the first write that ends after a read starts is the only one that needs checking, so the bisect is exact for byte spans. A refusal on one process still goes through the [digest, rejected] all-gather, so every process stops before data moves.

Each part is covered by the new tests: with in-place copies never recognised, 4 of the 8 fail; with an in-place region not counted as read, one fails; without the read/write check, the two single-process "a source and a target overlap" cases fail; without the write/write check, second_writer fails.

@Achyuthan-S

Copy link
Copy Markdown
Contributor

A correction to my note above: @LiRunGuo's point 1 is real, and I missed it. On CPU at 735b057, with base a complex64 tensor and a replicated map, execute_transfer from base.conj() into base completes and leaves base unchanged, and a float32 copy from torch._neg_view(target) into target does the same; master refuses both. With _flat refusing a buffer whose is_conj() or is_neg() is set, as Runguo suggests, both are refused again, and test_affine_transfer_executor.py and test_affine_transfer_executor_external_group.py still pass (62 passed, up to 4 processes).

@jinyouzhi

Copy link
Copy Markdown
Contributor

Reviewed 735b057 end to end (executor diff + the new TestB2RelayBandInPlace tests). The design is right: keeping master's storage-level check as a _buffers_alias fast path and falling back to region-level reasoning only when a batch actually aliases preserves master's behavior for the common case with identical error messages, and the in-place predicate (read[0] == write[0] plus stride equality wherever the extent is > 1) is a precise way to recognize the relay rank's own band. Write ordering + bisect_right over reads keeps the region check O(n log n). The tests genuinely cover the new behavior — confirmed on 2x3090 NCCL: TestB2RelayBandInPlace[False-2], test_a_band_kept_in_place_is_not_copied, and the second_writer case all fail on master and pass here.

Blocking: conj/neg views are now silently accepted (regression vs master)

Master refuses copies whose source or target is a conj/neg view, but only indirectly: base.conj() shares storage with base, so the storage-level alias check rejects it with a source and a target overlap within the transfer batch. In this branch those batches take the region-level path instead: the view has the same data_ptr and identical strides as the destination, so it hits the in-place predicate, the copy is dropped as a no-op, and the batch completes without moving any data (as reproduced by @Achyuthan-S on CPU):

base = torch.zeros(6, dtype=torch.complex64, device=_device())
# execute_transfers from {0: base.conj()} into {0: base}:
#   master:      AffineTransferExecutionError, cause 'a source and a target overlap ...'
#   735b057f4:   returns stats, base unchanged - the conj copy was treated as in-place
# same for a float32 copy from torch._neg_view(base) into base

Since peer transport moves raw bytes (batch_isend_irecv), conj/neg semantics cannot survive a cross-rank segment anyway, so the consistent fix is to refuse these views up front in _flat (which neither master nor this branch currently checks — master only catches them via the storage overlap):

if buffer.is_conj() or buffer.is_neg():
    raise AffineTransferExecutionError(
        f'{which} location {location}: a conj/neg view cannot be transferred; pass its materialization')

Adding that guard keeps test_affine_transfer_executor.py and the external-group suite green (62 passed, up to 4 processes). Please add both repro cases as tests before merge, in the existing assertion style:

@pytest.mark.parametrize('conj', [False, True])
def test_a_conj_or_neg_view_of_the_destination_is_refused(self, conj):
    _join_transfer_group()
    rank = dist.get_rank()
    world = dist.get_world_size()
    dtype = torch.complex64 if conj else torch.float32
    map_ = replicated_map((6, ), world)
    plan = plan_transfer(map_, map_)
    endpoints = TransferEndpoints({loc: loc for loc in range(world)}, {loc: loc for loc in range(world)})
    base = torch.zeros(6, dtype=dtype, device=_device())
    view = base.conj() if conj else torch._neg_view(base)
    sources = {loc: view for loc, proc in endpoints.source.items() if proc == rank}
    targets = {loc: base for loc, proc in endpoints.target.items() if proc == rank}
    with pytest.raises(AffineTransferExecutionError, match='rejected the transfer batch') as refused:
        execute_transfers([TransferRequest('transformed', plan, sources, targets)], endpoints, dtype)
    assert 'a source and a target overlap' in str(refused.value.__cause__)

This is a silent-wrong-result regression relative to master, so it should block merge.

Non-blocking

  • _validate_buffers now inlines reach = offset + sum(...), but current master has the address_range(offset, shape, strides) helper. This hunk will conflict on rebase anyway — prefer switching the reach computation to address_range while you are in there.
  • Letting disjoint same-buffer copies through the normal local-copy path is the right call, and the _validate_aliases docstring explains the relay-band case well.

Merge order

This conflicts with #9094 (both rewrite the same preflight region of execute_transfers on the same base). Suggested order: land this one first (after the conj/neg fix), then #9094 rebases and moves the _buffers_alias fast path + region-level _validate_aliases into its shared preflight/_agree — that direction is the smaller rebase. Note that if the rebase happens without the _flat guard, prepare_transfers would inherit this same silent conj/neg no-op.

0z5a added 2 commits November 18, 2026 16:49
_validate_aliases refused a batch whenever a target buffer overlapped a
source buffer. A relay rank in stage two holds its band as a view of its
own destination: it sends the band to its peers and receives the other
bands around it, so no element is both read and written, but the
buffers overlap and the batch was refused.

Keep the whole-buffer check as the fast path. When it finds an overlap,
check the regions this process's schedule reads and writes instead. A
local copy whose source and target are the same memory at the same
strides moves nothing: drop it and count its region as read, so another
write into it is still refused. Regions are compared by the bytes they
span, so strided regions that interleave are refused as before.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
The region check took a conjugate or negative view of a destination for
the destination itself: same address, same strides, so the copy was
dropped as in place and the batch returned without moving anything.
Master refused that batch only because the view shares the
destination's storage.

The lazy bit is not safe across processes either, on master too.
Point-to-point transport does not carry it. With gloo on CPU, a
conjugate shard fails inside the send on its own process after the
preflight has agreed, and the peer waits for it; a negative-view target
is never written. Refuse both views in the preflight, on the process
that holds one, so every process refuses before data moves.

Region extents now come from address_range, as in _validate_buffers.

schedule_transfers sees no buffers, so it still counts a copy dropped
as in place as a local copy; its docstring now says so.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@0z5a
0z5a force-pushed the affine-relay-self-alias branch from 735b057 to 66bad36 Compare November 18, 2026 17:12
@gpu-test-labeler gpu-test-labeler Bot removed the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 18, 2026
@0z5a

0z5a commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @jinyouzhi, @LiRunGuo and @Achyuthan-S. You're right, and it is fixed in 66bad36. I also rebased the branch onto current master. The first commit is the same patch as 735b057 (git range-diff shows no change). The fix is a separate commit.

conj/neg views. _flat now refuses a buffer whose is_conj() or is_neg() is set, as you and Runguo suggested: "shard is a conjugate or negative view; pass it resolved (resolve_conj, resolve_neg)". While testing this I found that master has the same problem across processes, at least with gloo on CPU, which is where I could run it. Master only refused the destination-view case because the view shares storage with the destination:

  • a conj shard on one side of a cross-process segment raises inside the send (view_as_real doesn't work on unresolved conjugated tensors), but only after the preflight has agreed, so the peer waits forever;
  • a neg-view target is never written: the receive completes, no error is raised, and the storage stays as it was.

A neg source happens to work across processes, and local copies from or into either view work, because copy_ resolves the lazy bit. The guard refuses those too, so a batch does not behave differently depending on where a shard lands. The check is in the preflight on the process that holds the view, so every process refuses before data moves.

Tests. I added TestB2LazyViewsRefused at 2 processes:

  • your destination-view case, for conj and neg;
  • a cross-process case with the view on the source or on the target, where both processes must refuse, the holder's cause must name the view, and the target must stay untouched.

One difference from your sketch: the cause no longer says "a source and a target overlap", because _flat refuses the view before the alias check runs. The test asserts the new message. On master the destination tests fail on that message, the cross-process conj cases hang, and the neg-target case completes with nothing written.

Non-blocking. _validate_buffers and the region extents in _validate_aliases both use address_range now.

@LiRunGuo, on 3: schedule_transfers now says in its docstring that it still counts a copy dropped as in place as a local copy (local_copy_chunks / local_copy_elements), and that TransferStats.local_copy_bytes does not.

Merge order. I agree: this PR first, then #9094 rebases and moves the _buffers_alias fast path and _validate_aliases into its preflight, with the in-place decision made in prepare (Runguo's point 4). prepare_transfers checks its buffers through _flat, so prepared runs get the conj/neg guard as well.

On CPU (gloo, up to 4 processes), test_affine_*.py, test_autoep_affine*.py, test_zero_affine.py and v1/moe/test_autoep_checkpoint.py give 970 passed and 4 skipped (956 on master daf175b).

@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 18, 2026
@delock
delock self-requested a review November 18, 2026 23:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants