Repository navigation
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 735b057f49
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| in_place = read[0] == write[0] and all(size <= 1 or source == target for size, source, target in zip( | ||
| operation.chunk_shape, operation.segment.source_strides, operation.segment.target_strides)) |
There was a problem hiding this comment.
Check lazy view bits before dropping in-place copies
When a local source and target view the same storage with identical strides but different lazy-value metadata, this predicate incorrectly treats the copy as a no-op. For example, with a complex base, using {0: base.conj()} as the source and {0: base} as the target passes _flat (same dtype and contiguous) and has equal pointers/strides, so the operation is deleted and base remains unconjugated even though the transfer should produce the conjugated values or reject the alias. Include is_conj()/is_neg() compatibility in the in-place check to avoid reporting success with incorrect checkpoint contents.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 66bad36: _flat refuses a buffer with is_conj() or is_neg() set, so this copy is refused before the alias check instead of being dropped as in place. TestB2LazyViewsRefused covers it, on the destination and across processes.
LiRunGuo
left a comment
There was a problem hiding this comment.
Thanks @0z5a, this is the check I had in mind, and the fast path keeps the common case free. A few points:
-
The lazy-bit case the bot raised is real. On CPU, with
basea complex64 tensor,execute_transfer(plan, {0: base.conj()}, {0: base}, ...)with a replicated map is refused on master ("a source and a target overlap within the transfer batch") but completes on this head withbaseunchanged, since the copy is taken as in place. It needs a complex tensor viewed through.conj(), so it is unlikely in practice. But it turns a refusal into a silent wrong result. Refusing buffers withis_conj()oris_neg()in_flatwould close it, and so would comparing the two buffers' bits in thein_placepredicate. Either way, a one-line test would pin it. -
Our relay fits the byte-span rule. A relay holder's band is a range of rows of its own destination shard, so it is one contiguous run of that buffer, whatever the TP layout. Interleaved regions being refused doesn't affect it.
-
schedule_transfersand in-place copies. The description saysschedule_transfersstill counts an in-place copy as a local copy. Could the docstring ofschedule_transfers(orProcessSchedule.local_copy_*) say so? A cost model reading it (#9026) would count, in stage two of a relay, one band per rank that never moves. -
Ordering with #9094. Both PRs change
execute_transfers. Whichever lands second will need the in-place decision made inprepare. It depends ondata_ptr, and #9094's buffer check already refuses a run whose buffers moved, so a prepared decision stays valid. -
On GPUs. On 2× RTX 3090 with NCCL (torch 2.13.0),
test_affine_transfer_executor.pyandtest_affine_transfer_executor_external_group.pyat 735b057 give 57 passed and 5 skipped (the four 4-process cases need 4 GPUs; the external-group case is CPU-only). The same tests on master (e35282b) give 3 failures:TestB2RelayBandInPlace[False-2]andtest_a_band_kept_in_place_is_not_copied, as you describe;second_writer, which master refuses with "a source and a target overlap" instead of the expected message.
I didn't run the 4-process cases on GPUs.
Achyuthan-S
left a comment
There was a problem hiding this comment.
Thanks @0z5a, this reads right to me. On CPU (macOS, gloo, up to 4 processes, torch 2.8.0), the suite you list gives 763 passed, 4 skipped, against 755 on master (e35282b).
Reading _validate_aliases: once the writes are sorted by start, comparing neighbours catches every pair of overlapping writes. With the writes disjoint, the first write that ends after a read starts is the only one that needs checking, so the bisect is exact for byte spans. A refusal on one process still goes through the [digest, rejected] all-gather, so every process stops before data moves.
Each part is covered by the new tests: with in-place copies never recognised, 4 of the 8 fail; with an in-place region not counted as read, one fails; without the read/write check, the two single-process "a source and a target overlap" cases fail; without the write/write check, second_writer fails.
|
A correction to my note above: @LiRunGuo's point 1 is real, and I missed it. On CPU at 735b057, with |
|
Reviewed 735b057 end to end (executor diff + the new Blocking: conj/neg views are now silently accepted (regression vs master)Master refuses copies whose source or target is a conj/neg view, but only indirectly: base = torch.zeros(6, dtype=torch.complex64, device=_device())
# execute_transfers from {0: base.conj()} into {0: base}:
# master: AffineTransferExecutionError, cause 'a source and a target overlap ...'
# 735b057f4: returns stats, base unchanged - the conj copy was treated as in-place
# same for a float32 copy from torch._neg_view(base) into baseSince peer transport moves raw bytes ( if buffer.is_conj() or buffer.is_neg():
raise AffineTransferExecutionError(
f'{which} location {location}: a conj/neg view cannot be transferred; pass its materialization')Adding that guard keeps @pytest.mark.parametrize('conj', [False, True])
def test_a_conj_or_neg_view_of_the_destination_is_refused(self, conj):
_join_transfer_group()
rank = dist.get_rank()
world = dist.get_world_size()
dtype = torch.complex64 if conj else torch.float32
map_ = replicated_map((6, ), world)
plan = plan_transfer(map_, map_)
endpoints = TransferEndpoints({loc: loc for loc in range(world)}, {loc: loc for loc in range(world)})
base = torch.zeros(6, dtype=dtype, device=_device())
view = base.conj() if conj else torch._neg_view(base)
sources = {loc: view for loc, proc in endpoints.source.items() if proc == rank}
targets = {loc: base for loc, proc in endpoints.target.items() if proc == rank}
with pytest.raises(AffineTransferExecutionError, match='rejected the transfer batch') as refused:
execute_transfers([TransferRequest('transformed', plan, sources, targets)], endpoints, dtype)
assert 'a source and a target overlap' in str(refused.value.__cause__)This is a silent-wrong-result regression relative to master, so it should block merge. Non-blocking
Merge orderThis conflicts with #9094 (both rewrite the same preflight region of |
_validate_aliases refused a batch whenever a target buffer overlapped a source buffer. A relay rank in stage two holds its band as a view of its own destination: it sends the band to its peers and receives the other bands around it, so no element is both read and written, but the buffers overlap and the batch was refused. Keep the whole-buffer check as the fast path. When it finds an overlap, check the regions this process's schedule reads and writes instead. A local copy whose source and target are the same memory at the same strides moves nothing: drop it and count its region as read, so another write into it is still refused. Regions are compared by the bytes they span, so strided regions that interleave are refused as before. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
The region check took a conjugate or negative view of a destination for the destination itself: same address, same strides, so the copy was dropped as in place and the batch returned without moving anything. Master refused that batch only because the view shares the destination's storage. The lazy bit is not safe across processes either, on master too. Point-to-point transport does not carry it. With gloo on CPU, a conjugate shard fails inside the send on its own process after the preflight has agreed, and the peer waits for it; a negative-view target is never written. Refuse both views in the preflight, on the process that holds one, so every process refuses before data moves. Region extents now come from address_range, as in _validate_buffers. schedule_transfers sees no buffers, so it still counts a copy dropped as in place as a local copy; its docstring now says so. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
735b057 to
66bad36
Compare
|
Thanks @jinyouzhi, @LiRunGuo and @Achyuthan-S. You're right, and it is fixed in 66bad36. I also rebased the branch onto current master. The first commit is the same patch as 735b057 ( conj/neg views.
A neg source happens to work across processes, and local copies from or into either view work, because Tests. I added
One difference from your sketch: the cause no longer says "a source and a target overlap", because Non-blocking. @LiRunGuo, on 3: Merge order. I agree: this PR first, then #9094 rebases and moves the On CPU (gloo, up to 4 processes), |
For open question 1 in #9026, as @LiRunGuo proposed in #9074 (comment).
In stage two of a relay, a relay rank holds its band as a view of its own destination. It sends the band to its peers and receives the other bands around it, so no element is both read and written.
_validate_aliasescompares whole buffers, though, soexecute_transfersrefuses the batch.Change.
_validate_aliasescollects the regions this process reads (sends and local-copy sources) and writes (receives and local-copy targets).schedule_transfersdoesn't see buffers, so it still counts an in-place copy as a local copy, and its docstring says so.TransferStats.local_copy_bytesdoes not count it._flat, on the process that holds one, before the agreement. Otherwise a lazy view of a destination would pass as in place and be dropped (raised by the Codex review, @LiRunGuo and @jinyouzhi). Point-to-point transport does not carry the lazy bit, on master either. With gloo on CPU, a conj shard fails inside the send after the agreement while its peer waits, and a neg-view target is never written. The docs say to pass such views resolved.address_range, as_validate_buffersdoes.Tests (14 new, in
test_affine_transfer_executor.py):TestB2RelayBandInPlace, at 2 and 4 processes: every rank's band is a view of its own full copy. Each rank completes its copy exactly, copies nothing locally, and sends and receives N − 1 bands. With the view moved to a peer's rows, every rank refuses before data moves, and the cause names the overlap.TestB2LazyViewsRefused, at 2 processes: a conj or neg view of the destination, and one on either side of a cross-process segment. Every process refuses, the holder's cause names the view, and nothing is written. On master (gloo, CPU) the destination cases fail on the old message, the cross-process conj cases hang, and the neg-target case completes with nothing written.On CPU (gloo, up to 4 processes),
test_affine_*.py,test_autoep_affine*.py,test_zero_affine.pyandv1/moe/test_autoep_checkpoint.pypass: 970 passed, 4 skipped (956 on master daf175b). yapf and flake8 at the pre-commit pins are clean.How it was found. By reading
_validate_aliasesagainst the stage-two shape of Runguo's relay on the affine IR (#9026, Q1): each relay rank's band is a view of its own destination, so the whole-buffer comparison refuses the batch although no element is both read and written. The conj/neg part came out of review, and the cross-process behaviour on master from probing each side and view on CPU.Production impact (estimate). Today a relay built on
execute_transferscannot run its second stage at all; with this change it runs, and copies nothing for the band it already holds. Batches whose buffers do not share storage see no change and pay nothing extra. Conj/neg views need a complex tensor viewed through.conj()or an explicit neg view, so we expect them to be rare in checkpoints; where they did occur across processes, master hung or left the target unwritten with gloo on CPU, and they are now refused up front.