test(distributed): cover the 8-rank read, and pin cross-mode byte parity

[jfischoff]

What

Two things this suite could not say before:

1. The 8-rank read works. Every distributed test here runs at _WORLD = 2, but the shape this path ships in is an 8-GPU node — and world size is not a free parameter of the sharded reader. It sets the shard plan (_shard_range / _plan_windows), how many ranks fall off the end of a short block with an empty shard, and how many collectives _replicate_contiguous issues per block. At world 2 a block only ever splits into "rank 0 takes it all" or "one shard each", so the interior ranks that exist only at larger worlds, and the geometric shard step-down in _plan_windows, were never exercised end to end. test_plan_windows_tiles_blocks_exactly covers worlds 2/4/8, but only as planning arithmetic — no read, no collectives.

2. The read modes agree with each other. The existing sharded tests compare each mode against the source state. Nothing compared the modes against each other, which is the claim a caller actually leans on when it flips distributed_sharded.

Both new tests run on gloo/CPU like the rest of the module, so the extra ranks cost processes, not GPUs.

test_sharded_read_is_byte_exact_at_world_8

Per strategy, over a pack built to hit all three shard shapes at once:

block	why
wide.bf16 (8 MiB)	≥ world * SHARD_ALIGN_BYTES — every rank owns a real shard
ragged.fp32	not a multiple of world * align — last shard runs long, windows must step its shard size down
short.fp32 (388 B)	smaller than world * align — high ranks must skip it identically or the collective sequence desynchronizes and the job hangs

test_all_read_modes_agree_byte_for_byte

Reads one pack four ways in a single process — whole-pack, broadcast, sharded/contiguous, sharded/windows — and requires identical bytes on every rank.

Why now

A downstream app hit a CUDA illegal memory access on its first forward after switching to a FlashPack load. The sharded read was named as the cause and the caller reverted off it. It was not the read. On 8×H200 with a 28.58 GB production bf16 pack:

• all four modes returned byte-identical payloads on all 8 ranks;
• the model built from them matched the loader it replaced on all 1095 parameters, with zero tensors left on meta;
• the fault still reproduced with distributed_sharded=False, with FlashPack disabled entirely, and finally on an unmodified checkout of the app's main branch — i.e. it was pre-existing and unrelated.

Nothing in this suite could have said that quickly, so clearing flashpack cost a multi-hour bisect on real hardware. These tests make it a two-second answer, and the README now carries the triage order (compare modes → diff the built model → only then look at the caller).

Test plan

Run on an 8×H200 node, torch 2.11, gloo/CPU ranks:

$ pytest tests/test_distributed_load.py -k 'world_8 or modes_agree'
3 passed

$ pytest tests/test_distributed_load.py
22 passed in 148.20s

No source changes — tests and README only.