feat(isobmff)!: model and write top-level boxes so a C2PA uuid box has somewhere to live

[justin13888]

Summary

• IsoBmffImage gains top_level_boxes: Vec<TopLevelBox> — every top-level box of the primary stream the model does not otherwise own (anything but ftyp/meta/mdat), in file order, with the uuid user type split off the payload exactly as RawBox does. This is where a C2PA ContentProvenanceBox lives; until now read silently discarded such boxes (reader.rs _ => {}) and write had nowhere to emit one.
• Each box carries a TopLevelPosition: AfterFtyp boxes are written between ftyp and meta (so before the first mdat, C2PA 2.4 §A.5.3); Trailing boxes after mdat. read assigns the position from where it met the box (after mdat → Trailing, otherwise AfterFtyp), so a file this crate wrote re-serialises byte-identically; a foreign box between meta and mdat is written back before meta — a move between two lawful §A.5.3 positions, documented in TopLevelPosition and STATUS.md.
• write rejects a top-level box typed ftyp/meta/mdat (InvalidInput), moov/trak (Unsupported, as on read), one whose user_type does not pair with the uuid type, one whose complete box (header included) does not fit the 32-bit size field, or a top_level_boxes list that interleaves positions (an AfterFtyp box after a Trailing one — the file cannot record that order, so read could not reproduce it); IsoBmffImage::push_top_level_box appends at the end of a box's position group so a parsed model never becomes interleaved (review round 1, decision 14/4a). A uuid inside meta is never promoted. walk_segments is unchanged and still tiles a file carrying such boxes.
• IsoBmffImage is now #[non_exhaustive]; IsoBmffImage::new(major_brand, compatible_brands, primary_item_id, items) plus with_minor_version/with_groups/with_top_level_boxes replace the struct literal, so the next field is a minor release. TopLevelBox and TopLevelPosition are #[non_exhaustive] too; TopLevelBox::new/::uuid/with_position build the boxes. Every literal site in the workspace (gamut-avif encoder/tests/example, gamut-heic tests/example, gamut-cli isobmff build, the isobmff bench and test fixtures) moved to the constructor in commits that each leave the workspace compiling.
• Acceptance pinned: (a) exact-byte — a C2PA uuid box lands right after ftyp, before meta/mdat (tests/top_level.rs); (b) write(&read(&f)) == f for a file carrying boxes at both positions; (c) walk_segments tiles the written file with no Trailer/unclassified run; (d) libavif decodes an AVIF carrying the C2PA box to identical pixels (gamut-avif/tests/remux_roundtrip.rs).

No human approved this plan: this is an unattended run, and the ## Decisions taken record below is what a human reads afterwards.

Validation

Every workspace-wide gate ran inside systemd-run --user --scope -p MemoryMax=16G -p MemorySwapMax=0 with CARGO_BUILD_JOBS=2 CMAKE_BUILD_PARALLEL_LEVEL=2 and ulimit -v 12000000, in the lane's worktree, logs under a lane-unique directory.

Round 3 (head c8b87cf, after the review-round-2 Lows, test/docs only):

• L1/L2: push_top_level_box_appends_within_the_position_group now pushes onto [a, t] → [a, new, t], then [a, a, t] + t → [a, a, t, new], then [a, a, t, t] + a → [a, a, new, t, t] — the "after the last AfterFtyp box" arm is reached (replacing it with index 0 fails) — and asserts list order only; the grouped-list round-trip stays pinned by boxes_keep_model_order_within_each_position and tests/roundtrip.rs.
• L3: reader.rs read docs now link IsoBmffImage::top_level_boxes instead of the dead self#top-level-boxes anchor (reader is private).
• cargo test -p gamut-isobmff --all-features — pass, 13 test binaries.
• RUSTDOCFLAGS=-D warnings cargo doc -p gamut-isobmff --no-deps — fails on one pre-existing link only: reader.rs:59 self#motion-photo-tolerance (present on master before this PR); the link this PR added is fixed. Out of scope here — see Unresolved review notes.
• mise run check-commits — "no errors in 14 commits"; mise run check-tests — conforms; __CARGO_TEST_ROOT=<worktree> mise run fmt-check — pass.
• CI on c8b87cf: observed by the lane to terminal state; reported in its final message.

Round 2 (head 2e7f2ef, after the review-round-1 repair):

• Diagnosis: a throwaway test reproduced the defect at d3be07a — write accepted [free Trailing, uuid AfterFtyp] and read returned [uuid AfterFtyp, free Trailing]; mechanism: write_top_level partitions by position, so the bytes never encode cross-group model order.
• cargo test -p gamut-isobmff --all-features — pass (13 test binaries; new writer::interleaved_top_level_positions_are_rejected, top_level::push_top_level_box_appends_within_the_position_group; boxes_keep_model_order_within_each_position fixture regrouped, same claim).
• cargo test -p gamut-avif --all-features --test remux_roundtrip — pass 2/2 (the oracle test now appends via push_top_level_box).
• cargo clippy -p gamut-isobmff --all-targets --all-features -- -D warnings — pass.
• mise run lint — exit 0.
• mise run test — exit 0, 203 test binaries green.
• mise run mutants-diff --budget 16 (diff vs origin/master) — 26 mutants tested in 3m: 20 caught, 6 unviable, 0 missed, 0 timeouts.
• mise run check-commits — "no errors in 12 commits"; mise run check-tests — conforms; __CARGO_TEST_ROOT=<worktree> mise run fmt-check — pass (incl. fmt-tooling-check).
• CI on 2e7f2ef: see the checks tab (observed to terminal state by the lane; reported in its final message).

Round 1 (head d3be07a):

• cargo test -p gamut-isobmff -p gamut-avif --all-features — pass. mise run lint — exit 0. mise run test — exit 0, 203 test binaries. mise run check-commits — 10 commits clean. mise run check-tests — conforms. __CARGO_TEST_ROOT=<worktree> mise run fmt-check — pass (without the prefix the task fails in this nested worktree before formatting anything: cargo metadata "current package believes it's in a workspace when it's not" for every tooling/* path-dep — unavailable there; per-member cargo +nightly fmt --check -p <32 members> also passed).
• mise run mutants-crate gamut-isobmff --shard 1/2 --budget 16 and --shard 0/2 — 171 + 171 mutants: 318 caught, 24 unviable, 0 missed, 0 timeouts (the whole crate).
• CI on d3be07a: Format & Metadata, Clippy & Doctests, Coverage (test gate), Incremental (PR diff) 0–3 — all pass.
• Not run: mise run coverage (no new src/ module; CI Coverage gate is the floor), check-release-deps/check-ffi-* (no Cargo.toml or C-surface change).

Risks and rollout

• Semver-major for gamut-isobmff (feat(isobmff)!: with a BREAKING CHANGE footer): struct literals of IsoBmffImage outside the crate no longer compile. release-plz will bump the major; gamut-avif/gamut-heic/gamut-cli are migrated in this PR. The constructor makes this the last literal break.
• Behaviour change on remux of foreign files: read now retains free/skip/vendor/uuid top-level boxes that it previously dropped, so write of a parsed foreign file carries them (and moves one found between meta and mdat to before meta). A libavif corpus file remuxes to identical pixels with and without an added uuid box (oracle test). gamut-avif/gamut-heic containers account bytes through walk_segments, which is untouched. A retained box is an owned copy (a motion-photo mpvd included), bounded by the input size — recorded in STATUS.
• Payload is opaque; nothing here parses box_purpose, the merkle offset or JUMBF — gamut-heic's c2pa module keeps that lens (gamut-heic: locate the C2PA manifest store and report its byte range #429).

Issue

Closes #443

Decisions taken

Issue 443 - gamut-isobmff: model and write top-level boxes, so a C2PA uuid box has somewhere to live
Plan:     v1
Branch:   feat/443-isobmff-top-level-box-list
Base:     origin/master (6a75ec4)
Cause:    -
Touches:  gamut-isobmff model/writer/reader/boxes/segments + tests + STATUS/README; struct-literal fallout in gamut-avif (and gamut-heic if any)
Will not: parse box_purpose / merkle offset / JUMBF LBox (that lens stays in gamut-heic, #429); model a C2PA store as an item; promote a uuid found inside meta to top level; touch EncodeImage
Lane:     parallel (root of the AVIF stack: #444 branches from this head)
Settled:  S2 - a new field on an exhaustive type is `feat(isobmff)!:` with a BREAKING CHANGE footer; S3 - docs/testing.md placement

Decisions taken.
1. Deliverable boundary
   Taken:    the whole issue - model field, writer placement, reader retention, byte-identical read->model->write round-trip, byte accounting unchanged; no split
   Rejected: read-side retention only - leaves #444 with nothing to write into
   Reverses: drop the writer commit
   Filed:    -
2. Representation
   Taken:    `IsoBmffImage.top_level_boxes: Vec<TopLevelBox>` in file order, where `TopLevelBox { ty: [u8; 4], user_type: Option<[u8; 16]>, payload: Vec<u8>, position: TopLevelPosition }`; `IsoBmffImage` gains `#[non_exhaustive]` plus a constructor/builder so this is the last struct-literal break; one `feat(isobmff)!:` commit with `BREAKING CHANGE: IsoBmffImage is #[non_exhaustive] and gains top_level_boxes` - consequence: gamut-avif/heic literals move to the constructor; adds model.rs, every literal site to the manifest
   Rejected: a `Vec<RawBox>`-shaped owned copy - discards the user-type split RawBox already models; a C2PA-specific field - the issue is a general container change
   Reverses: replace TopLevelBox with (ty, bytes) pairs
3. Placement on write
   Taken:    `TopLevelPosition::{AfterFtyp, Trailing}`; AfterFtyp boxes are written after ftyp and before meta and mdat (C2PA §A.5.3); Trailing boxes after mdat; the reader assigns the position from where it found the box (before meta -> AfterFtyp, after mdat -> Trailing); a box between meta and mdat is retained as AfterFtyp and the reordering is documented (a uuid box there is not a lawful C2PA placement anyway)
   Rejected: always after ftyp - breaks byte-identical round-trip for trailing boxes; a free-form offset - the writer recomputes offsets
   Reverses: drop the enum; write every retained box after ftyp
4. Which boxes are retained
   Taken:    every top-level box the model does not otherwise own (anything but ftyp, meta, mdat, the motion-photo second ftyp and what follows it); moov/trak stay Unsupported as today
   Reverses: retain uuid only

Appended by this lane (same shape):

5. Literal sites outside the predicted manifest
   Taken:    migrate crates/gamut-cli/src/commands/isobmff.rs (demo image) and crates/gamut-isobmff/benches/container.rs to the constructor — decision 2 already adds "every literal site to the manifest", and `grep -rn "IsoBmffImage {" crates/` found both
   Rejected: freeze and request a manifest revision — a `#[non_exhaustive]` struct leaves the workspace not compiling, so no pushed state would be coherent
   Reverses: nothing to reverse; both sites cannot compile as literals
6. Where the libavif acceptance test (d) lives
   Taken:    a new test in the existing crates/gamut-avif/tests/remux_roundtrip.rs (the file that already remuxes a libavif corpus file through gamut-isobmff and decodes it with libavif); a local C2PA_UUID constant rather than a dependency on gamut-heic
   Rejected: crates/gamut-isobmff/tests — no libavif oracle there (dev-dep topology); crates/gamut/tests — mutation-invisible
   Reverses: delete `remux_with_a_c2pa_uuid_box_preserves_decoded_pixels`
7. Commit series shape
   Taken:    an additive `feat(isobmff):` constructor commit first, then `refactor(avif|heic|cli):` literal migrations, then the single `feat(isobmff)!:` commit with the BREAKING CHANGE footer, then `test(avif):`; review repairs as four further `fix`/`refactor`/`test` commits — every commit leaves the workspace compiling
   Rejected: one commit across four crates — convco takes one crate per scope and a mid-series non-compiling state is not bisectable
   Reverses: squash on merge
8. `mise run fmt-check` in this worktree
   Taken:    the task fails before formatting anything in a worktree nested under the primary checkout (`cargo fmt --all` resolves the workspace-excluded tooling/* path-deps upward to the primary's Cargo.toml — no tooling manifest carries a `[workspace]` table); classified `unavailable`, substituted by per-member `cargo +nightly fmt --check -p …` over every workspace member, labelled as a substitute in Validation
   Rejected: skipping formatting checks; editing tooling manifests (outside the manifest)
   Reverses: none needed once CI runs the real task
   Update:   `__CARGO_TEST_ROOT=<worktree> mise run fmt-check` runs the real task in the nested worktree and passes (incl. fmt-tooling-check); recorded in Validation, the substitute kept only as corroboration
9. Top-level box size validation (superseded by 13)
   Taken:    mirror the item check (`u32::try_from(payload.len()).is_err()` → Unsupported "at or beyond 4 GiB"), the crate's existing precision for the 4 GiB edge
   Rejected: header-inclusive `checked_add(8 + 16)` — its only distinguishing input is a >4 GiB allocation, so its mutant is unkillable
   Reverses: restore the header-inclusive arithmetic
10. TopLevelBox constructors
   Taken:    `TopLevelBox::new(ty, payload)`, `TopLevelBox::uuid(user_type, payload)` (both AfterFtyp) and `with_position` — additive helpers over the record's struct shape, so #444 writes `TopLevelBox::uuid(C2PA_UUID, store)`
   Rejected: literal-only construction — every caller re-derives the user-type/type pairing `write` validates
   Reverses: delete the three helpers
11. `TopLevelBox` and `TopLevelPosition` are `#[non_exhaustive]` (review finding)
   Taken:    both new types carry `#[non_exhaustive]`, as every other enum in the crate and now `IsoBmffImage` do; `TopLevelPosition` keeps `#[repr(u8)]` with permanent append-only discriminants, so a finer position (e.g. between meta and mdat) is a minor release; tests build values through `TopLevelBox::new`/`::uuid`/`with_position`
   Rejected: exhaustive types — adding the obvious future variant would be a second major bump after this PR already pays one
   Reverses: drop the two attributes
12. Record premise corrected: a box between meta and mdat IS a lawful C2PA placement
   Taken:    behaviour unchanged (such a box is written back between ftyp and meta), but the docs (model.rs, STATUS.md) no longer claim the original spot was unlawful — C2PA 2.4 §A.5.3 (vendored text) requires only after `ftyp` and before the first `mdat`/any `moov`, which both positions satisfy; it is a move between two lawful positions
   Rejected: adding a third position now to make the round-trip byte-identical for such foreign files — the record fixes the two-position enum; `#[non_exhaustive]` (11) leaves the door open
   Reverses: none (documentation only)
13. Top-level box size bound is header-inclusive (supersedes 9; review finding)
   Taken:    `u32::try_from(payload.len().saturating_add(TOP_LEVEL_HEADER_MAX)).is_err()` with `TOP_LEVEL_HEADER_MAX = 24` (8-byte header + 16-byte uuid user type, applied to every box type so a non-uuid box is refused 16 bytes early) — no arithmetic operator, so no unkillable mutant, and `write`'s promise that a *box* at or beyond 4 GiB is rejected holds
   Rejected: payload-only bound (9) — a trailing payload within 24 bytes of 4 GiB wrote a wrapped size field; per-type exact header arithmetic — its `+`/`*` mutants need a 4 GiB allocation to kill
   Reverses: restore the payload-only `u32::try_from(payload.len())`
14. Review round 1 design questions (decided by the orchestrator on review)
   Taken:    1a keep `#[non_exhaustive]` on `TopLevelBox`/`TopLevelPosition` — AGENTS.md's C-portability rule wants append-only discriminants, and a third position must not be a major; 2a keep the constructor trio (`TopLevelBox::new`/`::uuid`/`with_position`) with the AfterFtyp default — the C2PA placement is the motivating case; 3a keep the uniform 24-byte header allowance in the 4 GiB bound — no arithmetic mutant, refuses 16 bytes early at an edge nobody reaches; 4a `validate` rejects a Trailing box that precedes an AfterFtyp box with `Error::InvalidInput` (the writer's cannot-round-trip policy), documented at writer.rs/reader.rs/model.rs/lib.rs, plus `IsoBmffImage::push_top_level_box` inserting at the end of the box's position group so a caller appending to a parsed model never builds an interleaved list (#444 will use it); both pinned by tests (`interleaved_top_level_positions_are_rejected`, `push_top_level_box_appends_within_the_position_group`)
   Rejected: 1b exhaustive types (a future position would be a second major); 2b literal-only construction; 3b per-type exact header arithmetic (unkillable `+`/`*` mutants); 4b normalise on read — weakens the `read(&write(&img)) == img` contract three docs state
   Reverses: 1 drop the two attributes; 2 drop the three helpers; 3 per-type header arithmetic; 4 accept and document normalisation (remove the `validate` rule and `push_top_level_box`)

Unresolved review notes

From the independent review pass before the PR opened; each declined with its evidence, none blocking.

• read now copies every retained top-level box, including a motion-photo mpvd (whole video). Declined: record decision 4 retains every box the model does not own, and an opt-in read_with(..) or an mpvd carve-out is a public-API fork the record did not open. Bounded by input size; documented under STATUS "Top-level boxes" as an accepted cost. A follow-up could add a read option if a consumer measures the cost.
• C2PA_UUID is spelled in gamut-heic and in three test/doc sites here; own it in gamut-isobmff. Declined: references/c2pa/README.md's clause map assigns the §A.5.1 user type to gamut-heic and only §A.5.3 placement to gamut-isobmff, and decision 2 rejected a C2PA-specific surface in the container crate; the test copies cannot depend on gamut-heic (dev-dep topology).
• An iloc extent addressing bytes inside a retained box is carried twice on write. Documented in STATUS as an accepted cost (no known encoder does this; the file decodes identically).
• gamut isobmff inspect does not print top_level_boxes. Out of this PR's manifest (gamut-cli is touched for constructor fallout only); a six-line follow-up in gamut-cli.
• Pre-existing dead rustdoc link (out of scope): reader.rs read docs link self#motion-photo-tolerance, but reader is a private module, so RUSTDOCFLAGS=-D warnings cargo doc -p gamut-isobmff fails on master and on this branch alike. Not touched here (it predates gamut-isobmff: model and write top-level boxes, so a C2PA uuid box has somewhere to live #443 and is not in the manifest); a one-line follow-up.
• Review round 1, Low (recorded, not repaired): the 4 GiB top-level box bound (writer.rs, TOP_LEVEL_HEADER_MAX check) has no test that reaches it. Reaching it needs a ~4 GiB Vec fixture, which is not portable across CI runners; the pre-existing item-payload and file-offset 4 GiB bounds in the same writer share exactly this status, so this matches crate convention. The check is a method call (u32::try_from(..).is_err()), so it contributes no mutation site; its body-replacement mutant (validate_top_level -> Ok(())) is caught by the type/user-type rejection tests.