perf(staging): batch push retirement and reduce edited-push latency

[forhappy]

Summary

• Batch staging file retirement into one SQLite transaction and run unowned payload inventory cleanup once per batch. Writer/shared-push publication, rollback and recovery share the canonical path.
• Preserve committed segment counts, prepared-payload leases, whole-batch rollback and commit-before-unlink ordering. No format/version, dependency or durability-setting changes.
• Split push timing into retirement, cache/index and manifest subphases.
• Create fresh smoke-test private cache roots with mode 0700; existing directories and production security checks remain unchanged.

Measured performance

Fresh local RustFS qualification: 50 × 2-GiB large files, ten shared content families, 500 small code files, three versions. Approximately 20 GiB unique base content, not 100 GiB unique entropy.

Phase	Before	After
First edited push	245.025 s	113.972 s
Second edited push	277.197 s	108.308 s
First post-success cleanup/cache	108.758 s	29.793 s
Second post-success cleanup/cache	131.536 s	30.215 s

Uploaded bytes are identical across the before/after scale runs: 57,579,824 and 57,285,483 bytes. The old aggregate timer included staging retirement, not just cache work. Same-snapshot SQL replay isolated per-file commits versus batching: 16.705 s → 1.686 s. The harness permission correction contributes separately to end-to-end improvements.

Verification

• Fresh-bucket 100-GiB RustFS lifecycle: 1,223 checks passed, including 1,100 SHA-256 comparisons, commit/push, cold clone/hydration, dehydration/rehydration and final Git fsck.
• Separate fresh-bucket lifecycle canary: 132 checks passed.
• Crab default-feature library: 4,145 passed, 3 ignored.
• Staging library: 217 passed, 1 ignored; new whole-batch failure rollback and shared prepared-payload retention regressions.
• Smoke-harness Python suite: 11 passed.
• Release CLI build, strict staging all-target clippy, Crab CI lint-category gate, workspace format and diff checks passed. Crab lint retains existing CI-allowed warnings.
• Earlier broad runs exposed unchanged timing-sensitive cache-lock/cancellation tests and an overly long temporary-path setup; final normal-temp run after live I/O completed passed without weakening assertions.
• Disposable qualification repositories/caches and all three task-created RustFS buckets removed; reports/logs retained locally.

Detailed evidence, ownership map, binary provenance and limitations: crab/docs/benchmarks/add-push-hardening-rustfs.md, section “Edited-push retirement follow-up”. Runtime source tested is this patch atop merged PR #156; the binary embeds its runtime-identical pre-squash revision, documented with SHA-256.

Design and remaining gaps

Is this the best fix rather than merely plausible? Batching removes repeated durable commits/global scans at the staging owner boundary, instead of weakening durability or deferring cleanup. It also removes duplicated writer/read-only retirement loops. The tradeoff is a longer individual write transaction and potentially more WAL retained until commit. Ownership publication remains a separate transaction with existing retry/recovery.

Retirement still takes approximately 29 seconds at scale; metadata work remains material. Initial pushes and add wall times were slower on the shared workstation/USB SSD, so this does not establish improved add latency, universal low latency, all historical-version reconstruction, or production S3/GCS/Azure performance. All formats remain v1; no compatibility path is added.

[forhappy]

Fix for branch-fanout protocol-v2 qualification failure

The failed CI reader had correct content and passed strict Git fsck, but the required protocol-v2 packet trace was absent. The original CI log does not record the owner probe, so the exact scheduling interleaving cannot be reconstructed from that artifact alone.

A controlled local RustFS probe reproduced the same symptom with only the generation-owner lease changed: v2 advertised before holding the lease, absent while held (clone content correct and fsck clean), and advertised again after restoring the lease. Eight- and 32-reader ungated baseline runs passed, confirming why an ordinary rerun alone was insufficient evidence.

Removing the owner gate alone passed the CI-shaped replay but a 32-reader run still failed once. A second deterministic gate then paused a reader after the compacted manifest CAS and before catalog materialization: another helper rejected the durable pending handoff and withheld v2. The matching Rust regression also failed before the second fix.

The complete fix removes the owner-activity gate and its diagnostic journal LIST, and recognizes a bounded pending visibility handoff bound to the compacted manifest. The shared metadata parser validates that binding for both admission discovery and handoff application. The existing published-proof predicate is unchanged: owners must not confuse pending evidence with completed publication and skip materialization. Actual fetch admission retains bounded owner/repair waiting, cancellation, complete base/edit validation, generation-pinned visibility authorization, and exact comparison with the discovered refs and pack inventory. No format, dependency, lock or staging-retirement changes.

Evidence map: dispatch_capabilities → snapshot_available / capability_snapshot_is_stable → manifest, published-proof or pending-handoff reads. read_catalog_pending is shared with apply_catalog_journal_edits; compact_ref_journal persists the handoff before manifest CAS. serve_admitted → open_repository_with_visibility_requirement remains the repair/authorization owner. Empty repositories, missing evidence, active journals, stale locators and replica-selected stores retain their existing boundaries. Current main has both admission gaps.

Is this the best fix rather than merely plausible? Protocol support should not depend on a one-read advisory owner hint or on whether another reader has finished materializing an already-published handoff. Recognizing the existing durable state fixes both gaps without sleeps, retries in the CI assertion, weaker object checks, or a compatibility fallback. Existing owner-aware fetch admission remains canonical.

Regression: the added active-owner assertion in upload_pack_repairs_locator_for_current_manifest_generation failed before the fix. The obsolete owner-suppression expectation in the handoff test now checks v2 remains available while retaining its deferred-locator checks. The missing-visibility tests remain unchanged.

Validation: full default-feature Crab library 4,145 passed, 3 ignored, using CI's 8-MiB test-thread stack. An initial default-stack replay overflowed; no production or test assertion change was used to bypass it. Release build and live replay results follow below.

Intermediate owner-only build 0fe980c59f24f1727e0c05ff1eaed40831b3b7149547de36e6fe4068dca5834b passed the held-owner clone, 23-check concurrent CI scenario and isolated 92-check partial-clone suite, but is not sufficient final proof because of the subsequent 32-reader failure. An initial partial-clone attempt shared a bucket with concurrent writes and invalidated its whole-bucket dry-run invariant; the isolated replay passed without changing the assertion. Final validation of both fixes follows below.

Final source: 4,145 Crab library tests passed, 3 ignored, including both new regression assertions and the unchanged missing-evidence rejection cases. 34 metadata visibility tests passed, including pending target-generation/pack-binding rejection. The metadata no-default-feature check passed with four existing unused-item warnings in ref_registry.rs. One broad replay stalled in the unchanged Unix filter test's worker join; its stack was sampled, only that test process was terminated, and the unchanged full replay passed. All temporary diagnostics were removed.

Final optimized binary SHA-256: ee3fcc6aac034f26f2a4a22c8ad29095dc777552959572d80d9255650aa95c0c. The held-owner clone and gated pending-handoff probe both passed. Fresh-bucket CI-shaped concurrency passed 23 checks (including crash/marker faults and request-cost budget), and the separate fresh-bucket partial-clone suite passed 92 checks. The first 32-reader metered round passed. A subsequent metered stress round hit the local Python proxy's socket-allocation failure (Can't assign requested address, surfaced as S3 502), not a protocol-evidence failure; larger follow-up rounds use the existing direct-RustFS mode without the optional meter. The original metered CI scenario is not bypassed.

Both direct 32-reader follow-up rounds passed: 96 successful branch-reader clones across three completed 32-reader rounds, with content and strict fsck checks. The final Crab CI lint-category gate and workspace format/diff checks passed (existing CI-allowed warnings retained). All disposable run data/caches and five task-created RustFS buckets were removed and bucket absence verified; reports, logs and clearly separated debug probes remain local. Commit: edb4c405ff4. Rebased on current main; hosted Linux CI must still run on this commit.