Skip to content

feat(sdk): add remote read foundation and owned cleanup - #160

Draft
forhappy wants to merge 7 commits into
mainfrom
codex/crab-sdk-remote-foundation
Draft

feat(sdk): add remote read foundation and owned cleanup#160
forhappy wants to merge 7 commits into
mainfrom
codex/crab-sdk-remote-foundation

Conversation

@forhappy

@forhappy forhappy commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Scope

Add an unpublished Rust SDK for pinned remote Git reads and bounded Crab/LFS content streaming. Applications can open repositories, refresh snapshots, traverse history and trees, read raw or hydrated files, and stream archives through typed requests with explicit cancellation, limits, progress and cleanup.

This draft implements the read foundation of crab/docs/architecture/crab-sdk.md. It does not deliver the entire plan or claim phase-one qualification. Publication/recovery extraction is started but incomplete; direct writes, local workflows, managed/native HTTP support, packaging and complete backend/performance qualification remain open.

Changes

  • Add crab-sdk with default, remote and content feature boundaries, explicit provider configuration, compiling examples and public API tests.
  • Put cancellation and admission in shared storage/read owners. Shared fetches preserve surviving readers, and the last reader cancels and drains its producer. Map Xet's successful cancellation return to cancellation before incomplete bytes become a corruption error.
  • Verify LFS delivery without remote receipts, reject unsupported extension hydration before delivering file bytes, retain pinned reconstruction metadata, and await cache finalization. Include the filesystem-locking correction required by the Linux fixture.
  • Bind persisted receipt bodies to the requested plan key, preventing another operation's valid receipt from being accepted after a same-repository copy. Stored formats and keys are unchanged.
  • Separate read-only historical commitment proof from receipt repair, sharing validation and preserving version-one serialized evidence. Generalize publication attribution types across journal, CLI/mirror and protected receive without aliases. This prepares the shared publication owner; lifecycle extraction remains open.
  • Start crab-remote with shared ref/internal lease renewal, GC admission and post-commit readiness outcomes. HTTP receive and default-branch publication use these scopes; abandoned owners stop renewing. Default-branch changes previously advanced HEAD during a GC sweep, and read-generation failure previously obscured known commits. Both regressions are fixed. Authorization, journal wiring and generation work remain caller-owned; CLI extraction and operation serialization remain open.
  • Restore completed read telemetry inside shared producers and prevent collectors from adding each participant's budget reservation to transfer totals. Add regression coverage and qualification tooling.

Validation

  • Post-rebase public SDK tests pass: four lifecycle and six remote-read cases on the combined owner changes; one explicit fixture publisher ignored.

  • All 36 hydrator tests, 14 receipt tests, 27 storage-only visibility tests and 33 catalog-enabled visibility tests pass. The receipt regression failed before the fix.

  • All ten journal tests pass after publication type generalization. CLI and protected-receive test targets compile with gix-transport on Rust 1.97.0; existing fixture warnings remain. Read-only receipt recovery is tested through journal compaction without creating a receipt, and both APIs reject mismatched operation evidence.

  • A real Git fixture verifies eight concurrent callers produce one backend pack read and one telemetry event; a warm read emits none. All 39 qualification report/tool tests pass.

  • Scoped SDK, remote-Git, read and storage-only metadata lint passed with warnings denied; formatting and whitespace checks pass.

  • Native RustFS public examples verified exact 4 KiB Git, 1 GiB Crab and 32 MiB LFS output through a rejecting proxy: 154 GETs, zero writes, zero proxy failures. The dedicated small SDK RustFS smoke passed and cleaned its bucket.

  • Native protocol lifecycle passed all 145 checks, including mirror cancellation and the previously failing filtered-transfer assertion: 28,095 filtered versus 206,754 full-clone pack/index payload bytes. This used a debug CLI with default plus gix-transport, Apple Git 2.50.1, Rust 1.97.0 and task-local AWS CLI 1.46.1. Its frozen source predates the final receipt/visibility fix; those fixes have separate owner and SDK consumer proof.

  • Feature checks remain 38 default, 411 remote and 438 content normal dependencies, with no forbidden edges. Earlier last-reader ownership tests and Linux qualification are recorded with their exact provenance in sdk-qualification.md.

  • New publication owner: five coordination tests, strict owner/HTTP library and test lint, three native HTTP receive/fault tests and one browser branch/default-branch regression pass. The GC-fencing regression failed against the previous publication path before passing through the extracted owner. Two RustFS tests remain explicitly ignored. Default and publication features compile on Rust 1.91.0; the default feature has no normal dependencies. The lockfile adds only the internal package and HTTP dependency edge, with no dependency upgrades.

  • Post-commit readiness: ten focused HTTP tests pass, including a five-case marker/readiness fault matrix, browser branch/release workflows and canonical merges. The new readiness-after-marker regression failed with HTTP 503 before the fix. Known committed refs are now acknowledged even while indexing is pending, consistent with Git report-status; uncertain commits still fail transport without per-ref rejection. Owner/HTTP strict lint and the Rust 1.91 owner check pass.

Remaining gates

Fresh CI on the updated commit, release-mode/other-Git lifecycle evidence, backend/cross-platform qualification and repeated performance comparisons remain required. Reader telemetry measures completed pack/index payload at the facade boundary, not hidden provider retries or HTTP framing. Local external-volume logs referenced by the qualification document are not uploaded CI artifacts.

The architecture checker false-positive and workspace inheritance fixes are included. The prepared SDK dependency-policy registration and metadata feature-budget inventory update remain unapplied pending explicit approval under the repository's inventory-change rule. The new publication crate and HTTP edge also require policy registration. Keep this PR in draft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant