feat(sdk): add remote read foundation and owned cleanup - #160
Draft
forhappy wants to merge 7 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Add an unpublished Rust SDK for pinned remote Git reads and bounded Crab/LFS content streaming. Applications can open repositories, refresh snapshots, traverse history and trees, read raw or hydrated files, and stream archives through typed requests with explicit cancellation, limits, progress and cleanup.
This draft implements the read foundation of
crab/docs/architecture/crab-sdk.md. It does not deliver the entire plan or claim phase-one qualification. Publication/recovery extraction is started but incomplete; direct writes, local workflows, managed/native HTTP support, packaging and complete backend/performance qualification remain open.Changes
crab-sdkwith default, remote and content feature boundaries, explicit provider configuration, compiling examples and public API tests.crab-remotewith shared ref/internal lease renewal, GC admission and post-commit readiness outcomes. HTTP receive and default-branch publication use these scopes; abandoned owners stop renewing. Default-branch changes previously advanced HEAD during a GC sweep, and read-generation failure previously obscured known commits. Both regressions are fixed. Authorization, journal wiring and generation work remain caller-owned; CLI extraction and operation serialization remain open.Validation
Post-rebase public SDK tests pass: four lifecycle and six remote-read cases on the combined owner changes; one explicit fixture publisher ignored.
All 36 hydrator tests, 14 receipt tests, 27 storage-only visibility tests and 33 catalog-enabled visibility tests pass. The receipt regression failed before the fix.
All ten journal tests pass after publication type generalization. CLI and protected-receive test targets compile with
gix-transporton Rust 1.97.0; existing fixture warnings remain. Read-only receipt recovery is tested through journal compaction without creating a receipt, and both APIs reject mismatched operation evidence.A real Git fixture verifies eight concurrent callers produce one backend pack read and one telemetry event; a warm read emits none. All 39 qualification report/tool tests pass.
Scoped SDK, remote-Git, read and storage-only metadata lint passed with warnings denied; formatting and whitespace checks pass.
Native RustFS public examples verified exact 4 KiB Git, 1 GiB Crab and 32 MiB LFS output through a rejecting proxy: 154 GETs, zero writes, zero proxy failures. The dedicated small SDK RustFS smoke passed and cleaned its bucket.
Native protocol lifecycle passed all 145 checks, including mirror cancellation and the previously failing filtered-transfer assertion: 28,095 filtered versus 206,754 full-clone pack/index payload bytes. This used a debug CLI with default plus
gix-transport, Apple Git 2.50.1, Rust 1.97.0 and task-local AWS CLI 1.46.1. Its frozen source predates the final receipt/visibility fix; those fixes have separate owner and SDK consumer proof.Feature checks remain 38 default, 411 remote and 438 content normal dependencies, with no forbidden edges. Earlier last-reader ownership tests and Linux qualification are recorded with their exact provenance in
sdk-qualification.md.New publication owner: five coordination tests, strict owner/HTTP library and test lint, three native HTTP receive/fault tests and one browser branch/default-branch regression pass. The GC-fencing regression failed against the previous publication path before passing through the extracted owner. Two RustFS tests remain explicitly ignored. Default and publication features compile on Rust 1.91.0; the default feature has no normal dependencies. The lockfile adds only the internal package and HTTP dependency edge, with no dependency upgrades.
Post-commit readiness: ten focused HTTP tests pass, including a five-case marker/readiness fault matrix, browser branch/release workflows and canonical merges. The new readiness-after-marker regression failed with HTTP 503 before the fix. Known committed refs are now acknowledged even while indexing is pending, consistent with Git report-status; uncertain commits still fail transport without per-ref rejection. Owner/HTTP strict lint and the Rust 1.91 owner check pass.
Remaining gates
Fresh CI on the updated commit, release-mode/other-Git lifecycle evidence, backend/cross-platform qualification and repeated performance comparisons remain required. Reader telemetry measures completed pack/index payload at the facade boundary, not hidden provider retries or HTTP framing. Local external-volume logs referenced by the qualification document are not uploaded CI artifacts.
The architecture checker false-positive and workspace inheritance fixes are included. The prepared SDK dependency-policy registration and metadata feature-budget inventory update remain unapplied pending explicit approval under the repository's inventory-change rule. The new publication crate and HTTP edge also require policy registration. Keep this PR in draft.