Merge origin/main (#257) into fix/profiling-correctness-bugs

#257 landed the X1b deferred pristine fetch, which already refuses a
not-installed gem in `--vendor-source build` with `gem_spec_missing`
BEFORE downloading it, under the same scope this branch's gate had
(verifiable lock entry, no ledger entry). Take main's gate and drop
this branch's copy; keep the scope-guard suite, adapted to main's
behavior (the backend's own detail text, and the already-vendored
fresh-clone re-run now makes no registry request at all).

The other three fixes still apply and are ported onto #257's code:

* js7: the in-memory stager's view fetches now run through
  `ordered_concurrent`; the per-file `needs_blob` rule and the
  per-package drop reasons fold into that loop unchanged. The dropped
  records never reach `vendor_records`, so they stay out of the exact
  service-download plan and the group commit too (new test).
* order: `filter_to_installed_releases` still drains a HashMap on main;
  the purl sort now also orders #257's view prefetch plan, which is
  built from the same selection.
* req: untouched by #257.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Author: mikolalysenko

.gitattributes

@@ -10,3 +10,9 @@ crates/socket-patch-core/tests/fixtures/pdm-native/*.lock -text
 # refuse CRLF by design (vendor_lockfile_crlf_unsupported), and the tests
 # derive their CRLF variants from the LF bytes themselves.
 crates/socket-patch-core/tests/fixtures/pnpm-hosted/** -text
+
+# The legacy vendor ledgers are what the base binary wrote, byte for byte:
+# vendor_ledger_schema_e2e replays them through `vendor --revert` and
+# compares the result byte for byte, so a CRLF checkout would change both
+# the replayed wiring files and the expected revert.
+crates/socket-patch-cli/tests/fixtures/legacy-ledgers/** -text

.github/workflows/ci.yml

@@ -1455,9 +1455,45 @@ jobs:
           SOCKET_PATCH_CARGO_E2E_LOCK_VERSION: ${{ matrix.lock }}
         run: |
           set -euo pipefail
-          cargo test -p socket-patch-cli --test e2e_redirect_cargo_build --test e2e_vendor_cargo_build --test mode_migration_cargo
+          cargo test -p socket-patch-cli --test e2e_redirect_cargo_build --test e2e_redirect_cargo_shapes --test e2e_vendor_cargo_build --test mode_migration_cargo
           cargo test -p socket-patch-cli --test e2e_safety_cargo_build -- --ignored
 
+  # Manifest `[patch]` + the tagged detached lock (the v5 vendored cargo
+  # wiring) on cargo 1.41 and 1.56 — below / at the 1.56 floor of
+  # config-file `[patch]`. The old-toolchain tests prefer the local
+  # `rust:1.41-slim` / `rust:1.56-slim` docker images (run `--network none`;
+  # they build AND run the consumer) and skip when no old cargo is
+  # available; this leg pulls both images and requires them.
+  cargo-old-toolchains:
+    name: cargo old toolchains (manifest [patch])
+    needs: test
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - name: Install Rust
+        run: rustup show
+      - name: Pull the old cargos under test
+        run: |
+          docker pull rust:1.41-slim
+          docker pull rust:1.56-slim
+      - name: Cache cargo
+        uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          key: cargo-old-toolchains
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+      - name: Vendored manifest [patch] on old cargo
+        shell: bash
+        env:
+          SOCKET_PATCH_CARGO_E2E_REQUIRED: '1'
+          SOCKET_PATCH_CARGO_OLD_TOOLCHAINS_REQUIRED: '1'
+        run: |
+          set -euo pipefail
+          cargo test -p socket-patch-cli --test e2e_vendor_cargo_build -- old_toolchain --nocapture
+
   # ----------------------------------------------------------------------
   # Experimental `setup`-flow matrix (NON-BLOCKING).
   #

CHANGELOG.md

@@ -21,17 +21,20 @@ sha1 = "=0.10.6"
 hex = "=0.4.3"
 reqwest = { version = "=0.12.28", features = ["rustls-tls", "json"], default-features = false }
 tokio = { version = "=1.50.0", features = ["full"] }
+futures-util = { version = "=0.3.32", default-features = false, features = ["std"] }
 thiserror = "=2.0.18"
 walkdir = "=2.5.0"
 uuid = { version = "=1.21.0", features = ["v4"] }
 console = "=0.15.11"
 dialoguer = "=0.11.0"
 tempfile = "=3.26.0"
 regex = "=1.12.3"
+aho-corasick = "=1.1.4"
 glob = "=0.3.4"
 toml_edit = "=0.25.12"
 once_cell = "=1.21.3"
 qbsdiff = "=1.4.4"
+rayon = "=1.12.0"
 tar = "=0.4.46"
 flate2 = "=1.1.9"
 zip = { version = "=8.6.0", default-features = false, features = ["deflate"] }

Cargo.lock

@@ -531,6 +531,40 @@ warning — it never breaks a command or pollutes `--json` output. `socket-patch
 cloned repo must never be able to redirect where patches come from or spend your token.
 (Full rationale: [docs/design/configuration.md](docs/design/configuration.md).)
 
+One more env-only knob tunes *pacing* rather than routing. `scan` queries the patch API
+with several requests in flight: against the authenticated endpoint, a quarter of the
+requests a step has to make, between 8 and 32 (so a step with 128 or more requests runs
+32 at once, one with 32 or fewer runs 8); against the public proxy, which shares one
+server-side limit across anonymous callers, 4. The patch-record fetches behind `vex` and
+`scan --vex` run up to 10 at once (4 on the proxy).
+`SOCKET_API_CONCURRENCY=<n>` overrides that, clamped to `1`-`32`; on the public proxy it
+can only lower it. Set it when an endpoint in front of the API caps in-flight requests
+per client — a self-hosted `--api-url`, a corporate reverse proxy, a WAF or a CDN — and
+a scan starts reporting fewer patches than it should because some requests are being
+rejected. `SOCKET_API_CONCURRENCY=1` sends one request at a time, the slowest and most
+conservative setting. An unset, empty or non-numeric value leaves the defaults in place.
+
+A throttled patch API is retried, within bounds. An HTTP `429` or `503` answer to any
+patch-API query (batch search, patch lists, patch views, VEX record fetches, hosted
+package references) is retried up to 3 times, waiting as long as the server's
+`Retry-After` asks (seconds or an HTTP date; a request asked to wait more than 30 s gives
+up at once) or, without one, 0.5 s, 1 s, 2 s with jitter. All retries in one run must
+finish within 60 s of the run's first retry (wall-clock: requests waiting in parallel
+don't add up), so a heavily throttled run gives up instead of hanging. `SOCKET_API_MAX_RETRIES=<n>` changes
+the per-request count (`0`-`10`; `0` turns retries off). Other errors are never retried.
+A query still throttled after its retries is reported, never dropped: a failed batch
+prints `Warning: API batch <n> of <total> failed: …` (under `--json`, a top-level
+`warnings[]` entry with code `api_batch_failed`), a failed patch-list lookup prints
+`Warning: could not fetch details for <purl>: …` (`--json`: `patch_details_failed`), and
+if every query fails the scan exits 1 with an error, as before.
+
+The crawl has a pacing knob too. Its directory walks (`node_modules`, and the Maven
+repository with its POM parse) run on a small pool of threads: 4 by default (fewer on a machine with fewer performance cores), because the walk is bound
+by the kernel's directory cache and more threads only add system time.
+`SOCKET_WALK_THREADS=<n>` overrides that, clamped to `1`-`16` and to the machine's CPU
+count; an unset, empty or non-numeric value leaves the default in place. A soft open-file
+limit below 128 still runs the walk on one thread, whatever the knob says.
+
 The sections below list only each command's **command-specific** flags.
 
 ### `scan`
@@ -566,7 +600,7 @@ socket-patch scan [options]
 | `--mode <hosted\|vendored\|agent>` | — | Selects one of the three [patch modes](#three-patch-modes), summarized above. Combining `--mode` with a legacy boolean flag of a *different* mode is an error (exit 2); the same mode spelled both ways is accepted. |
 | `--prune` | — | Garbage-collect after the scan: remove manifest entries for packages no longer present in the crawl (installed trees + lockfiles — a wiped `node_modules` alone doesn't prune lockfile-listed entries) and delete orphan blob/diff/package-archive files. Off by default. [Vendored](#vendor) packages are exempt from the crawl-based prune (an absent installed copy is their normal state), but a vendored entry whose dependency has left the lockfile is reverted (and any manifest entry it still had dropped). Orthogonal to `--mode` — combines with any mode. |
 | `--detached` | — | Hidden compatibility no-op. Vendored mode is manifest-free by default: the vendor ledger (`.socket/vendor/state.json`) embeds the patch records and `.socket/manifest.json` is never written, so this former opt-in changes nothing. Still an error without `--mode vendored`. |
-| `--batch-size <n>` | `SOCKET_BATCH_SIZE` | Packages per API request (default: `100`). |
+| `--batch-size <n>` | `SOCKET_BATCH_SIZE` | Packages per API request (default: `500` on the authenticated API, `100` on the public proxy). A request whose body would exceed 256 KiB is split into smaller ones. |
 | `--all-releases` | `SOCKET_ALL_RELEASES` | Store patches for every release/distribution variant, not just the installed one — PyPI wheel/sdist, RubyGems platform, Maven classifier. Makes the manifest portable across environments (e.g. cross-platform CI caches). |
 | `--vex <path>` | `SOCKET_VEX` | On a successful scan, also write an OpenVEX 0.2.0 document to this path. See [Inline VEX generation](#inline-vex-on-apply--scan--vendor). |
 | `--vex-product`, `--vex-no-verify`, `--vex-doc-id`, `--vex-compact` | `SOCKET_VEX_*` | Passthrough to the embedded VEX builder; mirror the standalone [`vex`](#vex) knobs. Inert unless `--vex` is set. |
@@ -1246,7 +1280,7 @@ Behavior worth knowing:
 | pnpm | `pnpm-lock.yaml` (all generations), `shrinkwrap.yaml` (pnpm 1/2), Rush locks | Aliased / nested `resolution` shapes are diagnosed, not attested; `overrides` alone prove nothing |
 | yarn | `yarn.lock` (classic + berry) | Berry vendored entries also need the root `package.json` `resolutions` mapping; member locks are not read |
 | bun | `bun.lock`, else `bun.lockb` | A hosted entry that Bun < 1.3.10 re-saved without its sha512 attests only after install |
-| cargo | `Cargo.lock`, `Cargo.toml`, `.cargo/config[.toml]` | Root manifest + project config only (no `$CARGO_HOME` / parent configs); a lockless hosted pin needs the redirect ledger's record |
+| cargo | `Cargo.lock`, `Cargo.toml`, `.cargo/config[.toml]` | Root manifest + project config only (no `$CARGO_HOME` / parent configs); vendored `[patch.crates-io]` entries are read from `Cargo.toml` first (v5), the project config for pre-v5 projects, and must agree with the detached lock entry's tagged version `<version>+socket.<uuid>` (a tag for another uuid — in the lock or in the copy's own `Cargo.toml` — is dead wiring; an untagged detached entry counts only beside an untagged, pre-tag copy); a manifest entry cargo ignores (a same-key project-config item, or a URL-spelled crates.io `[patch]` table) is not attested; a lockless hosted pin needs the redirect ledger's record |
 | golang | `go.mod`, `go.work`, `go.sum`, `go.work.sum` | A replace that `require` no longer selects is inert; `vendor/modules.txt` is not read |
 | pypi | `uv.lock`, `*.py.lock`, `pylock*.toml`, `poetry.lock`, `pdm.lock`, `Pipfile.lock`, `requirements.txt` (+ `-r` includes), `pyproject.toml` / `hatch.toml` | A `uv.lock` beside a `pyproject.toml` must agree with its `[tool.uv.sources]`; PDM 3.1 / 4.0–4.2 locks are refused; a Pipenv project needs `--product` (or a git remote) |
 | gem | `Gemfile.lock`, `gems.locked` | Platform gems unsupported; a Gemfile-only (pre-bundler-2.6, not yet locked) wiring needs the redirect ledger |