fix(smoke): split staging gate advisory-vs-fatal + O(1) snapshot sweep

[unforced]

Why

deploy-staging.yml (auto-runs on every push to main) has been red for 9 days — 25+ consecutive runs, the identical single failure each time:

SMOKE FAILED — 163 pass, 1 fail
FAIL  snapshots: live restore round-trip threw (non-fatal — sections continue) — TimeoutError

Two independent defects compounded:

1. A slow section hard-blocked prod. fail() increments a counter that process.exit(failures === 0 ? 0 : 1) reads. The §17 catch called fail(), so "non-fatal — sections continue" only meant the exception didn't abort the rest of the run — it was still fatal to the exit code. One timeout = red gate = merges to main don't deploy. Three finished changes sat undeployed.
2. The timeout was a fleet-size timebomb. The §17 snapshot sweep snapshotted every staging vault to R2 (O(fleet)); the fleet grew 167 → 260 as smoke debris accumulated and crossed the client timeout. Because the sweep runs first, the restore assertion behind it went dark for 9 days.

The rule this encodes — advisory vs fatal, as a category, not a list

Lives in scripts/smoke-report.ts (pure, side-effect-free, unit-tested). The next person classifies a new section by this rule without asking:

• FATAL — gates the deploy (exit 1). The checked thing is broken: a wrong answer is the failure. Contract shapes, status codes, auth refusals, scope boundaries, error types, note counts. Every assert() in the smoke is fatal. Ambiguous → fatal (safe default).
• ADVISORY — loud, but never gates (exit still 0). We couldn't verify it right now: a client-side timeout or network-unreachable while driving live third-party / fleet infra (Workers AI, R2, a cold DO, a fleet sweep). A timeout can't tell slow-but-healthy from stuck-broken, so it's never proof of breakage — only proof we didn't get to check. isUnverifiable() is the only downgrade path; any other throw (TypeError, bad-shape parse) stays fatal.

Why this holds per-section without a hand-maintained list: each live section's catch routes through liveCatch() — timeout/unreachable → advisory, everything else → fatal. The contract checks inside each section are assert()s, and assert() never throws (it records a fatal fail() and continues). So a wrong answer — e.g. account-MCP failing to refuse a vault creation (the case flagged as a genuine contract break) — stays fatal regardless of the shared try/catch. Only a couldn't-complete becomes advisory. A section that times out before reaching an assertion simply leaves it unverified (correctly advisory); one that fails an assertion then times out records the fatal first, and invariant 2 keeps it gating.

Three invariants, pinned in test-bun/smoke-report.test.ts:

1. Advisories never gate — exitCode is a function of fail alone.
2. Advisories can't hide a fatal — fail > 0 exits 1 and reads FAILED no matter how many advisories rode along (tested at 0/1/5/99).
3. Advisories are never silent — any advisory shows in the headline and on its own re-surfaced block; a run carrying advisories never reads as clean green. SMOKE PASSED — 163 pass, 0 fail, 2 advisory (UNVERIFIED — see below).

Section classification (all via the same rule): §17 snapshots, §18 voice, §18b semantic, §18c tickets, §16/19 mock-E2E, §account-mcp, tier-change — for all of them a wrong answer is fatal (assert()), and only a live-infra timeout is advisory. That's the category rule applied uniformly, not a per-section allowlist.

The slowness — scoped, not bumped

/__test/snapshot-run now takes an optional ?vault=<name> that scopes runSnapshotSweep to a single vault. Smoke §17 passes its own fresh arrival vault (which the section already depends on for the restore round-trip), so the sweep is O(1), removing the fleet-size dependency entirely. The nightly cron is untouched — it never sets onlyVault, so it still sweeps the whole fleet. The 120s timeout was deliberately NOT raised — raising it is what the last attempt did and why this became a 9-day outage.

scripts/staging-sweep.ts (the debris cleaner) is still wired into no workflow, so the fleet still grows — but with the sweep scoped, fleet size no longer gates. Wiring that cleaner is a recommended follow-up, not part of this gate fix (it mutates the fleet and deserves its own review).

Expect this on the first post-merge staging run

Once the sweep is fast, the restore assertion runs for the first time in 9 days. It may fail — if it does, that is a real finding to report, not a defect in this change.

Verified vs reasoned

Verified (ran locally, all green):

• bun run typecheck (root) · bun run test (root — 177 tests incl. the 10 new smoke-report invariant tests)
• workers/identity bun run typecheck · full identity vitest under workerd — 1050 tests across 38 files, incl. the new onlyVault sweep-scope + ?vault= trigger tests (run with singleWorker to sidestep a sibling worktree saturating workerd machine-wide; the multi-worker crashes were 100% environmental — zero test-logic failures; the affected file also passes standalone).

Verified (read against ground truth): rc bump 125→126; staging-sweep.ts unwired; the CI gate keys off the smoke's exit code; no throw inside any wrapped section; account-MCP create-refusal is an assert(); arrivalVault is a live, section-required variable.

Reasoned, NOT executed: scripts/smoke-staging.ts was not run (it creates real debris against live staging, per instruction). The live behavior — sweep now O(1), restore finally exercised — is reasoned from the code, not observed.

Scope

Identity worker + smoke scripts only; no workers/vault, no @openparachute/core. rc.125 → rc.126.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB