fix(vector-index): repair per-bank index coverage after restore/upgrade (#2645)

[nicoloboschi]

Fixes #2645.

Problem

Per-(bank, fact_type) partial vector indexes (idx_mu_emb_{obsv,expr,worl}_<internal_id16>) are created only at fresh-bank-creation time (gated on INSERT ... ON CONFLICT DO NOTHING RETURNING). A bank that becomes populated outside that path never gets them, so its bank-scoped recall silently falls back to the global index + a bank_id/fact_type post-filter — which is both slower and under-returns results (the ANN candidate set is drawn across all banks, then filtered to the target bank afterward). As measured by the reporter on a live 40-bank deployment: recall@10 of ~0.63–0.72 — a silent ~30% top-10 miss — plus 14×–100× slower plans.

A bank reaches this uncovered state via a logical restore, a cross-version upgrade, or a vector-extension switch (e.g. ScaNN→pgvector).

Two roots, two fixes

1. The one in-app leak: import-bank. importer.py restores the banks row directly, then calls get_or_create_bank_profile intending to "ensure the indexes exist" — but because the row now exists, that call takes the SELECT branch (created=False) and skips index creation. So every imported/restored bank landed uncovered. Fixed by creating the per-bank indexes explicitly right after the banks row is restored, while the bank is still empty (facts are imported afterward, so the build is instant).

2. The out-of-app routes: a re-runnable repair command. Raw pg_dump/restore and extension switches never run application code, so there is no hook to plug — the escape hatch has to be an operator command. A one-time migration can't cover these: a logical restore carries alembic_version at head, so a backfill migration is already stamped "applied" and skips. hindsight-admin repair-bank is re-runnable — run it once post-upgrade, and again after each restore/switch.

hindsight-admin repair-bank (--bank BANK_ID | --all) [--schema S] [--dry-run]

• Detects missing OR invalid coverage. The health check requires the index to be valid+ready, over memory_units, on a supported access method, and to carry the partial predicate. A name-only check (pg_indexes + IF NOT EXISTS) would treat an INVALID leftover from an interrupted build, or an index whose type drifted after a backend switch, as "present" and never repair it — this rejects both and rebuilds them.
• CREATE INDEX CONCURRENTLY on a raw autocommit connection — never takes ACCESS EXCLUSIVE on the shared memory_units, so repairing one bank can't stall the fleet.
• Invalid-leftover cleanup on failure so a re-run retries cleanly; per-schema isolation so one bad tenant can't abort the sweep.
• Idempotent and safe to run concurrently — every build is CREATE INDEX CONCURRENTLY IF NOT EXISTS gated by a valid/ready health check, so concurrency is handled by idempotency rather than a lock (per the project rule against advisory locks, which are unreliable behind poolers). --dry-run reports without touching anything, multi-tenant (base + discovered tenant schemas), no-op for global-index backends (ScaNN/Oracle).

Deliberately NOT included

No boot/periodic background reconcile and no retain-path self-heal hook. Those add a maintenance-loop job, an advisory-lock-gated worker connection, a PgBouncer/transaction-pool guard, and a hot-path catalog check — a much larger surface. The accepted tradeoff: a bank restored and then only ever read stays degraded until an operator runs repair-bank. For deployments that do routine bare-metal restores and want zero-touch convergence, that background layer can be a follow-up; this PR is the minimal, self-contained fix for the leak and the operator escape hatch.

Tests

tests/test_repair_bank_vector_indexes.py (deterministic, pg0 — index presence/shape via the catalog, no LLM):

• import-bank round-trip (export → delete → import) leaves the restored bank with its per-bank partial indexes (the leak regression);
• repair-bank --bank / --all rebuild dropped indexes;
• an invalid-shape name collision (a plain btree under the expected index name) is detected as unhealthy and rebuilt — the differentiator vs a name-only check;
• --dry-run creates nothing; re-run is idempotent (no dupes); exactly one of --bank/--all is required; no-op backend is a no-op.

Plus admin-CLI docs for the new command.

[nicoloboschi]

Relationship to the other open PRs on #2645:

• feat(admin): backfill-vector-indexes command to reconcile per-bank vector index coverage (#2645) #2712 (admin backfill command) and feat(maintenance): self-heal per-bank vector indexes at startup + on schedule (#2645) #2803 (stacked boot/periodic self-heal) implement the fuller design agreed in the issue thread. This PR is an intentionally smaller alternative: plug the one in-app leak (import-bank) + a re-runnable operator command, and deliberately drop the background reconcile / retain-path hook.
• It reuses the correct valid/ready + partial-predicate health check from feat(maintenance): self-heal per-bank vector indexes at startup + on schedule (#2645) #2803 (not the name-only pg_indexes check in feat(admin): backfill-vector-indexes command to reconcile per-bank vector index coverage (#2645) #2712, which — as @iRonin flagged — treats an INVALID leftover as "present" and never repairs it).
• One divergence forced by project standards: no advisory lock. The repo forbids pg_advisory_lock (unreliable behind poolers), so concurrency is handled by idempotency (CREATE INDEX CONCURRENTLY IF NOT EXISTS gated by the health check) instead of feat(maintenance): self-heal per-bank vector indexes at startup + on schedule (#2645) #2803's pg_try_advisory_lock.

Happy to fold in the background reconcile as a follow-up if maintainers want zero-touch convergence for restored-then-read-only banks; this PR closes the leak and gives operators the escape hatch.

Note: the pg0-backed integration tests could not run in my local worktree (embedded Postgres wouldn't boot here); they need CI to validate. The non-DB test and all lint/type/dead-code checks pass locally.