feat(maintenance): self-heal per-bank vector indexes at startup + on schedule (#2645)

[ijevin]

Summary

Stacked follow-up to #2712: turn the one-shot backfill-vector-indexes
CLI into a real self-heal. After this PR, the API:

• Runs the same reconciliation at startup and on a configurable schedule
  (HINDSIGHT_API_VECTOR_INDEX_RECONCILE_INTERVAL_SECONDS, default 3600s,
  0 disables).
• Reuses the per-bank partial-index catalog check and reconcile_vector_indexes()
  helper across CLI / boot / periodic paths, so a name-only false positive
  never silently satisfies reconciliation.
• Defers repair to the background loop from the hot retain path using a TTL
  cache (default 60s) and only signals request_vector_index_reconcile() —
  no DDL on the request connection.
• Holds a session-level pg_try_advisory_lock so two API replicas cannot
  rebuild the same index in parallel.
• Detects Postgres transaction-pool URLs (PgBouncer port 6543 /
  pool_mode=transaction) and refuses to reconcile unless the operator
  sets HINDSIGHT_API_MIGRATION_DATABASE_URL to a direct libpq URL —
  session-level advisory locks and CREATE/DROP INDEX CONCURRENTLY
  require a single backend session.
• Uses server-side quote_literal() for the bank_id predicate literal
  so escaping does not depend on standard_conforming_strings.
• Wraps each tenant schema in a try/except so a single bad schema cannot
  abort the rest of the sweep.
• Surfaces advisory-lock contention to the CLI as exit code 75
  (EX_TEMPFAIL) with a retry message rather than false-positive success.

The stacked commits, in order, are:

1. c0def121 — feat(maintenance): self-heal per-bank vector indexes (#2645)
2. f279ac4b — fix(reconcile): harden vector index self-heal per review (#2645)
3. 1d49aa73 — fix(reconcile): match pg_get_indexdef predicate shape and default new field (#2645)
4. a71c27e7 — fix(config): make new reconcile interval field keyword-only

Base

This PR is stacked on top of #2712 (8d45a64b,
fix/backfill-vector-indexes-2645) and is not intended to merge
until #2712 lands. Reviewing the four commits in order diff-by-diff against
8d45a64b will surface only the part-2 change.

Test plan

• uv run pytest tests/test_vector_index_reconcile.py tests/test_vector_index_retain_safety.py tests/test_maintenance_loop.py -k 'not (reconcile_submits_eligible or purge_expired)'
• uv run pytest tests/test_config_validation.py tests/test_main_module.py
• uv run ruff check ... and uv run ruff format --check ...
• uv run ty check hindsight_api/admin/cli.py hindsight_api/config.py hindsight_api/engine/maintenance.py hindsight_api/engine/memory_engine.py hindsight_api/engine/vector_index_reconcile.py
• Local root user cannot run pg0 initdb; the backfill integration tests
  (tests/test_backfill_vector_indexes.py) must be verified on CI or under
  a non-root user.

Notes for reviewers

• The retain-side _maybe_request_vector_index_reconcile() TTLs catalog
  checks per (schema, bank_id) using vector.monotonic() and never
  executes DDL on the request connection.
• bank_vector_indexes_healthy() and the full reconcile query reject
  indexes that are merely name-matching the expected relname; the check
  also requires valid/ready, memory_units as the indexed table, a
  recognised access method, and the canonical
  pg_get_indexdef partial-predicate shape
  (' WHERE ((fact_type = ').
• The kw_only field added in a71c27e7
  (vector_index_reconcile_interval_seconds) preserves all positional
  callers; the existing from_env() defaults (300 / 60 /
  ['consolidation.completed'] / 30) are unchanged.

[nicoloboschi]

Closing in favor of #2872 (merged as bd853be).

Thank you @ijevin — the health-check logic in #2872 is directly the correct detection approach from this PR (valid/ready + memory_units + supported access method + pg_get_indexdef partial-predicate shape), rather than the name-only check it replaced. That part was the load-bearing insight and it carried over.

#2872 is an intentionally smaller scope than this PR: it keeps the re-runnable command + the import-bank leak fix, but drops the boot/periodic background reconcile and the retain-path self-heal hook. Two reasons:

1. Project rule: no advisory locks. The repo forbids pg_advisory_lock/pg_try_advisory_lock (unreliable behind connection poolers — see the Database Locking standard). The session-level advisory lock this PR uses to serialize the reconcile pass isn't allowed, and the background reconcile leans on it. fix(vector-index): repair per-bank index coverage after restore/upgrade (#2645) #2872 handles concurrency by idempotency instead (CREATE INDEX CONCURRENTLY IF NOT EXISTS gated by the health check).
2. Scope: we opted to ship the leak fix + operator command first and treat always-on convergence as a separate decision.

If we later want zero-touch convergence for restored-then-read-only banks, the boot-reconcile can come back as a follow-up — but it'll need a lock-free coordination approach (per-process idempotency / row constraints) rather than the advisory lock. Really appreciate the thoroughness here; it shaped the merged result.