fix(ipc): connect reliability for spawned wsdb/bb-avm-sim services

[charlielye]

Hardens the spawn-and-connect path used by the generated IPC packages (@aztec/wsdb, @aztec/bb-avm-sim), and makes bb-avm-sim failures non-fatal to the prover. Reviewing this stack against the bb socket incident fixed in #24802 showed the specific shared-budget bug from that incident does not reproduce here, but the same failure family was present — in the connect path, and in how environmental failures were classified downstream.

Commit 1 — connect reliability

Servers listen before heavy init. aztec-wsdb previously constructed its entire WorldState (LMDB open, genesis prefill) before listen(), and bb-avm-sim connected to its upstream wsdb/CDB servers before creating its own socket — so the client-side connect timeout was a bet on how fast a loaded machine can do real, unbounded work. Both servers now listen first: clients connect straight into the kernel accept backlog and first requests wait in the socket buffer until the reactor starts, so the connect backstop only ever covers exec + linking + reaching listen(). A server that dies during init surfaces its real exit cause instead of a spurious connect timeout. bb-avm-sim also installs its SIGUSR1 cancellation handler before the socket is reachable (the default disposition is process termination) and its lifecycle handlers — including parent-death monitoring — before the potentially-long upstream waits, whose budget goes from a hard-coded 5s to a 60s backstop.

Client-side liveness-based connect. The connect wait is raced against child death (a dead server fails immediately with its exit code/signal); the backstop (60s) is purely a broken-process detector and now kills the wedged child instead of orphaning it (an orphaned wsdb holds LMDB locks on its data dir, poisoning any respawn). Every spawn-failure path reaps the child, unlinks the ipc path, and points at the child's captured log. Hard connect errors (EACCES etc.) fail immediately instead of being retried until the deadline; EAGAIN (momentarily full accept backlog under simultaneous pool spawns) is retryable. destroy() escalates SIGTERM → SIGKILL after 5s.

One behavioural consequence: spawn resolving no longer implies the server finished initializing. All consumers issue their first call immediately after spawn and simply block until init completes; an init failure surfaces on that first call with the child's exit code and log path.

Commits 2+3 — the backend owns process lifecycle; callers keep pre-IPC semantics

Previously a bb-avm-sim spawn failure or crash was fatal to the prover: the error was rewrapped into SimulationError, the tx reported failed, the checkpoint prover failed, and the session manager blocks that epoch until a prune replaces the failed prover — i.e. forever (hasFailedProver). The pool also never evicted dead processes, so one crash permanently poisoned a slot; and the sequencer drops failed txs from P2P, so environmental failures evicted innocent txs.

The design principle: process lifecycle belongs to the layer that owns the process, and never appears above it. Pre-IPC (in-process NAPI), callers could assume every failure was an actual tx failure, with the processor's deadline as the only environmental bound; that contract is preserved exactly.

• ipc-runtime gains SpawnedProcessBackend, extracted from the codegen template (which embedded ~200 lines of process machinery per generated package, untestable except through them). It owns spawn, connect, death detection, teardown — and opt-in lazy respawn: the next call() after a death transparently gets a fresh process (one shared respawn attempt, stable ipc path, lazy-only so a crashing binary can't respawn-loop unprompted). Errors are typed — IpcTransportError, IpcProcessExitedError, IpcSpawnError — and carry a retry flag distinguishing process death from configuration errors. Call failures that race the child's exit event (the socket breaks first) are attributed to the death after a short grace, so deaths are never misreported as bare transport errors.
• Generated packages shrink to binary resolution + backend config, plus a respawn?: boolean spawn option. The call surface gains nothing.
• The AVM pool spawns its services with respawn: true (each simulation is self-contained; state is routed via WSDB/CDB by fork id) and is the only interpreter of the backend's error flags. It absorbs environmental trouble outright: spawn failures retry indefinitely on a flat 1s cadence — no backoff: sequencers live on ~6s slots, so sleeping longer after a failure costs whole blocks while the machine may have recovered, and slow attempts self-pace inside the backend connect backstop, bounded by the caller's own deadline — the abort signal threads through checkout, the spawn-retry loop, and full-pool waits, all of which stop promptly on abort. For the sequencer that bound is the slot deadline (execWithSignal); for the prover, the epoch deadline — meaning a checkpoint prover riding out a load spike just waits (loudly logged) and the epoch fails only at its true deadline, rather than being permanently poisoned after a fixed budget. Configuration errors (missing binary) still fail fast, at the boot-time prewarm.
• Poison-tx bound: a simulation whose process dies is re-issued once on the respawned process; a second death for the same input is attributed to the input and surfaces as an ordinary failed tx — so a simulator-crashing tx is evicted from the mempool instead of burning a process per slot forever. (Pre-IPC, the same event killed the whole node.)
• Guaranteed cancellation cleanup: on deadline abort, SIGUSR1 asks the C++ process to cancel at its next checkpoint; if it doesn't respond within 5s, it is SIGKILLed. Killing is safe because the service respawns lazily — a wedged simulation can no longer leak a pool slot.
• Upstream layers are untouched relative to pre-IPC: no retry classification in the public tx simulator, public processor, or foundation. A simulation produces a result, fails on its own merits, or runs until deadlined out.

wsdb deliberately keeps fail-fast semantics: a respawned wsdb loses all forks, checkpoints, and uncommitted state, so clients holding forkIds would silently read wrong state; its death remains a node-level event, surfaced with typed cause and log path.

Deliberately out of scope: SHM readiness/call timeouts (shm has no readiness handshake; nothing deploys shm-wsdb today — it's exercised only by a parameterized world-state test).

Tests

• ipc-runtime: unit tests for SpawnedProcessBackend against script-based fake servers — death → typed exit error, lazy respawn gets a fresh pid, missing binary flagged as configuration, dies-before-listen fails promptly with the exit code, wedged process killed at the backstop, destroy during a pending respawn leaks nothing.
• Echo ts_package reliability tests (run in ipc-codegen CI, uds + shm): slow-to-listen, dies-before-listen, wedged-then-killed (fails on the pre-fix(ipc): connect reliability for spawned wsdb/bb-avm-sim services #25073 code by construction), missing-binary classification, death-without-respawn, respawn-recreates-process.
• Simulator: 9 pool tests — indefinite flat-cadence spawn retry, abort stops the retry loop and full-pool waits, config errors propagate immediately, re-issue-once on process death, second death attributed to the input (no environmental flag escapes), destroy semantics. Public processor suite unchanged from pre-IPC semantics; full public_tx_simulator suite green.
• Verified end-to-end locally: aztec-wsdb/bb-avm-sim compile, a manual wsdb run shows listening on before Creating WorldState with a clean socket unlink on failed init, both generated packages rebuilt, full yarn-project TS build passes.