feat(supervisor): prototype coordinated gateway-scoped HA supervision

[drew]

Summary

Prototype [openshell.gateway] supervisor_topology = "gateway" to run one shared two-replica supervisor Deployment for the Kubernetes gateway installation. A PostgreSQL-backed lease elects one gateway process to place sandboxes, recover failed owners, and drive explicit drain/rebalance operations. Both supervisor replicas host sessions, and every gateway serves supervisor heartbeats, grants, activation, and owner lease renewal.

Failed-owner recovery and planned movement fence the previous supervisor and resume the same agent process and workload generation, preserving memory. Healthy assignments stay where they are unless an operator requests movement. Dedicated supervision remains the default. This draft is a runnable feasibility prototype for estimating the production implementation.

Core surfaces

• Configuration and driver deployment: supervisor_topology = "gateway" is the only topology switch. The Kubernetes driver maintains an installation-wide supervisor pool and preserves it when individual sandboxes are deleted. Requires the in-process Kubernetes driver, shared workspace mode, and gateway JWT authentication.
• Coordinator persistence: unique per-process holder, monotonic term, 5-second lease, and graceful release. Store::write_fenced validates holder, term, and expiry in the same transaction as assignment CAS, operation updates, and gateway-token rotation. PostgreSQL locks the lease row before sampling database time; SQLite provides equivalent transaction semantics. Followers do not reconcile assignments.
• Durable operations: assign/recover/drain/rebalance records expose phase, message, attempts, moves, and deadlines. Successors adopt pending work without relaunching agents or extending recovery deadlines. The coordinator admits one maintenance operation at a time; queued drains do not exclude replicas, and a drain that would leave no eligible replica is refused.
• Operator API: GetSupervisorPoolStatus, StartSupervisorPoolOperation, and CancelSupervisorPoolOperation require bearer authentication and platform_admin with the appropriate config scope. Drain targets a member; rebalance targets the pool; max_parallel defaults to one and is bounded to 1–8. Cancelling a completed drain returns the member to service. No CLI command yet.
• Supervisor/session boundary: multiple sessions keep separate policy, credentials, token caches, logging, and cancellation. New RELINQUISH releases ownership without stopping the workload; RELEASE retains stop/delete behavior. Planned moves use the existing signed recovery grant, runtime fence, receipt, and activation protocol.
• Runtime recovery extension: supervisor-recovery.v1 preserves the process and checkpointed executable-identity pins. Stop/delete wins. Pool retries are bounded; single-use grant and activation calls are not automatically retried after a lost response.

Existing network, SSH, exec, and forwarding connections break during a handoff and are not replayed. Missing the runtime recovery deadline stops the workload and reports SupervisorRecoveryFailed.

Validation

• Full commit hook passed: workspace/E2E/example/perf clippy, formatting, lockfiles, SPDX, Helm lint/docs, protobuf lint, Python lint/formatting, TypeScript lint/formatting, and Markdown/Mermaid validation.
• Persistence tests passed on SQLite and PostgreSQL, including stale-term rejection, expiry without takeover, transaction rollback, and expiry during a PostgreSQL row-lock wait. The persistence suite passed 86 tests; mutation tests exposed the unsafe clock-before-lock ordering.
• Pool/coordinator suite passed 23 tests covering election, follower-gateway renewal, operation adoption, stable placement, bounded drain, rebalance convergence, deletion precedence, and concurrent drain admission. Final affected suites also passed: 158 supervisor tests, 66 sandbox-backend tests, and 46 sandbox boundary-server tests. These include planned live-owner handoff, session isolation, RPC retry behavior, and two simulated-time RecoveryHello budget tests; the budget test caught the old timeout in a mutation check.
• API inventory/fingerprint and authorization descriptor checks passed. TypeScript SDK CI passed. Go bindings were regenerated; Go build/vet passed. Go tests had three gateway-list failures involving this host's existing system gateway configuration; Go lint also reported generated-code/style/cache findings, so Go CI is not reported as fully passing.
• Markdown lint and Fern documentation/navigation validation passed.
• mise run e2e:kubernetes:supervisor-topology passed twice on the isolated gsp-proto-ae7f cluster, including CLI conformance. The final successful run used rebuilt supervisor/sandbox images with the bounded RecoveryHello retry and completed the topology test in 192 seconds.
• Final live measurements: coordinator failover 4.53 seconds, maximum agent-counter observation gap 1.83 seconds, drain of two sessions 8.62 seconds, rebalance 5.08 seconds, supervisor-crash recovery 10.37 seconds, paused-owner recovery 10.22 seconds. Coordinator change preserved healthy owners; planned handoffs and recovery preserved PIDs and workload Pod UIDs. Policy/credential isolation and deletion checks passed.
• A first topology run exposed a test snapshot taken before drain/rebalance; the test now refreshes ownership before crashing an owner. A later conformance rerun hit a 30-second stop/start timeout waiting for the controller to create a workload Pod; the final instrumented rerun restarted successfully in 4.2 seconds and passed the full lane. Instrumented logs show a stop/start shutdown race: a lease-renewal refusal can self-fence and abandon a session already terminating, leaving the suspended Pod to consume its full 30-second termination grace. A start during that interval can exhaust the driver’s 30-second wait for the replacement Pod. This remains an explicit prototype limitation; the successful rerun does not resolve it.

Previously recorded dedicated Kubernetes regression: 96/97 passed, three skips, one failure in canonical_main_connect_recovers_its_ssh_transport. The same failure reproduced on the unmodified base 61d42102c in the same environment. This follow-up does not claim a new dedicated-lane run.

Remaining production work

• Session context uses thread binding, two Tokio workers per session, and endpoint-scoped credential lookup. Density, quotas, fairness, and bounded retention need work.
• Pool member RPCs use handler-level authentication; a dedicated auth mode and the external compute-driver integration remain pending.
• Fix the stop/start shutdown race described above and add a regression test.
• Fixed timing assumes small clock skew. Database outage/failover, node loss, and scale need validation beyond the demonstrated scenarios.
• Candidate TLS identity, policy-revision validation during recovery, and security review remain pending.
• Maintenance triggers are raw gateway APIs; rollout integration, PDBs, and full upstream-proxy configuration parity remain pending.
• Managed/operator workspace modes and intentional backend suspend/resume are outside the demonstrated scope. RFC 0012/0018 alignment remains pending.

Closes #4428 (prototype scope). Refs #4311 and RFC draft #4377.

Checklist

• Conventional commits and DCO signoff
• Focused tests and Kubernetes E2E coverage added
• Configuration docs, crate READMEs, and related skills updated
• Production architecture/security review and remaining validation complete

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.