feat(services): add continuous HTTP readiness checks

[drew]

Summary

Exposed sandbox HTTP services now report continuously observed health. Services check listener responsiveness by default; configuring a readiness path requires a 2xx response, and the Codex app-server example uses /readyz.

Related Issue

Closes #4229

Changes

• Probe the service's existing loopback target through the supervisor relay every five seconds, with a one-second timeout, three failures to become unhealthy, and one success to recover.
• Return cached health from service APIs and CLI output. Keep observations ephemeral and return unknown for stale observations, changed configuration, and unavailable or replaced runtimes. Health does not gate routing or restart applications.
• Add optional HTTP readiness configuration to service exposure and sandbox creation, CLI flags, and Rust, Go, and TypeScript SDKs. Probes send no credentials, follow no redirects, and consume only response headers.
• Update the Codex app-server example, published service documentation, and CLI skill.

Service contract

A service remains an exposed loopback HTTP port identified by workspace, sandbox, and service name (empty name selects the unnamed endpoint). Readiness uses that same target port and relay; it does not require another listener or route.

Configuration and response

ExposeServiceRequest and create-time SandboxServiceExposure accept an optional readiness_check. The persisted ServiceEndpoint returns the effective configuration. Existing expose/get/list RPCs return health alongside the endpoint and URL:

message HttpReadinessCheck {
  string path = 1;
}

message ServiceEndpointResponse {
  ServiceEndpoint endpoint = 1;
  string url = 2;
  ServiceHealth health = 3;
}

message ServiceHealth {
  ServiceHealthState state = 1;
  google.protobuf.Timestamp last_checked_time = 2;
  string message = 3;
  optional uint32 http_status_code = 4;
}

enum ServiceHealthState {
  SERVICE_HEALTH_STATE_UNSPECIFIED = 0;
  SERVICE_HEALTH_STATE_UNKNOWN = 1;
  SERVICE_HEALTH_STATE_HEALTHY = 2;
  SERVICE_HEALTH_STATE_UNHEALTHY = 3;
}

Effective configuration	Probe	Passing result
No readiness check	GET /	Any HTTP response status; confirms listener responsiveness
readiness_check: { path: "/readyz" }	GET /readyz	HTTP 2xx; confirms application readiness

An empty configured path resolves to /. Paths must begin with /, be at most 1,024 bytes, and contain no query, fragment, whitespace, or alternate host. Omitting the check when updating an existing service preserves its configuration. Removing the check requires deleting and recreating the endpoint in this version.

Continuous observation

• Checks run while the sandbox is ready, on a five-second timer with a one-second deadline covering relay setup and response headers. They send no application credentials, follow no redirects, and do not consume response bodies.
• State starts unknown. One passing check makes it healthy; three consecutive failed checks make it unhealthy. One passing check recovers immediately. A healthy service stays healthy through the first two failures.
• Configuration changes, replaced runtimes, unavailable supervisor relays, stopped/non-ready sandboxes, and observations older than 15 seconds produce unknown. An inconclusive relay observation resets the consecutive-failure count.
• last_checked_time, message, and optional http_status_code describe the latest attempt. The code is present only when response headers arrived. Clients should branch on state, because thresholds can leave the state healthy while the latest attempt failed.
• Health is an ephemeral cache per gateway replica. Get/list reads do not launch probes or mutate the endpoint's resource version. Gateway restart clears observations; durable mutation replay omits health. Missing health from an older gateway, and an unspecified state, are interpreted as unknown.
• Health is observational: it does not gate routing, change sandbox phase, restart the service, or certify credentials/model access.

The CLI adds service expose --readiness-path PATH and sandbox create --expose-readiness-path PATH (requires --expose). Service output shows Ready / Not ready for configured readiness and Responsive / Unresponsive otherwise. The Codex example sets --expose-readiness-path /readyz.

Testing

• Checks appropriate to the affected code and behavior pass: mise run pre-commit, schema inventory, scoped Rust checks, TypeScript SDK CI, focused Go SDK tests and lint, and docs checks.
• Unit tests cover readiness validation and preservation, thresholds and recovery, stale/runtime/configuration invalidation, old wire payloads, and SDK conversions.
• OPENSHELL_E2E_DOCKER_TEST=service_bearer_passthrough mise run e2e:docker passes, including readiness transitions, closed targets, stopped sandboxes, unchanged routing, and authorization passthrough.
• Started the installed Codex app-server on loopback and confirmed /readyz returns HTTP 200 without credentials.

Full mise run go:ci reaches unrelated gateway tests that assume no system gateway configuration; this machine has /etc/openshell/gateways/default. Focused service/client/converter tests and Go lint pass.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4230.docs.buildwithfern.com/openshell

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.