feat(sandbox): explain failed calls to external tool servers (#3207)

Show configured tool server addresses and their last observed connection
results together in sandbox status. Keep sandbox lifecycle readiness
separate so an external connection failure does not mark the sandbox unready.

Expose direct endpoint records through the CLI and SDKs, with plain-language
failure explanations and gateway acceptance times. Keep observation tracking,
runtime reporting, and gateway validation in dedicated endpoint status modules.

Preserve bounded reporting, request attribution, retry ordering, and
configuration and supervisor authority checks. Clear obsolete observations
while retaining the configured addresses, and document the distinction
between an observed HTTP response, current availability, and tool success.

Signed-off-by: Shiju <shiju@nvidia.com>

Author: shiju-nv

Cargo.lock

@@ -334,16 +334,10 @@ The storage schema is intentionally narrow:
 
 ### Protobuf API and storage boundaries
 
-Public RPC contracts and durable protobuf formats have separate ownership. The
-`openshell.v1.OpenShell` service currently has 74 RPCs. Their request and
-response roots, streaming flags, and transitive message closure come from the
-public descriptor set generated by `openshell-core`; a fingerprint test in
-`openshell-server` requires this inventory to be reviewed whenever it changes.
-Compute-driver, credential-driver, gateway-interceptor, and
-supervisor-middleware services are compiled contracts for internal extension
-boundaries, not public gateway RPCs. The current public inventory has 74
-methods, 278 messages, and 12 enums
-(`8ac68c71d93e6a5e56406b8df1882ee40c6066270969e03eb99803f0e6396fc1`).
+Public RPC contracts and durable protobuf formats have separate ownership. The `openshell.v1.OpenShell` service's request and response roots, streaming flags, and transitive message closure come from the public descriptor set generated by `openshell-core`. The `public_and_durable_schema_inventories_are_complete` test in `openshell-server` owns the counts and fingerprints and requires this inventory to be reviewed whenever it changes. Compute-driver, credential-driver, gateway-interceptor, and supervisor-middleware services are compiled contracts for internal extension boundaries.
+
+`ReportEndpointStatus` is a sandbox-authenticated public gateway RPC. Its request, response, and `EndpointObservation` messages belong only to the public closure. `EndpointStatus` and `EndpointResult` also belong to the durable closure because `Sandbox.status.endpoint_statuses` persists them. The repeated status field uses a new wire tag; stored sandboxes without it decode with an empty endpoint list and retain their lifecycle fields. A fixed payload encoded with the earlier sandbox schema verifies that no database rewrite is required.
+
 The removed `NetworkBinary.harness` field remains reserved by number and name,
 so protobuf implementations cannot reuse its wire slot or source identifier.
 The durable-policy compatibility decoder reads the former boolean before Prost
@@ -363,12 +357,7 @@ Go, Python, and TypeScript client generation inputs do not advertise them.
 | Public messages used directly as encoded storage roots | `Sandbox`, `SandboxWorkloadTemplate`, `Provider`, `Workspace`, `WorkspaceMember`, `SshSession`, `ServiceEndpoint` | The generated public type is also the persisted payload. `SshSession` is not in the current public RPC message closure. |
 | Embedded encoded root | `SandboxPolicy` | Stored in policy rows and inside the JSON settings envelope. |
 
-The 12 encoded durable roots above have a closure of 81 messages and eight
-enums (`369b36511c2e38b9df9621704a00123516c7538d8ee89a499158d7de5cee1882`).
-Its intersection with the public RPC closure contains 71 messages and eight
-enums (`05add438ba041defc98d791038ae593d3f09352677cae43f2276d494205ce415`).
-The descriptor-derived test owns these full inventories; the tables here record
-the reviewed roots and classifications.
+The descriptor-derived test inventories the complete message and enum closure of the encoded durable roots and its intersection with the public RPC closure. The tables here record the reviewed roots and classifications.
 
 | Dual-purpose encoded root | Current decision |
 |---|---|

architecture/gateway.md

@@ -361,6 +361,26 @@ auth file without the acknowledgement, and the acknowledgement without an
 auth file, so credentials are never sent in cleartext without an explicit
 operator decision.
 
+## Tool server connection status
+
+A sandbox can be `Ready` while a call to an external tool server fails. For configured endpoints that use MCP over HTTP, OpenShell records the last observed network result beside the endpoint's address. Users can identify the server and distinguish policy denial, unavailable credentials, TLS or network failure, and an upstream HTTP rejection without combining client and supervisor logs. These observations do not affect sandbox lifecycle readiness.
+
+```mermaid
+flowchart LR
+    Traffic[Calls to configured tool servers] --> Observer[Observe network result]
+    Observer --> Reporter[Background reporter]
+    Reporter --> Gateway[Validate and store results]
+    Gateway --> Status[Endpoint address, last result, report time]
+```
+
+The gateway exposes one record per configured endpoint in `Sandbox.status.endpoint_statuses`. Each record contains its address, an opaque identifier, a typed result, and the time the gateway accepted that result. The address remains available when evidence resets to `NoObservedExchange`. Ordinary sandbox conditions continue to describe lifecycle and platform state.
+
+Network observers send only the endpoint identifier and a fixed result classification, without credentials, payloads, or raw upstream errors. Requests capture observation authority before selecting policy or credentials, then bind the selected policy hash and provider revision to that capture. Observation handles also identify the installed endpoint inventory and supervisor authority, so a concurrent update cannot attribute an old request to a new configuration, and reinstalling the same configuration cannot revive an obsolete request. The sandbox reporter coalesces observations and retries an immutable snapshot through `ReportEndpointStatus`. Bounded delivery can drop observations, so endpoint status is not a complete request history.
+
+The gateway validates the reporting supervisor's session, configuration revisions, and report sequence before atomically storing results. Global policy writes share the report's synchronization boundary. An effective policy change resets all endpoint evidence; repeated acknowledgements and metadata-only policy revisions preserve it. A provider environment change resets evidence for endpoints that depend on those credentials. Supervisor disconnection or replacement and gateway restart also invalidate observation authority. Identical report retries leave timestamps unchanged. After an inventory reset, still-valid pending evidence can be accepted again if an acknowledgement was lost, advancing the report time without another exchange.
+
+These are passive observations with no expiry. `HttpResponseReceived` means the server returned a final HTTP status below 400, including a protocol upgrade; that response can still contain an MCP error. Informational responses alone do not establish success. MCP protocol-version and request-body policy rejections produce `PolicyDenied`. A failure before the HTTP path is known updates status only when the host and port identify one distinct endpoint. Ambiguous failures remain in structured events and logs. Results combine callers and effective ports for an endpoint. Consumers that need current tool availability must verify an actual operation. The sandbox management guide explains the public fields and results.
+
 ## Credentials
 
 Provider credentials are stored at the gateway and fetched by the supervisor at