Keep credential references current after external rotation

[shiju-nv]

User Story

As an operator, I want a running sandbox client to use an externally rotated credential through the reference it already holds, so I can retire the old credential without interrupting its session.

Problem Statement

OpenShell already supports provider credential updates. PR #2780 also keeps references stable for gateway-managed refresh: when OpenShell refreshes a token, the client's existing reference resolves to the current token.

The remaining gap is for credentials supplied and rotated outside that managed-refresh lifecycle. These are static credentials in the provider model, even when an external service rotates their values. Their issued references identify a particular credential revision. After the supervisor applies an update, a reference held by an existing process can still resolve to the previous value while that revision is retained.

We reproduced this on upstream commit 1ad4e428a67c2a0869fd2043159acacea5d4683e on October 1, 2026: after B was installed and acknowledged, the same client process sent revoked A through its original reference and received HTTP 401. A fresh process used B successfully. The requested behavior is for the existing process to use B through the reference it already holds.

Concrete Use Case

Our motivating workload is a long-running Hermes agent calling MCP tools through AgentGateway and Agent Tool Gateway. A host service signs and rotates a short-lived caller assertion, then updates its OpenShell provider. Hermes holds an OpenShell reference in the x-openshell-sandbox-assertion request header. The issuer owns signing and rotation, so OpenShell treats the assertion as an externally supplied static credential.

When assertion A expires or is revoked after B is installed, the next tool call needs to use B without restarting Hermes. The agent keeps its session, and the real assertion stays outside the agent process.

Impact / Why This Matters

Revoking A can break an active session even though OpenShell has already installed B. Restarting the client gives it a fresh reference, but interrupts ongoing work and can discard in-memory state. Keeping A valid longer postpones the failure and delays retiring the old key.

Proposed Design

Extend the existing stable-reference behavior to externally supplied static credentials. The external system continues to own rotation and sends the new value through the existing provider update workflow. After the supervisor applies a value-only update, the client's next authorized request using its original reference uses the new value.

The reference remains bound to the same sandbox, provider instance, credential key, and authorized endpoint. Changes to those boundaries must revoke access to the replacement credential. The real credential stays outside the workload.

PR #3339 proposes credentials[].stable_placeholder: true and reuses the existing resolver. The opt-in preserves today's revision-scoped default; it is a compatibility choice, separate from the requirement to keep a running client working through external rotation.

Acceptance Criteria

• One persistent client keeps its original reference across a value-only update from A to B. After supervisor activation, its next request reaches a controlled HTTPS backend using B, without restarting or reloading its reference.
• Repeated rotations and reconstruction of supervisor credential state preserve this behavior for the same identity and endpoint binding.
• Changing the sandbox, provider instance, credential key, or authorized endpoint prevents the old reference from accessing the replacement credential.
• Expiry of the current credential or an applied provider detach stops resolution without falling back to an older value. An independent reachability check distinguishes revocation from a network outage.
• Existing endpoint and binary restrictions still apply, including host, port, and path checks. TLS verification remains enabled, and hand-written aliases cannot grant access.
• The workload never receives the real upstream credential. Tests and diagnostics do not print credentials or issued references.
• For the proposed opt-in, existing revision-scoped and managed-refresh behavior remain unchanged. Unsupported profile writes are rejected before mutation, and unsupported delivery or malformed credential bindings cannot grant access.

Alternatives Considered

Restarting clients or teaching each application to reload its reference adds disruption or application-specific work. Keeping both keys valid longer delays revocation without making the client use the new key.

Gateway-managed refresh already provides continuity when OpenShell owns token refresh. Externally issued credentials need that continuity while their existing issuer retains control of rotation.

Making current-value resolution the default for all static credentials would avoid an opt-in, but would change existing revision-scoped behavior. The candidate preserves that default.

Agent Investigation

The October 1 reproduction used Linux Docker runtimes built from upstream commit 1ad4e428a67c2a0869fd2043159acacea5d4683e. A persistent Python client sent a synthetic JSON-RPC tools/call for local_hello over verified HTTPS, using the caller-assertion header from the Hermes workload:

1. Start the client with an ordinary static-credential reference and confirm that its first request uses A.
2. Make the backend reject A, install B with one provider update, and wait for the supervisor to acknowledge the exact provider and policy revisions.
3. Send one request from the same process using the unchanged reference. The backend observes A and returns HTTP 401.
4. Start a separate client process in the same sandbox. Its fresh reference uses B successfully.

The backend identifies A and B independently, so the failure proves use of the retired credential after installation of its replacement. Endpoint and binary restrictions, TLS verification, and detach checks also passed. This replay tests the authentication path from our workload; it does not claim a fresh deployment of the complete Hermes, AgentGateway, and Agent Tool Gateway stack.

The exact upstream retained-generation unit test also passed. It deliberately expects the old static reference to resolve to the old value after an update. PR #3339 carries the persistent-client regression as ordinary_external_placeholder_keeps_revoked_assertion_after_rotation, alongside the proposed stable-reference case.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request

[johntmyers]

I thought #2780 was intended to solve this @mrunalp. At first look this feels like it puts more burden on the user for things that "should just work" - tagged for review @ community call.

[github-actions]

This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.