Skip to content

feat(gateway): broker stable sandbox SPIFFE provider tokens - #4732

Draft
drew wants to merge 15 commits into
feat/4700-shared-kubernetes-supervisor/drewfrom
feat/4731-stable-sandbox-spiffe/drew
Draft

feat(gateway): broker stable sandbox SPIFFE provider tokens#4732
drew wants to merge 15 commits into
feat/4700-shared-kubernetes-supervisor/drewfrom
feat/4731-stable-sandbox-spiffe/drew

Conversation

@drew

@drew drew commented Oct 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Let the Kubernetes shared supervisor use each sandbox's stable SPIFFE identity while keeping SPIRE access, JWT-SVIDs, and provider exchanges on the gateway. The supervisor authenticates with its gateway-issued sandbox session and receives only the final provider access token for injection into an admitted upstream request. Sandbox workloads receive no SPIFFE sockets or credentials.

Stacked on #4702, above #4701, #4695, and #4656. The previous supervisor-owned implementation is preserved on the backup branch.

Related Issue

Closes #4731

Changes

  • Add sandbox-authenticated GetSandboxProviderAccessToken. The caller supplies only a gateway-issued, endpoint-bound credential key. The gateway checks the signed session, persisted generation, credential epoch, token lineage, and active state; attachment, endpoint, audience, and scopes come from trusted configuration.
  • Own the SPIRE Delegated Identity API client in the gateway. Derive <prefix>/gateway/<gateway-id>/sandbox/<sandbox-id> from verified identity, require exactly the expected SVID, and verify signature, audience, and expiry against the agent's current trust bundle. Stop/start preserves identity; recreating the sandbox changes it.
  • Perform RFC 7523 grants or both RFC 8693 stages on the gateway. Cache final tokens by sandbox generation, credential epoch, provider revision, and grant configuration. Preserve exact TTL overrides and caps from SVID, session, and subject expiry.
  • Separate GatewayTokenGrantResolver from SpiffeTokenGrantResolver, both implementing the existing resolver trait. Gateway mode holds only an authenticated gateway endpoint, with no local SPIFFE client, HTTP client, or grant cache. Dedicated supervisors retain their Workload API path, intermediate exchange, and local cache.
  • Parse immutable delegation settings into gateway state at construction. CLI and embedded startup reject invalid settings. Keep provider configuration and SPIRE trust bundles live. Cache hits and misses share one final configuration and authorization check; new tokens enter the cache only after that check passes. Authorization checks around remote operations remain in place.
  • Configure delegation under openshell.gateway.experimental_spiffe_delegation. Helm mounts the private admin socket only on the gateway; its component Workload API setting remains available. Update the existing SPIRE fixture, generated Go bindings, and gRPC mocks for the additive RPC. Public documentation is deferred.

SPIRE trusts the gateway as a delegate: operators must provision sandbox registrations within its agent's scope and control node placement. This does not add assignment-scoped SPIRE authorization, automatic registration, HA, other-driver enablement, or validated Kata support. Already-issued provider bearer tokens retain their normal expiration. Existing transport/session identifiers remain compatible.

Release Notes

Testing

  • Existing supervisor-network library suite: 1,572 passed, 2 ignored.
  • Existing server tests filtered to token grants, delegation, provider configuration, and CLI: 155 passed, 2 ignored. Existing exact, fractional, zero, and missing-expiry TTL checks pass.
  • Final stacked all-targets Clippy passes for server, supervisor, supervisor-network, Kubernetes driver, and sandbox-backend with warnings denied; Rust formatting passes.
  • Gateway, sandbox, supervisor images and CLI build successfully from the current implementation.

The focused shared-supervisor, SPIFFE, and dedicated lifecycle scenarios compile. A fresh-cluster live run was blocked before OpenShell deployment: containerd could not create an inotify watcher (too many open files; host fs.inotify.max_user_instances=128), so the node never registered. No live scenarios ran; the temporary cluster was removed. No protobuf, SDK, or Helm files changed in this simplification follow-up; their earlier checks were not rerun.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Crate implementation documentation updated

@copy-pr-bot

copy-pr-bot Bot commented Oct 20, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@drew
drew added this pull request to stack #4698 October 20, 2026 02:05
@github-actions

github-actions Bot commented Oct 20, 2026

Copy link
Copy Markdown

🌿 Preview your docs: https://nvidia-preview-pr-4732.docs.buildwithfern.com/openshell

@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from db063a9 to 8dfbd78 Compare October 20, 2026 16:24
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch 2 times, most recently from a876a1a to 437619a Compare October 20, 2026 18:39
@drew drew changed the title feat(supervisor): delegate stable sandbox SPIFFE identities feat(gateway): broker stable sandbox SPIFFE provider tokens Oct 20, 2026
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 3da8000 to 5fbece5 Compare October 20, 2026 22:10
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 5fbece5 to 1899b5b Compare October 20, 2026 22:38
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch 2 times, most recently from 96ba1b7 to e0b6cec Compare October 20, 2026 22:53
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch 3 times, most recently from b4c1940 to b733887 Compare October 21, 2026 00:20
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from b733887 to 8e6786b Compare October 21, 2026 00:55
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 8e6786b to 6dd2cbb Compare October 21, 2026 01:05
@FrostGod FrostGod mentioned this pull request Oct 21, 2026
Draft
3 of 6 tasks
drew added 4 commits October 21, 2026 01:48
@drew
feat(supervisor): act as a sandbox's stable SPIFFE identity through S…
1f6eeb3 
…PIRE delegation

A shared supervisor's own Workload API identity names the supervisor, so
provider token grants failed closed for every hosted sandbox. With SPIRE
delegation configured, a hosted session now obtains the JWT-SVID registered
for its sandbox from the SPIRE agent's private admin socket through the
Delegated Identity API, and uses it in the existing RFC 7523 and RFC 8693
grant flows.

The SPIFFE ID derives from the immutable sandbox ID,
<prefix>/gateway/<gateway-id>/sandbox/<sandbox-id>, so it survives restarts
and runtime replacement. Each generation must earn its own authorization:

- The registration now carries the gateway ID and session verification keys
  from the gateway's launch authentication. The session's current gateway
  JWT must verify against them and name exactly the registered sandbox,
  runtime generation, and credential epoch.
- Before issuance, an authenticated gateway read confirms that the gateway
  still accepts the session's current credentials.
- The agent is asked for one selector naming the gateway and sandbox. Exactly
  one SVID with the expected ID must come back, verified against the agent's
  JWT bundle; nothing falls back to another identity.
- The session is re-authorized after every wait and before any cached token
  is served or a new one cached; a failed check discards the session's cache.

Dedicated supervisors keep their Workload API identity unchanged.
`mise run test:spire-delegation` runs these rules against a pinned SPIRE
server and agent.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
feat(helm): configure SPIRE delegation for the shared supervisor
f04b6ce 
The Kubernetes driver accepts an experimental_supervisor_spiffe_delegation
table with the gateway topology: the SPIRE agent admin socket, the prefix of
sandbox SPIFFE IDs, and the group that owns the admin socket. With it, the
driver also accepts provider_spiffe_workload_api_socket_path for the gateway's
own identity; without it, the topology still refuses provider SPIFFE identity.
Both socket paths must be canonical and in separate, non-nested directories,
the group must not be root, and the gateway ID must be a SPIFFE path segment.

The chart passes the settings to the shared supervisor, mounts only the admin
socket's directory from the node into the shared supervisor Pod, adds the
socket group to its supplemental groups, and runs it as a service account of
its own, so SPIRE can authorize exactly that Pod as a delegate. Gateway and
workload Pods receive nothing new. SPIRE registration stays with the operator.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
test(e2e): exchange provider tokens as SPIRE-delegated sandbox identi…
a6e8450 
…ties

The e2e:kubernetes:shared-supervisor-spiffe lane installs pinned SPIRE
hardened charts with the agent admin socket exposed for the shared
supervisor, enables SPIRE delegation, and registers sandbox identities with
ClusterStaticEntry resources as an operator would. Two sandboxes on one
shared supervisor exchange a stored subject token through the existing RFC
8693 flow, each under its own SPIFFE ID, with a token service that only lets
the identity an intermediate token names redeem it.

The test checks that an unregistered sandbox is refused rather than served
with the supervisor's identity, that a restart acquires again under the same
SPIFFE ID, that a sandbox recreated under the same name has a new identity,
and that no workload Pod or sandbox holds SPIFFE sockets, environment, or
SVIDs. The wrapper removes the SPIRE fixture, including its CRDs, on exit.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
docs: describe SPIRE-delegated sandbox identities on the shared super…
561d56a 
…visor

Document the experimental_supervisor_spiffe_delegation table, the stable
SPIFFE ID mapping, the SPIRE setup operators provide (admin socket and group,
authorized delegate, per-sandbox registration entries), and how restarts,
recreation, removal, and key rotation affect identities. Update the crate
READMEs and the cluster debugging and Helm development skills.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
drew added 6 commits October 21, 2026 01:48
@drew
fix: drop cross-source socket path checks and scope the SPIRE e2e fix…
7c3d720 
…ture

provider_spiffe_workload_api_socket_path names a CSI mount inside gateway
and dedicated supervisor Pods, while the SPIRE admin socket is a node
hostPath mounted only into the shared supervisor, so comparing the two
paths established nothing. The driver and chart now validate only the admin
socket path itself, and the docs no longer require separate directories.

The e2e SPIRE fixture refuses a cluster that already has SPIRE CRDs, creates
its own namespace (openshell-e2e-spire by default) before claiming
ownership, installs rather than upgrades, and removes only that namespace,
its CRDs, and registration entries labeled with that namespace. The SPIRE
delegation harness uses uv for host Python.

The configuration reference now states that gateway session JWTs must expire
and what cache hits check: the session's current JWT locally, with the
gateway confirming the generation only before new issuance.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
fix(e2e): match the SPIRE fixture's controller class for registration…
03196a7 
… entries

The SPIRE Controller Manager only reconciles entries of its own class,
<namespace>-<release>. The fixture namespace is now openshell-e2e-spire by
default and can be overridden, so the test derives the class from the fixture
namespace instead of assuming the old spire-spire.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
test(supervisor): generate the SPIRE fixture gateway key through open…
7dad785 
…shell-crypto

Main routes first-party key generation through the openshell-crypto facade.
Generate the SPIRE delegation fixture's Ed25519 gateway key with it instead
of calling rcgen's backend directly.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
feat(gateway): broker sandbox SPIFFE provider access tokens
d412e91 
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
test(helm): cover disabled gateway SPIRE delegation
4709a20 
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
refactor(supervisor): retain shared mode entry point with SPIFFE broker
85cba03 
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 6dd2cbb to 85cba03 Compare October 21, 2026 01:48
drew added 5 commits October 20, 2026 19:22
@drew
fix(gateway): share provider token cache TTL semantics
720a166 
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
chore(stack): carry workspace overlay fix into SPIFFE branch
11df558 
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
refactor(supervisor): carry sandbox instance names into SPIFFE delega…
ee8f1f4 
…tion

Merge the sandbox instance naming from the shared supervisor branch.
Resolve conflicts by keeping the gateway SPIRE broker, the
ProviderIdentitySource selection, and the shared OAuth cache TTL
semantics while applying the instance names to the token grants,
network inputs, shared host files, and READMEs. Describe the SPIFFE
end-to-end test's hosted placement as a sandbox instance.

Authentication sessions, session JWTs, the session-control protocol,
and the openshell.ai/sandbox-runtime-supervisor-session annotation
keep their names.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
refactor(supervisor): select provider token resolvers once and parse …
3fe7b15 
…delegation at startup

Merge the single hosted-instance registry from
refactor/4700-instance-lifecycle-owner/drew and adapt provider token grants.

Hosted instances now choose a token grant resolver once when their
networking starts. GatewayTokenGrantResolver holds only the instance's
authenticated gateway endpoint, forwards the credential key and request ID,
and validates the final token. SpiffeTokenGrantResolver keeps local
Workload API acquisition, the RFC 8693 intermediate exchange, and the
instance cache, and still fails closed without a socket. The duplicate
workload identity enum and the per-request gateway branch are gone.

The gateway parses its SPIRE delegation settings once into SpiffeDelegation,
held by every ServerState, instead of reparsing the SPIFFE ID prefix on each
request. Cache hits and fresh tokens now pass one final grant-configuration
and session check, and a fresh token is cached only after that check; the
session checks around each remote call remain.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
fix(server): report invalid SPIRE delegation as a startup error
590a0b5 
ServerState::new_with_credentials now returns a configuration error when the
SPIRE delegation settings are invalid, instead of panicking, so every gateway
startup path that builds state reports bad deployment settings. The CLI still
rejects them early with SpiffeDelegation::validate_config without building
the runtime value, and ServerState::new keeps its documented panic for
callers that skip validation.

Also correct the hosted instance admission docs: a failed admission retains
no instance, but deleting its directory is best effort and the next admission
deletes stale files before reuse.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew drew mentioned this pull request Oct 21, 2026
Draft
6 tasks done

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: delegate stable sandbox SPIFFE identities from the shared supervisor

1 participant

@drew