Skip to content

Commit a786dfb

Browse files
drewdrew
committed
feat(sandbox): add gVisor runtime adapter
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
1 parent f08322e commit a786dfb

25 files changed

Lines changed: 932 additions & 161 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -297,12 +297,15 @@ sandbox and can negate OpenShell workspace isolation and filesystem-policy
297297
controls. Driver-owned supervisor, token, and TLS bind mounts stay reserved.
298298

299299
Network features follow the driver/substrate split. Drivers own only the outer
300-
fence and protected channel. The sandbox owns seccomp notification, local DNS,
301-
socket virtualization, process observation, and binary identity. The supervisor
302-
owns DNS eligibility, policy authorization, destination filtering, upstream
303-
dials, relay behavior, credential rewriting, and OCSF decisions. No supported
304-
path requires nftables, a workload network namespace, proxy environment
305-
variables, added capabilities, or an unconfined AppArmor profile.
300+
fence and protected channel. The native Linux adapter owns seccomp notification,
301+
local DNS, socket virtualization, process observation, and binary identity. The
302+
Kubernetes gVisor adapter instead combines the sentry and zero-rule workload
303+
`NetworkPolicy` with a workload-local explicit proxy whose streams cross the
304+
protected channel. The supervisor owns DNS eligibility, policy authorization,
305+
destination filtering, upstream dials, relay behavior, credential rewriting,
306+
and OCSF decisions. Native mode does not require proxy environment variables;
307+
gVisor mode injects loopback HTTP proxy variables and applies endpoint-only
308+
policy without added capabilities or an unconfined AppArmor profile.
306309

307310
The Kubernetes deployment packaging has two ownership boundaries. The gateway
308311
chart owns the gateway workload, configuration, Services, PKI, and

‎architecture/sandbox.md‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ Each sandbox has three trust levels:
1111
| Component | Role |
1212
|---|---|
1313
| Supervisor | Owns gateway credentials, admitted policy, L7 proxying, SSH, and gateway relays. It never executes inside the agent workload. |
14-
| Sandbox | Runs as the same non-root identity as the agent, installs the workload seccomp listener, applies the Landlock baseline, owns child processes, and mediates the protected supervisor channel. |
15-
| Agent child | Inherits the sandbox network listener and runs with zero capabilities, `no_new_privs`, Landlock, and the final syscall filter. |
14+
| Sandbox | Runs as the same non-root identity as the agent, owns child processes, mediates the protected supervisor channel, and instantiates the driver-selected native Linux or gVisor adapter. |
15+
| Agent child | Runs with zero capabilities behind the selected adapter. Native Linux children inherit seccomp mediation, `no_new_privs`, Landlock, and the final syscall filter; gVisor children rely on the sentry, OCI mounts, and the outer network fence. |
1616

1717
The runtime grants neither trusted component nor agent child any Linux
1818
capability inside the workload. Drivers resolve one exact non-root UID, GID,
@@ -65,8 +65,8 @@ replacement from granting authority.
6565
run untrusted code yet.
6666
3. `openshell-supervisor` loads policy and runtime settings from the gateway,
6767
attaches to the sandbox, and verifies the driver's generation and evidence.
68-
4. The sandbox installs its seccomp notification broker and Landlock baseline,
69-
validates its mechanism-specific audit evidence, and reports backend-neutral
68+
4. The sandbox starts the selected runtime adapter, validates its mechanism-
69+
specific audit evidence, and reports backend-neutral
7070
enforcement properties. The supervisor must accept those properties and
7171
their immutable session and resource binding before it sends the launch
7272
permit. Other isolation backends may establish the same properties with
@@ -103,6 +103,16 @@ OpenShell uses overlapping controls rather than a single sandbox primitive:
103103
| Driver outer fence | Docker `network_mode=none`, a NIC-less VM, or Kubernetes NetworkPolicy prevents any missed or unsupported kernel path from escaping. |
104104
| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and inference interception. |
105105

106+
The Kubernetes gVisor adapter uses the same authenticated lifecycle and process
107+
backend with different enforcement mechanisms. The gVisor sentry and OCI mounts
108+
provide the workload boundary. A zero-rule Kubernetes egress `NetworkPolicy`
109+
blocks direct connections, while a workload-local HTTP/CONNECT listener reverse-
110+
tunnels streams to the existing supervisor proxy. The supervisor loads policy
111+
in endpoint-only mode because the tunnel authenticates the sandbox generation,
112+
not an individual executable. This adapter intentionally does not claim native
113+
Landlock path policy, nested child seccomp, transparent TCP, or per-binary
114+
network attribution.
115+
106116
The supervisor may enrich baseline filesystem allowances for runtime-required
107117
paths, such as proxy support files or GPU device paths when a GPU is present.
108118
These internal allowances must stay sandbox-scoped and avoid exposing host

‎crates/openshell-driver-docker/src/isolation.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ use std::path::PathBuf;
1414
use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity};
1515
use openshell_sandbox_backend::GPU_RESOURCE_CLAIM;
1616
use openshell_sandbox_backend::boundary_protocol::{
17-
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor,
18-
SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
17+
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeAdapter,
18+
SandboxRuntimeDescriptor, SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
1919
};
2020

2121
/// Driver-owned inputs that bind one Docker container to one boundary.
@@ -78,6 +78,7 @@ impl DockerBoundarySpec {
7878
resource_claims: resource_claims.clone(),
7979
resource_claim_files: BTreeMap::new(),
8080
workload_identity: self.workload_identity.clone(),
81+
adapter: SandboxRuntimeAdapter::default(),
8182
driver_fence: driver_fence.clone(),
8283
child_env: self.child_env,
8384
},
@@ -86,6 +87,7 @@ impl DockerBoundarySpec {
8687
generation: self.generation,
8788
session_id: self.session_id,
8889
workload_identity: self.workload_identity,
90+
adapter: SandboxRuntimeAdapter::default(),
8991
transport: SandboxTransport::Unix {
9092
socket_path: self.control_socket,
9193
},

‎crates/openshell-driver-kubernetes/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,16 @@ default seccomp profile. The sandbox installs a nested seccomp user-notification
6363
filter without requesting a capability in the Pod spec. Startup fails closed
6464
when the runtime blocks the required seccomp or Landlock operations.
6565

66+
An effective `pod.runtime_class_name` of `gvisor` selects the gVisor adapter in
67+
the same sandbox backend. The workload Pod keeps the non-root, drop-all-
68+
capabilities posture but omits the Kubernetes seccomp profile and custom sysctl
69+
that GKE Sandbox does not support. The sandbox qualifies the gVisor sentry,
70+
starts an explicit proxy on `127.0.0.1:3128`, and reverse-tunnels proxy streams
71+
to the supervisor over the authenticated boundary protocol. The empty-egress
72+
`NetworkPolicy` blocks direct workload connections. This mode applies endpoint-
73+
only network policy and does not apply Landlock path rules, the nested child
74+
seccomp filter, transparent TCP, or per-binary network attribution.
75+
6676
The supervisor Pod has a direct, non-controller owner reference to the Sandbox
6777
resource. This links its garbage-collection lifecycle to the sandbox without
6878
competing with the Agent Sandbox controller for workload-Pod ownership.

‎crates/openshell-driver-kubernetes/src/driver.rs‎

Lines changed: 125 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1962,6 +1962,7 @@ impl KubernetesComputeDriver {
19621962
return Err(fail("pair labels changed"));
19631963
}
19641964
let spec = pod.spec.as_ref().ok_or_else(|| fail("missing Pod spec"))?;
1965+
let gvisor = spec.runtime_class_name.as_deref() == Some("gvisor");
19651966
if spec.host_network == Some(true)
19661967
|| spec.host_pid == Some(true)
19671968
|| spec.host_ipc == Some(true)
@@ -2013,10 +2014,12 @@ impl KubernetesComputeDriver {
20132014
.supplemental_groups
20142015
.as_deref()
20152016
.is_some_and(|groups| !groups.is_empty())
2016-
|| security
2017-
.seccomp_profile
2018-
.as_ref()
2019-
.is_none_or(|profile| profile.type_ != "RuntimeDefault")
2017+
|| (!gvisor
2018+
&& security
2019+
.seccomp_profile
2020+
.as_ref()
2021+
.is_none_or(|profile| profile.type_ != "RuntimeDefault"))
2022+
|| (gvisor && security.seccomp_profile.is_some())
20202023
{
20212024
return Err(fail("numeric identity, groups, or seccomp profile changed"));
20222025
}
@@ -2046,9 +2049,17 @@ impl KubernetesComputeDriver {
20462049
&& sysctl.get("value").and_then(serde_json::Value::as_str) == Some("0")
20472050
})
20482051
});
2049-
if !unprivileged_port_sysctl {
2052+
if !gvisor && !unprivileged_port_sysctl {
20502053
return Err(fail("safe unprivileged-port sysctl changed"));
20512054
}
2055+
if gvisor
2056+
&& pod_json
2057+
.pointer("/spec/securityContext/sysctls")
2058+
.and_then(serde_json::Value::as_array)
2059+
.is_some_and(|sysctls| !sysctls.is_empty())
2060+
{
2061+
return Err(fail("gVisor workload must not request custom sysctls"));
2062+
}
20522063
let check_container = |container: &k8s_openapi::api::core::v1::Container,
20532064
name: &str|
20542065
-> Result<(), KubernetesDriverError> {
@@ -2265,6 +2276,16 @@ impl KubernetesComputeDriver {
22652276
agent_gid,
22662277
&names.sandbox_secret,
22672278
)?;
2279+
let runtime_adapter = if workload_pod
2280+
.spec
2281+
.as_ref()
2282+
.and_then(|spec| spec.runtime_class_name.as_deref())
2283+
== Some("gvisor")
2284+
{
2285+
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
2286+
} else {
2287+
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::NativeLinux
2288+
};
22682289
let workload_pod_uid =
22692290
workload_pod.metadata.uid.clone().ok_or_else(|| {
22702291
KubernetesDriverError::Message("workload Pod has no UID".to_string())
@@ -2308,6 +2329,22 @@ impl KubernetesComputeDriver {
23082329
child_env.extend(spec.environment.clone());
23092330
}
23102331
child_env.retain(|name, _| !name.starts_with("OPENSHELL_"));
2332+
if matches!(
2333+
runtime_adapter,
2334+
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
2335+
) {
2336+
for name in ["HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"] {
2337+
child_env.insert(name.to_string(), "http://127.0.0.1:3128".to_string());
2338+
}
2339+
child_env.insert(
2340+
"NO_PROXY".to_string(),
2341+
"localhost,127.0.0.1,::1".to_string(),
2342+
);
2343+
child_env.insert(
2344+
"no_proxy".to_string(),
2345+
"localhost,127.0.0.1,::1".to_string(),
2346+
);
2347+
}
23112348
let host_gateway_ip = self.config.host_gateway_ip.parse().ok();
23122349
let session_id = launch_authentication.supervisor.session_id;
23132350
let tls = generate_sandbox_tls_material(session_id)
@@ -2368,6 +2405,7 @@ impl KubernetesComputeDriver {
23682405
},
23692406
host_gateway_ip,
23702407
workload_identity,
2408+
adapter: runtime_adapter,
23712409
child_env,
23722410
}
23732411
.provision();
@@ -2470,7 +2508,7 @@ impl KubernetesComputeDriver {
24702508
supervisor_uid: &str,
24712509
agent_uid: u32,
24722510
agent_gid: u32,
2473-
child_env: std::collections::HashMap<String, String>,
2511+
mut child_env: std::collections::HashMap<String, String>,
24742512
launch_authentication: &openshell_core::jwt::SandboxLaunchAuthentication,
24752513
) -> Result<(), KubernetesDriverError> {
24762514
let namespace_uid = Api::<Namespace>::all(self.client.clone())
@@ -2525,6 +2563,32 @@ impl KubernetesComputeDriver {
25252563
agent_gid,
25262564
&names.sandbox_secret,
25272565
)?;
2566+
let runtime_adapter = if workload_pod
2567+
.spec
2568+
.as_ref()
2569+
.and_then(|spec| spec.runtime_class_name.as_deref())
2570+
== Some("gvisor")
2571+
{
2572+
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
2573+
} else {
2574+
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::NativeLinux
2575+
};
2576+
if matches!(
2577+
runtime_adapter,
2578+
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
2579+
) {
2580+
for name in ["HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"] {
2581+
child_env.insert(name.to_string(), "http://127.0.0.1:3128".to_string());
2582+
}
2583+
child_env.insert(
2584+
"NO_PROXY".to_string(),
2585+
"localhost,127.0.0.1,::1".to_string(),
2586+
);
2587+
child_env.insert(
2588+
"no_proxy".to_string(),
2589+
"localhost,127.0.0.1,::1".to_string(),
2590+
);
2591+
}
25282592
let workload_pod_uid =
25292593
workload_pod.metadata.uid.clone().ok_or_else(|| {
25302594
KubernetesDriverError::Message("workload Pod has no UID".to_string())
@@ -2606,6 +2670,7 @@ impl KubernetesComputeDriver {
26062670
},
26072671
host_gateway_ip: self.config.host_gateway_ip.parse().ok(),
26082672
workload_identity,
2673+
adapter: runtime_adapter,
26092674
child_env,
26102675
}
26112676
.provision();
@@ -5215,6 +5280,10 @@ fn apply_supervisor_sandbox_runtime_boundary(
52155280
else {
52165281
return;
52175282
};
5283+
let gvisor = spec
5284+
.get("runtimeClassName")
5285+
.and_then(serde_json::Value::as_str)
5286+
== Some("gvisor");
52185287
spec.insert("hostNetwork".to_string(), serde_json::json!(false));
52195288
spec.insert("hostPID".to_string(), serde_json::json!(false));
52205289
spec.insert("hostIPC".to_string(), serde_json::json!(false));
@@ -5238,20 +5307,22 @@ fn apply_supervisor_sandbox_runtime_boundary(
52385307
]
52395308
}),
52405309
);
5241-
spec.insert(
5242-
"securityContext".to_string(),
5243-
serde_json::json!({
5244-
"runAsUser": params.sandbox_uid,
5245-
"runAsGroup": params.sandbox_gid,
5246-
"runAsNonRoot": true,
5247-
"fsGroup": params.sandbox_gid,
5248-
"fsGroupChangePolicy": "OnRootMismatch",
5249-
"supplementalGroups": [],
5250-
"supplementalGroupsPolicy": "Strict",
5251-
"seccompProfile": {"type": "RuntimeDefault"},
5252-
"sysctls": [{"name": "net.ipv4.ip_unprivileged_port_start", "value": "0"}]
5253-
}),
5254-
);
5310+
let mut pod_security = serde_json::json!({
5311+
"runAsUser": params.sandbox_uid,
5312+
"runAsGroup": params.sandbox_gid,
5313+
"runAsNonRoot": true,
5314+
"fsGroup": params.sandbox_gid,
5315+
"fsGroupChangePolicy": "OnRootMismatch",
5316+
"supplementalGroups": [],
5317+
"supplementalGroupsPolicy": "Strict"
5318+
});
5319+
if !gvisor {
5320+
pod_security["seccompProfile"] = serde_json::json!({"type": "RuntimeDefault"});
5321+
pod_security["sysctls"] = serde_json::json!([
5322+
{"name": "net.ipv4.ip_unprivileged_port_start", "value": "0"}
5323+
]);
5324+
}
5325+
spec.insert("securityContext".to_string(), pod_security);
52555326
let volumes = spec
52565327
.entry("volumes")
52575328
.or_insert_with(|| serde_json::json!([]))
@@ -8504,6 +8575,40 @@ mod tests {
85048575
);
85058576
}
85068577

8578+
#[test]
8579+
fn gvisor_runtime_omits_incompatible_kernel_security_context() {
8580+
let pod_template = {
8581+
let params = SandboxPodParams {
8582+
default_runtime_class_name: "gvisor",
8583+
..SandboxPodParams::default()
8584+
};
8585+
sandbox_template_to_k8s(
8586+
&SandboxTemplate::default(),
8587+
false,
8588+
&std::collections::HashMap::new(),
8589+
true,
8590+
&params,
8591+
)
8592+
};
8593+
8594+
assert_eq!(
8595+
pod_template["spec"]["runtimeClassName"],
8596+
serde_json::json!("gvisor")
8597+
);
8598+
assert!(
8599+
pod_template["spec"]["securityContext"]["seccompProfile"].is_null(),
8600+
"GKE Sandbox rejects Kubernetes seccomp profiles"
8601+
);
8602+
assert!(
8603+
pod_template["spec"]["securityContext"]["sysctls"].is_null(),
8604+
"GKE Sandbox rejects custom sysctls"
8605+
);
8606+
assert_eq!(
8607+
pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["drop"],
8608+
serde_json::json!(["ALL"])
8609+
);
8610+
}
8611+
85078612
#[test]
85088613
fn template_runtime_class_name_overrides_config_default() {
85098614
let template = SandboxTemplate {

0 commit comments

Comments
 (0)