Skip to content

Commit 7fcd175

Browse files
committed
feat(docker): project CDI metadata into GPU workloads
Signed-off-by: Evan Lezar <elezar@nvidia.com>
1 parent 63bcdec commit 7fcd175

7 files changed

Lines changed: 184 additions & 17 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -357,6 +357,19 @@ Resource requirements enter the driver layer through `SandboxSpec.resource_requi
357357
can request a specific number of GPUs or the driver-specific default behaviour.
358358
For all in-tree drivers, this is equivalent to selecting a single GPU.
359359

360+
For Docker GPU sandboxes, the driver treats CDI specs as runtime metadata for
361+
both outer injection and inner sandbox policy. It selects opaque CDI device IDs,
362+
passes them to Docker, embeds a versioned CDI context in the protected workload
363+
boundary configuration, and mounts daemon-reported CDI spec directories
364+
read-only into the workload. The sandbox runtime resolves that context in the
365+
workload mount namespace and derives Landlock paths and supplemental groups
366+
from CDI `containerEdits` before launching agent processes.
367+
Host-side CDI spec paths are diagnostic only and are never treated as sandbox
368+
policy paths.
369+
Kubernetes must not infer CDI device IDs from the `nvidia.com/gpu` resource
370+
request; it needs a node-local selected-device handoff before using the same
371+
workload-side resolver.
372+
360373
VM runtime state paths are derived only from driver-validated sandbox IDs
361374
matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a
362375
private `run/` directory plus Unix peer UID/PID checks. Standalone

‎crates/openshell-core/src/cdi.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,12 @@ pub const CDI_CONTEXT_VERSION: u32 = 1;
1212
/// Base workload-runtime path under which compute drivers mount CDI specification directories.
1313
pub const CDI_SPEC_DIR_BASE: &str = "/run/openshell/boundary/cdi-specs";
1414

15+
/// Return the workload-runtime path used for a CDI specification directory.
16+
#[must_use]
17+
pub fn cdi_spec_mount_path(index: usize) -> String {
18+
format!("{CDI_SPEC_DIR_BASE}/{index}")
19+
}
20+
1521
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
1622
pub struct CdiContext {
1723
pub version: u32,

‎crates/openshell-driver-docker/README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ LSM decisions remain authoritative.
8585
| Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. |
8686
| In-memory `/run/openshell-supervisor-ca` tmpfs | Holds only the public supervisor CA certificate and trust bundle without making all of `/run` writable. |
8787
| CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. |
88+
| CDI workload projection | For GPU sandboxes only, embeds selected device metadata in the protected boundary config and mounts Docker daemon CDI spec directories read-only into the workload. |
8889

8990
## Stop, Start, and Delete
9091

@@ -98,6 +99,20 @@ Delete force-removes both containers, the driver-owned runtime volumes, and the
9899
host-private runtime descriptor. Missing or altered descriptor and channel resources
99100
fail closed; the driver does not run an older combined-supervisor layout.
100101

102+
## CDI GPU Metadata
103+
104+
Docker remains the source of truth for GPU injection. The driver selects opaque
105+
CDI device IDs from `driver_config.cdi_devices` or the daemon's discovered CDI
106+
inventory, then passes the same IDs to Docker with a CDI `DeviceRequest`.
107+
108+
For a GPU sandbox, the driver writes those IDs and projected CDI spec directory
109+
paths to a versioned context in the protected workload boundary configuration.
110+
It bind-mounts the daemon-reported spec directories read-only into the workload.
111+
The sandbox runtime resolves the selected devices in that mount namespace before
112+
it launches agent processes and derives Landlock paths and supplemental groups
113+
from CDI `containerEdits`. A missing context or spec directory fails workload
114+
startup closed. Non-GPU sandboxes receive no CDI context or spec mounts.
115+
101116
## Driver Config Mounts
102117

103118
The gateway forwards the `docker` block from `--driver-config-json`. Supported

‎crates/openshell-driver-docker/src/isolation.rs‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ use std::collections::{BTreeMap, HashMap};
1111
use std::net::IpAddr;
1212
use std::path::PathBuf;
1313

14+
use openshell_core::cdi::CdiContext;
1415
use openshell_isolation_interface::contract::{
1516
BackendError, OuterFenceGuarantee, OuterFenceGuarantees, ResolvedWorkloadIdentity,
1617
};
@@ -70,6 +71,7 @@ pub struct DockerBoundarySpec {
7071
pub container_id: String,
7172
pub image_identity: String,
7273
pub gpu_requested: bool,
74+
pub cdi_context: Option<CdiContext>,
7375
pub listener_socket: PathBuf,
7476
pub control_socket: PathBuf,
7577
pub sandbox_tls: SandboxTlsServerConfig,
@@ -118,7 +120,7 @@ impl DockerBoundarySpec {
118120
},
119121
resource_claims: resource_claims.clone(),
120122
resource_claim_files: BTreeMap::new(),
121-
cdi_context: None,
123+
cdi_context: self.cdi_context,
122124
workload_identity: self.workload_identity.clone(),
123125
outer_fence: outer_fence.clone(),
124126
child_env: self.child_env,
@@ -188,6 +190,13 @@ mod tests {
188190
container_id: "sha256:container".to_string(),
189191
image_identity: "sha256:image".to_string(),
190192
gpu_requested: true,
193+
cdi_context: Some(CdiContext::new(
194+
vec!["nvidia.com/gpu=0".to_string()],
195+
vec![openshell_core::cdi::CdiSpecDirectory::new(
196+
openshell_core::cdi::cdi_spec_mount_path(0),
197+
"/var/run/cdi",
198+
)],
199+
)),
191200
listener_socket: PathBuf::from("/run/openshell/boundary/control.sock"),
192201
control_socket: PathBuf::from("/host/control.sock"),
193202
sandbox_tls: SandboxTlsServerConfig {
@@ -224,6 +233,15 @@ mod tests {
224233
provisioned.runtime_descriptor.resource_claims[GPU_RESOURCE_CLAIM],
225234
"true"
226235
);
236+
assert_eq!(
237+
provisioned
238+
.boundary_config
239+
.cdi_context
240+
.as_ref()
241+
.unwrap()
242+
.selected_devices,
243+
["nvidia.com/gpu=0"]
244+
);
227245
assert_eq!(
228246
provisioned.boundary_config.outer_fence,
229247
provisioned.runtime_descriptor.outer_fence

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 51 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ use bollard::query_parameters::{
2323
};
2424
use bytes::Bytes;
2525
use futures::{Stream, StreamExt};
26+
use openshell_core::cdi::{CdiContext, CdiSpecDirectory, cdi_spec_mount_path};
2627
use openshell_core::config::DEFAULT_STOP_TIMEOUT_SECS;
2728
use openshell_core::driver_mounts;
2829
use openshell_core::driver_utils::{
@@ -311,12 +312,49 @@ struct DockerDriverRuntimeConfig {
311312
app_armor_profile: Option<AppArmorProfile>,
312313
}
313314

314-
#[derive(Debug, Clone, Copy)]
315+
#[derive(Debug, Clone)]
315316
struct DockerGpuRuntimeCapabilities {
316317
cdi_supported: bool,
318+
cdi_spec_dirs: Vec<String>,
317319
wsl_all_gpu_fallback_enabled: bool,
318320
}
319321

322+
impl DockerGpuRuntimeCapabilities {
323+
fn cdi_context(&self, gpu_device_ids: Option<&[String]>) -> Result<Option<CdiContext>, Status> {
324+
let Some(gpu_device_ids) = gpu_device_ids.filter(|device_ids| !device_ids.is_empty())
325+
else {
326+
return Ok(None);
327+
};
328+
self.require_cdi_spec_dirs()?;
329+
Ok(Some(CdiContext::new(
330+
gpu_device_ids.to_vec(),
331+
self.cdi_spec_dirs
332+
.iter()
333+
.enumerate()
334+
.map(|(index, source)| CdiSpecDirectory::new(cdi_spec_mount_path(index), source))
335+
.collect(),
336+
)))
337+
}
338+
339+
fn cdi_spec_bind_strings(&self) -> Result<Vec<String>, Status> {
340+
self.require_cdi_spec_dirs()?;
341+
Ok(self
342+
.cdi_spec_dirs
343+
.iter()
344+
.enumerate()
345+
.map(|(index, source)| format!("{source}:{}:ro,z", cdi_spec_mount_path(index)))
346+
.collect())
347+
}
348+
349+
fn require_cdi_spec_dirs(&self) -> Result<(), Status> {
350+
if self.cdi_spec_dirs.is_empty() {
351+
return Err(Status::failed_precondition(
352+
"docker GPU sandboxes require Docker CDI spec directories reported by the daemon",
353+
));
354+
}
355+
Ok(())
356+
}
357+
}
320358
#[derive(Clone)]
321359
pub struct DockerComputeDriver {
322360
docker: Arc<Docker>,
@@ -863,14 +901,13 @@ impl DockerComputeDriver {
863901
let info = docker.info().await.map_err(|err| {
864902
Error::execution(format!("failed to query Docker daemon info: {err}"))
865903
})?;
866-
let cdi_supported = info
867-
.cdi_spec_dirs
868-
.as_ref()
869-
.is_some_and(|dirs| !dirs.is_empty());
904+
let cdi_spec_dirs = info.cdi_spec_dirs.clone().unwrap_or_default();
905+
let cdi_supported = !cdi_spec_dirs.is_empty();
870906
let cdi_gpu_inventory = docker_cdi_gpu_inventory(&info);
871907
let wsl_all_gpu_fallback_enabled = docker_info_reports_wsl2(&info);
872908
let gpu = DockerGpuRuntimeCapabilities {
873909
cdi_supported,
910+
cdi_spec_dirs,
874911
wsl_all_gpu_fallback_enabled,
875912
};
876913
validate_sandbox_pids_limit(docker_config.sandbox_pids_limit)?;
@@ -941,7 +978,7 @@ impl DockerComputeDriver {
941978
supervisor_grpc_endpoint,
942979
ssh_socket_path: docker_config.ssh_socket_path.clone(),
943980
guest_tls,
944-
gpu,
981+
gpu: gpu.clone(),
945982
sandbox_pids_limit: docker_config.sandbox_pids_limit,
946983
enable_bind_mounts: docker_config.enable_bind_mounts,
947984
allow_driver_config: docker_config.allow_driver_config,
@@ -1761,7 +1798,7 @@ impl DockerComputeDriver {
17611798
&created.id,
17621799
&image,
17631800
&workload_identity,
1764-
gpu_devices.is_some(),
1801+
gpu_devices.as_deref(),
17651802
)
17661803
.await
17671804
{
@@ -4521,7 +4558,7 @@ async fn prepare_docker_boundary_files(
45214558
container_id: &str,
45224559
image: &DockerImageMetadata,
45234560
workload_identity: &ResolvedWorkloadIdentity,
4524-
gpu_requested: bool,
4561+
gpu_device_ids: Option<&[String]>,
45254562
) -> Result<(), Status> {
45264563
let directory = docker_boundary_state_dir(sandbox, config)?;
45274564
let workspace_root = driver_mounts::resolve_oci_workspace_root(&image.working_dir)
@@ -4551,7 +4588,8 @@ async fn prepare_docker_boundary_files(
45514588
verification_keys,
45524589
container_id: container_id.to_string(),
45534590
image_identity: image.id.clone(),
4554-
gpu_requested,
4591+
gpu_requested: gpu_device_ids.is_some_and(|device_ids| !device_ids.is_empty()),
4592+
cdi_context: config.gpu.cdi_context(gpu_device_ids)?,
45554593
listener_socket: PathBuf::from(BOUNDARY_SOCKET_MOUNT_PATH),
45564594
control_socket: PathBuf::from(BOUNDARY_SOCKET_MOUNT_PATH),
45574595
sandbox_tls: SandboxTlsServerConfig {
@@ -5694,7 +5732,10 @@ fn build_container_create_body_for_image(
56945732
}),
56955733
..Default::default()
56965734
});
5697-
let user_bind_strings = docker_driver_bind_strings(driver_config)?;
5735+
let mut user_bind_strings = docker_driver_bind_strings(driver_config)?;
5736+
if gpu_device_ids.is_some_and(|device_ids| !device_ids.is_empty()) {
5737+
user_bind_strings.extend(config.gpu.cdi_spec_bind_strings()?);
5738+
}
56985739
let device_requests = gpu_device_ids.map(|device_ids| {
56995740
vec![DeviceRequest {
57005741
driver: Some("cdi".to_string()),

‎crates/openshell-driver-docker/src/tests.rs‎

Lines changed: 69 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ use std::fs;
2323
use std::sync::Arc;
2424
use tempfile::TempDir;
2525

26+
const TEST_CDI_SPEC_DIR: &str = "/opt/openshell-test/cdi";
27+
const TEST_CDI_SPEC_DIR_ALT: &str = "/srv/openshell-test/cdi";
28+
2629
fn test_launch_authentication() -> Vec<u8> {
2730
serde_json::to_vec(&SandboxLaunchAuthentication {
2831
supervisor: SupervisorAuthBundle {
@@ -192,6 +195,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
192195
}),
193196
gpu: DockerGpuRuntimeCapabilities {
194197
cdi_supported: false,
198+
cdi_spec_dirs: Vec::new(),
195199
wsl_all_gpu_fallback_enabled: false,
196200
},
197201
sandbox_pids_limit: openshell_core::config::default_sandbox_pids_limit(),
@@ -202,6 +206,59 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
202206
}
203207
}
204208

209+
fn runtime_config_with_cdi_spec_dirs(cdi_spec_dirs: &[&str]) -> DockerDriverRuntimeConfig {
210+
let mut config = runtime_config();
211+
config.gpu.cdi_supported = !cdi_spec_dirs.is_empty();
212+
config.gpu.cdi_spec_dirs = cdi_spec_dirs
213+
.iter()
214+
.map(|path| (*path).to_string())
215+
.collect();
216+
config
217+
}
218+
219+
#[test]
220+
fn gpu_runtime_builds_cdi_context_from_selected_devices_and_daemon_dirs() {
221+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR, TEST_CDI_SPEC_DIR_ALT]);
222+
let selected_devices = vec![
223+
"nvidia.com/gpu=0".to_string(),
224+
"nvidia.com/gpu=1".to_string(),
225+
];
226+
227+
let context = config
228+
.gpu
229+
.cdi_context(Some(&selected_devices))
230+
.unwrap()
231+
.expect("GPU selection should produce a CDI context");
232+
233+
assert_eq!(context.selected_devices, selected_devices);
234+
assert_eq!(
235+
context.spec_dirs,
236+
vec![
237+
CdiSpecDirectory::new(cdi_spec_mount_path(0), TEST_CDI_SPEC_DIR),
238+
CdiSpecDirectory::new(cdi_spec_mount_path(1), TEST_CDI_SPEC_DIR_ALT),
239+
]
240+
);
241+
}
242+
243+
#[test]
244+
fn gpu_runtime_omits_cdi_projection_without_selected_devices() {
245+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
246+
assert_eq!(config.gpu.cdi_context(None).unwrap(), None);
247+
}
248+
249+
#[test]
250+
fn gpu_runtime_builds_read_only_cdi_bind_mounts() {
251+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR, TEST_CDI_SPEC_DIR_ALT]);
252+
253+
assert_eq!(
254+
config.gpu.cdi_spec_bind_strings().unwrap(),
255+
vec![
256+
format!("{TEST_CDI_SPEC_DIR}:{}:ro,z", cdi_spec_mount_path(0)),
257+
format!("{TEST_CDI_SPEC_DIR_ALT}:{}:ro,z", cdi_spec_mount_path(1)),
258+
]
259+
);
260+
}
261+
205262
fn test_workload_identity() -> ResolvedWorkloadIdentity {
206263
ResolvedWorkloadIdentity::new(
207264
1234,
@@ -2453,8 +2510,7 @@ fn validate_sandbox_auth_accepts_launch_authentication() {
24532510

24542511
#[test]
24552512
fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() {
2456-
let mut config = runtime_config();
2457-
config.gpu.cdi_supported = true;
2513+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
24582514
let mut sandbox = test_sandbox();
24592515
sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(None));
24602516

@@ -2479,6 +2535,15 @@ fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() {
24792535
request.device_ids.as_ref().unwrap(),
24802536
&vec!["nvidia.com/gpu=1".to_string()]
24812537
);
2538+
let binds = create_body
2539+
.host_config
2540+
.as_ref()
2541+
.and_then(|host_config| host_config.binds.as_ref())
2542+
.expect("GPU request should project CDI specs into the workload");
2543+
assert!(binds.contains(&format!(
2544+
"{TEST_CDI_SPEC_DIR}:{}:ro,z",
2545+
cdi_spec_mount_path(0)
2546+
)));
24822547
}
24832548

24842549
#[test]
@@ -2501,8 +2566,7 @@ fn build_container_create_body_omits_devices_without_resolved_default_cdi_device
25012566

25022567
#[test]
25032568
fn build_container_create_body_passes_explicit_cdi_device_id_through() {
2504-
let mut config = runtime_config();
2505-
config.gpu.cdi_supported = true;
2569+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
25062570
let mut sandbox = test_sandbox();
25072571
let spec = sandbox.spec.as_mut().unwrap();
25082572
spec.resource_requirements = Some(gpu_resources(None));
@@ -2572,8 +2636,7 @@ fn build_container_create_body_rejects_empty_cdi_devices() {
25722636

25732637
#[test]
25742638
fn driver_default_gpu_selection_consumes_distinct_devices_for_creates() {
2575-
let mut config = runtime_config();
2576-
config.gpu.cdi_supported = true;
2639+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
25772640
let driver = test_driver_with_config(config);
25782641
driver.gpu_selector.refresh(
25792642
CdiGpuInventory::new(["nvidia.com/gpu=0", "nvidia.com/gpu=1"]),

‎docs/how-it-works/sandboxes/runtimes.mdx‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,17 @@ Common options in `[openshell.drivers.docker]` are `socket_path`, `grpc_endpoint
9191

9292
Docker Desktop must have host networking enabled, and it cannot use Enhanced Container Isolation. Set `grpc_endpoint` when sandboxes cannot reach the gateway on host loopback. For GPU sandboxes, configure Docker CDI before starting the gateway.
9393

94+
For Docker GPU/CDI sandboxes, OpenShell uses Docker's selected CDI device IDs
95+
and daemon-reported CDI spec directories to build a workload CDI context. The
96+
driver embeds the context in the protected boundary configuration and
97+
bind-mounts the spec directories read-only into the workload. If context
98+
creation fails, the driver removes the created state. If workload or supervisor
99+
startup fails, it also removes the runtime resources before reporting the
100+
failure. The sandbox runtime derives filesystem and supplemental group
101+
requirements from CDI specs before launching agent processes. Non-GPU Docker
102+
sandboxes do not receive the CDI context, spec mounts, or CDI-derived policy
103+
changes.
104+
94105
### Docker Mounts
95106

96107
Mount existing named volumes or `tmpfs` through driver config. Label volumes so resource admission accepts them:

0 commit comments

Comments
 (0)