Skip to content

Commit 70044d0

Browse files
committed
feat(docker): project CDI metadata into GPU workloads
Signed-off-by: Evan Lezar <elezar@nvidia.com>
1 parent b69b35f commit 70044d0

6 files changed

Lines changed: 171 additions & 17 deletions

File tree

‎crates/openshell-core/src/cdi.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,12 @@ pub const CDI_CONTEXT_VERSION: u32 = 1;
1212
/// Base workload-runtime path under which compute drivers mount CDI specification directories.
1313
pub const CDI_SPEC_DIR_BASE: &str = "/run/openshell/boundary/cdi-specs";
1414

15+
/// Return the workload-runtime path used for a CDI specification directory.
16+
#[must_use]
17+
pub fn cdi_spec_mount_path(index: usize) -> String {
18+
format!("{CDI_SPEC_DIR_BASE}/{index}")
19+
}
20+
1521
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
1622
pub struct CdiContext {
1723
pub version: u32,

‎crates/openshell-driver-docker/README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,7 @@ LSM decisions remain authoritative.
111111
| Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. |
112112
| In-memory `/run/openshell-supervisor-ca` tmpfs | Holds only the public supervisor CA certificate and trust bundle without making all of `/run` writable. |
113113
| CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. |
114+
| CDI workload projection | For GPU sandboxes only, embeds selected device metadata in the protected boundary config and mounts Docker daemon CDI spec directories read-only into the workload. |
114115

115116
## Stop, Start, and Delete
116117

@@ -124,6 +125,20 @@ Delete force-removes both containers, the driver-owned runtime volumes, and the
124125
host-private runtime descriptor. Missing or altered descriptor and channel resources
125126
fail closed; the driver does not run an older combined-supervisor layout.
126127

128+
## CDI GPU Metadata
129+
130+
Docker remains the source of truth for GPU injection. The driver selects opaque
131+
CDI device IDs from `driver_config.cdi_devices` or the daemon's discovered CDI
132+
inventory, then passes the same IDs to Docker with a CDI `DeviceRequest`.
133+
134+
For a GPU sandbox, the driver writes those IDs and projected CDI spec directory
135+
paths to a versioned context in the protected workload boundary configuration.
136+
It bind-mounts the daemon-reported spec directories read-only into the workload.
137+
The sandbox runtime resolves the selected devices in that mount namespace before
138+
it launches agent processes and derives Landlock paths and supplemental groups
139+
from CDI `containerEdits`. A missing context or spec directory fails workload
140+
startup closed. Non-GPU sandboxes receive no CDI context or spec mounts.
141+
127142
## Driver Config Mounts
128143

129144
The gateway forwards the `docker` block from `--driver-config-json`. Supported

‎crates/openshell-driver-docker/src/isolation.rs‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ use std::collections::{BTreeMap, HashMap};
1111
use std::net::IpAddr;
1212
use std::path::PathBuf;
1313

14+
use openshell_core::cdi::CdiContext;
1415
use openshell_isolation_interface::contract::{
1516
BackendError, OuterFenceGuarantee, OuterFenceGuarantees, ResolvedWorkloadIdentity,
1617
};
@@ -70,6 +71,7 @@ pub struct DockerBoundarySpec {
7071
pub container_id: String,
7172
pub image_identity: String,
7273
pub gpu_requested: bool,
74+
pub cdi_context: Option<CdiContext>,
7375
pub listener_socket: PathBuf,
7476
pub control_socket: PathBuf,
7577
pub sandbox_tls: SandboxTlsServerConfig,
@@ -118,7 +120,7 @@ impl DockerBoundarySpec {
118120
},
119121
resource_claims: resource_claims.clone(),
120122
resource_claim_files: BTreeMap::new(),
121-
cdi_context: None,
123+
cdi_context: self.cdi_context,
122124
workload_identity: self.workload_identity.clone(),
123125
outer_fence: outer_fence.clone(),
124126
child_env: self.child_env,
@@ -188,6 +190,13 @@ mod tests {
188190
container_id: "sha256:container".to_string(),
189191
image_identity: "sha256:image".to_string(),
190192
gpu_requested: true,
193+
cdi_context: Some(CdiContext::new(
194+
vec!["nvidia.com/gpu=0".to_string()],
195+
vec![openshell_core::cdi::CdiSpecDirectory::new(
196+
openshell_core::cdi::cdi_spec_mount_path(0),
197+
"/var/run/cdi",
198+
)],
199+
)),
191200
listener_socket: PathBuf::from("/run/openshell/boundary/control.sock"),
192201
control_socket: PathBuf::from("/host/control.sock"),
193202
sandbox_tls: SandboxTlsServerConfig {
@@ -224,6 +233,15 @@ mod tests {
224233
provisioned.runtime_descriptor.resource_claims[GPU_RESOURCE_CLAIM],
225234
"true"
226235
);
236+
assert_eq!(
237+
provisioned
238+
.boundary_config
239+
.cdi_context
240+
.as_ref()
241+
.unwrap()
242+
.selected_devices,
243+
["nvidia.com/gpu=0"]
244+
);
227245
assert_eq!(
228246
provisioned.boundary_config.outer_fence,
229247
provisioned.runtime_descriptor.outer_fence

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 51 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ use bollard::query_parameters::{
2323
};
2424
use bytes::Bytes;
2525
use futures::{Stream, StreamExt};
26+
use openshell_core::cdi::{CdiContext, CdiSpecDirectory, cdi_spec_mount_path};
2627
use openshell_core::config::DEFAULT_STOP_TIMEOUT_SECS;
2728
use openshell_core::driver_mounts;
2829
use openshell_core::driver_utils::{
@@ -321,12 +322,49 @@ struct DockerDriverRuntimeConfig {
321322
app_armor_profile: Option<AppArmorProfile>,
322323
}
323324

324-
#[derive(Debug, Clone, Copy)]
325+
#[derive(Debug, Clone)]
325326
struct DockerGpuRuntimeCapabilities {
326327
cdi_supported: bool,
328+
cdi_spec_dirs: Vec<String>,
327329
wsl_all_gpu_fallback_enabled: bool,
328330
}
329331

332+
impl DockerGpuRuntimeCapabilities {
333+
fn cdi_context(&self, gpu_device_ids: Option<&[String]>) -> Result<Option<CdiContext>, Status> {
334+
let Some(gpu_device_ids) = gpu_device_ids.filter(|device_ids| !device_ids.is_empty())
335+
else {
336+
return Ok(None);
337+
};
338+
self.require_cdi_spec_dirs()?;
339+
Ok(Some(CdiContext::new(
340+
gpu_device_ids.to_vec(),
341+
self.cdi_spec_dirs
342+
.iter()
343+
.enumerate()
344+
.map(|(index, source)| CdiSpecDirectory::new(cdi_spec_mount_path(index), source))
345+
.collect(),
346+
)))
347+
}
348+
349+
fn cdi_spec_bind_strings(&self) -> Result<Vec<String>, Status> {
350+
self.require_cdi_spec_dirs()?;
351+
Ok(self
352+
.cdi_spec_dirs
353+
.iter()
354+
.enumerate()
355+
.map(|(index, source)| format!("{source}:{}:ro,z", cdi_spec_mount_path(index)))
356+
.collect())
357+
}
358+
359+
fn require_cdi_spec_dirs(&self) -> Result<(), Status> {
360+
if self.cdi_spec_dirs.is_empty() {
361+
return Err(Status::failed_precondition(
362+
"docker GPU sandboxes require Docker CDI spec directories reported by the daemon",
363+
));
364+
}
365+
Ok(())
366+
}
367+
}
330368
#[derive(Clone)]
331369
pub struct DockerComputeDriver {
332370
docker: Arc<Docker>,
@@ -870,14 +908,13 @@ impl DockerComputeDriver {
870908
let info = docker.info().await.map_err(|err| {
871909
Error::execution(format!("failed to query Docker daemon info: {err}"))
872910
})?;
873-
let cdi_supported = info
874-
.cdi_spec_dirs
875-
.as_ref()
876-
.is_some_and(|dirs| !dirs.is_empty());
911+
let cdi_spec_dirs = info.cdi_spec_dirs.clone().unwrap_or_default();
912+
let cdi_supported = !cdi_spec_dirs.is_empty();
877913
let cdi_gpu_inventory = docker_cdi_gpu_inventory(&info);
878914
let wsl_all_gpu_fallback_enabled = docker_info_reports_wsl2(&info);
879915
let gpu = DockerGpuRuntimeCapabilities {
880916
cdi_supported,
917+
cdi_spec_dirs,
881918
wsl_all_gpu_fallback_enabled,
882919
};
883920
validate_docker_app_armor_profile(docker_config.app_armor_profile.as_ref(), &info)?;
@@ -941,7 +978,7 @@ impl DockerComputeDriver {
941978
supervisor_grpc_endpoint,
942979
ssh_socket_path: docker_config.ssh_socket_path.clone(),
943980
guest_tls,
944-
gpu,
981+
gpu: gpu.clone(),
945982
sandbox_pids_limit: docker_config.sandbox_pids_limit,
946983
enable_bind_mounts: docker_config.enable_bind_mounts,
947984
allow_driver_config: docker_config.allow_driver_config,
@@ -1762,7 +1799,7 @@ impl DockerComputeDriver {
17621799
&created.id,
17631800
&image,
17641801
&workload_identity,
1765-
gpu_devices.is_some(),
1802+
gpu_devices.as_deref(),
17661803
)
17671804
.await
17681805
{
@@ -4522,7 +4559,7 @@ async fn prepare_docker_boundary_files(
45224559
container_id: &str,
45234560
image: &DockerImageMetadata,
45244561
workload_identity: &ResolvedWorkloadIdentity,
4525-
gpu_requested: bool,
4562+
gpu_device_ids: Option<&[String]>,
45264563
) -> Result<(), Status> {
45274564
let directory = docker_boundary_state_dir(sandbox, config)?;
45284565
let workspace_root = driver_mounts::resolve_oci_workspace_root(&image.working_dir)
@@ -4552,7 +4589,8 @@ async fn prepare_docker_boundary_files(
45524589
verification_keys,
45534590
container_id: container_id.to_string(),
45544591
image_identity: image.id.clone(),
4555-
gpu_requested,
4592+
gpu_requested: gpu_device_ids.is_some_and(|device_ids| !device_ids.is_empty()),
4593+
cdi_context: config.gpu.cdi_context(gpu_device_ids)?,
45564594
listener_socket: PathBuf::from(BOUNDARY_SOCKET_MOUNT_PATH),
45574595
control_socket: PathBuf::from(BOUNDARY_SOCKET_MOUNT_PATH),
45584596
sandbox_tls: SandboxTlsServerConfig {
@@ -5705,7 +5743,10 @@ fn build_container_create_body_for_image(
57055743
}),
57065744
..Default::default()
57075745
});
5708-
let user_bind_strings = docker_driver_bind_strings(driver_config)?;
5746+
let mut user_bind_strings = docker_driver_bind_strings(driver_config)?;
5747+
if gpu_device_ids.is_some_and(|device_ids| !device_ids.is_empty()) {
5748+
user_bind_strings.extend(config.gpu.cdi_spec_bind_strings()?);
5749+
}
57095750
let device_requests = gpu_device_ids.map(|device_ids| {
57105751
vec![DeviceRequest {
57115752
driver: Some("cdi".to_string()),

‎crates/openshell-driver-docker/src/tests.rs‎

Lines changed: 69 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,9 @@ fn startup_error_log_tails_fit_grpc_header_budget() {
4949
assert!(format_log_tail("").is_empty());
5050
}
5151

52+
const TEST_CDI_SPEC_DIR: &str = "/opt/openshell-test/cdi";
53+
const TEST_CDI_SPEC_DIR_ALT: &str = "/srv/openshell-test/cdi";
54+
5255
fn test_launch_authentication() -> Vec<u8> {
5356
serde_json::to_vec(&SandboxLaunchAuthentication {
5457
supervisor: SupervisorAuthBundle {
@@ -216,6 +219,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
216219
}),
217220
gpu: DockerGpuRuntimeCapabilities {
218221
cdi_supported: false,
222+
cdi_spec_dirs: Vec::new(),
219223
wsl_all_gpu_fallback_enabled: false,
220224
},
221225
sandbox_pids_limit: openshell_core::config::default_sandbox_pids_limit(),
@@ -389,6 +393,59 @@ fn sandbox_driver_config_cannot_override_proxy_ca_bundle() {
389393
assert!(error.message().contains("proxy_ca_bundle"), "{error}");
390394
}
391395

396+
fn runtime_config_with_cdi_spec_dirs(cdi_spec_dirs: &[&str]) -> DockerDriverRuntimeConfig {
397+
let mut config = runtime_config();
398+
config.gpu.cdi_supported = !cdi_spec_dirs.is_empty();
399+
config.gpu.cdi_spec_dirs = cdi_spec_dirs
400+
.iter()
401+
.map(|path| (*path).to_string())
402+
.collect();
403+
config
404+
}
405+
406+
#[test]
407+
fn gpu_runtime_builds_cdi_context_from_selected_devices_and_daemon_dirs() {
408+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR, TEST_CDI_SPEC_DIR_ALT]);
409+
let selected_devices = vec![
410+
"nvidia.com/gpu=0".to_string(),
411+
"nvidia.com/gpu=1".to_string(),
412+
];
413+
414+
let context = config
415+
.gpu
416+
.cdi_context(Some(&selected_devices))
417+
.unwrap()
418+
.expect("GPU selection should produce a CDI context");
419+
420+
assert_eq!(context.selected_devices, selected_devices);
421+
assert_eq!(
422+
context.spec_dirs,
423+
vec![
424+
CdiSpecDirectory::new(cdi_spec_mount_path(0), TEST_CDI_SPEC_DIR),
425+
CdiSpecDirectory::new(cdi_spec_mount_path(1), TEST_CDI_SPEC_DIR_ALT),
426+
]
427+
);
428+
}
429+
430+
#[test]
431+
fn gpu_runtime_omits_cdi_projection_without_selected_devices() {
432+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
433+
assert_eq!(config.gpu.cdi_context(None).unwrap(), None);
434+
}
435+
436+
#[test]
437+
fn gpu_runtime_builds_read_only_cdi_bind_mounts() {
438+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR, TEST_CDI_SPEC_DIR_ALT]);
439+
440+
assert_eq!(
441+
config.gpu.cdi_spec_bind_strings().unwrap(),
442+
vec![
443+
format!("{TEST_CDI_SPEC_DIR}:{}:ro,z", cdi_spec_mount_path(0)),
444+
format!("{TEST_CDI_SPEC_DIR_ALT}:{}:ro,z", cdi_spec_mount_path(1)),
445+
]
446+
);
447+
}
448+
392449
fn test_workload_identity() -> ResolvedWorkloadIdentity {
393450
ResolvedWorkloadIdentity::new(
394451
1234,
@@ -2640,8 +2697,7 @@ fn validate_sandbox_auth_accepts_launch_authentication() {
26402697

26412698
#[test]
26422699
fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() {
2643-
let mut config = runtime_config();
2644-
config.gpu.cdi_supported = true;
2700+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
26452701
let mut sandbox = test_sandbox();
26462702
sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(None));
26472703

@@ -2666,6 +2722,15 @@ fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() {
26662722
request.device_ids.as_ref().unwrap(),
26672723
&vec!["nvidia.com/gpu=1".to_string()]
26682724
);
2725+
let binds = create_body
2726+
.host_config
2727+
.as_ref()
2728+
.and_then(|host_config| host_config.binds.as_ref())
2729+
.expect("GPU request should project CDI specs into the workload");
2730+
assert!(binds.contains(&format!(
2731+
"{TEST_CDI_SPEC_DIR}:{}:ro,z",
2732+
cdi_spec_mount_path(0)
2733+
)));
26692734
}
26702735

26712736
#[test]
@@ -2688,8 +2753,7 @@ fn build_container_create_body_omits_devices_without_resolved_default_cdi_device
26882753

26892754
#[test]
26902755
fn build_container_create_body_passes_explicit_cdi_device_id_through() {
2691-
let mut config = runtime_config();
2692-
config.gpu.cdi_supported = true;
2756+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
26932757
let mut sandbox = test_sandbox();
26942758
let spec = sandbox.spec.as_mut().unwrap();
26952759
spec.resource_requirements = Some(gpu_resources(None));
@@ -2759,8 +2823,7 @@ fn build_container_create_body_rejects_empty_cdi_devices() {
27592823

27602824
#[test]
27612825
fn driver_default_gpu_selection_consumes_distinct_devices_for_creates() {
2762-
let mut config = runtime_config();
2763-
config.gpu.cdi_supported = true;
2826+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
27642827
let driver = test_driver_with_config(config);
27652828
driver.gpu_selector.refresh(
27662829
CdiGpuInventory::new(["nvidia.com/gpu=0", "nvidia.com/gpu=1"]),

‎docs/how-it-works/sandboxes/runtimes.mdx‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,17 @@ Common options in `[openshell.drivers.docker]` are `socket_path`, `grpc_endpoint
103103

104104
Docker Desktop must have host networking enabled, and it cannot use Enhanced Container Isolation. Set `grpc_endpoint` when sandboxes cannot reach the gateway on host loopback. For GPU sandboxes, configure Docker CDI before starting the gateway.
105105

106+
For Docker GPU/CDI sandboxes, OpenShell uses Docker's selected CDI device IDs
107+
and daemon-reported CDI spec directories to build a workload CDI context. The
108+
driver embeds the context in the protected boundary configuration and
109+
bind-mounts the spec directories read-only into the workload. If context
110+
creation fails, the driver removes the created state. If workload or supervisor
111+
startup fails, it also removes the runtime resources before reporting the
112+
failure. The sandbox runtime derives filesystem and supplemental group
113+
requirements from CDI specs before launching agent processes. Non-GPU Docker
114+
sandboxes do not receive the CDI context, spec mounts, or CDI-derived policy
115+
changes.
116+
106117
### Docker Corporate Proxy Egress
107118

108119
For proxy-required networks, the Docker driver accepts `https_proxy`,

0 commit comments

Comments
 (0)