Skip to content

Commit 5e6241d

Browse files
committed
feat(docker): project CDI metadata into GPU workloads
Signed-off-by: Evan Lezar <elezar@nvidia.com>
1 parent a39d182 commit 5e6241d

7 files changed

Lines changed: 186 additions & 17 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,6 +327,19 @@ Resource requirements enter the driver layer through `SandboxSpec.resource_requi
327327
can request a specific number of GPUs or the driver-specific default behaviour.
328328
For all in-tree drivers, this is equivalent to selecting a single GPU.
329329

330+
For Docker GPU sandboxes, the driver treats CDI specs as runtime metadata for
331+
both outer injection and inner sandbox policy. It selects opaque CDI device IDs,
332+
passes them to Docker, embeds a versioned CDI context in the protected workload
333+
boundary configuration, and mounts daemon-reported CDI spec directories
334+
read-only into the workload. The sandbox runtime resolves that context in the
335+
workload mount namespace and derives Landlock paths and supplemental groups
336+
from CDI `containerEdits` before launching agent processes.
337+
Host-side CDI spec paths are diagnostic only and are never treated as sandbox
338+
policy paths.
339+
Kubernetes must not infer CDI device IDs from the `nvidia.com/gpu` resource
340+
request; it needs a node-local selected-device handoff before using the same
341+
workload-side resolver.
342+
330343
VM runtime state paths are derived only from driver-validated sandbox IDs
331344
matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a
332345
private `run/` directory plus Unix peer UID/PID checks. Standalone

‎crates/openshell-core/src/cdi.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,12 @@ pub const CDI_CONTEXT_VERSION: u32 = 1;
1212
/// Base workload-runtime path under which compute drivers mount CDI specification directories.
1313
pub const CDI_SPEC_DIR_BASE: &str = "/run/openshell/boundary/cdi-specs";
1414

15+
/// Return the workload-runtime path used for a CDI specification directory.
16+
#[must_use]
17+
pub fn cdi_spec_mount_path(index: usize) -> String {
18+
format!("{CDI_SPEC_DIR_BASE}/{index}")
19+
}
20+
1521
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
1622
pub struct CdiContext {
1723
pub version: u32,

‎crates/openshell-driver-docker/README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ LSM decisions remain authoritative.
7979
| Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. |
8080
| In-memory `/run/openshell-supervisor-ca` tmpfs | Holds only the public supervisor CA certificate and trust bundle without making all of `/run` writable. |
8181
| CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. |
82+
| CDI workload projection | For GPU sandboxes only, embeds selected device metadata in the protected boundary config and mounts Docker daemon CDI spec directories read-only into the workload. |
8283

8384
## Stop, Start, and Delete
8485

@@ -92,6 +93,20 @@ Delete force-removes both containers, the driver-owned runtime volumes, and the
9293
host-private runtime descriptor. Missing or altered descriptor and channel resources
9394
fail closed; the driver does not run an older combined-supervisor layout.
9495

96+
## CDI GPU Metadata
97+
98+
Docker remains the source of truth for GPU injection. The driver selects opaque
99+
CDI device IDs from `driver_config.cdi_devices` or the daemon's discovered CDI
100+
inventory, then passes the same IDs to Docker with a CDI `DeviceRequest`.
101+
102+
For a GPU sandbox, the driver writes those IDs and projected CDI spec directory
103+
paths to a versioned context in the protected workload boundary configuration.
104+
It bind-mounts the daemon-reported spec directories read-only into the workload.
105+
The sandbox runtime resolves the selected devices in that mount namespace before
106+
it launches agent processes and derives Landlock paths and supplemental groups
107+
from CDI `containerEdits`. A missing context or spec directory fails workload
108+
startup closed. Non-GPU sandboxes receive no CDI context or spec mounts.
109+
95110
## Driver Config Mounts
96111

97112
The gateway forwards the `docker` block from `--driver-config-json`. Supported

‎crates/openshell-driver-docker/src/isolation.rs‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ use std::collections::{BTreeMap, HashMap};
1111
use std::net::IpAddr;
1212
use std::path::PathBuf;
1313

14+
use openshell_core::cdi::CdiContext;
1415
use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity};
1516
use openshell_sandbox_backend::GPU_RESOURCE_CLAIM;
1617
use openshell_sandbox_backend::boundary_protocol::{
@@ -30,6 +31,7 @@ pub struct DockerBoundarySpec {
3031
pub container_id: String,
3132
pub image_identity: String,
3233
pub gpu_requested: bool,
34+
pub cdi_context: Option<CdiContext>,
3335
pub listener_socket: PathBuf,
3436
pub control_socket: PathBuf,
3537
pub sandbox_tls: SandboxTlsServerConfig,
@@ -77,7 +79,7 @@ impl DockerBoundarySpec {
7779
},
7880
resource_claims: resource_claims.clone(),
7981
resource_claim_files: BTreeMap::new(),
80-
cdi_context: None,
82+
cdi_context: self.cdi_context,
8183
workload_identity: self.workload_identity.clone(),
8284
driver_fence: driver_fence.clone(),
8385
child_env: self.child_env,
@@ -123,6 +125,13 @@ mod tests {
123125
container_id: "sha256:container".to_string(),
124126
image_identity: "sha256:image".to_string(),
125127
gpu_requested: true,
128+
cdi_context: Some(CdiContext::new(
129+
vec!["nvidia.com/gpu=0".to_string()],
130+
vec![openshell_core::cdi::CdiSpecDirectory::new(
131+
openshell_core::cdi::cdi_spec_mount_path(0),
132+
"/var/run/cdi",
133+
)],
134+
)),
126135
listener_socket: PathBuf::from("/run/openshell/boundary/control.sock"),
127136
control_socket: PathBuf::from("/host/control.sock"),
128137
sandbox_tls: SandboxTlsServerConfig {
@@ -158,6 +167,15 @@ mod tests {
158167
provisioned.runtime_descriptor.resource_claims[GPU_RESOURCE_CLAIM],
159168
"true"
160169
);
170+
assert_eq!(
171+
provisioned
172+
.boundary_config
173+
.cdi_context
174+
.as_ref()
175+
.unwrap()
176+
.selected_devices,
177+
["nvidia.com/gpu=0"]
178+
);
161179
assert_eq!(
162180
provisioned.boundary_config.driver_fence,
163181
provisioned.runtime_descriptor.driver_fence

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 53 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ use bollard::query_parameters::{
2323
};
2424
use bytes::Bytes;
2525
use futures::{Stream, StreamExt};
26+
use openshell_core::cdi::{CdiContext, CdiSpecDirectory, cdi_spec_mount_path};
2627
use openshell_core::config::DEFAULT_STOP_TIMEOUT_SECS;
2728
use openshell_core::driver_mounts;
2829
use openshell_core::driver_utils::{
@@ -319,12 +320,50 @@ struct DockerDriverRuntimeConfig {
319320
app_armor_profile: Option<AppArmorProfile>,
320321
}
321322

322-
#[derive(Debug, Clone, Copy)]
323+
#[derive(Debug, Clone)]
323324
struct DockerGpuRuntimeCapabilities {
324325
cdi_supported: bool,
326+
cdi_spec_dirs: Vec<String>,
325327
wsl_all_gpu_fallback_enabled: bool,
326328
}
327329

330+
impl DockerGpuRuntimeCapabilities {
331+
fn cdi_context(&self, gpu_device_ids: Option<&[String]>) -> Result<Option<CdiContext>, Status> {
332+
let Some(gpu_device_ids) = gpu_device_ids.filter(|device_ids| !device_ids.is_empty())
333+
else {
334+
return Ok(None);
335+
};
336+
self.require_cdi_spec_dirs()?;
337+
Ok(Some(CdiContext::new(
338+
gpu_device_ids.to_vec(),
339+
self.cdi_spec_dirs
340+
.iter()
341+
.enumerate()
342+
.map(|(index, source)| CdiSpecDirectory::new(cdi_spec_mount_path(index), source))
343+
.collect(),
344+
)))
345+
}
346+
347+
fn cdi_spec_bind_strings(&self) -> Result<Vec<String>, Status> {
348+
self.require_cdi_spec_dirs()?;
349+
Ok(self
350+
.cdi_spec_dirs
351+
.iter()
352+
.enumerate()
353+
.map(|(index, source)| format!("{source}:{}:ro,z", cdi_spec_mount_path(index)))
354+
.collect())
355+
}
356+
357+
fn require_cdi_spec_dirs(&self) -> Result<(), Status> {
358+
if self.cdi_spec_dirs.is_empty() {
359+
return Err(Status::failed_precondition(
360+
"docker GPU sandboxes require Docker CDI spec directories reported by the daemon",
361+
));
362+
}
363+
Ok(())
364+
}
365+
}
366+
328367
#[derive(Debug, Clone, PartialEq, Eq)]
329368
enum DockerGatewayRoute {
330369
Bridge { bind_address: SocketAddr },
@@ -864,14 +903,13 @@ impl DockerComputeDriver {
864903
let info = docker.info().await.map_err(|err| {
865904
Error::execution(format!("failed to query Docker daemon info: {err}"))
866905
})?;
867-
let cdi_supported = info
868-
.cdi_spec_dirs
869-
.as_ref()
870-
.is_some_and(|dirs| !dirs.is_empty());
906+
let cdi_spec_dirs = info.cdi_spec_dirs.clone().unwrap_or_default();
907+
let cdi_supported = !cdi_spec_dirs.is_empty();
871908
let cdi_gpu_inventory = docker_cdi_gpu_inventory(&info);
872909
let wsl_all_gpu_fallback_enabled = docker_info_reports_wsl2(&info);
873910
let gpu = DockerGpuRuntimeCapabilities {
874911
cdi_supported,
912+
cdi_spec_dirs,
875913
wsl_all_gpu_fallback_enabled,
876914
};
877915
validate_sandbox_pids_limit(docker_config.sandbox_pids_limit)?;
@@ -973,7 +1011,7 @@ impl DockerComputeDriver {
9731011
ssh_socket_path: docker_config.ssh_socket_path.clone(),
9741012
guest_tls,
9751013
daemon_version: version.version.unwrap_or_else(|| "unknown".to_string()),
976-
gpu,
1014+
gpu: gpu.clone(),
9771015
sandbox_pids_limit: docker_config.sandbox_pids_limit,
9781016
enable_bind_mounts: docker_config.enable_bind_mounts,
9791017
upstream_proxy: docker_config.upstream_proxy.clone(),
@@ -1616,7 +1654,7 @@ impl DockerComputeDriver {
16161654
&created.id,
16171655
&image,
16181656
&workload_identity,
1619-
gpu_devices.is_some(),
1657+
gpu_devices.as_deref(),
16201658
)
16211659
.await
16221660
{
@@ -4387,7 +4425,7 @@ async fn prepare_docker_boundary_files(
43874425
container_id: &str,
43884426
image: &DockerImageMetadata,
43894427
workload_identity: &ResolvedWorkloadIdentity,
4390-
gpu_requested: bool,
4428+
gpu_device_ids: Option<&[String]>,
43914429
) -> Result<(), Status> {
43924430
let directory = docker_boundary_state_dir(sandbox, config)?;
43934431
let workspace_root = driver_mounts::resolve_oci_workspace_root(&image.working_dir)
@@ -4418,7 +4456,8 @@ async fn prepare_docker_boundary_files(
44184456
verification_keys,
44194457
container_id: container_id.to_string(),
44204458
image_identity: image.id.clone(),
4421-
gpu_requested,
4459+
gpu_requested: gpu_device_ids.is_some_and(|device_ids| !device_ids.is_empty()),
4460+
cdi_context: config.gpu.cdi_context(gpu_device_ids)?,
44224461
listener_socket: PathBuf::from(BOUNDARY_SOCKET_MOUNT_PATH),
44234462
control_socket: PathBuf::from(BOUNDARY_SOCKET_MOUNT_PATH),
44244463
sandbox_tls: SandboxTlsServerConfig {
@@ -4996,6 +5035,7 @@ async fn spawn_docker_control_process(
49965035
// fenced by network=none.
49975036
network_mode: Some(config.network_name.clone()),
49985037
mounts: Some(supervisor_mounts),
5038+
binds: None,
49995039
cap_drop: Some(vec!["ALL".to_string()]),
50005040
cap_add: None,
50015041
security_opt: Some(vec!["no-new-privileges:true".to_string()]),
@@ -5543,7 +5583,10 @@ fn build_container_create_body_for_image(
55435583
}),
55445584
..Default::default()
55455585
});
5546-
let user_bind_strings = docker_driver_bind_strings(driver_config)?;
5586+
let mut user_bind_strings = docker_driver_bind_strings(driver_config)?;
5587+
if gpu_device_ids.is_some_and(|device_ids| !device_ids.is_empty()) {
5588+
user_bind_strings.extend(config.gpu.cdi_spec_bind_strings()?);
5589+
}
55475590
let device_requests = gpu_device_ids.map(|device_ids| {
55485591
vec![DeviceRequest {
55495592
driver: Some("cdi".to_string()),

‎crates/openshell-driver-docker/src/tests.rs‎

Lines changed: 69 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ use std::net::{IpAddr, Ipv4Addr, SocketAddr};
2626
use std::sync::Arc;
2727
use tempfile::TempDir;
2828

29+
const TEST_CDI_SPEC_DIR: &str = "/opt/openshell-test/cdi";
30+
const TEST_CDI_SPEC_DIR_ALT: &str = "/srv/openshell-test/cdi";
31+
2932
fn test_launch_authentication() -> Vec<u8> {
3033
serde_json::to_vec(&SandboxLaunchAuthentication {
3134
supervisor: SupervisorAuthBundle {
@@ -150,6 +153,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
150153
daemon_version: "28.0.0".to_string(),
151154
gpu: DockerGpuRuntimeCapabilities {
152155
cdi_supported: false,
156+
cdi_spec_dirs: Vec::new(),
153157
wsl_all_gpu_fallback_enabled: false,
154158
},
155159
sandbox_pids_limit: openshell_core::config::default_sandbox_pids_limit(),
@@ -160,6 +164,59 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
160164
}
161165
}
162166

167+
fn runtime_config_with_cdi_spec_dirs(cdi_spec_dirs: &[&str]) -> DockerDriverRuntimeConfig {
168+
let mut config = runtime_config();
169+
config.gpu.cdi_supported = !cdi_spec_dirs.is_empty();
170+
config.gpu.cdi_spec_dirs = cdi_spec_dirs
171+
.iter()
172+
.map(|path| (*path).to_string())
173+
.collect();
174+
config
175+
}
176+
177+
#[test]
178+
fn gpu_runtime_builds_cdi_context_from_selected_devices_and_daemon_dirs() {
179+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR, TEST_CDI_SPEC_DIR_ALT]);
180+
let selected_devices = vec![
181+
"nvidia.com/gpu=0".to_string(),
182+
"nvidia.com/gpu=1".to_string(),
183+
];
184+
185+
let context = config
186+
.gpu
187+
.cdi_context(Some(&selected_devices))
188+
.unwrap()
189+
.expect("GPU selection should produce a CDI context");
190+
191+
assert_eq!(context.selected_devices, selected_devices);
192+
assert_eq!(
193+
context.spec_dirs,
194+
vec![
195+
CdiSpecDirectory::new(cdi_spec_mount_path(0), TEST_CDI_SPEC_DIR),
196+
CdiSpecDirectory::new(cdi_spec_mount_path(1), TEST_CDI_SPEC_DIR_ALT),
197+
]
198+
);
199+
}
200+
201+
#[test]
202+
fn gpu_runtime_omits_cdi_projection_without_selected_devices() {
203+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
204+
assert_eq!(config.gpu.cdi_context(None).unwrap(), None);
205+
}
206+
207+
#[test]
208+
fn gpu_runtime_builds_read_only_cdi_bind_mounts() {
209+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR, TEST_CDI_SPEC_DIR_ALT]);
210+
211+
assert_eq!(
212+
config.gpu.cdi_spec_bind_strings().unwrap(),
213+
vec![
214+
format!("{TEST_CDI_SPEC_DIR}:{}:ro,z", cdi_spec_mount_path(0)),
215+
format!("{TEST_CDI_SPEC_DIR_ALT}:{}:ro,z", cdi_spec_mount_path(1)),
216+
]
217+
);
218+
}
219+
163220
fn test_workload_identity() -> ResolvedWorkloadIdentity {
164221
ResolvedWorkloadIdentity::new(
165222
1234,
@@ -2638,8 +2695,7 @@ fn validate_sandbox_auth_accepts_launch_authentication() {
26382695

26392696
#[test]
26402697
fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() {
2641-
let mut config = runtime_config();
2642-
config.gpu.cdi_supported = true;
2698+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
26432699
let mut sandbox = test_sandbox();
26442700
sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(None));
26452701

@@ -2664,6 +2720,15 @@ fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() {
26642720
request.device_ids.as_ref().unwrap(),
26652721
&vec!["nvidia.com/gpu=1".to_string()]
26662722
);
2723+
let binds = create_body
2724+
.host_config
2725+
.as_ref()
2726+
.and_then(|host_config| host_config.binds.as_ref())
2727+
.expect("GPU request should project CDI specs into the workload");
2728+
assert!(binds.contains(&format!(
2729+
"{TEST_CDI_SPEC_DIR}:{}:ro,z",
2730+
cdi_spec_mount_path(0)
2731+
)));
26672732
}
26682733

26692734
#[test]
@@ -2686,8 +2751,7 @@ fn build_container_create_body_omits_devices_without_resolved_default_cdi_device
26862751

26872752
#[test]
26882753
fn build_container_create_body_passes_explicit_cdi_device_id_through() {
2689-
let mut config = runtime_config();
2690-
config.gpu.cdi_supported = true;
2754+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
26912755
let mut sandbox = test_sandbox();
26922756
let spec = sandbox.spec.as_mut().unwrap();
26932757
spec.resource_requirements = Some(gpu_resources(None));
@@ -2757,8 +2821,7 @@ fn build_container_create_body_rejects_empty_cdi_devices() {
27572821

27582822
#[test]
27592823
fn driver_default_gpu_selection_consumes_distinct_devices_for_creates() {
2760-
let mut config = runtime_config();
2761-
config.gpu.cdi_supported = true;
2824+
let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR]);
27622825
let driver = test_driver_with_config(config);
27632826
driver.gpu_selector.refresh(
27642827
CdiGpuInventory::new(["nvidia.com/gpu=0", "nvidia.com/gpu=1"]),

‎docs/reference/sandbox-compute-drivers.mdx‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,17 @@ idempotent start request. Explicitly stopped sandboxes remain stopped.
196196

197197
For GPU-backed Docker sandboxes, configure Docker CDI before starting the gateway so OpenShell can detect the daemon capability.
198198

199+
For Docker GPU/CDI sandboxes, OpenShell uses Docker's selected CDI device IDs
200+
and daemon-reported CDI spec directories to build a workload CDI context. The
201+
driver embeds the context in the protected boundary configuration and
202+
bind-mounts the spec directories read-only into the workload. If context
203+
creation fails, the driver removes the created state; if workload or supervisor
204+
startup fails, it also removes the runtime resources before reporting the
205+
failure. The sandbox runtime derives filesystem and supplemental group
206+
requirements from CDI specs before launching agent processes. Non-GPU Docker
207+
sandboxes do not receive the CDI context, spec mounts, or CDI-derived policy
208+
changes.
209+
199210
### Docker Driver Config Mounts
200211

201212
Docker driver config accepts user-supplied `volume` and `tmpfs` mounts. It also

0 commit comments

Comments
 (0)