fix(docker): retry transient runtime volume removal

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

Author: drew

architecture/sandbox.md

@@ -527,6 +527,8 @@ The Docker driver runs `openshell-supervisor` in a separate companion container.
 Its private named volume contains supervisor bootstrap and channel material.
 The workload container receives only `openshell-sandbox`, public interception
 CA material, and the other sandbox half of the authenticated channel.
+On deletion, the driver waits briefly for Docker to release the removed
+containers' private volume mounts before removing those volumes.
 
 For Kubernetes, the operator configures a Secret name and key rather than a
 gateway-host file path. Kubernetes projects that Secret only into the separate

crates/openshell-driver-docker/README.md

@@ -97,6 +97,8 @@ across gateway restarts.
 Delete force-removes both containers, the driver-owned runtime volumes, and the
 host-private runtime descriptor. Missing or altered descriptor and channel resources
 fail closed; the driver does not run an older combined-supervisor layout.
+Docker may briefly retain a volume mount after acknowledging container removal;
+the driver retries only that volume-in-use response for a bounded interval.
 
 ## Driver Config Mounts
 

crates/openshell-driver-docker/src/lib.rs

@@ -4140,17 +4140,28 @@ async fn remove_docker_channel_volume_by_id(
 }
 
 async fn remove_docker_volume(docker: &Docker, name: &str) -> Result<(), Status> {
-    docker
-        .remove_volume(name, None::<bollard::query_parameters::RemoveVolumeOptions>)
-        .await
-        .or_else(|error| {
-            if is_not_found_error(&error) {
-                Ok(())
-            } else {
-                Err(error)
+    // Docker can acknowledge removal of the last container before releasing
+    // its volume mount. Retry only that transient 409; other conflicts and
+    // daemon errors should still fail the sandbox deletion.
+    const MAX_ATTEMPTS: usize = 12;
+    for attempt in 1..=MAX_ATTEMPTS {
+        match docker
+            .remove_volume(name, None::<bollard::query_parameters::RemoveVolumeOptions>)
+            .await
+        {
+            Ok(()) => return Ok(()),
+            Err(error) if is_not_found_error(&error) => return Ok(()),
+            Err(error) if is_volume_in_use_error(&error) && attempt < MAX_ATTEMPTS => {
+                tokio::time::sleep(Duration::from_millis(250)).await;
             }
-        })
-        .map_err(|error| Status::internal(format!("remove Docker sandbox runtime volume: {error}")))
+            Err(error) => {
+                return Err(Status::internal(format!(
+                    "remove Docker sandbox runtime volume: {error}"
+                )));
+            }
+        }
+    }
+    unreachable!("the final volume removal attempt returns a result")
 }
 
 fn sandbox_token_host_path(
@@ -6525,6 +6536,16 @@ fn is_removal_in_progress_error(err: &BollardError) -> bool {
     )
 }
 
+fn is_volume_in_use_error(err: &BollardError) -> bool {
+    matches!(
+        err,
+        BollardError::DockerResponseServerError {
+            status_code: 409,
+            message,
+        } if message.contains("volume is in use")
+    )
+}
+
 fn is_not_modified_error(err: &BollardError) -> bool {
     matches!(
         err,

crates/openshell-driver-docker/src/tests.rs

@@ -3440,6 +3440,20 @@ fn concurrent_container_removal_is_idempotent() {
     assert!(!is_removal_in_progress_error(&other_conflict));
 }
 
+#[test]
+fn volume_removal_retries_only_mount_release_conflicts() {
+    let in_use = BollardError::DockerResponseServerError {
+        status_code: 409,
+        message: "remove sandbox-volume: volume is in use - [container-id]".to_string(),
+    };
+    let other_conflict = BollardError::DockerResponseServerError {
+        status_code: 409,
+        message: "volume has invalid options".to_string(),
+    };
+    assert!(is_volume_in_use_error(&in_use));
+    assert!(!is_volume_in_use_error(&other_conflict));
+}
+
 #[tokio::test]
 async fn missing_start_generation_is_adopted() {
     let directory = TempDir::new().expect("create temporary directory");