Skip to content

fix(gateway): report isolation backend mismatches in sandbox status - #4738

Open
shiju-nv wants to merge 2 commits into
NVIDIA:mainfrom
shiju-nv:fix/4737-backend-mismatch-status/shiju-nv
Open

shiju-nv wants to merge 2 commits into
NVIDIA:mainfrom
shiju-nv:fix/4737-backend-mismatch-status/shiju-nv

Conversation

@shiju-nv

Copy link
Copy Markdown
Collaborator

Summary

When a sandbox's pinned isolation backend no longer matched the gateway's selection, the automatic paths only logged a warning. A sandbox waiting for an automatic restart stayed in Starting with a stale "restart scheduled" reason and was retried every 500 ms. A sandbox caught mid-start later failed with ImagePreparationTimedOut. This PR writes the mismatch to the sandbox status as IsolationBackendMismatch, with the existing "restore the gateway selection or create a new sandbox" message.

Related Issue

Closes #4737

Changes

  • crates/openshell-server/src/compute/mod.rs: when the restart scan finds a mismatch, restart_sandbox_runtime moves the sandbox to Error with IsolationBackendMismatch. It writes under the lock it already holds, with a compare-and-swap pinned to the version it read, so a concurrent stop or readiness update wins and the next scan decides again. The sandbox leaves the restart scan, so the warning is logged once.
  • crates/openshell-server/src/compute/provisioning_deadline.rs: when the preparation deadline expires and the sandbox's backend no longer matches the selection, claim_provisioning_timeout reports IsolationBackendMismatch instead of ImagePreparationTimedOut. Cleanup is unchanged. An admission repair expiry (the supervisor registered) keeps ProvisioningTimedOut and its diagnostic.
  • docs/how-it-works/sandboxes/overview.mdx: names the new reason and the recovery steps.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

When a sandbox's pinned isolation backend no longer matched the gateway's
selection, the automatic restart scan only logged a warning every 500 ms and
left the sandbox in Starting with a stale restart reason. A sandbox that never
started later failed with ImagePreparationTimedOut.

The restart scan now moves the sandbox to Error with IsolationBackendMismatch,
writing under the lock it holds with a compare-and-swap on the version it
read. A preparation timeout reports the same reason when the backend does not
match; an admission repair timeout keeps ProvisioningTimedOut. Startup and
lifecycle recovery still skip mismatched sandboxes, because they run on every
replica and a skipped sandbox may be running on the backend it was created
with.

Closes NVIDIA#4737

Signed-off-by: Shiju <shiju@nvidia.com>
@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

For preparation expiry with an isolation-backend mismatch, Ready=False changes from reason ImagePreparationTimedOut to IsolationBackendMismatch.

  • Impact: Automations, CLI JSON consumers, and SDK clients matching conditions[].reason == &quot;ImagePreparationTimedOut&quot; for preparation timeouts will miss backend-mismatch failures after upgrading.
  • Action: Keep ImagePreparationTimedOut for preparation expiry in v0.1.5 and expose the mismatch additively, or defer the reason change to a minor release with consumer migration guidance.
  • PR: fix(gateway): report isolation backend mismatches in sandbox status #4738
Review details
  • Execution: complete
  • Candidate: PR #4738 (a645ad7215f92b8df36a5d2bb187fe553e7e66c7)
  • Baseline: 7417f75384a02a12b30b736cc185bb08b814bd40 (7417f75384a02a12b30b736cc185bb08b814bd40)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

Comment thread docs/how-it-works/sandboxes/overview.mdx Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(gateway): a sandbox whose isolation backend selection changed stays in Starting and never reports the mismatch

2 participants