Skip to content

docs(sandboxes): describe failure modes and recovery behavior - #4274

Draft
drew wants to merge 3 commits into
mainfrom
docs/failure-and-recovery/drew
Draft

drew wants to merge 3 commits into
mainfrom
docs/failure-and-recovery/drew

Conversation

@drew

@drew drew commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a draft Failure and Recovery page that models what happens to sandbox workloads when the gateway goes down or the supervisor or its link to the sandbox goes down, how a sandbox responds to policy violations, and how operators configure that behavior. The page is forward looking: the gateway outage limit, violation responses, and failure_handling configuration describe intended behavior that is not implemented yet.

Related Issue

None yet.

Changes

  • Add docs/how-it-works/sandboxes/failure-and-recovery.mdx covering:
    • Gateway outages: the agent keeps running on its installed policy and stops after 2 minutes without gateway contact, recoverable with openshell sandbox start.
    • Supervisor crash or supervisor–sandbox link loss: pause, 30 second recovery window, then shutdown.
    • Policy violation response levels: block, quarantine, pause, stop.
    • Configuration: gateway defaults in [openshell.gateway.failure_handling], with sandbox policy overrides that can only tighten them.
  • Add the page to the Sandboxes navigation in docs/index.yml.

Testing

  • mise run docs:build:strict passes (0 errors)
  • Pre-commit checks pass
  • Unit tests added/updated (not applicable)
  • E2E tests added/updated (not applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Add a Failure and Recovery page covering gateway outages, supervisor and
supervisor-sandbox link loss, responses to policy violations, and the
gateway and sandbox configuration that controls them.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 7, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

Describe three failure modes (gateway unreachable, supervisor
unreachable, sandbox process or compute runtime failure), drop the
policy-violation responses and per-sandbox failure_handling override,
and document which recovery limits are configurable.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@github-actions

Copy link
Copy Markdown

This pull request has had no activity for 14 days and is now marked stale.

@github-actions github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Oct 21, 2026
@FrostGod
FrostGod self-requested a review October 28, 2026 06:23
#4993)

Correct the failure and recovery page for the shared supervisor pool:
a pool member takes over a sandbox whose supervisor crashed, within a
60 second window set by [openshell.gateway.supervisor_pool]
recovery_timeout_secs. Note that losing one gateway replica is not an
outage, and document the pool table in the gateway configuration page.

Signed-off-by: divesh <dgude@nvidia.com>
@github-actions github-actions Bot removed the state:stale Inactive item at risk of automatic closure. label Oct 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants