Skip to content

perf(docker): slow steady-state supervisor healthcheck - #4802

Open
lunarwhite wants to merge 1 commit into
NVIDIA:mainfrom
lunarwhite:perf/4729-docker-supervisor-health-interval/lunarwhite
Open

lunarwhite wants to merge 1 commit into
NVIDIA:mainfrom
lunarwhite:perf/4729-docker-supervisor-health-interval/lunarwhite

Conversation

@lunarwhite

Copy link
Copy Markdown
Contributor

Summary

Keep the Docker supervisor healthcheck's 250 ms start_interval for fast readiness detection and raise the steady-state interval from 250 ms to 30 s, as proposed in the triage.

Related Issue

Closes #4729

Changes

  • Give the start interval its own constant (250 ms) and set SUPERVISOR_HEALTH_INTERVAL_NS to 30 s. Timeout, retries, and the 60 s start period are unchanged.
  • Move the HealthConfig into docker_supervisor_healthcheck() and pin all three timing values in a unit test.
  • Note: Docker uses the start interval only within the 60 s start period, so a supervisor that becomes ready later (for example after a policy repair) is detected up to 30 s later. Public Ready comes from the supervisor session and is unaffected.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Additionally, one idle sandbox, Docker 29.7.2 on WSL2, gateway built from main vs this branch:

main This PR
Health probes per minute 185 2
Supervisor container CPU 16.3% 0.4%
dockerd CPU 4.7% 0.3%
sandbox start (3 runs) 1.7–2.8 s 1.8–2.7 s
Policy repaired 91 s into startup: to Ready / to Docker healthy 1.4 s / 1.2 s 1.0 s / 29.1 s

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(driver-docker): 250 ms supervisor healthcheck costs ~12% of a CPU core per idle sandbox

1 participant