Skip to content

ci: bound the validate job so a stalled step fails fast - #106

Merged
dubadub merged 1 commit into
mainfrom
ci/validate-job-timeout
Aug 20, 2026
Merged

ci: bound the validate job so a stalled step fails fast#106
dubadub merged 1 commit into
mainfrom
ci/validate-job-timeout

Conversation

@dubadub

@dubadub dubadub commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

#104's first run was killed at 6h0m14s — GitHub's default job limit — without ever reaching compile or test. It had hung in apt-get update: all four azure.archive.ubuntu.com entries were Ign'd, the fallback to archive.ubuntu.com stalled mid-fetch at 21:14:26, and the job sat there in silence until cancellation at 03:11:58.

A green validate run takes ~10 minutes. The job had no timeout-minutes, so it inherited the 6h default.

Changes

  • 45m job timeout. Generous headroom over a cold-cache run (a warm run is ~10m; a cold npm/cargo cache roughly triples that), while cutting the worst case from 6h to 45m.
  • 10m timeout on the apt step. This is the step actually observed to hang, and it normally completes in seconds — even the slow run spent only ~2m negotiating mirrors before stalling.

Notes

This makes stalls visible fast; it does not make them stop happening. A flaky mirror will still fail the run — it will just do so in 10 minutes instead of occupying a runner for six hours. If these become frequent, the next step would be retry logic or dropping the apt dependency, but that's more invasive than warranted for something seen once.

Verified the workflow still parses and both timeouts land where intended (yaml.safe_load → job timeout-minutes: 45, apt step timeout-minutes: 10, all 15 steps intact). The 6h-hang path itself can't be reproduced on demand — it depends on Ubuntu mirror health.

Out of scope, worth a look separately: the build job in release-please.yml has step-level timeouts (25m, 60m — the SwiftPM backstops) but likewise no job-level timeout-minutes. I left it alone because that job legitimately runs long across the platform matrix and I'd be guessing at a safe ceiling.

PR #104's first run was killed at 6h0m14s, GitHub's default job limit. It had
hung in `apt-get update`: all four azure.archive.ubuntu.com entries were
`Ign`'d, the fallback to archive.ubuntu.com stalled mid-fetch, and the job sat
there for six hours before reporting anything. A green run takes ~10 minutes.

Adds a 45m job timeout (generous headroom over a cold-cache run) and a 10m
timeout on the apt step itself, which is the step actually observed to hang.
A stall now surfaces in minutes instead of occupying a runner for six hours.
@dubadub
dubadub merged commit 97be051 into main Aug 20, 2026
2 checks passed
@dubadub
dubadub deleted the ci/validate-job-timeout branch August 20, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant