Repository navigation
Bound the sbt warm-cache update so a stalled resolver can't hang the image build - #1742
Merged
Mikola Lysenko (mikolalysenko) merged 3 commits intoOct 27, 2026
Merged
Conversation
The warm `sbt update` loop had no deadline. On 2026-10-22 a stalled resolver connection (Ivy has no read timeout) hung the layer 4 times in 4 hours, each until the job's 45-minute timeout (#1665), with no diagnostic. Give each warm update a 600 s deadline (timeout -k 30) and one retry, and fail the layer if both attempts fail. timeout(1) signals its whole process group, so the JVM under the sbt script dies too and leaves no Ivy lock for the retry. This also fixes a silent failure: the loop sits inside a `set -eu && ...` AND-list, where errexit does not apply, so a failed update was ignored and the image shipped unwarmed. The new loop exits on failure explicitly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011f9aePV6aRxWWuVgK81qyD
Collaborator
Author
|
bugbot run Generated by Claude Code |
unearth 0.19.0 (published 2026-10-27 03:05 UTC) moved to httpx2. PDM 2.x declares an unbounded unearth, so a fresh bootstrap now resolves 0.19 and every `pdm lock` fails with "PDMPyPIClient is not an instance or subtype of type (Client)". That turns the required `e2e_vex_build` pdm leg red on every PR and in the merge queue, before any socket-patch code runs. Pin unearth<0.19 for PDM 2.x in the e2e bootstrap and in scripts/backtest-pdm.py, which resolves the 0.18 line these releases ran with until today. Assisted-by: Claude Code:claude-opus-5-5
The `[ -n "$ok" ]` check was a no-op: errexit does not apply inside the for loop, which is a non-final operand of the RUN's `&&` list, so the loop still exited 0 after two failed attempts. Exit explicitly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011f9aePV6aRxWWuVgK81qyD
Collaborator
Author
|
Generated by Claude Code |
Tanmay Singla (Tanmay182003)
approved these changes
Oct 27, 2026
Collaborator
Author
|
bugbot run Generated by Claude Code |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit b9e018c. Configure here.
Collaborator
Author
|
Ready for review at
Generated by Claude Code |
Mikola Lysenko (mikolalysenko)
deleted the
ci-janitor/sbt-warm-update-deadline
branch
October 27, 2026 14:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#1665: the warm-cache
sbt updateloop intests/docker/Dockerfile.sbthas no deadline. On 2026-10-22 it hung 4 times between 05:53 and 09:47 UTC: nightly 39730006970coverage-docker (sbt), and dispatch run 39745825228 attempts 1 and 2 (coverage-docker (sbt)ande2e-docker (sbt)). Each time it sat onUpdating ProjectRef(... "warm-1-2-8-build")...until the 45-min job timeout. That cost about 180 runner-minutes, turned the nightly red, and left no diagnostic. It has not recurred since, but nothing bounds it. These jobs run nightly and on dispatch only (lean scope), so this does not block merges today.Root cause
sbt 1.2.8 resolves its build definition through Ivy, which has no socket read timeout, so one stalled resolver connection blocks
updateforever. Everycurlin the same Dockerfile has--retry. This step had neither a retry nor a deadline.I also found a second bug: the loop sits inside a
RUN set -eu && for ... done && ...AND-list, and POSIX doesn't apply errexit inside an AND-list. So a failed warmupdatewas silently ignored, and the image shipped unwarmed (the 0.13.18 hosted group takes ~1380 s cold vs ~420 s warm). I reproduced this under dash:set -eu && for v in a b; do (false-sbt update); echo cleanup; done && echo nextprintsnextand exits 0.Fix
Each warm update now runs under
timeout -k 30 600with one retry. If both attempts fail, the build stops withexit 1and logswarm sbt <v> update failed or ran past 600 s (attempt N). Without--foreground,timeoutsignals its whole process group, so the JVM started by the sbt script dies too and leaves no Ivy lock behind for the retry. Normally each update finishes well under a minute: the whole sbt image builds in about 3-4 min.One file changed: Dockerfile only. No tests were added, removed or moved.
Also in this PR
Commit 9758d4f ports #1738 (pins
unearth<0.19for PDM 2.x). The PDM e2e leg is red on every PR until #1738 merges; once it does, this commit is a no-op.Proof
There's no Docker daemon in this sandbox, so I ran the exact loop body under
dashwith a fakesbton PATH (2 s deadline):&&step runs, rc=0sleep) process survives the kill.The real image build only runs nightly and on
workflow_dispatch(coverage-docker/e2e-docker;sbt-compatibility.ymlis disabled), so PR CI does not exercise this layer. Aworkflow_dispatchof CI on this branch would validate it before merge. Otherwise the next nightly does.Where tests run
Unchanged.
Generated by Claude Code
Note
Low Risk
Changes affect CI/docker image build and test bootstrap pins only, not production runtime or auth/data paths.
Overview
sbt Docker image: Each cache-warming
sbt updatenow runs under a 600stimeoutwith one retry, and the loop fails the build if both attempts fail (fixing silent failures underset -euin an AND-list). This addresses Ivy resolver hangs that could block image builds until the job timeout (#1665).PDM test harness: For PDM 2.x bootstrap venvs, the same
unearth<0.19pin is added in the Rust e2e VEX fixture helper,scripts/backtest-pdm.py, and mirrors the existinghishel<1workaround—unearth 0.19 switched to httpx2 and breakspdm lockwith a PyPI client type error.Reviewed by Cursor Bugbot for commit b9e018c. Configure here.
Generated by Claude Code