Skip to content

Bound the sbt warm-cache update so a stalled resolver can't hang the image build - #1742

Merged
Mikola Lysenko (mikolalysenko) merged 3 commits into
mainfrom
ci-janitor/sbt-warm-update-deadline
Oct 27, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 3 commits into
mainfrom
ci-janitor/sbt-warm-update-deadline

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

#1665: the warm-cache sbt update loop in tests/docker/Dockerfile.sbt has no deadline. On 2026-10-22 it hung 4 times between 05:53 and 09:47 UTC: nightly 39730006970 coverage-docker (sbt), and dispatch run 39745825228 attempts 1 and 2 (coverage-docker (sbt) and e2e-docker (sbt)). Each time it sat on Updating ProjectRef(... "warm-1-2-8-build")... until the 45-min job timeout. That cost about 180 runner-minutes, turned the nightly red, and left no diagnostic. It has not recurred since, but nothing bounds it. These jobs run nightly and on dispatch only (lean scope), so this does not block merges today.

Root cause

sbt 1.2.8 resolves its build definition through Ivy, which has no socket read timeout, so one stalled resolver connection blocks update forever. Every curl in the same Dockerfile has --retry. This step had neither a retry nor a deadline.

I also found a second bug: the loop sits inside a RUN set -eu && for ... done && ... AND-list, and POSIX doesn't apply errexit inside an AND-list. So a failed warm update was silently ignored, and the image shipped unwarmed (the 0.13.18 hosted group takes ~1380 s cold vs ~420 s warm). I reproduced this under dash: set -eu && for v in a b; do (false-sbt update); echo cleanup; done && echo next prints next and exits 0.

Fix

Each warm update now runs under timeout -k 30 600 with one retry. If both attempts fail, the build stops with exit 1 and logs warm sbt <v> update failed or ran past 600 s (attempt N). Without --foreground, timeout signals its whole process group, so the JVM started by the sbt script dies too and leaves no Ivy lock behind for the retry. Normally each update finishes well under a minute: the whole sbt image builds in about 3-4 min.

One file changed: Dockerfile only. No tests were added, removed or moved.

Also in this PR

Commit 9758d4f ports #1738 (pins unearth<0.19 for PDM 2.x). The PDM e2e leg is red on every PR until #1738 merges; once it does, this commit is a no-op.

Proof

There's no Docker daemon in this sandbox, so I ran the exact loop body under dash with a fake sbt on PATH (2 s deadline):

  • ok/ok → both versions warm, the next && step runs, rc=0
  • hang, then ok → attempt 1 is killed, the retry succeeds, the build continues, rc=0
  • fail/fail → two logged attempts, rc=1 (the build fails instead of shipping a cold image)
  • ok, then hang/hang on the second version → rc=1
  • after a hang, no orphaned grandchild (sleep) process survives the kill.

The real image build only runs nightly and on workflow_dispatch (coverage-docker / e2e-docker; sbt-compatibility.yml is disabled), so PR CI does not exercise this layer. A workflow_dispatch of CI on this branch would validate it before merge. Otherwise the next nightly does.

Where tests run

Unchanged.


Generated by Claude Code


Note

Low Risk
Changes affect CI/docker image build and test bootstrap pins only, not production runtime or auth/data paths.

Overview
sbt Docker image: Each cache-warming sbt update now runs under a 600s timeout with one retry, and the loop fails the build if both attempts fail (fixing silent failures under set -eu in an AND-list). This addresses Ivy resolver hangs that could block image builds until the job timeout (#1665).

PDM test harness: For PDM 2.x bootstrap venvs, the same unearth<0.19 pin is added in the Rust e2e VEX fixture helper, scripts/backtest-pdm.py, and mirrors the existing hishel<1 workaround—unearth 0.19 switched to httpx2 and breaks pdm lock with a PyPI client type error.

Reviewed by Cursor Bugbot for commit b9e018c. Configure here.


Generated by Claude Code

The warm `sbt update` loop had no deadline. On 2026-10-22 a stalled
resolver connection (Ivy has no read timeout) hung the layer 4 times
in 4 hours, each until the job's 45-minute timeout (#1665), with no
diagnostic.

Give each warm update a 600 s deadline (timeout -k 30) and one retry,
and fail the layer if both attempts fail. timeout(1) signals its whole
process group, so the JVM under the sbt script dies too and leaves no
Ivy lock for the retry.

This also fixes a silent failure: the loop sits inside a `set -eu &&
...` AND-list, where errexit does not apply, so a failed update was
ignored and the image shipped unwarmed. The new loop exits on failure
explicitly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011f9aePV6aRxWWuVgK81qyD
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Oct 27, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread tests/docker/Dockerfile.sbt Outdated
unearth 0.19.0 (published 2026-10-27 03:05 UTC) moved to httpx2.
PDM 2.x declares an unbounded unearth, so a fresh bootstrap now
resolves 0.19 and every `pdm lock` fails with "PDMPyPIClient is not
an instance or subtype of type (Client)". That turns the required
`e2e_vex_build` pdm leg red on every PR and in the merge queue,
before any socket-patch code runs.

Pin unearth<0.19 for PDM 2.x in the e2e bootstrap and in
scripts/backtest-pdm.py, which resolves the 0.18 line these releases
ran with until today.

Assisted-by: Claude Code:claude-opus-5-5
The `[ -n "$ok" ]` check was a no-op: errexit does not apply inside
the for loop, which is a non-final operand of the RUN's `&&` list, so
the loop still exited 0 after two failed attempts. Exit explicitly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011f9aePV6aRxWWuVgK81qyD
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

ci-ok was red on ff17d0e because e2e (ubuntu-latest, e2e_vex_build, pdm:: --ignored, 2.29.2) failed. This PR didn't cause it: unearth 0.19.0 (released 2026-10-27 03:05 UTC) breaks pdm lock on every PR and on main's nightly. I ported #1738's fix (unearth<0.19, commit 9758d4f) into this branch; it becomes a no-op once #1738 merges.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit b9e018c. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 27, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Ready for review at b9e018c.


Generated by Claude Code

Merged via the queue into main with commit 33dba71 Oct 27, 2026
56 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the ci-janitor/sbt-warm-update-deadline branch October 27, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants