Skip to content

Commit ebe88a2

Browse files
authored
ci: make the matrix retry budget fit inside the job cap (#245)
The retry budget was larger than the job cap on both engines, so the last attempt was always killed partway and the cell reported `cancelled`. Measured on master run 32643567648 (post-#243), the 5 JRuby cells: jruby-10.0 rails81 suite 713s, no hang, 1 attempt -> 12m02 pass jruby-10.0 rails71 suite 545s + 6m hang -> 15m t/o; attempt 2 clean at 543s -> 24m20 pass jruby-10.0 rails80 attempt 1 hung -> 15m t/o; attempt 2 killed at the 25m cap -> cancelled jruby-10.0 rails72 same -> cancelled jruby-head rails81 same -> cancelled 3 of 5 JRuby cells gate nothing. The cause is arithmetic, not the driver: max_attempts 3 x timeout_minutes 15 = 45 min against timeout-minutes 25. Attempt 3 could never start, and attempt 2 had only 25 - 15 - 0.7 = 9.3 min to finish a run that measures 9-12 min -- so whether a cell survived a hang came down to which gemfile it drew. MRI has the same shape, smaller: 3x3 = 9 against a cap of 8. - JRuby job cap 25 -> 31, so 1 + 15 + 15 fits. - max_attempts 3 -> 2 on both engines, since 3 was never reachable. A doubly-hung JRuby cell now costs 31 min instead of 25, but today's 25 min buys no verdict at all. Coverage, drivers and the full-ci/cron opt-in model are unchanged. Also adds the concurrency group Lint never had, so superseded PR pushes stop running the linter to completion.
1 parent f9c8ca6 commit ebe88a2

2 files changed

Lines changed: 24 additions & 9 deletions

File tree

.github/workflows/lint.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,12 @@ on:
1313
env:
1414
RUBY_YJIT_ENABLE: 1
1515

16+
# Test has had this since it was written; Lint never did, so every push to a
17+
# PR ran a fresh linter to completion alongside the superseded ones.
18+
concurrency:
19+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
20+
cancel-in-progress: true
21+
1622
jobs:
1723
lint:
1824
name: Ruby & YAML

.github/workflows/test.yml

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,10 @@ jobs:
108108
contains(github.event.pull_request.labels.*.name, 'full-ci')
109109
needs: [ functional-test ]
110110
runs-on: ubuntu-latest
111-
timeout-minutes: ${{ contains(matrix.ruby-version, 'jruby') && 25 || 8 }}
111+
# Must fit `max_attempts * timeout_minutes` below, plus ~1 min of setup,
112+
# or the last attempt gets killed mid-run and the cell reports `cancelled`
113+
# -- a dead gate. JRuby: 1 + 15 + 15 = 31. MRI: 1 + 3 + 3 = 7, under 8.
114+
timeout-minutes: ${{ contains(matrix.ruby-version, 'jruby') && 31 || 8 }}
112115
continue-on-error: ${{ matrix.experimental }}
113116
strategy:
114117
matrix:
@@ -151,17 +154,23 @@ jobs:
151154
ruby-cache-version: ${{ matrix.ruby-version }}-${{ matrix.gemfile }}-1
152155
cache-apt-packages: true
153156

154-
- name: Run tests (with 2 retries)
157+
- name: Run tests (with 1 retry)
155158
uses: nick-fields/retry@v4
156159
with:
157-
# 7 minutes stopped being enough for JRuby once the suite grew past
158-
# ~600 tests: every JRuby cell on master now burns all three attempts
159-
# on `Timeout of 420000ms hit` and dies at the job cap, so the cell
160-
# has stopped gating anything. Locally the full suite is 390s on
161-
# JRuby with vips and 434s with chunky_png -- the budget is the
162-
# problem, not the driver.
160+
# Measured on master run 32643567648: a clean JRuby attempt is
161+
# 545-713s depending on the gemfile, so 15 min is ~26% headroom over
162+
# the slowest. MRI is 128s against 3 min.
163163
timeout_minutes: ${{ contains(matrix.ruby-version, 'jruby') && 15 || 3 }}
164-
max_attempts: 3
164+
# Two, not three. A third attempt never fit inside the job cap on
165+
# either engine (JRuby 3x15=45 > 25; MRI 3x3=9 > 8), so it only ever
166+
# got killed partway and reported the cell as `cancelled`. Keeping
167+
# the retry budget inside the cap is what makes the cell a real gate.
168+
#
169+
# The retry exists solely for the intermittent JRuby teardown hang
170+
# (#244), which strands the process for minutes *after* the suite
171+
# prints `Finished in ...`. Once #244 is fixed, drop this to a single
172+
# attempt and the JRuby cap to 16 -- that halves the JRuby bill.
173+
max_attempts: 2
165174
command: bin/rake test
166175

167176
matrix-screenshot-driver:

0 commit comments

Comments
 (0)