Skip to content

Commit dfcd595

Browse files
authored
Add bin/ci with gh signoff (#7)
* Add bin/ci with gh signoff, matching our other repos No cloud CI here, so bin/ci runs the checks locally and signs the commit off on success, the same shape as bcx and highrise. Bash rather than their Ruby CI class, deliberately: this repo exists because a working Ruby is precisely what you don't have yet, so its own CI must not need one. house-skills/bin/ci sets the in-house precedent. Only a full run signs off. Passing a platform or version narrows the matrix and explicitly declines to sign — a green tick covering one platform is worse than no tick. The lint pass costs nothing and catches the two mistakes that otherwise surface ten minutes into a Docker build: a syntax error in a definition, since ruby-build sources these, and an install_package URL with no #sha256, since ruby-build silently skips verification when the checksum is absent. Shellcheck runs when present and says so out loud when it isn't, rather than passing silently. It's gated at warning severity because the info tier here is all intentional. Fixing what it did flag: two declare-and-assign warnings, and a note on the one deliberate unquoted expansion, which holds two words and must split — an array would be tidier but expanding an empty one under set -u breaks on macOS's Bash 3.2. * Handle bin/ci --help before the lint pass, not after * Fix two silent false-greens in the lint pass, and stop under-scheduling builds Both lint findings are the same failure mode this repo keeps producing: a check that reports success because it inspected nothing. sort -V is GNU-only, and macOS — which we explicitly support — ships BSD sort. The failure doesn't trip set -e in bin/ci because the call sits in a command substitution inside a `for` list, so definitions() returned empty and both lint passes sailed over zero files and printed "CI passed". Reproduced with a stub sort that rejects -V: six definitions silently unchecked, exit 0. Plain sort here, since ordering is cosmetic when linting, plus a hard guard so an empty list can never be a pass whatever the cause — wrong directory, failed glob, broken sort. test/build had the same -V dependency but failed loudly instead, because a failing command substitution in a direct assignment *does* trip set -e. Still broken on macOS, just noisily, so it now detects -V support and falls back. The checksum lint only matched double-quoted URLs, so a definition written with single quotes — valid shell — would extract nothing and pass with no digest at all. Widening the pattern alone would have flagged 1.8.7's `curl '<savannah>'` calls, which fetch config.guess and have no checksum to carry, so extraction is now scoped to install_package lines and accepts either quote style. Verified both directions: single-quoted without a digest fails, with one passes, and the savannah URLs stay unflagged. Also fails when a definition yields no install_package URLs at all, since that means the extractor stopped matching rather than that the file is clean. Separately, JOBS was budgeting MAKE_JOBS cores per container. Measured against a real run, each container averages ~1.0 core: these builds are mostly single-threaded — miniruby bootstrapping and generating exts.mk, the dependency-serialized tail of make, gem install bundler — with brief parallel bursts, and 1.9.3 forces make -j1 in its own definition to dodge a race. The old default left ~75% of the machine idle and could queue the long pole (1.9.3, ~190s vs ~110s) behind short builds. Default to cores/2. * Inspect line-continued install_package calls too Scoping checksum extraction to lines starting with install_package missed a call spelled across continued lines: only the first line matched, so a URL on a continuation went uninspected. Latent rather than live — no definition uses continuations today — but it's the same vacuous-pass hole one level down, and the "found no URLs" guard wouldn't have caught it either, since another well-formed call in the same file keeps the count nonzero. Fold continuations before matching. Pure bash rather than sed or awk: the usual line-joining one-liners differ between BSD and GNU, and macOS portability is what this whole section is about. * Match package URLs regardless of quoting `install_package "x" https://…` is valid shell, and requiring quotes meant the extractor found nothing there — passing a definition with no digest at all rather than complaining. The "found no URLs" guard doesn't help when another quoted call in the same file keeps the count nonzero. This is the third input shape to slip past this check, after single quotes and line continuations, so stop enumerating quote styles: match the URL itself and let whitespace or either quote terminate it. One pattern now covers double-quoted, single-quoted and bare, and the `tr -d` goes away with it. Scoping to install_package lines stays, and stays load-bearing — over the whole file this pattern would flag 1.8.7's `curl '<savannah url>'` calls, which fetch config.guess and carry no checksum by nature. Verified all four forms with digests pass, each without one fails, and the savannah URLs stay unflagged. * Check every URL for a digest, not just ones that look like install_package Fifth report of the same defect, so stop fixing shapes. Matching invocations means matching shell syntax, and shell spells the same call unboundedly many ways — quoted, unquoted, line-continued, after `then` or `;` or `&&`, inside a function. Each form the pattern doesn't know is a download that goes uninspected, and it fails silently: reports success having looked at nothing. Four such forms turned up in a row, each fix addressing the shape rather than the class, so the next one was always waiting. Inverted: every URL in a definition must carry a digest. That fails closed. A download written in a syntax nobody anticipated is flagged rather than skipped, and the only way to exempt one is to say so explicitly — right friction, given an unverified download is what this exists to prevent. One exemption, GNU's git web view for config.guess/config.sub: a moving HEAD with no release tarball and no published digest, fetched only to teach ancient configure scripts about modern architectures, never linked into the built Ruby. Comment lines are skipped so a URL in prose isn't treated as a download. Verified against six forms with no digest — including `&&` chaining and a curl inside a function body, neither of which review had raised — all caught. Real definitions still pass, savannah URLs stay exempt, commented URLs ignored. * Exempt the two config files by name, not the whole savannah domain The comment claimed a config.guess/config.sub exemption but the pattern was a domain wildcard, so any other unverified download from that host would have been skipped — with the checksummed Ruby URL keeping the count nonzero, silently. Intent and implementation disagreed, and the implementation was the permissive one. Narrowing it first required fixing an extraction bug underneath. Excluding shell metacharacters from the URL pattern also truncated URLs that legitimately contain them: both savannah links were being cut at the first semicolon, down to `?p=config.git`, discarding the `f=config.guess` / `f=config.sub` that says which file is fetched. Nothing to match on. Metacharacters are now allowed inside the match and trimmed from the end instead, which is where they actually signal shell syntax rather than URL content. Verified: real definitions still pass, a different artifact from the same host is now caught, so is the same gitweb config.git path requesting another file, and unquoted URLs trailed by `;` or `&&` are trimmed and still checked. * Match the exempted config URLs as exact literals `*f=config.guess*` also matches `f=config.guess.backdoor`, so the exemption still skipped verification on unrelated downloads. Second time a wildcard in this exemption has been wider than intended — a domain glob before, a filename prefix now — so drop wildcards entirely and match the two URLs as literals. There is nothing left to widen: the definitions reference exactly these two fixed strings, anything else is checked. Quoted so `?` and `;` are matched literally rather than as glob and case-clause syntax. If the URLs ever change shape, this list has to be updated by hand, which is the intended cost of skipping verification on a download. Verified the two real URLs still pass and four bypass shapes are caught: config.guess.backdoor, config.subversion, an altered hb parameter, and the same path on another host. * Join continued lines with nothing, as shell does A backslash-newline is removed entirely in shell; it does not become whitespace. Joining with a space split tokens that bash keeps together, so `"https\` + `://host/pkg.tar.gz"` — one URL to bash, confirmed by sourcing it — arrived at the extractor as `https ://host/pkg.tar.gz` and matched nothing. Another download skipped silently. Contrived to write by hand, but the joiner was approximating shell semantics rather than following them, and that gap is what the check keeps getting caught by. Ordinary continuations already carry whitespace around the backslash, so nothing else changes. Verified: real definitions pass, the split-scheme URL is now caught, and continuations with digests still pass in all three forms. * Ignore URLs in shell comments Only full-line comments were stripped, so a reference link in a trailing comment was treated as a download and failed for lacking a digest — blocking CI and signoff over a URL that is never fetched. First false positive here rather than a false negative, and the more disruptive direction: it stops work rather than letting something through. Stripping comments has to leave `pkg.tar.gz#<sha256>` alone, since cutting at the first `#` would turn every checksummed URL into a failure. The `#` opening a comment is always preceded by whitespace or starts the line; the `#` before a digest never is, it follows the last character of the URL. Keying on that separates them without parsing quotes. Verified: a reference URL in a trailing comment is ignored, the digest on the same line still registers, a full-line comment URL is ignored, and a genuinely undigested download on a commented line is still caught. * Recognise comments that open straight after a shell operator `install_package ...;# note` is a comment to bash — verified, not assumed: `bash -c 'echo one;# echo two'` prints only "one". The previous rule required whitespace or line start before the `#`, so the trailing reference URL was read as a download and failed the lint despite the real package URL being checksummed. Same disruptive direction as the last one: it blocks CI over a URL nothing fetches. A `#` opens a comment when it starts a word, which includes straight after an operator that terminated the previous one. Digests stay safe because a digest's `#` never starts a word — it follows the last character of the URL, and no URL character is in the operator set. Verified against `;#`, `&&#` and a spaced `#`, all ignored, with the digests on those same lines still registering and a genuinely undigested download after a `;#` still caught. * Track quote state when stripping comments A regex over a line cannot know whether a `#` is inside quotes, so the previous rule fired on the literal hash in `"download #1"` and truncated the rest of the line — dropping a real install_package and its unverified URL. Successive regexes here were each wrong in a new way, in both directions: one blocks CI over a URL nothing fetches, the other skips a genuine unverified download. So scan the line and track quoting instead of guessing. A `#` opens a comment only when it starts a word and is unquoted. Digests are untouched because a digest's `#` follows the last character of the URL, so it never starts a word. Worth keeping this check despite the churn: ruby-build's verify_checksum returns success for an empty checksum, so a missing digest means the download is never verified and the build still passes. Nothing else catches that — not the build matrix, not ruby-build itself. Also splits comment_index's `local` in two. `local line="$1" n=${#line}` leaves n at 0 in bash; it only scanned correctly because dynamic scoping resolved ${#line} to the caller's identically-named variable. Renaming either would have silently disabled stripping. Verified by renaming the caller's variable and confirming comments are still stripped. * Fail safe on quoting forms the scanner doesn't model $'...' has its own escape rules, so an escaped apostrophe inside it looked like the end of a quoted string and the following `#` like a comment — truncating a real install_package off the line and passing its unverified URL. Modelling $'...', then $"...", then heredocs, is writing a shell lexer in bash, and every incomplete version of one is wrong in some new way. This is the fourth report in that sequence, so stop extending the model and bound it instead: the scanner handles '...' and "..." and, on encountering anything else, declines to strip that line at all. The failure then lands in the safe direction by construction. Comments on such a line get scanned for URLs, which can only produce a false positive — a loud complaint about a URL that needed no digest. Guessing risks the silent direction, dropping a real download and passing it unverified. No definition here uses these forms, so the cost is theoretical while the safety isn't. Verified the reported case now fails as it should, and ordinary quoted hashes and trailing comments still strip with no false positives. * Carry quote state between lines Quoting is not a per-line property — a string can contain a literal newline, and everything up to the closing quote is data. Scanning each line from a clean state made a `#` inside such a string look like a comment, truncating the rest of the line and taking any real download on it along too. Unlike the last few reports this isn't an unmodelled construct; it's ordinary `"..."` quoting that the scanner claimed to handle and got wrong by chunking the file into lines. Thread the state through instead. When a comment is found the line necessarily ends unquoted, since a comment can only open outside quotes. Verified the reported case now fails as it should, with no false positives on trailing comments, `;#` comments, or a quoted hash on the same line as a checksummed download. * Latch closed when quote state becomes unknowable Interaction between my own last two commits: the fallback for unmodelled quoting returned before updating the quote state that cross-line tracking had just started depending on, so state could carry forward desynced and drop a real download on a later line. Fixing the interaction directly would leave the same shape of bug available again. Instead, once a construct we can't track appears, stop stripping for the rest of that file. State can then never be wrong, only absent, and the failure is confined to over-reporting: URLs in comments get flagged loudly rather than real ones dropped quietly. The latch is per file, so one awkward definition can't affect the others. Verified the reported case is caught and a clean definition in the same run still strips its comments normally. * Skip heredoc bodies instead of scanning them as shell The comment listed heredocs as unmodelled but nothing acted on that, so their bodies were scanned as code: a `#` in one looked like a comment, a quote in one desynced quote state for the rest of the file. Not hypothetical — 1.8.7-p374 contains a heredoc, and further down has a URL inside a comment that depends on stripping still working. Latching off on `<<` would therefore have failed our own definition, so this needs real handling rather than another fail-closed shortcut: track the delimiter and skip the body. Anything in it is patch content, not a download. Openers are looked for in the code part of a line only, so one mentioned in a comment doesn't start swallowing lines. `<<<` is excluded as a herestring with no body, and arithmetic `<< 2` can't match since a delimiter must start with a letter or underscore. Verified: real definitions pass, a heredoc body containing a quote no longer hides a later undigested download, `<<-` with an indented terminator ends correctly, and neither `<<<` nor arithmetic shifts swallow what follows. * Identify downloads by shape, deleting the shell scanner Both reports are interactions with the heredoc handling added two commits ago: an opener detected inside a quoted string, and join_continuations merging a delimiter line because it runs before heredocs are recognised. Fixing them means more lexer, and more lexer has produced more findings every time — quoting, continuations, `;#`, ANSI-C strings, cross-line state, heredocs, now these. The premise was wrong. All that machinery existed to answer "is this URL an argument, a comment, or data", which needs a shell tokeniser. But every download here is a release tarball and every reference URL is an issue or a repo, so the question answers itself from the URL: check the ones ending in an archive extension, ignore the rest. Position stops mattering, and 113 lines of scanner go with it — comment_index, strip_comments, the unknown-state latch and the heredoc tracking, along with the bug surface that kept generating reports. The savannah config.guess/config.sub fetches no longer need an exemption list either; they aren't archives, so they fall outside the rule naturally. Cost, documented in place: a tarball URL written in a comment is flagged. Rare, self-explanatory, and loud rather than silent. No definition has one. Verified every undigested form from the whole review sequence is still caught — double and single quoted, unquoted, continued, split mid-scheme, after `if`, after `&&`, before `;`, after ANSI-C quoting, after a string spanning a newline, after a heredoc, and after a quoted `<<HIDE` — with real definitions passing and no false positives on their reference links. * Cut unquoted URLs at the first operator, not just trailing ones grep stops at whitespace, so `…tar.gz;echo done` arrived with `;echo` attached. Trimming only trailing punctuation left it there, the archive extension went unrecognised, and the download was skipped — silently, the direction that matters. Cutting at the first operator rather than the last is safe now that archive shape decides what gets checked. The savannah gitweb links were the reason for preserving mid-string semicolons, and they aren't archives, so nothing depends on that any more. Deleting the exemption made this fix a one-liner. Verified `;`, `&&`, `|` and `)` attached with no space are all caught, real definitions still pass, and every form from the earlier rounds still fails as it should. * Classify on the URL path, ignoring the query string `https://host/pkg.tar.gz?download=1` didn't end in an archive extension, so it was skipped and its missing digest went unreported — silently. Classification now uses the path with the query removed. Fixing that surfaced a hazard in the previous commit: trimming at the first shell operator cuts a legitimate `?a=1&b=2` query too, taking the digest after it and reporting a checksummed URL as unchecksummed. `&` is both a shell operator and query syntax, and no amount of trimming distinguishes them. So the two questions are now asked of different strings. Whether a digest is present is asked of the untrimmed URL, where `&` is harmless. What kind of URL it is gets asked of the trimmed, query-stripped path, where an attached command or query can't hide the extension. Neither answer depends on the other. Verified a query-with-ampersand URL carrying a digest passes, query-string and attached-command URLs without one fail, and every form from the earlier rounds still fails as it should.
1 parent fb80a75 commit dfcd595

3 files changed

Lines changed: 312 additions & 8 deletions

File tree

‎README.md‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,10 +153,37 @@ Whichever tool you use, the definitions are only found if ruby-build can actuall
153153
If a build fails in a way that looks nothing like the notes in this repo, check that the
154154
definition was actually picked up before debugging the compiler error.
155155

156+
### CI
157+
158+
There's no cloud CI here. Run `bin/ci` before merging, and it signs off the commit for you
159+
on success:
160+
161+
```bash
162+
bin/ci # lint + the full build matrix, then gh signoff
163+
bin/ci --lint # lint only, no Docker — seconds, good for a quick check
164+
bin/ci arch # lint + Arch only
165+
bin/ci arch 2.7.8 # lint + a single build
166+
```
167+
168+
Only a full run signs off. Anything narrower reports its results and explicitly declines to
169+
sign, because a green tick that covered one platform is worse than no tick.
170+
171+
The lint pass is cheap and catches the two mistakes that otherwise surface ten minutes into
172+
a Docker build: a syntax error in a definition (ruby-build sources these, so a stray quote
173+
is a build failure), and an `install_package` URL with no `#sha256` (ruby-build silently
174+
skips verification when the checksum is missing).
175+
176+
Sign-off needs the extension:
177+
178+
```bash
179+
gh extension install basecamp/gh-signoff
180+
```
181+
156182
### Testing
157183

158184
`test/build` builds definitions in throwaway Docker containers, so a clean-machine build
159-
is checked without touching your own toolchain.
185+
is checked without touching your own toolchain. `bin/ci` runs it for you; use it directly
186+
when you want a specific slice.
160187

161188
```bash
162189
test/build arch 1.8.7-p374 # one version on one platform

‎bin/ci‎

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
#!/usr/bin/env bash
2+
# Local CI. There is no cloud CI for this repo — run this, then `gh signoff`
3+
# marks the commit green so the PR can merge.
4+
#
5+
# Bash rather than the Ruby CI class the app repos use, deliberately: this repo
6+
# exists because a working Ruby is the thing you don't have yet. Its own CI must
7+
# not need one.
8+
#
9+
# bin/ci # lint + full build matrix, then sign off
10+
# bin/ci arch # lint + Arch only (no signoff — partial run)
11+
# bin/ci arch 2.7.8 # lint + one build (no signoff — partial run)
12+
# bin/ci --lint # lint only, no Docker (no signoff — partial run)
13+
#
14+
# Only a full run signs off. A partial run deliberately won't: signing off on a
15+
# subset is worse than not signing off at all.
16+
set -euo pipefail
17+
18+
cd "$(dirname "$0")/.."
19+
20+
# Before anything else, so --help doesn't sit through a lint pass first.
21+
case "${1:-}" in
22+
-h|--help) sed -n '2,15p' "$0" | sed 's/^#\{1,\} \{0,1\}//'; exit 0 ;;
23+
esac
24+
25+
BANNER=$'\033[1;32m'; TITLE=$'\033[1;35m'; SUBTITLE=$'\033[1;90m'
26+
ERROR=$'\033[1;31m'; SUCCESS=$'\033[1;32m'; RESET=$'\033[0m'
27+
28+
failures=()
29+
30+
# A URL runs to the next whitespace or quote. Shell metacharacters are captured here and
31+
# cut afterwards by trim_url, rather than excluded outright, so that an unquoted URL with
32+
# a command attached (`…tar.gz;echo`) is seen whole before being trimmed — grep alone
33+
# can't tell where the URL ends and the next command begins.
34+
#
35+
# Built here rather than inline because embedding a single quote inside a single-quoted
36+
# grep pattern is unreadable.
37+
SQ="'"
38+
URL_RE="https?://[^[:space:]\"$SQ\`]+"
39+
40+
# Cut an unquoted URL where the shell command continues. grep stops at whitespace, so
41+
# `…tar.gz;echo done` arrives with `;echo` still attached — and trimming only trailing
42+
# punctuation left it there, hiding the archive extension and skipping the download.
43+
#
44+
# Cutting at the *first* operator rather than the last is safe now that archive shape
45+
# decides what gets checked: the savannah gitweb links are the only URLs here with
46+
# mid-string semicolons, and they aren't archives either way.
47+
trim_url() {
48+
local u="$1"
49+
printf '%s' "${u%%[;\)\(\&\|\<\>]*}"
50+
}
51+
52+
echo "${BANNER}🚀 Local CI for ruby-dev${RESET}"
53+
54+
heading() { printf '\n%s%s%s\n' "$TITLE" "$1" "$RESET"; [ $# -gt 1 ] && printf '%s%s%s\n' "$SUBTITLE" "$2" "$RESET"; return 0; }
55+
pass() { printf '%s ✓ %s%s\n' "$SUCCESS" "$1" "$RESET"; }
56+
fail() { printf '%s ✗ %s%s\n' "$ERROR" "$1" "$RESET"; failures+=("$1"); }
57+
58+
# Plain `sort`, not `sort -V`: macOS ships BSD sort, which rejects -V. That failure would
59+
# not trip set -e here (it's a command substitution inside a `for` list), so the list would
60+
# come back empty and every check below would silently pass over nothing — a green run that
61+
# linted zero definitions. Ordering is cosmetic for linting, so portability wins.
62+
definitions() { ls -1 [0-9]* 2>/dev/null | sort; }
63+
64+
# And belt-and-braces: whatever the cause — wrong directory, failed glob, broken sort — an
65+
# empty list must be a hard error, never a quiet pass.
66+
require_definitions() {
67+
if [ -z "$(definitions)" ]; then
68+
printf '%serror: no definition files found in %s — refusing to report success%s\n' \
69+
"$ERROR" "$PWD" "$RESET" >&2
70+
exit 1
71+
fi
72+
}
73+
74+
# --- Shell syntax -----------------------------------------------------------
75+
# Definitions are sourced by ruby-build, so a syntax error in one is a build
76+
# failure several minutes into a Docker run. Catch it in milliseconds instead.
77+
lint_syntax() {
78+
heading "Syntax" "bash -n over scripts and definitions"
79+
local f
80+
for f in bin/ci test/build $(definitions); do
81+
if bash -n "$f" 2>/dev/null; then pass "$f"; else fail "$f has a syntax error"; bash -n "$f" || true; fi
82+
done
83+
}
84+
85+
# --- Checksums --------------------------------------------------------------
86+
# Fold backslash-continued lines into one, so an install_package spelled across several
87+
# lines is inspected whole. Matching raw lines would see only the first, and a missing
88+
# checksum on a continuation would pass unnoticed — with another well-formed call in the
89+
# file, even the "found nothing" guard below wouldn't fire.
90+
#
91+
# Pure bash rather than sed/awk: the usual line-joining one-liners differ between BSD and
92+
# GNU, and macOS portability is the entire point of this section.
93+
join_continuations() {
94+
local line acc=""
95+
while IFS= read -r line || [ -n "$line" ]; do
96+
if [ "${line%\\}" != "$line" ]; then
97+
# Join with nothing, matching shell: a backslash-newline is removed entirely, it
98+
# does not become whitespace. Inserting a space would split tokens shell keeps
99+
# together — `"https\` + `://host/x"` is one URL to bash but would arrive here as
100+
# `https ://host/x` and match nothing, silently skipping that download. Ordinary
101+
# continuations already carry their own whitespace around the backslash.
102+
acc="${acc}${line%\\}"
103+
else
104+
printf '%s%s\n' "$acc" "$line"
105+
acc=""
106+
fi
107+
done < "$1"
108+
[ -n "$acc" ] && printf '%s\n' "$acc"
109+
return 0
110+
}
111+
112+
# ruby-build only verifies a download when the URL carries a #checksum. Without one it
113+
# fetches and builds whatever it got, silently. A missing checksum is the kind of thing
114+
# that survives review, so assert it here.
115+
#
116+
# Every URL in the file is considered, wherever it appears, and the ones that look like
117+
# source archives must carry a digest. No attempt is made to work out which are arguments
118+
# and which are prose — that question is what made earlier versions of this wrong.
119+
#
120+
# The cost is that a tarball URL written in a comment gets flagged too. That is rare, it
121+
# says exactly what to do about it, and it errs loud rather than quiet. Reference links in
122+
# these definitions point at issues and repos, not tarballs, so it doesn't arise today.
123+
#
124+
# Not checked, deliberately: the config.guess and config.sub fetches in 1.8.7 and 1.9.3.
125+
# They come from GNU's git web view, which serves a moving HEAD with no release tarball
126+
# and no digest to cite. They aren't archives, so they fall outside this rule naturally
127+
# rather than needing an exemption list. The same is true of any future non-archive
128+
# download, which is the honest limitation of matching on shape.
129+
lint_checksums() {
130+
heading "Checksums" "every source archive carries a #sha256"
131+
local f raw url base path ok found
132+
for f in $(definitions); do
133+
ok=true
134+
found=0
135+
while IFS= read -r raw; do
136+
# Two questions, deliberately answered from different strings.
137+
#
138+
# Whether a digest is present is asked of the untrimmed URL, because `&` is both a
139+
# shell operator and ordinary query syntax. Trimming first would cut
140+
# `…tar.gz?a=1&b=2#<digest>` at the ampersand and report a missing digest that is
141+
# right there.
142+
#
143+
# What kind of URL it is gets asked of the trimmed path, so an attached command
144+
# (`…tar.gz;echo`) or a query string (`…tar.gz?download=1`) can't hide the extension.
145+
url=$(trim_url "$raw")
146+
base=${url%%#*} # up to the digest — what to report
147+
path=${base%%\?*} # and without the query — what to classify on
148+
# Only archives are checked, and that is what removes the need to understand the
149+
# shell around them. install_package fetches release tarballs, so a URL ending in
150+
# an archive extension is a download; anything else is a reference.
151+
#
152+
# The alternative was deciding by position — is this URL an argument, or inside a
153+
# comment, or inside a heredoc — which means tokenising shell, and that is where
154+
# every bug in this check came from. Quoting, continuations, `;#`, ANSI-C strings,
155+
# heredocs: each one handled, each one exposing the next. Matching on what the URL
156+
# *is* needs none of it.
157+
case "$path" in
158+
*.tar.gz|*.tar.bz2|*.tar.xz|*.tgz|*.tbz2|*.tar.Z|*.zip) ;;
159+
*) continue ;;
160+
esac
161+
found=$(( found + 1 ))
162+
# Unanchored, with a boundary, since the digest may be followed by a query
163+
# remnant or an attached command rather than ending the string.
164+
[[ $raw =~ \#[0-9a-f]{64}([^0-9a-f]|$) ]] || { fail "$f: no sha256 on $base"; ok=false; }
165+
done < <(join_continuations "$f" | grep -oE "$URL_RE")
166+
# Every definition fetches at least Ruby itself as a tarball. Finding none means the
167+
# extractor stopped matching, not that the file is clean — don't call that a pass.
168+
if [ "$found" -eq 0 ]; then
169+
fail "$f: no source archives found — checksum lint could not inspect this file"
170+
ok=false
171+
fi
172+
# Explicit if, not `$ok && pass`: that leaves the loop's exit status at 1 when
173+
# the *last* definition fails, and set -e then kills the run before the later
174+
# checks and the summary — failures reported, no verdict.
175+
if $ok; then pass "$f"; fi
176+
done
177+
}
178+
179+
# --- Shellcheck -------------------------------------------------------------
180+
# Optional: not everywhere, and not worth blocking a build matrix over. Report
181+
# the skip out loud rather than passing silently, so nobody reads a green run as
182+
# "shellcheck is clean" when it never ran.
183+
lint_shellcheck() {
184+
heading "Shellcheck" "optional static analysis"
185+
if ! command -v shellcheck >/dev/null; then
186+
printf '%s – skipped: shellcheck not installed%s\n' "$SUBTITLE" "$RESET"
187+
return 0
188+
fi
189+
# Scripts only. Definitions are ruby-build fragments, not standalone scripts —
190+
# they call install_package et al from their sourcing shell, so shellcheck reads
191+
# every one of those as an unknown command.
192+
#
193+
# --severity=warning on purpose: the info tier here is all intentional (ls over
194+
# find on version-numbered filenames, deliberate word splitting). Gating on info
195+
# would mean either noisy failures or a scattering of disable comments, and both
196+
# train people to ignore the step.
197+
if shellcheck -s bash --severity=warning bin/ci test/build; then
198+
pass "scripts"
199+
else
200+
fail "shellcheck"
201+
fi
202+
}
203+
204+
# --- Build matrix -----------------------------------------------------------
205+
build_matrix() {
206+
heading "Builds" "test/build ${*:-all}"
207+
if ! docker info >/dev/null 2>&1; then
208+
fail "Docker isn't available — the build matrix can't run"
209+
return 0
210+
fi
211+
if test/build "${@:-all}"; then pass "build matrix"; else fail "build matrix"; fi
212+
}
213+
214+
# --- Signoff ----------------------------------------------------------------
215+
signoff() {
216+
heading "📋 Signoff" "gh signoff"
217+
if ! command -v gh >/dev/null || ! gh extension list 2>/dev/null | grep -q gh-signoff; then
218+
printf '%s – skipped: gh signoff not installed (gh extension install basecamp/gh-signoff)%s\n' \
219+
"$SUBTITLE" "$RESET"
220+
return 0
221+
fi
222+
gh signoff
223+
}
224+
225+
started=$SECONDS
226+
227+
require_definitions
228+
lint_syntax
229+
lint_checksums
230+
lint_shellcheck
231+
232+
partial=false
233+
case "${1:-}" in
234+
--lint) partial=true ;;
235+
"") build_matrix ;;
236+
*) partial=true; build_matrix "$@" ;;
237+
esac
238+
239+
elapsed=$(( SECONDS - started ))
240+
241+
if [ ${#failures[@]} -eq 0 ]; then
242+
printf '\n%s✅ CI passed in %ds%s\n' "$SUCCESS" "$elapsed" "$RESET"
243+
if $partial; then
244+
printf '%s📋 Partial run — not signing off. Run bin/ci with no arguments to sign off.%s\n' \
245+
"$SUBTITLE" "$RESET"
246+
else
247+
signoff
248+
fi
249+
else
250+
printf '\n%s❌ CI failed in %ds%s\n' "$ERROR" "$elapsed" "$RESET"
251+
for f in "${failures[@]}"; do printf '%s • %s%s\n' "$ERROR" "$f" "$RESET"; done
252+
printf '%s📋 No sign-off. Fix the issues and try again.%s\n' "$SUBTITLE" "$RESET"
253+
exit 1
254+
fi

‎test/build‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,31 @@ set -euo pipefail
44
cd "$(dirname "$0")/.."
55

66
PLATFORMS="ubuntu-noble arch"
7-
VERSIONS=$(ls -1 [0-9]* 2>/dev/null | sort -rV)
7+
8+
# macOS ships BSD sort, which rejects -V. Fall back to plain sort there rather than dying
9+
# on an unsupported flag — the order is cosmetic, it only sets which builds start first.
10+
if printf '1\n' | sort -V >/dev/null 2>&1; then
11+
VERSIONS=$(ls -1 [0-9]* 2>/dev/null | sort -rV)
12+
else
13+
VERSIONS=$(ls -1 [0-9]* 2>/dev/null | sort -r)
14+
fi
15+
[ -n "$VERSIONS" ] || { echo "error: no definition files found in $PWD" >&2; exit 1; }
816

917
# Builds run concurrently. JOBS caps how many containers are in flight; MAKE_JOBS caps
10-
# make parallelism inside each one. The product is what actually hits the CPU, so the
11-
# defaults aim for a mild oversubscribe rather than JOBS x nproc meltdown.
18+
# make parallelism inside each one.
19+
#
20+
# JOBS is deliberately generous relative to core count. Measuring a real run, each
21+
# container averages ~1.0 core, not the MAKE_JOBS it's allowed: these old Ruby builds
22+
# spend most of their wall time single-threaded — miniruby bootstrapping and generating
23+
# exts.mk, the dependency-serialized tail of make, `gem install bundler` — with only brief
24+
# bursts of parallel compilation. 1.9.3 is stricter still, forcing make -j1 in its own
25+
# definition to dodge a parallel-make race. Budgeting MAKE_JOBS cores per container left
26+
# the machine ~75% idle, and the long pole (1.9.3, ~190s vs ~110s for the rest) could sit
27+
# queued behind short builds instead of starting immediately.
1228
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
13-
JOBS=${JOBS:-$(( CORES / 4 ))}
29+
JOBS=${JOBS:-$(( CORES / 2 ))}
1430
(( JOBS < 1 )) && JOBS=1
15-
(( JOBS > 12 )) && JOBS=12
31+
(( JOBS > 16 )) && JOBS=16
1632
MAKE_JOBS=${MAKE_JOBS:-4}
1733

1834
# Use multi-platform builder if available (faster cross-arch builds)
@@ -58,7 +74,8 @@ build_image() {
5874
local platform=$1
5975
local dockerfile="test/${platform}.dockerfile"
6076
local image="ruby-build-test:${platform}"
61-
local target_platform=$(platform_for "$platform")
77+
local target_platform
78+
target_platform=$(platform_for "$platform")
6279

6380
local build_args=(-f "$dockerfile" -t "$image" --load .)
6481

@@ -109,7 +126,8 @@ test_ruby() {
109126
local platform=$1
110127
local version=$2
111128
local image="ruby-build-test:${platform}"
112-
local target_platform=$(platform_for "$platform")
129+
local target_platform
130+
target_platform=$(platform_for "$platform")
113131
local platform_flag=""
114132

115133
[[ -n "$target_platform" ]] && platform_flag="--platform $target_platform"
@@ -140,6 +158,11 @@ test_ruby() {
140158
"
141159

142160
local started=$SECONDS output elapsed
161+
# $platform_flag is deliberately unquoted: it holds two words ("--platform
162+
# linux/amd64") and must split into two arguments. An array would be the tidier
163+
# idiom, but expanding an empty one under `set -u` is an error on macOS's Bash
164+
# 3.2, and this script has to keep working there.
165+
# shellcheck disable=SC2086
143166
if output=$(docker run --rm $platform_flag -e MAKE_OPTS="-j${MAKE_JOBS}" \
144167
"$image" bash -c "$build_script" 2>&1); then
145168
elapsed=$(( SECONDS - started ))

0 commit comments

Comments
 (0)