Repository navigation
Expand file tree
/
Copy pathnextbsd-ci-testing-strategy-plan.html
More file actions
349 lines (343 loc) · 68.7 KB
/
Copy pathnextbsd-ci-testing-strategy-plan.html
File metadata and controls
349 lines (343 loc) · 68.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>NextBSD CI: testing coupled cross-repo changes — strategy options</title>
<style>
:root {
--fg: #1a1a1a;
--fg-muted: #555;
--bg: #fafaf7;
--accent: #b8472a;
--accent-soft: #f3e7df;
--border: #d8d4c8;
--code-bg: #f0ece2;
--table-stripe: #f4efe5;
--warn: #8a5a00;
--good: #2d6f3b;
--bad: #a23030;
}
* { box-sizing: border-box; }
html { -webkit-text-size-adjust: 100%; }
body {
font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", Helvetica, sans-serif;
color: var(--fg);
background: var(--bg);
line-height: 1.55;
margin: 0;
padding: 0;
}
.wrap {
max-width: 880px;
margin: 0 auto;
padding: 48px 32px 96px;
}
h1 {
font-size: 2.1rem;
line-height: 1.2;
margin: 0 0 8px;
letter-spacing: -0.01em;
}
h2 {
font-size: 1.45rem;
margin: 56px 0 12px;
padding-top: 18px;
border-top: 2px solid var(--border);
letter-spacing: -0.005em;
}
h3 {
font-size: 1.15rem;
margin: 32px 0 10px;
color: var(--accent);
}
h4 {
font-size: 1rem;
margin: 24px 0 6px;
color: var(--fg);
}
p { margin: 0 0 14px; }
ul, ol { margin: 0 0 14px 22px; padding: 0; }
li { margin: 0 0 6px; }
code {
font-family: "SF Mono", Menlo, Consolas, monospace;
font-size: 0.9em;
background: var(--code-bg);
padding: 1px 5px;
border-radius: 3px;
}
pre {
font-family: "SF Mono", Menlo, Consolas, monospace;
font-size: 0.86em;
line-height: 1.5;
background: var(--code-bg);
border: 1px solid var(--border);
border-radius: 4px;
padding: 14px 16px;
overflow-x: auto;
margin: 0 0 14px;
}
pre code {
background: none;
padding: 0;
border-radius: 0;
font-size: inherit;
}
.lede {
font-size: 1rem;
color: var(--fg-muted);
margin: 0 0 28px;
}
.meta {
font-size: 0.85rem;
color: var(--fg-muted);
margin: 0 0 24px;
}
table {
border-collapse: collapse;
width: 100%;
margin: 12px 0 22px;
font-size: 0.94rem;
}
th, td {
text-align: left;
padding: 10px 12px;
border: 1px solid var(--border);
vertical-align: top;
}
th {
background: var(--accent-soft);
font-weight: 600;
}
tr:nth-child(even) td { background: var(--table-stripe); }
.callout {
border-left: 3px solid var(--accent);
background: var(--accent-soft);
padding: 14px 18px;
margin: 18px 0 22px;
border-radius: 0 4px 4px 0;
}
.callout p:last-child { margin-bottom: 0; }
.callout-warn {
border-left-color: var(--warn);
background: #fbf3df;
}
.callout-good {
border-left-color: var(--good);
background: #e8f1e3;
}
.pill {
display: inline-block;
font-size: 0.78rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.04em;
padding: 2px 8px;
border-radius: 10px;
margin-right: 8px;
}
.pill-good { background: #d6ead0; color: var(--good); }
.pill-warn { background: #f4dfbf; color: var(--warn); }
.pill-bad { background: #f0c8c8; color: var(--bad); }
.pill-neutral { background: #ddd; color: #333; }
.toc {
background: white;
border: 1px solid var(--border);
border-radius: 4px;
padding: 18px 24px 14px 36px;
margin: 0 0 36px;
font-size: 0.95rem;
}
.toc h2 {
margin: 0 0 8px;
padding-top: 0;
border-top: none;
font-size: 1rem;
text-transform: uppercase;
letter-spacing: 0.04em;
color: var(--fg-muted);
margin-left: -14px;
}
.toc ol { margin: 0 0 0 6px; }
.toc li { margin-bottom: 4px; }
.toc a { color: var(--fg); text-decoration: none; }
.toc a:hover { text-decoration: underline; }
.footnote {
font-size: 0.85rem;
color: var(--fg-muted);
border-top: 1px solid var(--border);
margin-top: 48px;
padding-top: 16px;
}
.pros-cons {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 14px;
margin: 14px 0 18px;
}
.pros, .cons {
border: 1px solid var(--border);
border-radius: 4px;
padding: 12px 16px;
background: white;
}
.pros { border-left: 3px solid var(--good); }
.cons { border-left: 3px solid var(--bad); }
.pros h4, .cons h4 { margin-top: 0; }
.verdict {
font-size: 1.1rem;
font-weight: 600;
margin: 8px 0 14px;
}
.verdict-go { color: var(--good); }
.verdict-warn { color: var(--warn); }
.verdict-no { color: var(--bad); }
.twocol {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 14px;
margin: 14px 0 18px;
}
.twocol > div {
border: 1px solid var(--border);
border-radius: 4px;
padding: 12px 16px;
background: white;
}
.twocol h4 { margin-top: 0; }
@media (max-width: 600px) {
.pros-cons, .twocol { grid-template-columns: 1fr; }
.wrap { padding: 24px 18px 64px; }
}
</style>
</head>
<body>
<div class="wrap">
<h1>NextBSD CI: testing coupled cross-repo changes — strategy options</h1>
<p class="lede">How to boot-test <strong>coupled changes that span two repos before either merges</strong> — the motivating case being a <code>nextbsd-kernel</code> PR that raises FreeBSD’s dynamic syscall-slot limit plus a <code>nextbsd</code> PR that drops <code>mach.ko</code>’s syscall multiplexor in favour of individually-numbered Mach trap slots. Five CI strategies compared with realistic <strong>time-to-test</strong> for each, in today’s <code>vmactions</code> VM world and the near-future native-cross world.</p>
<p class="meta">Companion to the <a href="nextbsd-linux-crossbuild-image-plan.html">native-Linux cross-build plan</a> (which the after-state assumes). Synthesized 2026-06-02 from an 8-agent analysis: verified the current 5-repo CI topology + 5 testing gaps, grounded the syscall/multiplexor change in the real <code>src/mach_kmod</code> code, and modelled time-to-test per stage. Repos: <code>nextbsd-kernel-toolchain</code> / <code>-kernel</code> / <code>-kernel-modules</code> / <code>-freebsd-compat</code> / <code>nextbsd</code>.</p>
<div class="callout callout-good">
<p><span class="pill pill-good">Status — June 2026</span> <strong>What shipped (and how it diverged from the recommendation below).</strong></p>
<ul>
<li><strong>The continuous-release backbone is live</strong> — Option A's <em>mechanism</em>, built as a <strong>rolling <code>continuous</code> release</strong> per producer (kernel / modules / compat) rather than the <code>versions.lock</code> immutable-pin variant. <code>nextbsd</code>'s three ingests now <code>gh release download continuous</code> instead of the <code>gh run list --branch main --status success --limit 1</code> scrape, which closes the broken-/half-cascaded-main ingest hole (<strong>GAP#6</strong>): only a green <code>main</code> build ever updates <code>continuous</code>. (No per-SHA <code>versions.lock</code> pin yet — the rolling tag is the pointer.)</li>
<li><strong>Every repo now validates pull requests</strong> (toolchain build-only, <code>push:false</code>) — closing <strong>GAP#1</strong>'s “kernel has no <code>pull_request</code> trigger”. Invariant: a PR never updates <code>continuous</code> anywhere (publish/release require <code>refs/heads/main</code>).</li>
<li><strong>A kernel-only PR boot smoke test shipped</strong> — a <em>lighter</em> realization of Step 1's “boot/sysctl smoke on <code>nextbsd-kernel</code>”: instead of rebuilding a full ISO via override inputs, it injects the PR's freshly-built kernel into the latest <code>continuous</code> NextBSD ISO (FreeBSD-VM <code>mdconfig</code> mount of the UFS root) and boots it under qemu with nextbsd's own <code>boot-test.sh</code>. PR-only; does not gate publish.</li>
<li><strong>Scope divergence from this doc's premise.</strong> The coupled <em>syscall-limit × mux-drop</em> scenario that motivates B/C/D is <strong>not</strong> a current concern: NextBSD pins <strong>production <code>releng/15.0</code></strong>, so kernel KBI is stable and the shipped <code>mach.ko</code> + modules keep loading against a freshly-built kernel. The live question became “does my kernel patch still boot?” — answered by the single-repo smoke test above, not cross-repo coupled pre-merge testing.</li>
<li><strong>Not built / deferred:</strong> B's artifact-override <code>*_run_id</code>/<code>*_ref</code> inputs, the <code>versions.lock</code> immutable pin + pin-bump bot, the <code>/integrate</code> bot, the monorepo (C1/C2), and the D meta-workflow. Revisit if/when a genuinely coupled cross-repo change lands. The options analysis below is preserved as the decision record.</li>
</ul>
</div>
<div class="toc">
<h2>Contents</h2>
<ol>
<li><a href="#the-scenario-this-must-serve">The scenario this must serve</a></li>
<li><a href="#the-five-options-at-a-glance">The five options at a glance</a></li>
<li><a href="#timetotest-perstage-cost-model">Time-to-test: per-stage cost model</a></li>
<li><a href="#timetotest-endtoend-per-case">Time-to-test: end-to-end per case</a></li>
<li><a href="#the-options-in-detail">The options in detail</a></li>
<li><a href="#ratings-matrix">Ratings matrix</a></li>
<li><a href="#recommendation">Recommendation</a></li>
</ol>
</div>
<h2 id="the-scenario-this-must-serve">The scenario this must serve</h2>
<div class="callout"><p>Two changes must be boot-tested together before either merges. PR #1 in nextbsd-redux/nextbsd-kernel adds a patch (to patches/series) that widens FreeBSD's fixed band of dynamically-claimable syscall slots (today only the ~10 lkmnosys slots 210-219 exist, and NextBSD's mach.ko already burns all of them). PR #2 in nextbsd-redux/nextbsd drops the multiplexor: it deletes the mach_trap_mux_sysent / sys_mach_trap_mux_trap switch in src/mach_kmod/src/mach_traps.c (which today folds task_set_special_port=op2, host_set_special_port=op1, mach_port_move_member=op3, event-bell ops behind one slot), adds per-trap wire_one() calls in src/mach_syscall_wire.c, and rewrites src/libmach/mach_traps.c to resolve_syscall(\"task_set_special_port\") directly instead of resolve_syscall(\"mach_trap_mux\")+op. The coupling is mechanical, not conceptual: if the userland/module side ships individual syscalls but the kernel slot limit is still 10, the extra wire_one() calls fail (kern_syscall_register returns ENFILE), the sysctls publish -1, resolve_syscall returns -1, and syscall(-1) faults — every task_set_special_port / host bootstrap / port_move_member call breaks. The kernel patch alone is inert (stock kernel boots fine via the mux). The failure mode is invisible to compilation and to single-repo CI: only a live kernel booting the combined ISO proves the raised cap actually granted runtime slots. So the strategy must build {nextbsd-kernel@syscall-limit} x {nextbsd@mux-drop} into ONE ISO and boot it under QEMU, asserting TASK-SPECIAL-PORT-OK + HOST-BOOTSTRAP-OK + BOOTSTRAP-REMOTE-OK (with smoke-sysctls.sh as a cheaper pre-gate: each of mach.syscall.task_set_special_port/.host_set_special_port/.mach_port_move_member returns a dedicated >=0 slot and mach.syscall.mach_trap_mux is gone). The complication: the IMG only builds today inside a vmactions FreeBSD VM (~17 min, the long pole and the only repo that boots), while kernel/modules/compat already cross-build fast on Linux. A crossbuild migration making the IMG build natively on Linux (~5 min + ccache) is in-flight, which flips the economics — so every option is evaluated in BOTH today's VM world and the near-future native-cross world.</p></div>
<h2 id="the-five-options-at-a-glance">The five options at a glance</h2>
<table><thead><tr><th>Option</th><th>Family</th><th>Verdict</th><th>Best when</th></tr></thead><tbody>
<tr><td><strong>A.</strong> Continuous releases on every repo + PRs consume latest RELEASED artifact (versions.lock latest-green pin)</td><td><span class="pill pill-neutral">multi-repo</span></td><td><span class="pill pill-warn">Partial — keep the pin</span></td><td>Adopt as the STABILITY BACKBONE, not the pre-merge coupling answer: when you want to harden the cascade against ingesting broken/half-cascaded main artifacts and gain immutable, reproducible, quarantined consumption. Best paired with B (or later D) which supply the actual pre-merge coupling. The versions.lock latest-green pin is the piece worth adopting NOW even if you skip the rest of A.</td></tr>
<tr><td><strong>B.</strong> Artifact-override workflow_dispatch inputs (nextbsd gains kernel_run_id/base_run_id/modules_run_id; downstream PR points at an upstream PR/branch build)</td><td><span class="pill pill-neutral">multi-repo</span></td><td><span class="pill pill-good">Adopt now</span></td><td>Adopt NOW, in today's VM world, as the lowest-effort path to pre-merge coupled validation — idiomatic to these repos and ~80% present. Ideal when coupled changes are occasional and a 2-ref (kernel+nextbsd, optionally +modules/base) override covers the case; graduate to D once combinations grow beyond a couple of refs, and layer A's latest-green pin + a merge queue as the safety rail.</td></tr>
<tr><td><strong>C1.</strong> Full monorepo (toolchain + kernel + modules + compat + nextbsd in one repo, path-filtered workflows)</td><td><span class="pill pill-good">monorepo</span></td><td><span class="pill pill-neutral">Alt (after crossbuild)</span></td><td>Adopt as the LONG-TERM TARGET strictly AFTER the crossbuild migration lands and the IMG builds natively on Linux. It is the cleanest answer to coupled changes precisely because the only reason the repos were split — the VM/artifact handoff boundary — is dissolved by native cross-build. Sequence: ship crossbuild -> use B as the bridge meanwhile -> migrate to this monorepo + path filters + merge queue. Prefer C1 over C2 only once the external toolchain image boundary proves more annoying than valuable.</td></tr>
<tr><td><strong>C2.</strong> Partial monorepo (kernel + modules + compat + nextbsd merged; toolchain stays a separate GHCR image, pinned by digest)</td><td><span class="pill pill-warn">hybrid</span></td><td><span class="pill pill-good">Target (after crossbuild)</span></td><td>Prefer over C1 when the toolchain genuinely rebuilds on a different (slower, source-bump-driven) cadence than the OS source — which is the case here (toolchain only rebuilds on releng/15.0 bumps) — and when you want the cross-compiler pinned by immutable digest as the reproducibility anchor. Also the lower-risk FIRST monorepo step: migrate to C2, and only fold the toolchain in later (-> C1) if the external image boundary proves more annoying than valuable.</td></tr>
<tr><td><strong>D.</strong> Hybrid integration meta-workflow over {repo:ref} pairs (+ latest-green pin + merge queue)</td><td><span class="pill pill-warn">hybrid</span></td><td><span class="pill pill-warn">Keep thin / on-demand</span></td><td>You want pre-merge boot-test of arbitrary coupled cross-repo branch combinations WITHOUT a monorepo migration and WITHOUT giving up per-repo release isolation — the realistic near-term answer that handles the syscall+mux pair today and scales to N-ref coupling, then layers a latest-green pin (adopt now, cheap, fixes the broken-main-ingest hole) and a merge queue (adopt on the ISO repo once boot tests are fast post-crossbuild) as safety rails on top. Adopt D when 2-ref B overrides stop sufficing, or directly if you anticipate N-ref coupling and want the recorded-ref-set ergonomics from day one.</td></tr>
</tbody></table>
<h2 id="timetotest-perstage-cost-model">Time-to-test: per-stage cost model</h2>
<p>Wall-clock per build stage in three worlds: today’s <code>vmactions</code> VM, native cross-build cold (no ccache), and native cross-build warm (ccache hot). These are the building blocks behind every option’s time-to-test.</p>
<table><thead><tr><th>Stage</th><th>Today (VM)</th><th>Cross cold</th><th>Cross warm</th></tr></thead><tbody>
<tr><td>Toolchain container (GHCR pull, cached; full rebuild only on releng source bump)</td><td>~1m pull in-VM; full rebuild ~6m (rare)</td><td>~1m docker pull + type=gha layer cache; full rebuild ~6m (rare)</td><td>~0.3m warm layer cache, no pull</td></tr>
<tr><td>Kernel build (nextbsd-kernel, Linux x86_64 cross via toolchain image — same job in every world)</td><td>7m45s cold / 3m09s warm (already a Linux cross job, NOT in the VM)</td><td>7m45s (465s) cold</td><td>3m09s (189s) warm, ccache hot</td></tr>
<tr><td>mach.ko relink (module half of the mux-drop, built against extracted /usr/src/sys)</td><td>~1.5m (on the FreeBSD host at ISO-build time, inside the VM)</td><td>~1m (Linux cross compile of one kmod, no ccache)</td><td>~0.3m (ccache hot; tiny incremental relink)</td></tr>
<tr><td>Userland build — 32 comps incl. libmach (userland half of the mux-drop)</td><td>~7m (dominant part of VM source-compile ~8.5m; libicu -j2 OOM-capped)</td><td>~5m (native Linux cross, no ccache)</td><td>~1.5m (ccache hot)</td></tr>
<tr><td>Image assembly (pkg install + makefs/mkimg + zip)</td><td>~6.2m (pkg ~2.3m + zip ~3.9m, in-VM)</td><td>~4m (native makefs/mkimg + zip)</td><td>~3m (zip compression dominates; ccache-immune)</td></tr>
<tr><td>Boot test (qemu + expect, ~80 markers incl. TASK-SPECIAL-PORT-OK / HOST-BOOTSTRAP-OK / BOOTSTRAP-REMOTE-OK)</td><td>~3m (qemu q35 nested inside the FreeBSD VM)</td><td>~2.5m (qemu on Linux host, KVM)</td><td>~2.5m (boot time fixed; ccache irrelevant)</td></tr>
<tr><td>VM-only overhead (vmactions spin-up + guest boot + copyback) — DISAPPEARS in native world</td><td>~4m (VM boot ~2m + copyback ~2m)</td><td>0m (no VM)</td><td>0m (no VM)</td></tr>
</tbody></table>
<h2 id="timetotest-endtoend-per-case">Time-to-test: end-to-end per case</h2>
<table><thead><tr><th>Case</th><th>Today (VM)</th><th>Cross cold</th><th>Cross warm</th></tr></thead><tbody>
<tr><td>Kernel-only PR — A (continuous releases + pin)</td><td>NOT pre-merge testable end-to-end; release-then-pin-bump path ~20m+ AND post-merge only</td><td>~13.5m but still post-merge only (release+pin dance)</td><td>~7m but still post-merge only</td></tr>
<tr><td>Kernel-only PR — B (artifact-override inputs)</td><td>~28m: kernel cold 7m45s + dispatch nextbsd ISO reusing cached base/modules ~17m boot; one extra full ISO+boot</td><td>~13.5m (kernel 7m45s + assembly 4m + boot 2.5m; toolchain cached ~1m)</td><td>~6.5m (kernel 3m09s + assembly 3m + boot 2.5m)</td></tr>
<tr><td>Kernel-only PR — C1/C2 (monorepo + path filters)</td><td>N/A — blocked until crossbuild lands (no FreeBSD VM on the monorepo Linux runner)</td><td>~13m (C1) / ~13m (C2): one PR, one run — kernel 7m45s + mach.ko 1m + (userland path-skipped) + assembly 4m + boot 2.5m</td><td>~6m (C1) / ~5.5m (C2): kernel 3m09s + mach.ko 0.3m + assembly 3m + boot 2.5m; C2 slightly faster (toolchain never content-hash-evaluated, fixed pinned digest pull)</td></tr>
<tr><td>Kernel-only PR — D (meta-workflow {repo:ref})</td><td>~28m on-demand (same single-ISO cost as B; run by label/comment to bound VM cost)</td><td>~13.5m (same as B; meta-workflow builds kernel ref + assembles + boots)</td><td>~6.5m (same as B, generalized to N refs)</td></tr>
<tr><td>Coupled PR (syscall-limit kernel + mux-drop userland/mach.ko) — A</td><td>NOT possible pre-merge: requires RC tags of BOTH branches + pin bump before any combined boot — post-merge validation only</td><td>Still post-merge only; ~13.5m once both released and pinned</td><td>Still post-merge only; ~7m once released and pinned</td></tr>
<tr><td>Coupled PR — B (artifact-override inputs)</td><td>~28m: kernel PR branch 7m45s (prior) -> dispatch nextbsd ISO with kernel_run_id + mux-drop in nextbsd PR -> VM build ~17m + boot ~3m. One extra dispatch. Needs GAP#4 fix (ref-aware module patch clone)</td><td>~14.5m: kernel 7m45s ∥ userland 5m -> assembly 4m -> boot 2.5m</td><td>~7m: kernel 3m09s ∥ userland 1.5m -> assembly 3m -> boot 2.5m</td></tr>
<tr><td>Coupled PR — C1/C2 (monorepo, one PR + one run)</td><td>N/A — blocked until crossbuild lands</td><td>~14.5m: single PR rebuilds kernel 7m45s + mach.ko + userland (path-filtered) -> IMG -> boot. No cross-repo refs; coupling eliminated by construction</td><td>~7m: kernel 3m09s ∥ userland 1.5m -> mach.ko 0.3m -> assembly 3m -> boot 2.5m. Lowest steady-state, best ergonomics</td></tr>
<tr><td>Coupled PR — D (meta-workflow over both refs)</td><td>~28m on-demand (label/comment-triggered): builds kernel_ref + nextbsd_ref together, one recorded run, full VM ISO+boot</td><td>~14.5m (same critical path as B/C; generalizes to a 3rd/4th coupled ref)</td><td>~7m (cheap enough to run on every cross-repo-labeled PR)</td></tr>
</tbody></table>
<div class="callout callout-warn"><p>Time-to-test strings on each option are made consistent with this table. MODEL BASIS (real measurements): nextbsd VM build ~17m = userland ~7m + mach.ko ~1.5m + pkg ~2.3m + zip ~3.9m + VM boot ~2m + copyback ~2m; kernel 7m45s cold / 3m09s warm (already a Linux cross job); compat ~4m; boot test ~3m. Native cold/warm derived by removing VM/emulation overhead and applying ccache to compile stages only (zip and boot are fixed-cost, ccache-immune). KEY ASSUMPTIONS: (1) kernel/modules/compat ALREADY cross-build on the x86_64 Linux runner today; only the nextbsd ISO leg is VM-bound, so the entire 'today VM' penalty (~17m) lives in image-assembly+boot, and the kernel row is identical across all worlds. (2) Coupled critical path = kernel (7.75m cold / 3.15m warm) IN PARALLEL with userland (5m / 1.5m); the longer feeds assembly (4m/3m) -> boot (2.5m); hence coupled ~14.5m cold / ~7m warm (stages are NOT summed where they parallelize). (3) Boot is a fixed ~2.5-3m floor in every world (qemu+expect over ~80 markers, wall-clock bound by guest boot) — the irreducible floor of any boot-tested option; the cheapest sub-boot signal for THIS change is smoke-sysctls.sh. OPTION-LEVEL CONCLUSIONS: A never delivers pre-merge coupled testing at any speed (validates only after both halves are released) — keep it as the stability backbone (latest-green pin), not the coupling answer. B is the cheapest path to the goal and ~80% built (modules already has run_id+fbsd_sha; toolchain already publishes :<branch> tags and gates cascade on main); coupled cost = one extra full ISO+boot (~28m today, ~14.5m native cold, ~7m native warm); HARD CAVEAT GAP#4 — modules 'Apply patches to source' clones kernel MAIN with no ref, so for a coupled kernel-SOURCE change the patch source must be made ref-aware or mach.ko is KBI-desynced. C1/C2 give the ideal coupled ergonomics (one PR, one run, ~7m warm, lowest steady-state) but are BLOCKED in today's VM world. D matches B's per-run cost, generalizes to N coupled refs, and records the exact ref-set on each PR. Merge-queue + latest-green pin are a safety rail, not a coupling mechanism: adopt the pin NOW (replaces fragile 'gh run list --branch main --status success --limit 1' that can ingest a half-broken/half-cascaded main, fixing GAP#6 sha-coherence at the three nextbsd ingests); add the merge queue on the ISO repo once boot tests are fast (post-crossbuild), since it adds one re-validation boot per merge (~3m, noticeable today). ARM64 CAVEAT (GAP#5): every number is amd64 — compat is amd64-only and nextbsd ingests amd64-only, so arm64 has ZERO end-to-end/boot coverage today regardless of option. FORK-PR CAVEAT (GAP#7): all cross-repo reads use secrets.DISPATCH_TOKEN, unavailable to fork pull_request runs, so B/D auto-trigger is limited to same-repo branches until a GitHub App token / pull_request_target path is added.</p></div>
<h2 id="the-options-in-detail">The options in detail</h2>
<div style="border:1px solid var(--border);border-radius:6px;padding:18px 22px;margin:18px 0;background:white;">
<h3 style="margin-top:0;border:0;">A. Continuous releases on every repo + PRs consume latest RELEASED artifact (versions.lock latest-green pin) <span class="pill pill-neutral">multi-repo</span> <span class="pill pill-warn">Partial — keep the pin</span></h3>
<p>Every repo in the cascade cuts an immutable, versioned release on each main merge instead of leaving artifacts as ephemeral 90-day run outputs. toolchain already publishes immutable :<arch>-fbsd-<sha> GHCR tags and a mutable :<arch>-latest; under A you add a date/semver release tag so the image is addressable by a stable name. kernel, modules and compat gain a 'release' job (modules already has a dormant one gated client_payload.release=='true' — you flip it on for every main build) publishing per-arch GH releases (kernel-<date>-<sha>-<arch>, etc.). nextbsd keeps its rolling 'continuous' release. The keystone is a checked-in versions.lock in nextbsd recording the last fully-integrated boot-green set {toolchain_tag, kernel_release, modules_release, base_release}. The nextbsd ISO build's three ingest steps STOP doing 'gh run list --branch main --status success --limit 1' and instead 'gh release download <tag-from-versions.lock>'. A pin-bump bot opens a PR editing versions.lock whenever a new upstream release passes a green integration boot; merging that PR promotes the artifact set. Downstreams are always pinned to an immutable, known-good release rather than racing latest-main-success scraping (which can ingest a half-cascaded/freshly-broken main artifact — GAP#2/GAP#6).</p>
<p><strong>Coupled change (syscall + multiplexor) flow:</strong> The syscall-limit (kernel) + mux-drop (nextbsd src/mach_kmod + src/libmach) pair is INHERENTLY post-merge under pure A: pinning consumes only RELEASED tags, and you cannot release an unmerged branch without polluting the release namespace. To co-test pre-merge you must cut throwaway PRE-RELEASE tags of each branch and a temporary pin: (1) branch+push the kernel patch so a path-filtered push on patches/** builds it; (2) dispatch a release-flagged build to promote that branch build to a prerelease the pin can name (requires adding a release/prerelease input to kernel); (3) because the modules patch clone is hardcoded to kernel MAIN (GAP#4), cut a matching modules prerelease pointing its patch clone at the kernel branch — modules has no such ref input, so pure A cannot produce a coherent mach.ko for an unreleased kernel without that fix; (4) open the nextbsd mux-drop PR and temporarily edit versions.lock to point kernel_release/modules_release at the prerelease tags; the PR boot test gh-release-downloads the prerelease set, boots the combined ISO and asserts TASK-SPECIAL-PORT-OK + HOST-BOOTSTRAP-OK + BOOTSTRAP-REMOTE-OK; (5) after green, UN-pin (revert versions.lock) before merge, re-cut REAL releases on each merge, then bump versions.lock for real. Net: >=3 repos and 2-3 PRs plus prerelease bookkeeping — clumsy, multi-PR, and still hits GAP#4.</p>
<div class="twocol"><div><h4>⏱ Kernel-only PR</h4><p>Today (VM): NOT pre-merge end-to-end — a kernel-only PR is observable only by promoting a prerelease and re-pinning the ISO build, so realistically ~20m+ wall clock dominated by the vmactions ISO build+QEMU boot, PLUS manual prerelease+pin ceremony, and still post-merge in spirit. Native-cross: ~7m compute once a prerelease+pin exists, but the human prerelease/pin ceremony still dominates and keeps kernel-PR iteration slow.</p></div><div><h4>⏱ Coupled PR</h4><p>Today (VM): NOT possible pre-merge without throwaway RC tags of BOTH branches + a temporary pin edit/revert; the combined boot itself is one ~28m-class VM ISO+boot, but the prerelease+pin dance makes real iteration high-friction and post-merge in nature. Native-cross: ~7m once both are released and pinned, but the multi-prerelease + pin-edit/revert overhead keeps real coupled iteration high-friction and still post-merge.</p></div></div>
<div class="pros-cons"><div class="pros"><h4>Pros</h4><ul><li>EXCELLENT post-merge isolation/quarantine: every consumer pinned to an immutable release, so a bad kernel/base/modules main merge is quarantined until versions.lock is bumped — blast radius of a bad main merge is zero until the pin moves.</li><li>Fixes the real correctness hole (GAP#2/GAP#6): replaces fragile 'gh run list --branch main --status success --limit 1' scraping with a coherent, SHA-aligned {toolchain,kernel,modules,base} set, eliminating the interleaved/half-cascaded-main ingestion race.</li><li>Perfect reproducibility & forensics: a boot failure names the exact released artifact set; re-download and re-boot the identical combination any time within retention.</li><li>Low-moderate infra; reuses existing machinery (modules + nextbsd already release; toolchain already publishes immutable :fbsd-<sha> tags). No new per-PR compute.</li><li>Provides the stability backbone the other strategies layer on (the latest-green pin is the safety rail for B/D and a future merge queue).</li></ul></div><div class="cons"><h4>Cons</h4><ul><li>WEAK for the actual stated goal: coupling is naturally POST-merge. Pre-merge coupled validation requires throwaway prereleases + a temporary pin edit/revert — multi-PR, multi-repo, error-prone ceremony.</li><li>Does NOT by itself let a kernel PR be co-tested with a userland PR; you must bolt on prerelease+pin or pair with B/D.</li><li>Still blocked by GAP#4: modules patch clone hardcoded to kernel MAIN, so a prerelease against an unreleased kernel yields a source/KBI-desynced mach.ko unless that clone is made ref-aware first.</li><li>Release/tag namespace churn and storage growth (per-arch releases on every main merge across 4 repos); branch-testing prereleases pollute the namespace.</li><li>kernel still has no pull_request trigger and no boot test, so A gives kernel authors no local pre-merge signal — they only learn via the ISO pin path.</li><li>Fork-PR hostile (GAP#7): release-download from fork PRs is unavailable and prerelease-cuts require push rights, so external contributors can't drive the coupled flow.</li></ul></div></div>
<p><strong>Ratings:</strong> coupling ease <span class="pill pill-bad">low</span> speed <span class="pill pill-bad">low</span> infra simplicity <span class="pill pill-warn">medium</span> blast-radius control <span class="pill pill-good">high</span> release flexibility <span class="pill pill-good">high</span></p>
<p class="meta" style="margin:0;"><strong>Effort to adopt:</strong> Low-moderate. Reuse modules' dormant release path (flip on), nextbsd's continuous release, toolchain's immutable :fbsd-<sha> tags. New: add a 'release' job to kernel and compat; add versions.lock + a 'gh release download <pinned-tag>' rewrite of nextbsd's three ingest steps; add a pin-bump bot/PR on green integration. To make A usable for coupled PRE-merge testing you ALSO need a release input on kernel and a ref-aware modules patch clone (GAP#4) — those extras push effort toward moderate.</p>
</div>
<div style="border:1px solid var(--border);border-radius:6px;padding:18px 22px;margin:18px 0;background:white;">
<h3 style="margin-top:0;border:0;">B. Artifact-override workflow_dispatch inputs (nextbsd gains kernel_run_id/base_run_id/modules_run_id; downstream PR points at an upstream PR/branch build) <span class="pill pill-neutral">multi-repo</span> <span class="pill pill-good">Adopt now</span></h3>
<p>Generalize the override seam that ALREADY exists on nextbsd-kernel-modules (workflow_dispatch run_id + fbsd_sha, resolving client_payload.run_id || inputs.run_id and downloading cross-repo artifacts with DISPATCH_TOKEN) to the whole cascade. Three edits: (1) add a pull_request trigger AND a short boot/sysctl smoke to nextbsd-kernel so a kernel branch/PR builds the kernel + (on source change) mach.ko and produces nextbsd-kernel-<arch> / kernel-obj-<arch> under a known run_id — toolchain already publishes branch-scoped :<arch>-<branch> tags and gates all downstream dispatch on ref_name=='main', so branch builds are already addressable AND non-cascading; (2) add kernel_run_id / base_run_id / modules_run_id (+ matching *_ref) inputs to the nextbsd ISO build: when set, each replaces the 'gh run list --branch main --status success --limit 1' lookup with 'gh run download --repo <repo> --run-id <id>'; when unset, today's latest-main behavior is preserved; (3) FIX GAP#4: make the modules 'Apply patches to source' step clone the kernel at a passed kernel_ref instead of hardcoded main, so an override run patches module SOURCE against the same kernel branch the obj came from. Optionally add a /integrate kernel#NN PR-comment bot that resolves the head run_id, dispatches the ISO build, and posts boot results to both PRs. Override runs never publish 'continuous' and never cascade, so a pinned PR build can't touch prod artifacts.</p>
<p><strong>Coupled change (syscall + multiplexor) flow:</strong> Boot-test the syscall-limit (kernel) + mux-drop (nextbsd) pair BEFORE either merges: (1) create the kernel branch with the lkmnosys-band patch and push (path-filtered push on patches/** auto-builds; or PR once pull_request is added); (2) grab that build's KERNEL_RUN_ID; (3) with the GAP#4 fix in effect, dispatch modules against the SAME kernel branch (gh workflow run build.yml --repo nextbsd-kernel-modules -f run_id=$KERNEL_RUN_ID -f kernel_ref=syscall-limit) so mach.ko's source matches the obj, capture MODULES_RUN_ID; (4) open the nextbsd mux-drop PR (wire_one() per-trap + deletion of the mach_trap_mux switch in mach_traps.c; libmach resolve_syscall per-name); (5) dispatch the nextbsd ISO build pointing at the unmerged runs (gh workflow run build.yml --repo nextbsd --ref mux-drop -f kernel_run_id=$KERNEL_RUN_ID -f modules_run_id=$MODULES_RUN_ID; base_run_id omitted -> latest-main); the build downloads the pinned kernel/modules, builds libmach+mach.ko from the branch, bakes the ISO, boots it; (6) watch the boot assert TASK-SPECIAL-PORT-OK + HOST-BOOTSTRAP-OK + BOOTSTRAP-OK + BOOTSTRAP-REMOTE-OK + MACH-SMOKE-OK/LIBSYSTEM-KERNEL-OK; a cheaper pre-gate is smoke-sysctls.sh asserting each of mach.syscall.task_set_special_port/.host_set_special_port/.mach_port_move_member returns a dedicated >=0 slot and mach.syscall.mach_trap_mux is GONE. One extra dispatch (or one /integrate comment) yields the full boot-tested coupled ISO.</p>
<div class="twocol"><div><h4>⏱ Kernel-only PR</h4><p>Today (VM): kernel compile+smoke alone ~3m09s warm / ~7m45s cold on the Linux runner (the new local kernel signal B introduces); a full end-to-end boot of a kernel-only change via the ISO override costs the nextbsd VM leg, ~28m. Native-cross: kernel compile+smoke ~3m09s warm; full boot-tested kernel PR ~6.5m warm / ~13.5m cold.</p></div><div><h4>⏱ Coupled PR</h4><p>Today (VM): ~28m wall clock, dominated by the single vmactions ISO build+QEMU boot (~17m + ~3m boot); the kernel and modules override legs run fresh but cheap (kernel 7m45s, modules ~3m) and precede the ISO dispatch, which downloads their pinned artifacts rather than rebuilding the kernel — exactly ONE slow VM boot per iteration. Native-cross: ~14.5m cold / ~7m warm end-to-end (kernel ∥ userland -> assembly -> boot) — cheap enough to run on every cross-repo-labeled PR and the natural stepping stone to D.</p></div></div>
<div class="pros-cons"><div class="pros"><h4>Pros</h4><ul><li>BEST IMMEDIATE WIN for the stated goal: directly enables pre-merge boot testing of the syscall-limit + mux-drop pair with one extra dispatch, and matches the repos' existing idiom (modules already has run_id+fbsd_sha).</li><li>~80% already built: toolchain branch tags + non-cascading branch builds + modules override inputs all exist; remaining work is incremental, not greenfield.</li><li>Closes GAP#1 (adds pull_request + boot/sysctl smoke to kernel), GAP#2/#3 (adds the three override inputs to the nextbsd ISO build), and motivates GAP#4 (ref-aware modules patch clone) as part of the same change.</li><li>Strong isolation: override runs are explicitly non-cascading and never publish 'continuous'; a pinned PR build can't touch prod artifacts, and a failure points cleanly at the exact run_ids passed in.</li><li>Low standing infra and one extra full build+boot per coupled test; a /integrate kernel#NN bot makes the run_id wiring ergonomic and records the combination on the PR.</li><li>Natural on-ramp to D (generalize 2-ref overrides to N {repo:ref} pairs) and composes with A's latest-green pin and a future merge queue.</li></ul></div><div class="cons"><h4>Cons</h4><ul><li>Manual run_id wrangling is error-prone and not self-recorded on the PR unless the /integrate bot is built; a stale or wrong run_id silently tests the wrong combination.</li><li>Without the GAP#4 fix the seam is only half-complete: the obj is from the kernel PR but the module patch series is from kernel main, producing a source/KBI-desynced mach.ko for a coupled kernel-source change — so the ref-aware clone is a HARD prerequisite, not optional, for THIS scenario.</li><li>Fork-PR limitation (GAP#7): DISPATCH_TOKEN is unavailable to fork pull_request runs, so external contributors can't pull cross-repo artifacts without a GitHub App token or pull_request_target.</li><li>Today the nextbsd VM ISO leg still serializes every coupled boot (~28m), so high-frequency coupled iteration is bounded until crossbuild lands.</li><li>Override consumption bypasses the latest-green coherence guarantee (you can pin a genuinely-broken upstream run); pairing with A's versions.lock for the default/main path is advisable.</li><li>Retention asymmetry (GAP#8): override run_ids must be consumed within the upstream 90d / image 14d ceilings.</li></ul></div></div>
<p><strong>Ratings:</strong> coupling ease <span class="pill pill-good">high</span> speed <span class="pill pill-warn">medium</span> infra simplicity <span class="pill pill-good">high</span> blast-radius control <span class="pill pill-good">high</span> release flexibility <span class="pill pill-warn">medium</span></p>
<p class="meta" style="margin:0;"><strong>Effort to adopt:</strong> Low. (1) add pull_request + a boot/sysctl smoke job to nextbsd-kernel; (2) add kernel_run_id/base_run_id/modules_run_id (+ *_ref) inputs to the nextbsd ISO build, each gating its 'gh run list' lookup behind inputs.X || latest-main; (3) make the modules 'Apply patches to source' honor a kernel_ref input (GAP#4); (4) optional /integrate kernel#NN bot. No standing infra; reuses build.sh/boot-test.sh/toolchain image. Add a fork-safe token path only when external-contributor coupled PRs are needed.</p>
</div>
<div style="border:1px solid var(--border);border-radius:6px;padding:18px 22px;margin:18px 0;background:white;">
<h3 style="margin-top:0;border:0;">C1. Full monorepo (toolchain + kernel + modules + compat + nextbsd in one repo, path-filtered workflows) <span class="pill pill-good">monorepo</span> <span class="pill pill-neutral">Alt (after crossbuild)</span></h3>
<p>After the IMG cross-builds on Linux, collapse all five repos into one tree and let path-filtered workflows decide what rebuilds. Layout: /toolchain/ (Dockerfile.amd64/arm64), /kernel/patches/ (series + the syscall-limit patch) + /kernel/config/NEXTBSD, /src/ (the ~33 userland comps incl. mach_kmod, libmach, libdispatch, launchd, libxpc...), /compat/, /image/ (build.sh, overlays, pkglist), /tests/ (boot-test.sh + the freebsd-launchd-mach markers), /.github/workflows/, /versions/freebsd.lock (FREEBSD_BRANCH=releng/15.0 + pinned SHA — freebsd-src stays EXTERNAL, cloned at build time exactly as toolchain's Dockerfile does today, NOT vendored, so the repo stays ~325M not multi-GB). ONE build.yml with dorny/paths-filter computes the affected set, all jobs chained via needs:: toolchain-image (on toolchain/<strong> OR freebsd.lock else resolves to current :latest digest); kernel (on kernel/</strong> -> build inside toolchain image, emits kernel-obj as in-run artifact); modules (on kernel/<strong> OR src/mach_kmod/</strong> -> build mach.ko against THIS run's kernel-obj, killing GAP#4 by construction); userland (on src/<strong> -> rebuild only affected comps, ccache-scoped); compat (on compat/</strong> OR freebsd.lock); image (ALWAYS, the integration point: assemble from whichever of {kernel-obj, modules, userland, compat} this run produced, falling back to cached latest-green for untouched legs) -> boot-test asserts TASK-SPECIAL-PORT-OK + HOST-BOOTSTRAP-OK + BOOTSTRAP-REMOTE-OK. All cross-repo gh download / DISPATCH_TOKEN / repository_dispatch machinery is DELETED — artifacts flow via in-run needs:/download-artifact, so GAP#6 and GAP#7 both vanish.</p>
<p><strong>Coupled change (syscall + multiplexor) flow:</strong> The syscall-limit + mux-drop pair becomes ONE atomic PR touching three subtrees in one commit graph: /kernel/patches/ (lkmnosys-band increase + series entry), /src/mach_kmod/ (drop mach_trap_mux_sysent in mach_traps.c, add per-trap wire_one() in mach_syscall_wire.c), and /src/libmach/mach_traps.c (resolve_syscall('task_set_special_port') instead of the mux op-dispatch). On push, paths-filter flags kernel/<strong> AND src/mach_kmod/</strong> AND src/libmach/** -> kernel rebuilds with the new slot band, modules rebuilds mach.ko against THAT kernel-obj (same run, same SHA), userland rebuilds libmach, image assembles them into one ISO, boot-test asserts the markers. NO ref-pinning, NO run_id wrangling, NO patch-clone-against-main desync (GAP#4 cannot occur — modules build against the PR's own kernel tree). One PR, one CI run, one boot test proves the slots are claimed at runtime — the textbook case the monorepo collapses into a single unit of change.</p>
<div class="twocol"><div><h4>⏱ Kernel-only PR</h4><p>~6m warm / ~13m cold (POST-crossbuild only). kernel/<strong> triggers: kernel rebuild (~3m09s warm / 7m45s cold; toolchain image a cache hit), modules rebuild (~0.3m warm, kernel/</strong> affects mach.ko), image assemble from cached-green userland/compat (~3m warm / 4m cold), QEMU boot (~2.5m). Userland/compat SKIPPED by path filter. BLOCKED in today's VM world (the IMG still needs the vmactions FreeBSD VM, which the monorepo Linux runner does not have).</p></div><div><h4>⏱ Coupled PR</h4><p>~7m warm / ~14.5m cold (POST-crossbuild only). The coupled PR triggers kernel + modules + libmach(userland) rebuilds (kernel ∥ userland), then the single image assemble + boot — still ONE run, ONE boot, because all legs converge in-run. C's win is not raw minutes (similar to B/D native) but that it is a single self-contained run with ZERO cross-repo coordination. N/A in today's VM world.</p></div></div>
<div class="pros-cons"><div class="pros"><h4>Pros</h4><ul><li>Coupled changes are ELIMINATED as a problem, not managed: the syscall+mux pair is one commit, one PR, one CI run, one boot test — no run_id pins, no {repo:ref} sets, no /integrate bot.</li><li>Kills GAP#4 by construction: mach.ko builds against the PR's own kernel tree (in-run needs:), so the modules-patched-against-kernel-main desync cannot happen.</li><li>Kills GAP#6: artifacts flow in-run via needs:/download-artifact, so kernel<->modules<->base SHA are coherent by definition.</li><li>Kills GAP#7 (fork PRs): no cross-repo gh download / DISPATCH_TOKEN needed for the normal build, so external-contributor PRs can build and boot-test end-to-end.</li><li>Closes GAP#1/#2/#3: a kernel change is now boot-tested per-PR through the always-on image job; the nextbsd PR no longer ingests frozen latest-main.</li><li>Atomic bisect across one history; lower steady-state compute (one run vs a 4-5 run dispatch cascade); repo stays ~325M because freebsd-src is cloned at build time from the pinned SHA, not vendored.</li></ul></div><div class="cons"><h4>Cons</h4><ul><li>HARD-BLOCKED until the IMG cross-builds on Linux — useless in today's VM world because the image job still needs the vmactions FreeBSD VM, which would re-serialize every PR.</li><li>LOSES independent release cadence: toolchain/kernel/modules/compat can no longer version on their own timelines; everything releases at the monorepo's pace.</li><li>Path-filter fragility: a mis-scoped filter can skip a needed rebuild (false green — e.g. a sys/-touching change not mapped to the kernel job) or rebuild everything (slow). Filters become safety-critical config.</li><li>Blast radius of a bad main merge now spans all subsystems at once; the per-repo quarantine release pins gave you is gone unless a merge queue guards main.</li><li>HIGH one-time migration cost: unify 5 git histories + permissions + CONTRIBUTING, rewrite cross-repo artifact handoff into in-run needs:, re-scope ccache, port and test every path filter, delete the dispatch cascade, rebuild the continuous release around one tag stream.</li><li>Toolchain image rebuild coupling: a toolchain/** or freebsd.lock change sits in the same tree and can trigger a heavy image rebuild for PRs that conceptually only bump the toolchain — needs careful content-hash gating.</li><li>arm64 asymmetry (GAP#5) is inherited, not fixed: still amd64-only unless the compat/image arm64 legs are added during migration.</li></ul></div></div>
<p><strong>Ratings:</strong> coupling ease <span class="pill pill-good">high</span> speed <span class="pill pill-good">high</span> infra simplicity <span class="pill pill-good">high</span> blast-radius control <span class="pill pill-bad">low</span> release flexibility <span class="pill pill-bad">low</span></p>
<p class="meta" style="margin:0;"><strong>Effort to adopt:</strong> HIGH one-time, LOW steady-state. Blocked on crossbuild. Work: (1) merge five histories (git subtree/filter-repo to preserve blame); (2) write build.yml with dorny/paths-filter + the needs:-chained job graph; (3) replace every cross-repo gh download / repository_dispatch with in-run download-artifact; (4) re-scope ccache per subsystem; (5) content-hash gate the toolchain image so it only rebuilds on toolchain/**|freebsd.lock; (6) rebuild the continuous-release job around the single tag stream; (7) add a merge queue on main once boot tests are fast. Estimate: a few engineer-weeks after crossbuild, dominated by getting path filters and cache scoping correct.</p>
</div>
<div style="border:1px solid var(--border);border-radius:6px;padding:18px 22px;margin:18px 0;background:white;">
<h3 style="margin-top:0;border:0;">C2. Partial monorepo (kernel + modules + compat + nextbsd merged; toolchain stays a separate GHCR image, pinned by digest) <span class="pill pill-warn">hybrid</span> <span class="pill pill-good">Target (after crossbuild)</span></h3>
<p>Same as C1 but the toolchain stays its OWN repo, still publishing the GHCR cross-toolchain image (:<arch>-latest, :<arch>-fbsd-<sha>) exactly as today. The monorepo consumes that image as a job container, pinned by /versions/toolchain.lock (image digest) + /versions/freebsd.lock (releng SHA). Layout drops the /toolchain/ subtree; kernel/, src/, compat/, image/, tests/ are identical to C1. The path-filtered job graph is the same EXCEPT the toolchain-image job is replaced by a 'resolve-toolchain' step that reads versions/toolchain.lock and pulls the pinned digest — a toolchain bump is a one-line lockfile PR (or a repository_dispatch from the toolchain repo that opens a bump PR), not an in-tree image rebuild. kernel/modules/userland/compat/image jobs are byte-identical to C1. Rationale: the toolchain rebuilds rarely (only on releng/15.0 source bumps), is the slowest single artifact (per-arch docker build), and has the cleanest API boundary (a versioned container digest), so keeping it external avoids dragging heavy infrequent image builds into every coupled PR's tree and preserves the one boundary that genuinely benefits from independent cadence + GHCR's immutable digest addressing.</p>
<p><strong>Coupled change (syscall + multiplexor) flow:</strong> IDENTICAL to C1 for the syscall+mux pair: it spans kernel/patches/, src/mach_kmod/, src/libmach/ — all inside the monorepo — so it is still ONE atomic PR + ONE CI run + ONE boot test, with the same GAP#4/#6/#7 elimination. The toolchain being external is irrelevant to this pair because the pair never touches the toolchain. The ONLY coupled change this variant does NOT collapse into a single PR is a kernel change requiring a SIMULTANEOUS toolchain bump (e.g. a new clang feature): that becomes a two-step — land/dispatch the toolchain image, then bump versions/toolchain.lock in the monorepo PR. Rare; the syscall scenario is unaffected.</p>
<div class="twocol"><div><h4>⏱ Kernel-only PR</h4><p>~5.5m warm / ~13m cold (POST-crossbuild only) — slightly FASTER than C1 because the toolchain image is never content-hash-evaluated in-run; it is a fixed pinned digest pull (cache hit). kernel ~3m09s warm + modules ~0.3m + image assemble ~3m + boot ~2.5m, userland/compat path-skipped. BLOCKED in today's VM world (same hard prerequisite as C1).</p></div><div><h4>⏱ Coupled PR</h4><p>~7m warm / ~14.5m cold (POST-crossbuild only), essentially the same as C1 minus any toolchain-image evaluation overhead. The coupled syscall+mux PR rebuilds kernel + mach.ko + libmach against the pinned toolchain digest, assembles one ISO, boots once. N/A in today's VM world.</p></div></div>
<div class="pros-cons"><div class="pros"><h4>Pros</h4><ul><li>All of C1's coupling wins for the syscall+mux pair (one atomic PR, one run, one boot; GAP#4/#6/#7 eliminated) WITHOUT pulling heavy infrequent toolchain image builds into every PR's tree.</li><li>PARTIALLY PRESERVES independent release cadence: the toolchain keeps its own versioning + immutable GHCR digest, and a toolchain bump is a reviewable one-line lockfile PR — the one boundary that most benefits from independent cadence stays intact.</li><li>Smaller, more focused monorepo (no Dockerfiles / docker-buildx CI in the main tree); the per-arch docker matrix doesn't coexist with the source build graph.</li><li>Toolchain digest pinning makes the cross-compiler the cleanest reproducibility anchor — versions/toolchain.lock fully determines it for any monorepo commit.</li><li>Lower migration risk: merging 4 source repos is simpler than 5, and the toolchain (most different CI shape — docker build-push vs source build) is left untouched.</li><li>Same atomic bisect / lower steady-state compute / ~325M footprint benefits as C1 for the merged repos.</li></ul></div><div class="cons"><h4>Cons</h4><ul><li>A kernel change that genuinely needs a coordinated toolchain bump is NOT a single PR — it reintroduces a (small, lockfile-shaped) two-step cross-repo coordination for that specific case.</li><li>Still BLOCKED until the IMG cross-builds on Linux — same hard prerequisite as C1.</li><li>Keeps ONE cross-repo seam alive: the toolchain->monorepo edge still needs a bump mechanism (lockfile PR or repository_dispatch), so a sliver of the dispatch/token machinery survives (far less than today's 5-repo cascade).</li><li>Loses independent cadence for kernel/modules/compat/nextbsd (same as C1) — only the toolchain's cadence is preserved.</li><li>Same path-filter fragility and main-blast-radius concerns as C1 for the merged subsystems.</li><li>Slight conceptual asymmetry: 'mostly monorepo, one external image' is a less clean mental model; contributors must know the toolchain lives elsewhere.</li></ul></div></div>
<p><strong>Ratings:</strong> coupling ease <span class="pill pill-good">high</span> speed <span class="pill pill-good">high</span> infra simplicity <span class="pill pill-warn">medium</span> blast-radius control <span class="pill pill-bad">low</span> release flexibility <span class="pill pill-warn">medium</span></p>
<p class="meta" style="margin:0;"><strong>Effort to adopt:</strong> HIGH one-time but slightly LESS than C1 (4 histories not 5; no docker-build CI to port into the main tree; the toolchain repo is left as-is). Blocked on crossbuild. Extra vs C1: define versions/toolchain.lock + a toolchain-bump path (repository_dispatch opening a bump PR, or a scheduled lockfile updater). Everything else (path filters, needs: graph, ccache scoping, in-run artifact handoff, merge queue) is identical to C1. Estimate: a few engineer-weeks after crossbuild, marginally cheaper than C1.</p>
</div>
<div style="border:1px solid var(--border);border-radius:6px;padding:18px 22px;margin:18px 0;background:white;">
<h3 style="margin-top:0;border:0;">D. Hybrid integration meta-workflow over {repo:ref} pairs (+ latest-green pin + merge queue) <span class="pill pill-warn">hybrid</span> <span class="pill pill-warn">Keep thin / on-demand</span></h3>
<p>Keep all five repos separate exactly as today, but add ONE new dispatchable workflow, integration.yml, in nextbsd (or a tiny nextbsd-ci meta-repo for clean token scope and no path-filter collisions). It is a workflow_dispatch (+ repository_dispatch type: integrate, + PR-comment bot) workflow whose inputs are a SET of {repo:ref} pins: toolchain_ref, kernel_ref, modules_ref, compat_ref, nextbsd_ref (each a branch name, PR head ref, or sha; blank = current main/latest-green). It runs the WHOLE chain on demand for that exact ref-set, fresh and non-cascading: job toolchain (builds from toolchain_ref or reuses the ghcr :<arch>-<branch> tag, exporting an image the later jobs use as their container); job kernel (needs: toolchain — checks out kernel@kernel_ref, cross-builds, uploads kernel-<arch> + kernel-obj-<arch> AS ARTIFACTS OF THIS RUN, so everything is addressed by one github.run_id, killing GAP#6); job modules (needs: kernel — checks out modules@modules_ref and CRITICALLY clones the kernel patch source at kernel_ref not main, fixing GAP#4; consumes the kernel obj from this run's artifacts via needs); job compat (needs: toolchain); job assemble+boot (needs: kernel, modules, compat — checks out nextbsd@nextbsd_ref, builds mach.ko + libmach from THAT ref where the mux-drop lives, assembles the IMG consuming the three in-run artifacts via the override path NOT the latest-main scrape, runs boot-test.sh). NEVER publishes continuous, NEVER dispatches downstream. The run records the full {repo:ref} set in its inputs and posts a status check/comment back to every participating PR (integration/coupled => required check). Generalizes the ONE existing override seam (modules' run_id+fbsd_sha) from a single pinned upstream run to an arbitrary N-branch combination built fresh and coherent in a single recorded run.</p>
<p><strong>Coupled change (syscall + multiplexor) flow:</strong> The coupled pair is exactly {nextbsd-kernel: syscall-limit} x {nextbsd: mux-drop} (the slot-limit patch lives in nextbsd-kernel/patches/; mach.ko + libmach both live in the nextbsd repo). Steps: (1) open PR #NN in nextbsd-kernel adding the lkmnosys-band-widening patch to patches/series; (2) open PR #MM in nextbsd dropping the mux: add wire_one(task_set_special_port/host_set_special_port/mach_port_move_member/event-bell) as dedicated sysents in mach_syscall_wire.c, delete the mach_trap_mux_sysent + sys_mach_trap_mux_trap switch in mach_traps.c, switch libmach/mach_traps.c to per-name resolve_syscall; (3) comment '/integrate kernel#NN nextbsd#MM' (the bot maps PR numbers to head refs and repository_dispatch's integration.yml with kernel_ref=<#NN head>, nextbsd_ref=<#MM head>, the other three blank => latest-green pin); (4) integration.yml builds toolchain (latest-green), cross-builds kernel@#NN (widened slot band -> kernel-obj), builds modules cloning kernel patch source AT #NN (GAP#4-correct), builds compat (latest-green base), assembles the IMG from nextbsd@#MM consuming THIS run's artifacts, boots it; (5) required green markers TASK-SPECIAL-PORT-OK (was mux op=2), HOST-BOOTSTRAP-OK (op=1), BOOTSTRAP-REMOTE-OK (depends on mach_port_move_member op=3 -> launchd), plus MACH-SMOKE-OK/LIBSYSTEM-KERNEL-OK/LIBXPC-OK; smoke-sysctls.sh additionally asserts each of mach.syscall.task_set_special_port/.host_set_special_port/.mach_port_move_member returns a DEDICATED >=0 slot and mach.syscall.mach_trap_mux is GONE; (6) the integration status check posts back to BOTH PRs as a required 'coupled-integration' check — neither can merge red. On green the merge queue merges them together: kernel#NN first (so its main artifact exists), then nextbsd#MM, re-running integration against the up-to-date target before each lands; on kernel#NN merge a pin-bump PR updates versions.lock to the new kernel artifact so subsequent nextbsd main builds ingest the coherent slot-raised kernel.</p>
<div class="twocol"><div><h4>⏱ Kernel-only PR</h4><p>Today (VM): a kernel-only /integrate that actually boots drags in the nextbsd vmactions VM leg -> ~28m on-demand (run by label/comment to bound VM cost); if you only need 'does it compile + do modules wire', skip the boot leg for ~6m warm / ~10m cold. Native-cross: full boot-tested kernel PR ~6.5m warm / ~13.5m cold (same as B, generalized to N refs).</p></div><div><h4>⏱ Coupled PR</h4><p>Today (VM): one /integrate run builds toolchain (reused tag ~0-2m) + kernel cross (~7m45s) + modules cross (~3m, parallel w/ compat) + compat (~3m) then the dominant vmactions VM leg (livecd ISO + mach.ko/libmach + QEMU boot) ~17m+3m => ~28m wall-clock, fully serialized on the single VM leg; run on-demand (label/comment) to bound cost. Native-cross: the whole coupled {kernel#NN, nextbsd#MM} boot-tested run drops to ~14.5m cold / ~7m warm and becomes cheap enough to run automatically on every cross-repo-labeled PR.</p></div></div>
<div class="pros-cons"><div class="pros"><h4>Pros</h4><ul><li>Strongest coupling answer for arbitrary N-branch combinations: tests {syscall-limit, mux-drop} AND any third/fourth coupled branch (e.g. a matching toolchain bump) in ONE recorded, reproducible run with the {repo:ref} set captured in inputs.</li><li>Reuses ~80% of existing machinery: toolchain already publishes :<arch>-<branch> tags and gates downstream dispatch on ref_name=='main' (branch builds addressable AND non-cascading); modules already accepts run_id+fbsd_sha. D generalizes the one existing seam rather than inventing a new model.</li><li>Fixes GAP#6 (sha coherence) by construction: everything is addressed by one integration run_id with in-run needs: hand-off, instead of three independent latest-main-success scrapes that can interleave/mismatch.</li><li>Fixes GAP#4 inside the integration path: the modules leg clones the kernel patch source at kernel_ref, so module SOURCE matches the kernel PR obj.</li><li>Zero blast radius on prod: never publishes continuous, never dispatches downstream; a failure names the precise ref-set and isolates which repo's branch broke the combination.</li><li>No repo migration, no rewrite of the artifact-handoff model, no merge of five histories/permissions — strictly additive, low-regret. Survives even a later monorepo if any repo (toolchain/freebsd-src) stays split.</li><li>The latest-green pin refinement directly closes the real correctness hole where a nextbsd main build ingests a freshly-broken or half-cascaded main artifact via 'gh run list --branch main --status success --limit 1'.</li></ul></div><div class="cons"><h4>Cons</h4><ul><li>Builds each leg fresh per integration run, so it does more work than artifact-reuse strategy B for a simple 2-ref override; in today's VM world the nextbsd boot leg dominates and serializes, so on-demand-only (label/comment) is required to bound cost.</li><li>Needs a small dispatcher/bot (PR-comment -> repo_dispatch with mapped head refs) and DISPATCH_TOKEN; fork PRs can't carry DISPATCH_TOKEN (GAP#7), so cross-repo reads need an app token or pull_request_target path before D works for outside branches.</li><li>The {repo:ref} input set is manual surface area; a wrong/omitted ref silently falls back to latest-green and can mask a needed coupling unless the bot echoes the resolved ref-set back to the PR.</li><li>Until the IMG cross-build lands, the integration run inherits the vmactions VM as the long pole, so per-PR auto-runs are too slow to gate every push — it is on-demand integration, not continuous, in the VM world.</li><li>Merge-queue re-validation adds one boot-test run per merge; noticeable in the VM world, so scope the queue to the nextbsd ISO repo first and enable broadly only post-crossbuild.</li><li>Latest-green pin adds a versions.lock + pin-bump PR/bot to maintain, and must respect retention ceilings (upstream 90d, image 14d — GAP#8) so a pin can't point at an evicted run.</li></ul></div></div>
<p><strong>Ratings:</strong> coupling ease <span class="pill pill-good">high</span> speed <span class="pill pill-warn">medium</span> infra simplicity <span class="pill pill-warn">medium</span> blast-radius control <span class="pill pill-good">high</span> release flexibility <span class="pill pill-good">high</span></p>
<p class="meta" style="margin:0;"><strong>Effort to adopt:</strong> Moderate, mostly additive. (1) write integration.yml in nextbsd or nextbsd-ci taking 5 *_ref inputs, jobs toolchain->{kernel,compat}->modules->assemble+boot wired by needs: and in-run artifacts; (2) make the modules leg clone kernel patch source at kernel_ref (GAP#4) and the nextbsd assemble step consume in-run artifacts instead of the three latest-main scrapes; (3) add a /integrate PR-comment bot mapping PR#->head ref -> repository_dispatch, echoing the resolved ref-set and posting the result as a required check on each PR; (4) add a versions.lock latest-green pin + a pin-bump-on-green-integration bot, and switch nextbsd main ingest from latest-main-success to the pin; (5) enable merge_group on the nextbsd ISO repo re-running integration against the up-to-date target; (6) add an app token / pull_request_target path for fork PRs (GAP#7). Steps 1-4 are the must-do core; 5-6 are incremental hardening. No history merge, no path-filter rewrite, no VM-removal dependency for the core (only auto-on-every-PR economics wait on crossbuild).</p>
</div>
<h2 id="ratings-matrix">Ratings matrix</h2>
<p>Higher is better on every axis (green = high, amber = medium, red = low).</p>
<table><thead><tr><th>Option</th><th>Coupling ease</th><th>Speed</th><th>Infra simplicity</th><th>Blast-radius control</th><th>Release flexibility</th></tr></thead><tbody>
<tr><td><strong>A</strong> Continuous releases on every repo + PRs consum</td><td><span class="pill pill-bad">low</span></td><td><span class="pill pill-bad">low</span></td><td><span class="pill pill-warn">medium</span></td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-good">high</span></td></tr>
<tr><td><strong>B</strong> Artifact-override workflow_dispatch inputs </td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-warn">medium</span></td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-warn">medium</span></td></tr>
<tr><td><strong>C1</strong> Full monorepo </td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-bad">low</span></td><td><span class="pill pill-bad">low</span></td></tr>
<tr><td><strong>C2</strong> Partial monorepo </td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-warn">medium</span></td><td><span class="pill pill-bad">low</span></td><td><span class="pill pill-warn">medium</span></td></tr>
<tr><td><strong>D</strong> Hybrid integration meta-workflow over {repo:re</td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-warn">medium</span></td><td><span class="pill pill-warn">medium</span></td><td><span class="pill pill-good">high</span></td><td><span class="pill pill-good">high</span></td></tr>
</tbody></table>
<h2 id="recommendation">Recommendation</h2>
<div class="callout callout-good"><p class="verdict verdict-go">Pick: NOW (today's VM world): adopt B (artifact-override inputs) as the coupling mechanism + the latest-green pin from A as the stability backbone. AFTER crossbuild lands: migrate to C2 (partial monorepo, toolchain stays an external pinned image) + a merge queue, keeping a thin D-style integration.yml for any ref combination that still spans a non-merged repo.</p><p>The syscall-limit + mux-drop pair must be built into ONE ISO and booted under QEMU before either merges — the failure mode (slot exhaustion -> sysctl -1 -> syscall(-1) faults) is invisible to compilation and to single-repo CI, so a boot-tested integration build is mandatory. Against that hard requirement: A NEVER delivers pre-merge coupled testing at any speed (it validates only after both halves are released), so it is the stability backbone, not the answer. B is the cheapest path to the goal and ~80% already built (modules already has run_id+fbsd_sha; toolchain already publishes :<branch> tags and gates the cascade on main), needs no standing infra, is idiomatic to these repos, and works TODAY in the VM world at ~28m on-demand — immediately unblocking the pair with one extra ISO dispatch. Its one hard prerequisite for THIS scenario is the GAP#4 fix (make the modules 'Apply patches to source' clone ref-aware), because the syscall-limit is a kernel-SOURCE change and without it mach.ko is KBI-desynced; that fix is small and is also required by D, so it is never wasted work. C1/C2 give the ideal ergonomics (one PR, one run, ~7m warm, GAP#4/#6/#7 eliminated by construction) but are HARD-BLOCKED until the IMG cross-builds on Linux, so they cannot be the now-answer. Once crossbuild lands, C2 is preferred over C1 because the toolchain genuinely rebuilds on a slower, source-bump-driven cadence and benefits from an immutable GHCR digest as the reproducibility anchor, and merging 4 histories is lower-risk than 5; fold the toolchain in later (-> C1) only if the external image boundary proves more annoying than valuable. D is the right generalization when 2-ref overrides stop sufficing or when a third coupled ref (e.g. a matching toolchain bump) appears, and it stays useful even after a monorepo for any combination spanning a still-split repo (toolchain/freebsd-src). Net cost-to-goal ranking: B (lowest, mostly built) < D (moderate, reuses B) < merge-queue/pin (low config) ; C highest one-time and blocked until crossbuild but best long-term steady-state. The pick threads these: B+pin now, C2+merge-queue (with residual D) later.</p></div>
<h3>Sequencing</h3><ol>
<li>Step 0 (prereq, do first): fix GAP#4 — make nextbsd-kernel-modules' 'Apply patches to source' clone the kernel at a passed kernel_ref instead of hardcoded main. This is required for any coupled kernel-SOURCE change (the syscall-limit patch) and is shared by B and D, so it is never wasted.</li>
<li>Step 1 (NOW, VM world — adopt B): add a pull_request trigger + a short boot/sysctl smoke job to nextbsd-kernel (closes GAP#1, gives kernel authors a local signal); add kernel_run_id/base_run_id/modules_run_id (+ *_ref) override inputs to the nextbsd ISO build, each gating its 'gh run list' lookup behind inputs.X || latest-main (closes GAP#2/#3). This alone delivers pre-merge boot testing of the syscall-limit + mux-drop pair at ~28m on-demand today.</li>
<li>Step 2 (NOW, cheap, parallel — adopt A's pin only): add a versions.lock latest-green pin and switch nextbsd's three main ingests from 'gh run list --branch main --status success --limit 1' to 'gh release download <pinned-tag>', with a pin-bump-on-green-integration bot. This fixes the real correctness hole (GAP#6 — ingesting a half-cascaded/freshly-broken main) and restores per-repo quarantine. Adopt the pin without the rest of A.</li>
<li>Step 3 (NOW, ergonomics — optional but recommended): add a /integrate kernel#NN PR-comment bot that resolves head run_ids, dispatches the ISO build, echoes the resolved combination, and posts the boot result as a required check on both PRs — making the run_id wiring self-recorded and hard to get wrong.</li>
<li>Step 4 (WHEN 2-ref overrides stop sufficing — generalize B to D): write integration.yml taking {repo:ref} pairs (toolchain/kernel/modules/compat/nextbsd), building fresh and coherent in one recorded run; run on-demand (label/comment) in the VM world to bound the ~28m cost. Reuses the Step-0 GAP#4 fix and the Step-3 bot.</li>
<li>Step 5 (POST-CROSSBUILD — migrate to C2): once the IMG cross-builds natively on Linux (~5m + ccache), merge kernel+modules+compat+nextbsd into one repo with dorny/paths-filter + a needs:-chained job graph; keep the toolchain external, pinned by /versions/toolchain.lock digest. Coupled changes become one PR + one run + one boot (~7m warm), with GAP#4/#6/#7 eliminated by construction.</li>
<li>Step 6 (POST-CROSSBUILD — add the safety rail): enable a GitHub merge queue (merge_group) on the monorepo/ISO repo re-running the boot test against the up-to-date target, so two independently-green PRs can't break main when combined. Keep a thin D-style integration.yml for any ref combination still spanning a non-merged repo (toolchain / freebsd-src). Add a GitHub App token / pull_request_target path (GAP#7) when external-contributor coupled PRs are needed, and add the arm64 compat + ISO legs (GAP#5) if/when arm64 boot coverage is in scope — every estimate above is amd64-only.</li>
</ol>
<p class="footnote">Generated from an 8-agent workflow (3 investigators: current topology + gaps, the real syscall/multiplexor change, coupled-change CI patterns; 4 option-designers: continuous-release, artifact-override, monorepo, hybrid integration + a shared time-to-test cost model; 1 synthesizer). Time-to-test figures derive from measured runs (nextbsd VM build ~17 min, kernel 3m09s warm / 7m45s cold, compat ~4 min, boot test ~3 min).</p>
</div>
</body>
</html>