Repository navigation
Expand file tree
/
Copy pathgershwin-livecd-unionfs-plan.html
More file actions
539 lines (508 loc) · 38.5 KB
/
Copy pathgershwin-livecd-unionfs-plan.html
File metadata and controls
539 lines (508 loc) · 38.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Gershwin · init_chroot + unionfs/tmpfs port — Implementation Plan</title>
<style>
:root {
--bg:#f6f7f9; --card:#fff; --ink:#1c2128; --muted:#5b6570; --line:#e3e6ea;
--accent:#2f6f4f; --accent-soft:#e7f1ec; --warn:#8a5a00; --warn-soft:#fbf1dd;
--danger:#9b2c2c; --danger-soft:#fbe8e8; --code-bg:#1e2228; --code-ink:#e6e6e6;
--blue:#2b5d8a; --blue-soft:#e6eef5;
}
* { box-sizing:border-box; }
body { margin:0; background:var(--bg); color:var(--ink);
font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
-webkit-font-smoothing:antialiased; }
.wrap { max-width:920px; margin:0 auto; padding:48px 24px 96px; }
header.page { border-bottom:3px solid var(--accent); padding-bottom:20px; margin-bottom:32px; }
header.page h1 { margin:0 0 6px; font-size:27px; letter-spacing:-0.01em; }
header.page .sub { color:var(--muted); font-size:15px; }
header.page .repos { margin-top:14px; display:flex; gap:8px; flex-wrap:wrap; }
.pill { font:12px/1 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;
background:var(--accent-soft); color:var(--accent); padding:6px 10px; border-radius:999px;
border:1px solid #cfe2d8; }
h2 { font-size:20px; margin:40px 0 14px; letter-spacing:-0.01em; display:flex;
align-items:baseline; gap:10px; }
h2 .n { font:13px/1 ui-monospace,monospace; color:#fff; background:var(--accent);
padding:5px 9px; border-radius:6px; flex:none; }
h3 { font-size:16px; margin:22px 0 8px; }
p { margin:10px 0; }
.card { background:var(--card); border:1px solid var(--line); border-radius:10px;
padding:20px 22px; margin:16px 0; }
.card.tight { padding:16px 18px; }
code { font:13.5px/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;
background:#eef0f2; padding:1.5px 5px; border-radius:4px; color:#2a2f36; }
pre { background:var(--code-bg); color:var(--code-ink); padding:14px 16px; border-radius:8px;
overflow-x:auto; font-size:13px; line-height:1.55; margin:12px 0; }
pre code { background:none; padding:0; color:inherit; }
ul,ol { margin:10px 0; padding-left:24px; }
li { margin:5px 0; }
.muted { color:var(--muted); }
.tag { display:inline-block; font:11px/1 ui-monospace,monospace; padding:3px 7px;
border-radius:5px; vertical-align:middle; margin-right:6px; font-weight:600;
letter-spacing:0.02em; }
.tag.keep { background:var(--accent-soft); color:var(--accent); }
.tag.new { background:var(--blue-soft); color:var(--blue); }
.tag.change { background:var(--warn-soft); color:var(--warn); }
.tag.delete { background:var(--danger-soft); color:var(--danger); }
.tag.locked { background:var(--accent); color:#fff; }
.decision { border-left:4px solid var(--accent); background:var(--accent-soft);
border-radius:0 8px 8px 0; padding:14px 18px; margin:12px 0; }
.decision .q { font-weight:600; margin-bottom:4px; }
.decision .a { color:var(--accent); font-weight:600; }
.callout { border-left:4px solid var(--warn); background:var(--warn-soft);
border-radius:0 8px 8px 0; padding:12px 16px; margin:14px 0; font-size:14.5px; }
.callout.danger { border-left-color:var(--danger); background:var(--danger-soft); }
.callout.info { border-left-color:var(--blue); background:var(--blue-soft); }
table { border-collapse:collapse; width:100%; margin:14px 0; font-size:14px; }
th,td { text-align:left; padding:9px 12px; border-bottom:1px solid var(--line); vertical-align:top; }
th { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:0.04em; }
td code { font-size:12.5px; }
.filehdr { font:13px/1.4 ui-monospace,monospace; background:#eef0f2; border:1px solid var(--line);
border-radius:6px 6px 0 0; padding:8px 12px; margin:18px 0 0; font-weight:600; border-bottom:none; }
.filebody { border:1px solid var(--line); border-radius:0 0 6px 6px; padding:14px 18px; margin:0 0 8px; }
.filebody ul { margin-top:4px; }
footer { margin-top:56px; padding-top:18px; border-top:1px solid var(--line);
color:var(--muted); font-size:13px; }
.legend { display:flex; gap:14px; flex-wrap:wrap; margin:8px 0 0; font-size:13px; }
</style>
</head>
<body>
<div class="wrap">
<header class="page">
<h1>Port <code>init_chroot</code> + unionfs/tmpfs full read-write architecture into Gershwin</h1>
<div class="sub">Fixes "admin can't log in after install" at its source, and makes the installed
system a faithful copy of the pristine live ISO image.</div>
<div class="repos">
<span class="pill">gershwin-on-freebsd @ unionfs-full-readwrite</span>
<span class="pill">gershwin-system @ unionfs-full-readwrite</span>
<span class="pill">reference: freebsd-livecd-unionfs</span>
</div>
</header>
<h2><span class="n">Status</span> Implementation outcome & known unionfs copy issues</h2>
<div class="card">
<p><strong>Both branches were implemented, tested on real hardware, and merged
(~May 2026).</strong> The installed system now faithfully captures the running live
session — full metadata, hardlinks, in-session <code>pkg</code> installs. But getting there
surfaced three concrete defects in FreeBSD 15's in-kernel <code>unionfs</code>: the first
forced a change to decision <strong>D2</strong>, and the second shipped as a
<strong>known unfixed defect</strong>. They are documented here so the next person doesn't
rediscover them from scratch. The plan body below is left as the original (pre-implementation)
plan; this section records where reality diverged.</p>
<h3>1 · Lower-layer symlink corruption — <span class="tag change">forced D2 change: bsdtar → cpio</span></h3>
<p>Reading a lower-layer (uzip) symlink through the union intermittently fails:
<code>readlink()</code> returns <code>EINVAL</code>; <code>lstat()</code> returns
<code>EBADF</code> or a garbage mode/size. It is a kernel bug — <em>every</em> userland tool
(<code>tar</code>, <code>cpio</code>, <code>cp</code>, <code>pax</code>, <code>rsync</code>,
<code>cpdup</code>) is handed the same garbage, because they all read via the same
<code>lstat</code>/<code>readlink</code> syscalls.</p>
<p>What differs is the <em>reaction</em>: <code>tar</code>/libarchive treats it as
<strong>fatal</strong> and aborts the whole archive — on the live ISO this consistently
truncated a 155k-file system to ~137 files, every run. <code>cpio</code> treats it as a
<strong>per-file warning</strong>: it skips that entry and continues. So D2's
<code>bsdtar</code> pipe was replaced with <code>find -x / <prunes> | cpio -pdmu</code>
(both base tools), plus a <strong>stage-2 symlink-repair sweep</strong> that re-walks
<code>find -type l</code> and recreates any symlink cpio dropped (a dedicated
<code>readlink</code> sweep with a short retry reliably reads links that are flaky under
cpio's own access pattern). <code>cpio -pdmu</code> was verified on hardware to preserve
hardlinks (<code>/rescue</code> stays 149-linked, ~19 MB not ~2.9 GB),
<code>schg</code> flags, setuid, ownership, perms and mtime. <em>Status: fixed and merged.</em></p>
<h3>2 · Directory copy-up corruption — <span class="tag delete">KNOWN UNFIXED DEFECT</span></h3>
<p><code>avahi-daemon</code>'s startup touches <code>/usr/local/etc/avahi</code>, which triggers
a buggy unionfs <strong>copy-up</strong>: a corrupt node is left in the tmpfs
<strong>upper</strong> layer, shadowing the intact directory in the uzip
<strong>lower</strong>. The corrupt node's <code>stat()</code> reads back inconsistently over
time — a garbage multi-KB regular file, a 19-byte symlink, or a zero-size directory, all with
garbage size/mtime. The installer faithfully copies whatever the union shows it, so an
installed system ends up with a corrupt <code>/usr/local/etc/avahi</code> (a garbage ~34 KB
file instead of the config directory). avahi / mDNS is then misconfigured on installs;
everything else about the install is correct. Reproduced on a clean boot: avahi was a healthy
directory before the install and corrupt after.</p>
<div class="callout danger" style="margin:10px 0;"><strong>Shipped as a known defect.</strong>
The <code>cpio</code> pivot was the priority and avahi is one non-critical config directory, so
the branches were merged without the fix.</div>
<p><strong>Root cause is understood and the fix is identified</strong> (not yet implemented):
the corruption is 100% in the upper layer — <code>rm -rf /usr/local/etc/avahi</code> on the
live union removes the corrupt upper node and the clean uzip lower shows through (verified
in-session). The intended fix is a <strong>pre-copy repair step</strong> in
<code>installer.sh</code>: before the <code>find | cpio</code>, detect a corrupt upper node
(e.g. a <code>/usr/local/etc</code> child that exists but is not a directory) and
<code>rm -rf</code> it to expose the clean lower, then copy. This keeps full live-session
capture and needs no separate uzip read.</p>
<h3>3 · The <code>/usr/local/sbin</code> tmpfs shadow — <span class="tag change">fixed</span></h3>
<p>Not a unionfs bug, but it surfaced in the same work and is tool-agnostic.
<code>installer.sh</code> mounts an empty tmpfs over <code>/usr/local/sbin</code> at startup to
suppress devd automounting during partitioning. The normal-install copy walks the live
<code>/</code>, so <code>find</code> walked that <em>empty tmpfs</em> and the installed system
was missing all ~73 binaries there (<code>cupsd</code>, <code>avahi-daemon</code>,
<code>blkid</code>, <code>automount</code>, …). Fixed by unmounting the workaround tmpfs
immediately before the copy — partitioning is done and the target is mounted by then, so devd
is no longer a concern. <em>Status: fixed and merged.</em></p>
<h3>Why not read the uzip directly, or use a different tool?</h3>
<ul>
<li><strong>The uzip lower can't be read in isolation.</strong> <code>/dev/md0.uzip</code> is
already mounted as the union's lower layer; a second mount fails with FFS
<code>Device busy</code>, and re-mounting the cd9660 to fresh-<code>mdconfig</code> the
backing file was also unreliable. The only readable view of the live session is the union
itself — so the copy must go through it, bugs and all.</li>
<li><strong>The copy tool isn't the lever.</strong> <code>rsync</code> and <code>cpdup</code>
read via the same <code>lstat</code>/<code>readlink</code> syscalls and get the same kernel
garbage; they behave like <code>cpio</code> (tolerate & continue), not like
<code>tar</code> (abort) — exactly where the <code>cpio</code> pivot already landed. Neither
is in base, so they would add a dependency for no benefit.</li>
<li><strong>The real long-term fix is upstream</strong> — the <code>unionfs</code> symlink
and copy-up behaviours are FreeBSD kernel defects. Failing that, an architecture that
doesn't layer a writable tmpfs over the uzip via <code>unionfs</code> would sidestep both.</li>
</ul>
</div>
<h2><span class="n">Why</span> Context & root cause</h2>
<div class="card">
<p>The live ISO boots via <code>resources/overlays/boot/init_script</code>, which nullfs-mounts
each top-level dir from the uzip over the cd9660 root and layers tmpfs/unionfs piecemeal.
Because that hand-built layout fights the normal rc boot, the script
<strong>overwrites <code>/etc/rc.d/{cleanvar,var,cleartmp,tmp,hostname}</code> with neutered
stubs</strong>.</p>
<p>The installer (<code>gershwin-system/Library/Scripts/installer.sh</code>) copies <code>/</code>
wholesale with a broken <code>cp -a</code> fallback, so those neutered stubs — plus stale
<code>/var/run</code> cruft — land on installed disks. Diagnosed downstream effect: a stale
<code>/var/run/dshelper.pid</code> is never cleared (the real <code>cleanvar</code> is gone), so
<code>dshelper</code> never starts, and <code>admin</code> (a Directory Services user) can't log in.</p>
<p><strong>The fix:</strong> replace the piecemeal <code>init_script</code> with the
<code>init_chroot</code> + in-kernel-unionfs pivot from <code>freebsd-livecd-unionfs</code> — the
whole root becomes one writable union, the real rc.d scripts run normally, and
<strong>nothing is neutered</strong>. Port the <code>/boot/firmware</code> symlink; split
<code>loader.conf</code> into a live-only tier and an "everywhere" tier; and change the installer
to copy the <strong>running live session with <code>bsdtar</code></strong> from the live union
<code>/</code> (explicit mount-point excludes, no <code>--one-file-system</code>) instead of the
broken <code>cp -a</code> fallback.</p>
<div class="callout info" style="margin-bottom:4px;"><strong>Design north star:</strong> the
installed system <em>is</em> the live session — achieved by the installer making a faithful
file-level copy of the <em>running live union</em> (so in-session pkg installs / config changes
land on disk — hard requirement), baking all system config into the uzip so the live ISO is
already a working desktop, and keeping the one genuinely live-only setting
(<code>root_rw_mount="NO"</code>) out of both the uzip and the installer copy. No new live-only
rc.d script; no <code>SystemPrepare.sh</code> hook.</div>
</div>
<h2><span class="n">Decisions</span> Locked during planning</h2>
<div class="decision">
<div class="q"><span class="tag locked">D1</span> Pivot mechanism & single-user mode</div>
<div class="a">→ D1a — fallback-only (<code>init_chroot</code> kenv).</div>
<p class="muted" style="margin:6px 0 0;"><code>init_script</code> runs as a child of
<code>/sbin/init</code>, sets <code>kenv init_chroot=/sysroot</code>, exits; init chroots and
continues. Single-user (<code>boot -s</code>) works natively — init keeps its own argv and honors
<code>-s</code> after the chroot. The <code>freebsd-launchd-mach</code> <code>"$@"</code>-forwarding
fix is only needed for the dual-mode PID-1 design, which we are not using.</p>
</div>
<div class="decision">
<div class="q"><span class="tag locked">D2</span> Installer copy method</div>
<div class="a">→ <code>bsdtar</code> of the live union root, with explicit excludes — not <code>cp -a</code>, not <code>rsync</code>, no <code>SystemPrepare.sh</code>.</div>
<p class="muted" style="margin:6px 0 0;"><strong>Hard requirement:</strong> the installer copies the
<em>running live session</em> — in-session pkg installs / config changes must land on the installed
disk — so copying the pristine uzip is out (it drops all of that). <code>cp -a</code> is unfit (no
hardlink preservation → <code>/rescue</code> explodes; no exclude mechanism); <code>rsync</code> isn't
in base. The installer <code>bsdtar</code>s straight from <code>/</code>, the live in-kernel unionfs.
It does <strong>not</strong> use <code>--one-file-system</code> — on a unionfs the <code>st_dev</code>
detection isn't trustworthy; an earlier attempt relied on it, the create-side tar wandered into
<code>/mnt</code> and read the target the extract side was writing, truncating the archive. Instead
every mount point + bit of live-only cruft is named explicitly: <code>./mnt</code> (the target —
<strong>critical</strong>), <code>./dev ./proc ./tmp ./media</code>, the linprocfs/linsysfs/devfs
submounts under <code>./compat/linux</code>, <code>./var/{run,tmp,cache}</code>, and
<code>./etc/rc.conf.local</code> (the live-only <code>root_rw_mount</code> override).
<code>bsdtar --acls --xattrs --fflags</code> carries the full metadata set (hardlinks, ownership,
perms, timestamps, setuid/sticky); excluded dirs are recreated empty. Walking the union is a proven
technique (NomadBSD / GhostBSD / mfsBSD do it). <em>Rejected along the way:</em> <code>dd</code> of
the uzip (maintainer wants a file-level tool); <code>bsdtar</code> from a second mount of
<code>/dev/md0.uzip</code> (impossible — FFS <code>Device busy</code>); and <code>bsdtar</code> from
a fresh <code>mdconfig</code> of the uzip (works, but copies the <em>pristine</em> image — drops
live-session changes).</p>
</div>
<div class="decision">
<div class="q"><span class="tag locked">D3</span> No <code>gershwin_live</code> rc.d script</div>
<div class="a">→ Redistribute the old per-boot tweaks into baked, universal config.</div>
<p class="muted" style="margin:6px 0 0;">The old "Gershwin specific" tweaks go into rc.conf /
sysctl.conf / loader.conf / <code>/etc/rc.local</code> — all identical on live and installed.
Debug aids are dropped or made universal, never <code>is_live()</code>-gated. The one genuinely
live-only setting, <code>root_rw_mount="NO"</code>, is written by <code>init_script</code> into the
ephemeral tmpfs upper, so the uzip stays byte-identical.</p>
</div>
<div class="decision">
<div class="q"><span class="tag locked">D4</span> Zero live-only behavioral branches</div>
<div class="a">→ Drop the SMBIOS hostname override; make full console-mute universal.</div>
<p class="muted" style="margin:6px 0 0;">The old <code>init_script</code> quietly set the live
ISO's hostname from the SMBIOS product name — that's dropped; both live and installed use
<code>hostname="gershwin"</code>. <code>conscontrol mute on</code> becomes universal via
<code>kern.consmute=1</code> in the Tier-2 <code>loader.conf.d/gershwin.conf</code> — full quiet
boot on live <em>and</em> installed, no "1 line of text" on either. Result: <strong>no
<code>is_live()</code>-gated behavior anywhere.</strong> The only live-only setting at all,
<code>root_rw_mount="NO"</code>, is written to the tmpfs upper by <code>init_script</code> and
never touches the uzip.</p>
</div>
<h2><span class="n">Layers</span> The cd9660 / uzip boundary</h2>
<div class="card">
<ul>
<li><strong>cd9660 boot layer</strong> — staged in <code>CD_ROOT</code> by
<code>prepare_boot_env()</code>: <code>init_script</code>, the Tier-1 <code>loader.conf</code>,
the <code>/sbin/init</code> + <code>/boot/firmware</code> symlinks, <code>/sysroot</code>,
<code>/upper</code>, <code>/rescue</code>. Kernel's real root, hidden from the post-pivot
chroot, not part of the uzip — never reaches an install.</li>
<li><strong>uzip layer</strong> — built from <code>RELEASE_DIR</code>: the actual system, the
lower layer of the live union. Everything that should reach installs lives here (the installer
copies the live union <code>/</code>, whose lower layer this is — see D2).</li>
<li><strong>unionfs tmpfs upper</strong> — live-session writable state: in-session pkg installs /
config changes (which the installer <strong>does</strong> copy — that's the requirement), plus
a little genuinely live-only state. <code>init_script</code> writes
<code>root_rw_mount="NO"</code> into <code>/etc/rc.conf.local</code> here; the installer excludes
that one file so it never reaches an install.</li>
</ul>
</div>
<h2><span class="n">Audit</span> Live → installed propagation</h2>
<div class="card">
<h3 style="margin-top:4px;">loader.conf — split into two tiers</h3>
<p><strong>Tier 1 — live-only, stays in cd9660 <code>loader.conf</code>:</strong>
<code>init_script</code>, <code>init_shell</code>, <code>unionfs_load</code>,
<code>geom_uzip_load</code>, <code>geom_ventoy_load</code>, <code>vfs.root.mountfrom=cd9660:…</code>,
<code>vfs.mountroot.timeout</code>, <code>vm.kmem_size*</code> (LiveCD low-RAM ARC cap — harmful on
installs), <code>kern.geom.label.debug</code>, <code>loader_conf_dirs="/boot/loader.conf.d"</code>.</p>
<p><strong>Tier 2 — everywhere, new <code>loader.conf.d/gershwin.conf</code></strong> (baked into
<code>RELEASE_DIR/boot/loader.conf.d/</code> → uzip → installs, read via the base-default
<code>loader_conf_dirs</code>; also staged onto cd9660 for the live loader): quiet boot
(<code>boot_mute</code>, <code>kern.vt.color.*</code>, <code>splash=""</code>),
<code>beastie_disable</code>, <code>autoboot_delay</code>, <code>screen.font</code>,
<code>hw.vga.textmode</code>, <code>screen.textmode</code>, the <code>hw.usb.quirk.*</code>
SD-card-reader block, <code>hw.psm.*</code> touchpad tap, <code>hint.pcm.*.eq</code>,
<code>hw.syscons.bell=0</code>, <code>hw.usb.no_shutdown_wait</code>, <code>dev.bge.0.msi=0</code>,
<code>firewire_load</code>, <code>aio_load</code>, <strong><code>kern.consmute=1</code></strong>
(full console mute — replaces the old runtime <code>conscontrol mute on</code>; falls back to
<code>/etc/sysctl.conf</code> if not loader-tunable). The separate <code>loader.mute.d/</code> dir
is merged in here and removed.</p>
<h3>rc.conf / sysctl.conf — uzip → installs (everywhere)</h3>
<p>All <code>configure_system</code> sysrc settings stay everywhere (services, networking,
<code>kld_list</code>, <code>sendmail_*="NO"</code>), plus <code>allscreens_kbdflags="-b quiet.off"</code>
(replaces the old <code>kbdcontrol</code> call). <code>kern.panic_reboot_wait_time</code> →
<code>/etc/sysctl.conf</code> if kept.
<strong><code>root_rw_mount="NO"</code> is NOT baked</strong> — live-only, written by
<code>init_script</code> to the tmpfs upper (an installed UFS root needs the rw remount).</p>
<h3>init_script "Gershwin specific" block — redistribution</h3>
<table>
<thead><tr><th>Item</th><th>Decision</th><th>Mechanism</th></tr></thead>
<tbody>
<tr><td>rc.d neutering (cleanvar/var/cleartmp/tmp/hostname)</td><td><span class="tag delete">DELETED</span></td><td>—</td></tr>
<tr><td><code>LoginWindow.plist</code></td><td>everywhere</td><td><code>configure_system</code> copies overlay → <code>RELEASE_DIR/Local/…</code></td></tr>
<tr><td>rc.d <code># REQUIRE:</code> reordering (ldconfig/dbus/initgfx/slim)</td><td><span class="tag delete">DROPPED</span></td><td>replacing ldconfig's stock <code># REQUIRE: … cleanvar</code> lets it run before the now-real <code>cleanvar</code>, which purges <code>/var/run</code> and wipes <code>ld-elf.so.hints</code> → every <code>/usr/local/lib</code> lib "not found". FreeBSD's default order is correct.</td></tr>
<tr><td>initgfx <code>__wait 3→1</code></td><td>everywhere</td><td><code>configure_system</code> build-time sed of <code>RELEASE_DIR/usr/local/etc/rc.d/initgfx</code> (non-ordering speedup)</td></tr>
<tr><td>sysrc service / sendmail-off writes; <code>allscreens_kbdflags</code></td><td>everywhere</td><td><code>configure_system</code> heredoc</td></tr>
<tr><td><code>kbdcontrol -b quiet.off</code></td><td>everywhere</td><td>folded into <code>allscreens_kbdflags</code> — no script</td></tr>
<tr><td><code>kern.panic_reboot_wait_time=30</code></td><td>everywhere, or drop</td><td><code>/etc/sysctl.conf</code> baked into uzip</td></tr>
<tr><td>VirtualBox kmod load + sysrc</td><td>everywhere</td><td><code>/etc/rc.local</code> (base; runs on live + installed; wanted on both)</td></tr>
<tr><td><code>conscontrol mute on</code></td><td>everywhere</td><td><code>kern.consmute=1</code> in Tier-2 <code>loader.conf.d/gershwin.conf</code></td></tr>
<tr><td>SMBIOS hostname override</td><td><span class="tag delete">DROPPED</span></td><td>both use <code>hostname="gershwin"</code></td></tr>
<tr><td><code>start-hello</code> verbose logging sed</td><td>drop (dev aid)</td><td>—</td></tr>
<tr><td>monkey-patch</td><td>dropped (user-chosen)</td><td>—</td></tr>
</tbody>
</table>
<p class="muted" style="margin-bottom:0;">Other: <code>fstab.extra</code> / <code>devfs.rules.extra</code>
→ everywhere (baked into the uzip). <code>local.lua</code> boot menu → live-only (cd9660 only,
inherent). <code>resources/config/Gershwin.conf</code> → build-time pkg-repo config, not
live-system state.</p>
</div>
<h2><span class="n">Repo A</span> gershwin-on-freebsd — boot architecture</h2>
<div class="legend">
<span><span class="tag new">NEW</span>new file</span>
<span><span class="tag change">CHANGE</span>modified</span>
<span><span class="tag keep">KEEP</span>unchanged</span>
<span><span class="tag delete">DELETE</span>removed</span>
</div>
<div class="filehdr"><span class="tag change">CHANGE</span> resources/overlays/boot/init_script — rewrite contents (name & location unchanged)</div>
<div class="filebody">
<p>Minimal pre-pivot script (modeled on <code>freebsd-livecd-unionfs/ramdisk/init.sh</code>). It
lives only on the cd9660 boot layer, so it is <strong>not</strong> copied to installed systems:</p>
<pre><code>PATH=/rescue ; silence stdout/stderr
kldload geom_uzip ; kldload unionfs
mdconfig -a -t vnode -o readonly -f /boot/rootfs.uzip -u 0
wait-loop for /dev/md0.uzip (halt after ~30s)
mount -t ufs -o ro /dev/md0.uzip /sysroot
mount -t tmpfs tmpfs /upper
mount -t unionfs /upper /sysroot
mount -t devfs devfs /sysroot/dev
echo 'root_rw_mount="NO"' >> /sysroot/etc/rc.conf.local # live-only, tmpfs upper
kenv init_chroot=/sysroot ; kenv -u init_script init_shell ; exit 0</code></pre>
<p><span class="tag delete">DELETE</span><strong>Dropped vs. the old script:</strong> all per-directory
nullfs mounts, the <code>/etc</code> & <code>/var</code> tmpfs+cp dance, <strong>the rc.d
neutering</strong>, the whole "Gershwin specific" block (redistributed — see audit), the
<code>ps -o command 1</code> single-user shell-out (single-user works natively now), and the
monkey-patch.</p>
</div>
<div class="filehdr"><span class="tag change">CHANGE</span> resources/overlays/boot/loader.conf (Tier 1) + <span class="tag new">NEW</span> loader.conf.d/gershwin.conf (Tier 2)</div>
<div class="filebody">
<ul>
<li><strong>Tier 1</strong> — <code>loader.conf</code> reduced to the live-only set.
<code>init_script="/boot/init_script"</code> unchanged; add <code>init_shell="/rescue/sh"</code>
and <code>unionfs_load="YES"</code>; keep <code>loader_conf_dirs="/boot/loader.conf.d"</code>
(drop the <code>loader.mute.d</code> entry).</li>
<li><span class="tag new">NEW</span><strong>Tier 2</strong> —
<code>resources/overlays/boot/loader.conf.d/gershwin.conf</code>: quiet boot + hardware
quirks/tuning, including everything merged in from <code>loader.mute.d</code>.</li>
<li><span class="tag delete">DELETE</span><code>resources/overlays/boot/loader.mute.d/</code>.</li>
</ul>
</div>
<div class="filehdr"><span class="tag change">CHANGE</span> build.sh — <code>prepare_boot_env()</code> (~lines 325–395)</div>
<div class="filebody">
<ul>
<li><span class="tag keep">KEEP</span><code>tar -cf - boot | tar … -C CD_ROOT</code>, COPYRIGHT
copy, <code>rm -rf boot/modules/*</code>, the <code>.ko</code> prune, kernel + module gzip,
<code>cp …/login.conf CD_ROOT/etc/</code>, the <code>/rescue</code> tar-pipe, <code>fdupes</code>,
the <code>splash</code> and <code>tmpfs_load</code> seds, <code>rm tmpfs.ko*</code>.</li>
<li><span class="tag change">CHANGE</span>replace the ~25-entry <code>mkdir -p</code> block with
<code>mkdir -p CD_ROOT/{sysroot,upper,dev,etc,sbin}</code>.</li>
<li><span class="tag keep">KEEP</span><code>cp -R OVERLAYS_DIR/boot CD_ROOT</code> still merges
<code>loader.conf</code>/<code>lua</code> and the rewritten <code>init_script</code> into
CD_ROOT/boot. Keep <code>chmod 755 CD_ROOT/boot/init_script</code>.</li>
<li><span class="tag new">NEW</span>stage the Tier-2 drop-in:
<code>cp RELEASE_DIR/boot/loader.conf.d/gershwin.conf CD_ROOT/boot/loader.conf.d/</code>.</li>
<li><span class="tag new">NEW</span><code>ln -sf /rescue/init CD_ROOT/sbin/init</code></li>
<li><span class="tag new">NEW</span><code>ln -sf /sysroot/boot/firmware CD_ROOT/boot/firmware</code>
— kernel-context firmware loads resolve via the cd9660 namespace; Repo A ships
drm-kmod / gpu-firmware-kmod. Rock Ridge preserves both symlinks.</li>
</ul>
</div>
<div class="filehdr"><span class="tag keep">KEEP</span> build.sh — <code>generate_iso()</code></div>
<div class="filebody">
<p>No change. (The <code>dd</code>-era note about trimming the <code>makefs -b 75%</code> slack no
longer applies — <code>bsdtar</code> copies file content, not the UFS free space, so the slack
doesn't affect the installed-system size.)</p>
</div>
<div class="filehdr"><span class="tag change">CHANGE</span> build.sh — <code>configure_system()</code> (~lines 226–283)</div>
<div class="filebody">
<p>Add to the existing <code>sysrc</code> heredoc: <code>sendmail_*="NO"</code> (×4),
<code>allscreens_kbdflags="-b quiet.off"</code>,
<code>kld_list+="cuse ig4 iicbus iichid utouch asmc if_urndis if_cdce if_ipheth"</code>.</p>
<div class="callout danger" style="margin:8px 0;"><strong>Do NOT</strong> put
<code>root_rw_mount="NO"</code> here — it would bake into the uzip and reach installs, where a
UFS root <em>needs</em> the rw remount. It's live-only, written by <code>init_script</code> into
the unionfs tmpfs upper instead.</div>
<p>After <code>dscli init</code>:</p>
<ul>
<li><span class="tag new">NEW</span><code>mkdir -p RELEASE_DIR/Local/Library/Preferences</code> +
copy in the existing <code>resources/overlays/Local/Library/Preferences/LoginWindow.plist</code>.</li>
<li><span class="tag new">NEW</span><code>mkdir -p RELEASE_DIR/nvidia</code>.</li>
<li><span class="tag change">CHANGE</span>the initgfx <code>__wait 3</code>→<code>1</code> sed (a
non-ordering speedup). The old init_script's rc.d <code># REQUIRE:</code> reordering
(ldconfig/dbus/initgfx/slim) is <strong>deliberately not ported</strong> — it was only safe when
<code>cleanvar</code> was a neutered stub; with the real <code>cleanvar</code> running, rewriting
ldconfig's stock <code># REQUIRE: … cleanvar</code> lets ldconfig run before cleanvar purges
<code>/var/run</code>, wiping <code>ld-elf.so.hints</code>. FreeBSD's default rc order is correct.</li>
<li><span class="tag new">NEW</span><code>mkdir -p RELEASE_DIR/boot/loader.conf.d</code> + write
the Tier-2 <code>gershwin.conf</code> into it.</li>
<li><span class="tag new">NEW</span>append <code>kern.panic_reboot_wait_time=30</code> to
<code>RELEASE_DIR/etc/sysctl.conf</code> (if kept).</li>
<li><span class="tag new">NEW</span>add the VirtualBox-detection block to
<code>RELEASE_DIR/etc/rc.local</code> (kenv product check → kldload vboxguest + sysrc enables);
create <code>rc.local</code> if absent.</li>
</ul>
<p class="muted" style="margin-bottom:0;">No <code>gershwin_live</code> rc.d script is created.</p>
</div>
<h2><span class="n">Repo B</span> gershwin-system — installer</h2>
<div class="filehdr"><span class="tag change">CHANGE</span> Library/Scripts/installer.sh — install with <code>bsdtar</code> of the live union</div>
<div class="filebody">
<p>Replace the tree-copy logic (the <code>rsync</code> path and the broken <code>cp -a</code>
fallback) for the normal "install this live system" path:</p>
<ul>
<li><span class="tag keep">KEEP</span><code>gpart</code> partitioning (EFI + freebsd-ufs, or
freebsd-boot + freebsd-ufs).</li>
<li><span class="tag change">CHANGE</span><code>newfs -U ${ROOT_PART}</code>, then mount the target (+ EFI).</li>
<li><span class="tag new">NEW</span><code>bsdtar</code> pipe straight from <code>/</code> — the
live in-kernel union (this is what captures the live session). <strong>No
<code>--one-file-system</code></strong> (unionfs <code>st_dev</code> isn't trustworthy); every
mount point + live-only cruft named explicitly. <code>tar</code> is bsdtar on FreeBSD:<br>
<code>( cd / && tar --acls --xattrs --fflags --exclude=./mnt --exclude=./dev --exclude=./proc --exclude=./tmp --exclude=./media --exclude=./compat/linux/proc --exclude=./compat/linux/sys --exclude=./compat/linux/dev --exclude=./var/run --exclude=./var/tmp --exclude=./var/cache --exclude=./etc/rc.conf.local -cf - . ) | ( cd $MNT && tar --acls --xattrs --fflags -xpf - )</code><br>
<code>./mnt</code> is the critical exclude — never read the target the extract side is writing.
<strong>Hardlinks</strong> (<code>/rescue</code>), ACLs, xattrs, BSD flags, ownership, perms,
timestamps, setuid/sticky all preserved by bsdtar.</li>
<li><span class="tag new">NEW</span>recreate the excluded mount-point dirs as empty
(<code>mnt dev proc tmp media compat/linux/{proc,sys,dev/shm} var/{run,tmp,cache}</code>);
<code>chmod 1777 $MNT/tmp</code>.</li>
<li><span class="tag keep">KEEP</span>the post-copy <code>chroot "$MNT" …/dscli init</code>;
bootcode / EFI loader staging; the <code>loader.conf</code> append
(<code>vfs.root.mountfrom="ufs:..."</code> / <code>nvme_load</code>).</li>
<li><span class="tag change">CHANGE</span><code>/etc/fstab</code> — on this path, <strong>prepend</strong>
the per-install root + EFI entries to the copied fstab (which carries the
<code>fstab.extra</code> mounts) rather than overwriting.</li>
</ul>
<p>Detect <code>/dev/md0.uzip</code>; if absent (installer run outside a live boot), error clearly.
The separate image-based install path (copying from a mounted <code>da0</code>) keeps
<code>rsync</code> with a hardlink-safe <code>tar</code>-pipe fallback — the <code>cp -a</code>
fallback is removed either way.</p>
</div>
<h2><span class="n">Scope</span> Follow-up & out of scope</h2>
<div class="card tight">
<ul style="margin:0;">
<li><strong>Native GNUstep installer backend</strong> — a future, separate PR: a Foundation-based
<code>Copier.framework</code> + CLI as a fully-native, cross-platform file-level copier, scoped
in its own pkgdemon.github.io plan doc. The current <code>bsdtar</code> step is the interim;
both are file-level walkers, so the swap is clean.</li>
<li><code>dshelper.rc</code>'s <code>dshelper_start()</code> checks <code>[ -f pidfile ]</code>
rather than process liveness — a latent weakness in the DirectoryServices component
(<code>gershwin-developer</code>), not these two repos. With the architecture + bsdtar installer
the stale pidfile is never present on an install, so it won't be triggered. Flag it for a
separate follow-up.</li>
</ul>
</div>
<h2><span class="n">Verify</span> How this gets tested end-to-end</h2>
<div class="card">
<ol>
<li><strong>Build.</strong> <code>build.sh</code> on a FreeBSD 15 host produces the ISO. Inspect
the cdroot: <code>boot/init_script</code> rewritten, <code>sbin/init</code> +
<code>boot/firmware</code> are symlinks, <code>sysroot</code>/<code>upper</code> present,
<code>boot/loader.conf.d/gershwin.conf</code> present, <code>boot/rootfs.uzip</code> present.</li>
<li><strong>Live boot</strong> (VM + bare metal). Reaches the login window; <code>admin</code>
logs in. <code>mount</code> shows the <code>unionfs</code> overlay;
<code>ls -la /etc/rc.d/cleanvar</code> is the <em>real</em> FreeBSD script;
<code>sysrc -n root_rw_mount</code> is <code>NO</code>; quiet boot active; in VirtualBox
<code>kldstat | grep vbox</code>.</li>
<li><strong>Firmware.</strong> On hardware needing it, <code>dmesg | grep -i firmware</code>
shows drm / iwlwifi firmware loaded.</li>
<li><strong>Install + the actual bug.</strong> On the live ISO, <strong><code>pkg install</code>
a marker package</strong> (e.g. <code>nano</code>) so there's a known live-session change to
verify. Then clone <code>gershwin-system</code> @ <code>unionfs-full-readwrite</code>, run its
<code>installer.sh</code> against a target disk; reboot the installed system. Verify:
<strong>the marker package (<code>nano</code>) is installed</strong> — the live session was
captured; <code>/etc/rc.d/cleanvar</code> is the real script; <code>/var/run</code> has no stale
<code>dshelper.pid</code>; <code>service dshelper status</code> running; <code>admin</code>
logs in; <code>sysrc -n root_rw_mount</code> is empty/<code>YES</code> and root is rw;
<code>loader.conf</code> has the installer's <code>ufs:</code> mountfrom and <strong>no</strong>
<code>unionfs_load</code>/<code>init_script</code>; <code>loader.conf.d/gershwin.conf</code> present.</li>
<li><strong>Metadata + divergence check.</strong> On the installed system, <code>/rescue</code>
is still hardlinked (<code>stat -f %l /rescue/sh</code> shows a high link count, not ~200
copies) and setuid bits are intact (<code>ls -l /usr/bin/su</code> → <code>-r-sr-xr-x</code>);
<code>/etc/rc.conf.local</code> is <strong>absent</strong>; <code>/var/run</code> has no stale
pidfiles. Confirm quiet boot on the <em>installed</em> system (<code>kern.consmute</code> set)
and <code>hostname</code> = <code>gershwin</code> on both — no <code>is_live()</code>-gated
behavior should exist.</li>
</ol>
</div>
<h2><span class="n">Files</span> Critical files touched</h2>
<table>
<thead><tr><th>Repo</th><th>Path</th><th>Action</th></tr></thead>
<tbody>
<tr><td>gershwin-on-freebsd</td><td><code>build.sh</code> — <code>prepare_boot_env()</code>, <code>configure_system()</code></td><td><span class="tag change">CHANGE</span></td></tr>
<tr><td>gershwin-on-freebsd</td><td><code>resources/overlays/boot/init_script</code></td><td><span class="tag change">CHANGE</span> rewrite (name/location unchanged)</td></tr>
<tr><td>gershwin-on-freebsd</td><td><code>resources/overlays/boot/loader.conf</code></td><td><span class="tag change">CHANGE</span> reduce to Tier 1</td></tr>
<tr><td>gershwin-on-freebsd</td><td><code>resources/overlays/boot/loader.conf.d/gershwin.conf</code></td><td><span class="tag new">NEW</span> Tier 2</td></tr>
<tr><td>gershwin-on-freebsd</td><td><code>resources/overlays/boot/loader.mute.d/</code></td><td><span class="tag delete">DELETE</span> merged into Tier 2</td></tr>
<tr><td>gershwin-system</td><td><code>Library/Scripts/installer.sh</code></td><td><span class="tag change">CHANGE</span> bsdtar-based install</td></tr>
<tr><td><span class="muted">reference</span></td><td><code>freebsd-livecd-unionfs/ramdisk/init.sh</code>, <code>.../build.sh</code></td><td><span class="muted">read-only</span></td></tr>
</tbody>
</table>
<footer>
Implementation plan — both <code>unionfs-full-readwrite</code> branches were implemented, tested
on hardware, and merged (~May 2026). See <strong>Status</strong> at the top for the three
unionfs copy issues found along the way: issue #1 changed decision D2 (<code>bsdtar</code> →
<code>find | cpio</code>), and issue #2 — the avahi copy-up corruption — shipped as a known
unfixed defect with the fix identified. Decisions D1, D3, D4 hold as written; D2's copy
<em>method</em> is the part that changed.
</footer>
</div>
</body>
</html>