[jerryfane]

Closes #74. Assignment: owner "okay start the work go" (2026-10-22), on phobos-dev's owner-approved request in #74.

What

-fc-balloon (Firecracker driver, off by default) makes running VMs give memory back to the host:

1. Balloon with free page reporting. Each VM's vm.json gets balloon: {amount_mib: 0, deflate_on_oom: true, stats_polling_interval_s: 0, free_page_reporting: true}. The guest kernel reports freed pages; Firecracker MADV_DONTNEEDs them. Target 0: the host never takes memory the guest is using.
2. Idle reclaim in the guest agent. The guest is told on its command line (sandboxd.balloon=1). PID 1 samples /proc/stat every 5 s; after 30 s under 5% of one CPU it syncs, writes drop_caches=3 and compact_memory=1, once per quiet period.

Why (2): measured, free page reporting alone returned only ~90 MiB of ~1 GiB. Most of what a busy Phobos guest holds is page cache (omp, the whisper model), which is not free memory, so the balloon never sees it.

Without the flag, vm.json and the boot args are byte-identical to today (TestFirecrackerBalloon off-case, TestFirecrackerBootArgs).

Choices

• Daemon flag, not a template key. A per-template key would have to pass through templates, the ledger and the worker protocol (like hardened). Phobos runs its own instance (-fc-instance phobos), so turning the flag on there is per-tenant in practice. The docs say how to scope it: one instance per template set.
• Guest-side idle detection. The guest knows when it is quiet; the host would need guest stats (untrusted) or cgroup CPU plus a root command channel per profile.
• Old images. An image with a v0.4.0 agent ignores sandboxd.balloon=1 and gets only (1). The Phobos image builder takes its agent from the release tarball, so (2) reaches Phobos after the next release and an image rebuild.

Evidence (real KVM, fleet host, scratch instance b74, live services untouched)

Phobos image phobos-omp-amd64-42437b9ac771 with this PR's agent installed into a scratch copy. 1536 MiB guest, 2 vCPUs. Workload = phobos-dev's probe: omp --help, then whisper.cpp small-q8_0 on 60 s of audio, twice, then idle. No manual cache drop.

	VMM RSS after work	VMM RSS idle +50 s	guest used	whisper, 2nd round
flag off	1137 MiB	1137 MiB (until the VM ends)	202 MiB	19.2 s
flag on	1138 MiB	270 MiB	129 MiB	20.0 s

Second round with the flag on: 861 → 263 MiB. Balloon alone (stock v0.4.0 agent): 1138 → 1036 MiB.

Isolation re-probe, flag on, from inside the guest: 1.1.1.1:443 open; 10/8, 192.168/16, 172.16/12, 100.100.100.100, 169.254.169.254, the gateway's ports, and IPv6 are all blocked. The virtio_balloon driver is bound. The guest user cannot write drop_caches.

Instruments and raw output: /root/fleet-tools/state/sandboxd-74-balloon/ (probe-1..3.txt, isolation-on.txt, run-probe.sh, probe-memory.py, isolation.py).

Tests

• TestFirecrackerBalloon: the balloon device and boot arg are present exactly when the flag is on, with target 0 and reporting on.
• TestIdleReclaim: one reclaim per quiet period, the 5% threshold boundary, and a reset on activity.
• TestParseCPUBusy: guest/guest_nice are not double-counted, idle/iowait excluded, malformed input rejected.

test-check prove: inconclusive (old_code_build_failed) for all four. Expected for a new feature: the tests use new symbols. Manual mutants instead, each red:

• free_page_reporting: false → TestFirecrackerBalloon FAIL
• never resetting done on activity → TestIdleReclaim FAIL
• counting guest/guest_nice twice → TestParseCPUBusy FAIL

--pieces lists the flag wiring and the PID 1 goroutine as not_tested. Both are exercised by the KVM run above, where the flag-on numbers depend on them.

go vet ./..., gofmt, go test ./... pass locally.

[jerryfane]

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e0eeb5d82d7e31-1

Decision: approved
Summary: Reviewed PR #75 at exact head fde82d7 (1 commit vs base 5586940, 9 files +300/-6). No prior verdicts exist for PR #75 (GITMOOT_PRIOR_VERDICTS covers PRs 12-73; the PR-59/60/71/72/73 findings reviewed are in files untouched by this diff). (1) Security: the balloon config {amount_mib:0, deflate_on_oom:true, stats_polling_interval_s:0, free_page_reporting:true} is verified field-for-field against Firecracker v1.17.0's BalloonDeviceConfig (serde deny_unknown_fields; free_page_hinting correctly absent). amount_mib=0 means the host never inflates; free page reporting only discards ranges inside the guest's own memory (discard_range on guest mem, device.rs:618-637); cfg.Balloon is read only at firecracker.go:811,843 — jailer args, cgroups, nft, claims untouched. The reclaim is guest-local: PID 1 already runs as guest root, and a compromised guest could always write drop_caches itself or stay busy to defeat reclaim — no new host trust. Reporting (not hinting) is the safe choice. (2) Correctness: IdleReclaim fires exactly once after 30 s of <5%-of-one-CPU quiet; boundary is '>' so exactly 5% is quiet (tested); ParseCPUBusy sums user/nice/system/irq/softirq/steal and correctly skips idle/iowait/guest/guest_nice — verified against this host's real /proc/stat. Post-reclaim cpuBusy re-read correctly excludes reclaim CPU from the next interval. (3) Flag-off: scratch probe proved vm.json byte-identical to the base-revision marshal for plain (857 B) and hardened+volume (1096 B) specs; boot args unchanged. (4) PID 1: idleReclaim cannot panic and every error path is log+return inside the goroutine; halt() unreachable; spawned only in initGuest, never in serve/envd children. One P3 nit below. Could not run: real-VM KVM boot (needs /var/lib/sandboxd-fc images, sandbox-denied) and -race (cgo blocked); PR-body KVM measurements not re-verified.
Severity: P3

Findings

• idleReclaim comment overstates retry: a /proc/stat failure ends reclaim for the VM's life (P3)
  • detail: The doc comment says 'a failure is logged and retried at the next quiet period', but a cpuBusy() read or parse error makes the goroutine return permanently (reclaim_linux.go:26-27,36-37,53-54); only the drop_caches/compact_memory write failures are actually retried. Harmless direction — reclaim is advisory and the guest can always keep it from firing — and /proc/stat failure is near-impossible since initGuest always mounts proc, but either retry on read errors or fix the comment.
  • file: cmd/sandboxd-guest-agent/reclaim_linux.go
  • id: Firecracker: -fc-balloon returns idle guest memory to the host (#74) #75-f1
  • line: 26

Tests Run

• CGO_ENABLED=0 go build ./... — pass (plain go build fails in sandbox: /usr/include/stdc-predef.h unreadable, cgo blocked)
• CGO_ENABLED=0 go vet ./... — pass
• CGO_ENABLED=0 go test ./... — pass (control, envd, firewall, guestagent, helpersvc, store, vm, worker, cmd/sandboxd all ok)
• go test ./internal/vm -run TestFirecrackerBalloon — pass (balloon key absent/present, exact fields, boot-arg presence both ways)
• scratch-copy probe TestProbeFlagOffByteIdentical — PASS: flag-off vm.json byte-identical to base-revision marshal for plain (857B) and hardened+volume (1096B) specs
• scratch-copy probe TestProbeBalloonOnShape — PASS: balloon-on vm.json has exactly the six expected top-level keys; boot args end with ' sandboxd.balloon=1'
• scratch-copy probe TestProbeRealProcStat — PASS: ParseCPUBusy parses real /proc/stat; cumulative busy monotone; delta 570ms over 120ms wall on 12 CPUs
• scratch-copy probe TestProbeIdleReclaimEdges — PASS: zero-elapsed tick resets quiet period; fire lands exactly at step 6; done blocks refire until a busy interval; 5%-exact boundary counts as quiet
• static check: Firecracker v1.17.0 vmm_config/balloon.rs + device.rs fetched — all four emitted keys valid under deny_unknown_fields; reporting path discards ranges within guest memory only
• NOT RUN: SANDBOXD_FC_KVM boot test (needs /var/lib/sandboxd-fc images — sandbox-denied); go test -race (cgo toolchain blocked)

Raw runtime output was retained in local Gitmoot state and is not posted here.