Fix serial console attachment silently disabled by wrong error check

[jlagedo]

Summary

newVZFileHandleSerialPortAttachment checked the error out-parameter pointer instead of the returned file handle, which silently disabled the file-handle serial console.

NSFileHandle *fileHandleForReading = newFileHandleDupFd(readFileDescriptor, error);
if (error != nil) {   // <-- always true: `error` is void** out-param (&caller's var)
    return nil;
}

Because callers always pass the address of their error variable (&nserrPtr in serial_console.go), error != nil is always true, so the function returned nil before ever building the attachment. On the Go side this produced a non-nil *FileHandleSerialPortAttachment wrapping a NULL Objective-C object with a nil error — a silent failure. The guest's serial console (console=hvc0) had nowhere to write.

Fix

Check the duplicated handle returned by newFileHandleDupFd, which returns nil and sets *error only on dup() failure. This is the same pattern already used by newVZFileHandleNetworkDeviceAttachment (virtualization_11.m) and newVZDiskBlockDeviceStorageDeviceAttachment (virtualization_14.m).

NSFileHandle *fileHandleForReading = newFileHandleDupFd(readFileDescriptor, error);
if (fileHandleForReading == nil) {
    return nil;
}

This also aligns with Apple's Cocoa convention: success/failure is signaled by the return value, and the NSError out-parameter is only consulted on failure.

Why existing tests didn't catch it

The bootable-VM integration test (TestRun) talks to the guest over vsock/SSH, not the serial console. A serial port with a nil attachment is valid config, so Validate() and Start() succeed; only the kernel boot log was lost, and nothing asserted on it. Added a focused unit test that asserts the constructor returns a non-NULL attachment.

Test evidence

Run via make test (codesigns each test binary with the com.apple.security.virtualization entitlement; bare go test aborts with an uncaught NSInvalidArgumentException when creating a VM).

New regression test:

=== RUN   TestNewFileHandleSerialPortAttachment
--- PASS: TestNewFileHandleSerialPortAttachment (0.00s)

Verified the test actually catches the bug — temporarily reverting the checks to if (error != nil) makes it fail:

serial_console_test.go:25: attachment wraps a NULL pointer: constructor reported success but built nothing
--- FAIL: TestNewFileHandleSerialPortAttachment (0.00s)

Real VM boot now shows serial console output (console=hvc0) and completes the full SSH-over-vsock flow:

=== RUN   TestRun
[    0.080305] Run /init as init process
chpasswd: password for 'root' changed
udhcpc: lease of 192.168.64.19 obtained from 192.168.64.1, lease time 3600
Run socat as vsock ssh proxyserver (port=62026)
localhost login: ... container setup done
[    5.067368] reboot: Power down
--- PASS: TestRun (5.21s)

Full suite:

ok  github.com/Code-Hex/vz/v3                    55.311s
ok  github.com/Code-Hex/vz/v3/internal/progress   0.554s
ok  github.com/Code-Hex/vz/v3/internal/sliceutil  0.348s

(TestRunIssue124 SKIP — gated behind TEST_ISSUE_124=1.)

Local environment

macOS	26.5 (build 25F71)
Hardware	Apple M4 (arm64)
Go	go1.26.3 darwin/arm64
Toolchain	Apple clang 21.0.0 (clang-2100.1.1.101)
Test kernel	puipui-linux v1.0.3 (aarch64)

Test plan

• make test passes (full suite, incl. VM boot tests)
• New regression test passes with the fix
• New regression test fails when the bug is reintroduced
• go vet ./... clean

[cfergeau]

Not sure this comment is required once the bug is fixed, there’s a test case to prevent regressions, and git blame will also give an explanation.
Apart from this, looks good to me, great catch!

[jlagedo]

Done, comment removed in d77a0dd — thanks for the review!

Funny story: I wasn't even hunting this bug. I was trying to debug a boot error and turned on the serial console to see what was happening… except nothing ever showed up. Turns out the console was being silently disabled the whole time by this broken error check. So I went looking for my boot bug and accidentally found a different one. 😅