Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1089,15 +1089,15 @@ occurrence wins (`crashkernel=`) takes the user's value. `root=`,
ltvm's values, and rc.local configures the guest from `fc_*`. Fixed at
create time like the disks; `cluster create` applies it to every node.

**Nested KVM on Apple Silicon:** a Linux host on an Apple CPU (`CPU
implementer : 0x61`) is a VM under Apple Virtualization -- UTM, M3 or
later, nested virtualization on -- and `qemu_machine_for_arch` gives its
KVM guests `its=off` (`vm_state._nested_kvm_on_apple`). With the ITS,
virtio-pci MSIs lose wakeups there and the guest idles in `cpu_do_idle`
forever a few seconds after init; wired INTx works. It changes nothing
on any other host, macOS's HVF included. KVM is 3-40x faster than TCG
there; the benchmarks, against TCG and against ltvm on the Mac itself,
are in [docs/NESTED_VIRTUALIZATION.md](docs/NESTED_VIRTUALIZATION.md).
**Nested KVM on Apple Silicon:** on macOS, `qemu_run` checks the host's
EL2 capability and the selected QEMU binary at boot. Native Arm HVF
guests get EL2 on macOS 15+ with compatible hardware and QEMU 11.1+.
`create --disable-nesting` saves `DISABLE_NESTING=1` in the VM's
`.info` file. Inside an Apple-hosted Linux VM, `qemu_machine_for_arch`
gives KVM guests `its=off` (`vm_state._nested_kvm_on_apple`). This
uses wired INTx to avoid a virtio interrupt hang observed under
UTM's Apple Virtualization backend. See the setup and benchmarks in
[docs/NESTED_VIRTUALIZATION.md](docs/NESTED_VIRTUALIZATION.md).

**Naming:** `<hh><N>-<role>`, where `<hh>` is the first two consonants
of the host's short name (`bl` on buildhost, `ts` on testnode) and N is unused
Expand Down
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,11 @@ a podman-managed Linux VM + QEMU microvms. A few things differ from Linux:
ends), and `export LTVM_PODMAN_AUTOSTOP=0` keeps it up for good.
A machine that lists as running but does not answer -- a failed
start leaves it so -- is restarted by the next build.
- **To confine an agent to a VM**, run ltvm inside an aarch64 Linux VM
on the Mac instead: UTM on the Apple Virtualization backend with
nested virtualization on, which needs an M3 or later and macOS 15+.
ltvm then gets KVM there; see
- **Native Arm guests can run KVM by default** on a Mac with hardware
EL2 support, macOS 15+, and QEMU 11.1+ (included in `./ltvm install`).
Pass `--disable-nesting` at create time to opt out. To confine an agent
to a Linux VM, run ltvm inside that guest or inside a UTM Linux guest
with nested virtualization enabled on the Apple Virtualization backend. See
[docs/NESTED_VIRTUALIZATION.md](docs/NESTED_VIRTUALIZATION.md#apple-silicon----aarch64).

```bash
Expand Down
45 changes: 36 additions & 9 deletions docs/NESTED_VIRTUALIZATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,9 +153,36 @@ gcloud compute instances update <name> \

### Apple Silicon -- aarch64

ltvm runs on macOS itself, with HVF. To run it inside a Linux VM on
the Mac instead -- so an agent driving ltvm is confined to that VM --
the Linux VM needs nested virtualization, and that needs all of:
#### Expose KVM in a VM created by ltvm

ltvm exposes Arm EL2 to native aarch64 guests by default on compatible
Apple Silicon (M3 or later) with macOS 15+ and QEMU 11.1+. A Linux guest
kernel with KVM enabled then provides `/dev/kvm`. On a capable Mac,
`ltvm doctor` flags an older QEMU. `ltvm doctor --fix` or
`ltvm install --qemu` installs the signed Apple Silicon QEMU bundle
from [ltvm's GitHub releases](https://github.com/lustre-tools/lustre-test-vms/releases).

```bash
ltvm create nested-host --target rocky10 --arch aarch64 \
--mem 6144 --vcpus 4 --root-size 16G --mdt-disks 0 --ost-disks 0
ltvm vm exec nested-host -- ls -l /dev/kvm
ltvm vm exec nested-host -- dmesg | grep -E 'started at EL2|kvm \['
```

ltvm checks support at each cold boot, including for existing VMs.
On an older host or QEMU, the VM boots with ordinary HVF. To opt out,
pass `--disable-nesting` to `ltvm create` or `ltvm cluster create`.
ltvm saves the opt-out for each VM. QEMU suppresses Arm SME when it
exposes EL2 through HVF, so opt out for workloads that need SME.

The example allocates space for inner VMs and their images.
For an inner QEMU, use
`-accel kvm -cpu host -machine virt,gic-version=max,its=off`.

#### Run ltvm inside a Linux VM on the Mac

Use the Linux guest above, or use UTM to create the outer VM. For the
UTM path, use:

- an **M3 or later** chip (M1 and M2 have no nested virtualization);
- **macOS 15** or later;
Expand All @@ -176,13 +203,13 @@ sudo usermod -aG kvm $USER # if you are not in the kvm
it: `accel=kvm` and `-cpu host`. `ps -o args -C qemu-system-aarch64`
shows what a running VM got.

**ltvm turns off the GIC ITS here.** On an Apple CPU (`CPU
**ltvm turns off the GIC ITS for the inner VM.** On an Apple CPU (`CPU
implementer : 0x61` in `/proc/cpuinfo`), ltvm launches KVM guests with
`virt,...,its=off`. With the ITS, virtio-pci MSIs lose wakeups under
this nesting: the guest boots past init, then sits in `cpu_do_idle`
for good while its vCPU thread shows 100% CPU. Without the ITS,
virtio falls back to wired INTx interrupts, which work. Trapping WFI
in the L1 KVM does not help, so it is the MSI delivery itself.
`virt,...,its=off`. This avoids a hang observed under UTM's Apple
Virtualization backend. With the ITS, virtio-pci MSIs lose wakeups and
the guest sits in `cpu_do_idle` while its vCPU thread shows 100% CPU.
Without the ITS, virtio falls back to wired INTx interrupts.
Trapping WFI in the L1 KVM does not help.
`LTVM_FORCE_TCG=1` still forces emulation, if you ever need it.

Parallels Desktop and VMware Fusion have not been tested with ltvm.
Expand Down
13 changes: 11 additions & 2 deletions ltvm
Original file line number Diff line number Diff line change
Expand Up @@ -1595,7 +1595,16 @@ def build_parser() -> argparse.ArgumentParser:
target_parser.set_defaults(func=_target_no_action)

# -- VM lifecycle: create --
_vm_create_args = argparse.ArgumentParser(add_help=False)
_nesting_args = argparse.ArgumentParser(add_help=False)
_nesting_args.add_argument(
"--disable-nesting",
action="store_true",
help="Keep Arm EL2 unavailable to guests on macOS HVF. "
"Fixed at create time.",
)
_vm_create_args = argparse.ArgumentParser(
add_help=False, parents=[_nesting_args]
)
_vm_create_args.add_argument(
"--vcpus", type=int, default=2, help="vCPU count (default: 2)"
)
Expand Down Expand Up @@ -2568,7 +2577,7 @@ def build_parser() -> argparse.ArgumentParser:

ccp = cluster_sub.add_parser(
"create",
parents=[common],
parents=[common, _nesting_args],
help="Create a cluster from node specs (needs root)",
)
ccp.add_argument("name", help="Cluster name (e.g. co2)")
Expand Down
1 change: 1 addition & 0 deletions ltvm_pkg/cli/cluster.py
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@ def cmd_cluster_create(args: argparse.Namespace) -> int:
eth0_speed=args.eth0_speed,
nic=list(args.nic or []),
kernel_args=args.kernel_args,
disable_nesting=getattr(args, "disable_nesting", False),
numa_nodes=args.numa_nodes,
numa_dist=args.numa_dist,
numa_dist_file=args.numa_dist_file,
Expand Down
54 changes: 51 additions & 3 deletions ltvm_pkg/qemu_run.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import functools
import math
import os
import platform
import pwd
import re
import secrets
Expand Down Expand Up @@ -51,6 +52,40 @@ def die(msg: str, code: int = EXIT_ERROR) -> NoReturn:
sys.exit(code)


def hvf_el2_supported() -> bool:
"""Return whether this Apple Silicon host can expose Arm EL2."""
if not is_macos() or platform.machine() != "arm64":
return False

import ctypes

try:
hvf = ctypes.CDLL(
"/System/Library/Frameworks/Hypervisor.framework/Hypervisor"
)
query = hvf.hv_vm_config_get_el2_supported
except (OSError, AttributeError):
return False
query.argtypes = [ctypes.POINTER(ctypes.c_bool)]
query.restype = ctypes.c_int
supported = ctypes.c_bool()
return query(ctypes.byref(supported)) == 0 and supported.value


def qemu_supports_hvf_vgic(binary: str) -> bool:
"""QEMU 11.1 introduced both the HVF platform GIC and EL2 support."""
try:
result = run([binary, "--version"], check=True, timeout=5)
except (OSError, subprocess.SubprocessError):
return False
version = re.search(r"\bversion (\d+)\.(\d+)\.(\d+)\b", result.stdout)
return version is not None and tuple(map(int, version.groups())) >= (
11,
1,
0,
)


def _read_meminfo_mb(key: str) -> int:
"""Return /proc/meminfo's <key> value in MiB, or 0 if unreadable.

Expand Down Expand Up @@ -1294,11 +1329,23 @@ def _start_qemu(vm: VMInfo) -> None:
f"{str(_nic)!r} on VM {vm.name!r}"
)

import platform as _platform

arch = vm.arch
qemu_bin = qemu_binary_for_arch(arch)
machine = qemu_machine_for_arch(arch)
accel_args = []
if arch == "aarch64" and "accel=hvf" in machine:
macos_major = int(platform.mac_ver()[0].partition(".")[0] or 0)
if macos_major < 15:
if qemu_supports_hvf_vgic(qemu_bin):
# Newer QEMU defaults to a GIC that needs macOS 15.
machine = machine.replace(",accel=hvf", "")
accel_args = ["-accel", "hvf,kernel-irqchip=off"]
elif (
not vm.disable_nesting
and hvf_el2_supported()
and qemu_supports_hvf_vgic(qemu_bin)
):
machine += ",virtualization=on"

# q35 (x86) and virt (aarch64) both have a PCI bus, so virtio
# devices attach as virtio-*-pci. Previously x86 used microvm and
Expand All @@ -1317,7 +1364,7 @@ def _start_qemu(vm: VMInfo) -> None:
# those, so /sbin/init aborts with "Fatal glibc error: CPU does not
# support x86-64-v2" and the kernel panics. Nehalem (Intel 2008) is
# the baseline CPU model that satisfies v2 in full.
host_arch = _platform.machine()
host_arch = platform.machine()
import os as _os

_force_tcg = _os.environ.get("LTVM_FORCE_TCG") == "1"
Expand Down Expand Up @@ -1345,6 +1392,7 @@ def _start_qemu(vm: VMInfo) -> None:
qemu_bin,
"-name",
vm.name,
*accel_args,
"-machine",
machine,
"-cpu",
Expand Down
9 changes: 9 additions & 0 deletions ltvm_pkg/vm_cluster.py
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,7 @@ def _create_one_node(
disk_cache: str = "",
disk_prealloc: str = "",
eth0_speed: int | None = None,
disable_nesting: bool = False,
) -> tuple[str, int, str]:
"""Create a single cluster VM via ltvm subprocess.

Expand Down Expand Up @@ -679,6 +680,8 @@ def _create_one_node(
if kernel_args:
# One argv word, so a value starting with '-' is not an option.
cmd += [f"--kernel-args={kernel_args}"]
if disable_nesting:
cmd.append("--disable-nesting")
if numa_nodes:
cmd += ["--numa-nodes", str(numa_nodes)]
if numa_dist:
Expand Down Expand Up @@ -880,6 +883,7 @@ def _print_cluster_plan(
disk_dir: str = "",
disk_cache: str = "",
disk_prealloc: str = "",
disable_nesting: bool = False,
) -> None:
"""Report what `cluster create` would do, for --dry-run.

Expand Down Expand Up @@ -930,6 +934,8 @@ def _print_cluster_plan(
print(f" nics: eth0 (mgmt) + {', '.join(nics)}")
if kernel_args:
print(f" cmdline: + {kernel_args}")
if disable_nesting:
print(" nesting: disabled for every node (macOS HVF)")
print(f" owner: {owner_id}")
print("Nothing was written. Re-run without --dry-run to create it.")

Expand Down Expand Up @@ -1140,6 +1146,7 @@ def cmd_cluster_create(args: argparse.Namespace) -> None:
die(network_issue)
nics = [str(n) for n in nic_specs]
kernel_args = _validate_kernel_args(getattr(args, "kernel_args", None))
disable_nesting = getattr(args, "disable_nesting", False)
wait_seconds: int = getattr(args, "wait", 0)
numa_nodes: int = getattr(args, "numa_nodes", 0) or 0
problem = nic_numa_problem(nic_specs, numa_nodes)
Expand Down Expand Up @@ -1202,6 +1209,7 @@ def cmd_cluster_create(args: argparse.Namespace) -> None:
nics=nics,
kernel_args=kernel_args,
owner_id=owner_id,
disable_nesting=disable_nesting,
**disk_opts,
)
return
Expand Down Expand Up @@ -1256,6 +1264,7 @@ def create_wave(wave: list[ClusterNode]) -> list[str]:
variant="base" if node_targets[node.name] else variant,
share_disks=node.name in shared,
eth0_speed=getattr(args, "eth0_speed", None),
disable_nesting=disable_nesting,
**disk_opts,
): node
for node in wave
Expand Down
38 changes: 38 additions & 0 deletions ltvm_pkg/vm_commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,12 @@
from .qemu_run import (
die,
host_node_problem,
hvf_el2_supported,
is_running,
kill_qemu,
launch_qemu,
numa_problem,
qemu_supports_hvf_vgic,
run,
)
from .vm_net import (
Expand Down Expand Up @@ -85,6 +87,7 @@
drop_orphan_clusters,
lustre_libdir,
nic_ip6,
qemu_binary_for_arch,
resolve_os_artifacts,
)

Expand Down Expand Up @@ -708,6 +711,7 @@ def _validate_kernel_args(raw: str | None) -> str:
"disk_size": "--disk-size",
"root_size": "--root-size",
"kernel_args": "--kernel-args",
"disable_nesting": "--disable-nesting",
"numa_nodes": "--numa-nodes",
"numa_dist": "--numa-dist",
"disk_dir": "--disk-dir",
Expand Down Expand Up @@ -950,6 +954,7 @@ def _allocate_and_persist_vm(
os_id=os_id,
kver=kver,
arch=os_arts.arch,
disable_nesting=getattr(args, "disable_nesting", False),
# Track the invoking human. SUDO_USER preserves the legacy
# ``sudo ltvm create`` path; the unified privilege model runs
# the command as the user and elevates only individual writes.
Expand Down Expand Up @@ -1057,6 +1062,7 @@ def _print_create_plan(
"target": os_target,
"variant": variant,
"arch": getattr(args, "arch", None) or os_arch_of(image),
"disable_nesting": getattr(args, "disable_nesting", False),
"kernel": kernel,
"kernel_version": kver,
"image": image,
Expand Down Expand Up @@ -1096,6 +1102,8 @@ def _print_create_plan(
print(f" {kernel}")
print(f" image: {image}")
print(f" cpu/mem: {args.vcpus} vcpus, {args.mem} MB")
if plan["disable_nesting"]:
print(" nesting: disabled (macOS HVF)")
if getattr(args, "numa_nodes", 0):
print(f" numa: {args.numa_nodes} nodes")
if getattr(args, "numa_dist", ""):
Expand Down Expand Up @@ -4005,6 +4013,33 @@ def _unreadable_vm_taps(name: str) -> set[str]:
return taps_for_vm(name, count)


def _check_macos_nesting_qemu(fix: bool) -> tuple[int, int]:
if not hvf_el2_supported():
return 0, 0
binary = qemu_binary_for_arch("aarch64")
if qemu_supports_hvf_vgic(binary):
return 0, 0

print(
f"nested virtualization needs QEMU 11.1 or newer at {binary}; "
"run `ltvm install --qemu` to install ltvm's macOS bundle from GitHub"
)
if not fix:
return 1, 0

from .host_setup import install_qemu_macos_vhost

try:
install_qemu_macos_vhost()
if not qemu_supports_hvf_vgic(qemu_binary_for_arch("aarch64")):
raise RuntimeError("selected QEMU is still missing or too old")
except (RuntimeError, OSError, subprocess.CalledProcessError) as exc:
print(f" FAILED to update macOS QEMU: {exc}")
return 1, 1
print(" fixed: installed QEMU with nesting support")
return 1, 0


def cmd_doctor(args: argparse.Namespace) -> int:
issues = 0
# Counts repairs that were attempted and did not work, so --fix
Expand Down Expand Up @@ -4293,6 +4328,9 @@ def cmd_doctor(args: argparse.Namespace) -> int:

if is_macos():
print("note: `ltvm target export` needs a Linux host; not checked")
found, failed = _check_macos_nesting_qemu(args.fix)
issues += found
fix_failures += failed
else:
from ltvm_pkg import image_export

Expand Down
Loading
Loading