Large scratch regions on KVM with hw-interrupts fail with EEXIST due to APIC access page overlap

[ppenna]

Summary

See nanvix/nanvix#2082 for more context on why this is relevant for Nanvix on Hyperlight.

When a guest configures a large scratch region via SandboxConfiguration::set_scratch_size(), HyperlightVm::new() fails on KVM with EEXIST (Error(17)):

UpdateRegion(MapMemory(Hypervisor(KvmError(Error(17)))))

The root cause is that the scratch memory slot (KVM slot 1) overlaps with an internal KVM memory slot created by create_irq_chip() for the LAPIC/APIC access page at GPA 0xFEE00000.

Details

Scratch region placement

Hyperlight places the scratch region at the top of the 32-bit GPA space via scratch_base_gpa():

// hyperlight_common::layout
pub fn scratch_base_gpa(size: usize) -> u64 {
    (MAX_GPA - size + 1) as u64
}

With MAX_GPA = 0xFFFF_FFFF, a scratch size of e.g. 0x6882000 (~104 MB) yields:

• scratch_base = 0xF977E000
• Scratch KVM slot covers GPA range [0xF977E000, 0xFFFFFFFF]

KVM irqchip APIC access page

When the hw-interrupts feature is enabled, KvmVm::new() calls create_irq_chip(). On Intel hardware with APICv (or on AMD with AVIC), KVM automatically creates an internal APIC access page at GPA 0xFEE00000. This is a non-removable, non-movable memory slot managed internally by KVM.

Since 0xFEE00000 falls inside [0xF977E000, 0xFFFFFFFF], KVM rejects the set_user_memory_region call for the scratch slot with EEXIST — the two regions overlap.

Maximum safe scratch size

The maximum scratch size that avoids the APIC page is:

max_scratch = MAX_GPA - 0xFEE00000 = 0x11FFFFF ≈ 18 MB

Any set_scratch_size() value above ~18 MB will fail on KVM with hw-interrupts enabled on Intel (APICv) or AMD (AVIC) hosts.

Why this does not affect Windows WHP

On Windows, WhpVm::new() does not create an explicit interrupt controller memory slot at a fixed GPA. The WHP API (WHvMapGpaRange2) maps guest physical address ranges independently, and the platform's LAPIC emulation does not reserve a GPA slot that conflicts with user-mapped regions. This is why the same scratch configuration works on Windows.

Why this does not affect small scratch sizes

The default scratch size (DEFAULT_SCRATCH_SIZE = 0x48000 = 288 KB) places scratch at 0xFFFB8000, which is above 0xFEE00000, so there is no overlap.

Reproduction

This can be reproduced with the Nanvix project, which uses Hyperlight as a VMM backend:

1. Clone and checkout branch enhancement-uservm-hyperlight at commit b9c50ed28 (uses Hyperlight rev 4b57b84):

   git clone https://github.com/nanvix/nanvix.git
   cd nanvix
   git checkout enhancement-uservm-hyperlight

2. Build with Hyperlight machine target:

   ./z build -- all MACHINE=hyperlight DEPLOYMENT_MODE=standalone

3. Run the integration test on a machine with KVM and APICv enabled (Intel bare-metal):

   ./bin/mkimage.elf -o nanvix.img \
       "bin/procd.elf;procd" \
       "bin/memd.elf;memd" \
       "bin/testd.elf;testd"
   bash scripts/run-nanvixd.sh hyperlight nanvix.img 120 \
       --wait-for-string "hello, world!"

   On bare-metal Intel with APICv, nanvixd fails immediately with the EEXIST error. On machines with APICv disabled (e.g., WSL2) or on Windows WHP, it succeeds.

   The failing CI run: https://github.com/nanvix/nanvix/actions/runs/24616269717

Proposed solutions

1. Split the scratch KVM slot around the APIC page: When creating the scratch memory mapping on KVM with hw-interrupts, detect whether [scratch_base, scratch_end] contains 0xFEE00000 and split it into two KVM memory slots: [scratch_base, 0xFEDFFFFF] and [0xFEF00000, scratch_end]. The APIC page itself (0xFEE00000–0xFEEFFFFF) would be left for KVM's internal slot. The host-side mmap backing would remain contiguous; only the KVM slot registration would be split.

2. Validate scratch_size against known reserved GPAs: In SandboxMemoryLayout::new(), reject scratch sizes that would cause scratch_base_gpa() to fall below 0xFEE00000 when hw-interrupts is enabled on KVM. This would at least provide a clear error message instead of an opaque KvmError(Error(17)).

3. Document the maximum scratch size constraint: Add a note to SandboxConfiguration::set_scratch_size() and DEFAULT_SCRATCH_SIZE explaining the ~18 MB upper bound on KVM with hw-interrupts.

4. Disable APICv on the host: Consumers running KVM on Intel can work around this by disabling APICv (sudo modprobe kvm_intel enable_apicv=0), which prevents KVM from allocating the APIC access page. This eliminates the overlap but comes at a performance cost — APIC accesses fall back to VM-exit based emulation instead of hardware-accelerated handling. This is a viable short-term workaround but does not require any Hyperlight changes.

Option 1 would be the most flexible, allowing scratch regions of arbitrary size on all platforms. Options 2 and 3 are simpler but limit the usable scratch space. Option 4 is a host-side workaround that does not require any Hyperlight changes.

Environment

• Hyperlight revision: 4b57b8416114c489083922afa3dd9716127278fb
• Features: kvm, hw-interrupts, nanvix-unstable, executable_heap
• Host: bare-metal Intel x86_64, Linux, KVM with APICv enabled
• Fails: Intel bare-metal runners (prometheus28, prometheus30, prometheus43)
• Works: WSL2 (APICv disabled), Windows 11 WHP

[syntactically]

Thanks for pointing this out---this is a good point especially with 32-bit PAs where the address space is a little congested. The absolute simplest quick-and-dirty fix could be to move the whole region below this access page when it's large---do you know if you have enough PA space for that to be practical?

It also looks like one can set the IA32_APIC_BASE msr to move the lapic page---but possibly this also ends up disabling the hardware-accelerated virtual APIC? I'm not sure on i686 whether that is likely to have a significant performance cost.

For a perhaps more general solution, splitting around the APIC page is an option. The main issues with that will be

• The guest scratch allocator needs to be modified not to return the APIC page as an allocation result
• (slightly larger) if, as suggested, the hole for the apic page does not exist on the host, all the host code that figures out how to read from a GPA needs to take account of that. Most of that should be centralised in a couple of functions that check whether a GPA is snapshot or scratch, but there may be some other functions that know they are looking at scratch addresses and will need to be changed to do the translation, including the scratch region bookkeeping updates on snapshot restore.

Pinging @ludfjig @andreiltd who were working on #1385 as well, in case a solution to this should be folded into that.