SR: Phase 1: decouple VM templating from SaveVM, RestoreVM by Camelron · Pull Request #537 · microsoft/kata-containers

Cameron E Baird (Camelron) · 2026-08-24T17:27:30Z

In its current form, the only use of Snapshot/Restore api for a given hypervisor is for VM templating, configured via the BootToBeTemplate and BootFromTemplate configs.

Replace these hypervisor-level configs with a new FileBackedMemory abstraction so that other usecases for SaveVM/RestoreVM can use it.

Here we update the CLH api to expose memory_restore_mode in restore PUTs.

Also wire up a new hypervisor.clh config which exposes this setting in the kata config.

Expose one more hypervisor.clh config enable_virtio_mem so that we can easily turn off Virtio-Mem resizable zones, which is incompatible with the mode copyonwrite.

Cameron E Baird (Camelron) · 2026-08-24T17:29:21Z

Two notes for the reviewers,

Checked in here for review is the AI design document, describing phases 1-4. See stacked PRs for the remaining phases. I will drop the document from the branch before final merge.
I had to cherry-pick in Sudipta's rs VM templating work into msft-main. It is not present until we set msft-main to the 4.1 rebase Saul Paredes (@Redent0r)

Copilot

Pull request overview

Decouples runtime-rs VM snapshot/restore operations from templating and adds Cloud Hypervisor factory support.

Changes:

Introduces generic file-backed memory and explicit save/restore APIs.
Adds CLH restore modes, network FD handoff, and lifecycle handling.
Extends factory tests, configuration, and design documentation.

Reviewed changes

Copilot reviewed 34 out of 35 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`tests/integration/kubernetes/runtimeclass_workloads/busybox-vm-templating.yaml`	Adds templating workload fixture.
`tests/integration/kubernetes/k8s-vm-templating.bats`	Extends factory testing to runtime-rs CLH.
`src/runtime-rs/crates/runtimes/virt_container/src/sandbox.rs`	Uses explicit restore and deferred networking.
`src/runtime-rs/crates/runtimes/virt_container/src/lib.rs`	Configures clone memory backing.
`src/runtime-rs/crates/runtimes/virt_container/src/factory/vm.rs`	Adds CLH factory and path-based save support.
`src/runtime-rs/crates/runtimes/virt_container/src/factory/template.rs`	Generalizes template creation and cleanup.
`src/runtime-rs/crates/runtimes/virt_container/src/factory/mod.rs`	Selects hypervisor-specific state artifacts.
`src/runtime-rs/crates/resource/src/rootfs/erofs_rootfs.rs`	Propagates block queue settings.
`src/runtime-rs/crates/hypervisor/src/remote/mod.rs`	Returns explicit unsupported snapshot errors.
`src/runtime-rs/crates/hypervisor/src/remote/inner.rs`	Removes snapshot stub.
`src/runtime-rs/crates/hypervisor/src/qemu/mod.rs`	Implements generalized save/restore methods.
`src/runtime-rs/crates/hypervisor/src/qemu/inner.rs`	Decouples QEMU migration from template flags.
`src/runtime-rs/crates/hypervisor/src/qemu/cmdline_generator.rs`	Adds generic file-backed memory arguments.
`src/runtime-rs/crates/hypervisor/src/lib.rs`	Extends the hypervisor snapshot contract.
`src/runtime-rs/crates/hypervisor/src/firecracker/mod.rs`	Reports unsupported snapshot operations.
`src/runtime-rs/crates/hypervisor/src/firecracker/inner_hypervisor.rs`	Removes obsolete save stub.
`src/runtime-rs/crates/hypervisor/src/dragonball/mod.rs`	Reports unsupported snapshot operations.
`src/runtime-rs/crates/hypervisor/src/dragonball/inner_hypervisor.rs`	Removes obsolete save stub.
`src/runtime-rs/crates/hypervisor/src/ch/mod.rs`	Exposes CLH snapshot lifecycle operations.
`src/runtime-rs/crates/hypervisor/src/ch/inner_hypervisor.rs`	Implements CLH save, paused restore, and state handling.
`src/runtime-rs/crates/hypervisor/src/ch/inner_device.rs`	Aligns vsock CID typing.
`src/runtime-rs/crates/hypervisor/README.md`	Documents the updated trait.
`src/runtime-rs/crates/hypervisor/ch-config/src/lib.rs`	Aligns CLH API model types.
`src/runtime-rs/crates/hypervisor/ch-config/src/errors.rs`	Adds file-backed memory errors.
`src/runtime-rs/crates/hypervisor/ch-config/src/convert.rs`	Converts file-backed memory and wider CPU types.
`src/runtime-rs/crates/hypervisor/ch-config/src/ch_api.rs`	Adds snapshot, restore, and FD-aware API calls.
`src/runtime-rs/crates/hypervisor/ch-config/Cargo.toml`	Adds test dependency.
`src/runtime-rs/config/configuration-clh-runtime-rs.toml.in`	Exposes CLH restore and factory settings.
`src/runtime-rs/config/configuration-clh-azure-runtime-rs.toml.in`	Mirrors settings for Azure CLH.
`src/libs/kata-types/src/config/hypervisor/mod.rs`	Adds restore mode and file-backed memory types.
`src/libs/kata-types/src/config/hypervisor/ch.rs`	Validates CLH restore configuration.
`src/libs/kata-types/src/capabilities.rs`	Adds network-hotplug capability.
`RUNTIME-RS-SNAPSHOT-RESTORE-PORT-DESIGN.md`	Documents the phased runtime-rs design.
`Cargo.lock`	Records the test dependency.
`.github/workflows/run-k8s-tests-on-free-runner.yaml`	Adds runtime-rs CLH templating CI.

Suppressed comments (1)

src/runtime-rs/crates/hypervisor/src/qemu/inner.rs:575

The caller-provided destination is interpolated into an exec: migration URI without shell escaping. Paths containing spaces fail, while shell metacharacters can execute arbitrary commands as the shim during snapshot. Use QEMU's non-shell file transport or rigorously encode/escape the destination.

        let state_path = snapshot_dir.join("state");
        let uri = format!("exec:cat >{}", state_path.display());

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Cameron E Baird (Camelron) · 2026-08-24T18:19:58Z

+fn validate_memory_restore_config(ch: &super::Hypervisor) -> Result<()> {
+    if ch.memory_info.memory_restore_mode == MemoryRestoreMode::CopyOnWrite
+        && ch.memory_info.enable_virtio_mem
+    {


In its current state, I have chosen not to port the full VM templating memory resize feature. We expect to not use it in its current state for snapshot/restore as memory hotplug is costly.

Makes sense. Just posting it here for reference but CloudHypervisor's resize disallows it when memory is provided as user defined zones.

But only for Acpi, yes? I was able to get zone resizing working in golang with virtio-mem.

Harshit Gupta (harshitgupta1337)

Overall looks good. Just some minor improvements.

Aurélien Bombo (sprt) · 2026-09-14T21:58:28Z

+# Cloud Hypervisor memory population strategy for VM snapshot restore:
+#   - copy
+#   - ondemand
+#   - copyonwrite
+memory_restore_mode = "copyonwrite"
+
+# Enable virtio-mem for guest memory resize. This cannot be combined with
+# memory_restore_mode = "copyonwrite".
+enable_virtio_mem = false
+


Do we ever want to support anything else than copyonwrite? Or is having these configs helpful for testing?

If we keep, would be nice to explain what enable_virtio_mem does.

memory_restore_mode is a helpful tool for testing. We could instead opt to hard-code kata to use copyonwrite, the only practical option.

ondemand is also viable for users of kvm. Mshv does not currently support it.

Regarding enable_virtio_mem, you are right to call it out as it isn't fully implemented yet. To support virtio_mem-based resizing is a hard problem we aren't attempting to solve here. So we can:

Remove enable_virtio_mem in this change. We'll add it back later when we need it.

Keep enable_virtio_mem and the related compatibility check against memory_restore_mode. It doesn't materially affect the VM in any way until we actually implement it.

Ack that memory_restore_mode is helpful for testing, let's leave it then.

But I would remove enable_virtio_mem from the config if not implemented, confusing otherwise.

nit: adding dashes would be more readable: on-demand, copy-on-write

Dropped enable_virtio_mem in the latest push.

Oops, missed the second half, the nit. I think it is okay as-is given I've already force-pushed my branches?

In its current form, the only use of Snapshot/Restore api for a given hypervisor is for VM templating, configured via the BootToBeTemplate and BootFromTemplate configs. Replace these hypervisor-level configs with a new FileBackedMemory abstraction so that other usecases for SaveVM/RestoreVM can use it. Here we update the CLH api to expose memory_restore_mode in restore PUTs. Also wire up a new hypervisor.clh config which exposes this setting in the kata config. Expose one more hypervisor.clh config enable_virtio_mem so that we can easily turn off Virtio-Mem resizable zones, which is incompatible with the mode copyonwrite. Signed-off-by: Cameron Baird <cameronbaird@microsoft.com> Assisted-by: Sol 5.6

Copilot AI balanced review requested due to automatic review settings August 24, 2026 17:27

Copilot started reviewing on behalf of Cameron E Baird (Camelron) August 24, 2026 17:28 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes