ZFS underneath

Every VM lives in its own ZFS dataset: disks, configuration, firmware variables and the firmware itself. One snapshot captures all of it at the same instant.

What you get

One dataset per VM, laid out like SmartOS. Its configuration is a single config.json next to its disks.
Disks are raw files (the default) or zvols, thin-provisioned, with lz4 compression.
The VM sees them as virtio-blk, NVMe or AHCI; older systems can use IDE.
Each VM keeps its own copy of the UEFI firmware, so a restored VM never meets a firmware that changed underneath it.
TRIM works end to end: space the guest frees is returned to the pool.

Snapshots

A snapshot is zfs snapshot -r over the VM's dataset tree. ZFS takes it atomically, so several disks, the configuration and the firmware variables all come from the same moment. It copies no data and the VM keeps running. A snapshot of a running VM is crash-consistent, like pulling the power: modern file systems recover, and databases may replay their logs.

Rolling back is done dataset by dataset, because ZFS has no recursive rollback; rollback -r alone left a zvol unchanged in testing. keelOS rolls back each one.

What was measured

Raw files with a 16K record size perform like zvols: random reads and writes within about 10%.
virtio-blk and NVMe are within about 10% of each other, so NVMe costs nothing.
Record size matters most: with 128K records, 4K random writes drop to about 4,000 IOPS against 10,000–16,000 at 16K.
TRIM: after writing 4 GiB in the guest, deleting it and running fstrim, the disk image went back to under 1.5 MB on the host, for all four combinations of file or zvol and virtio-blk or NVMe.
Snapshot, change data, roll back: both the root disk and an extra zvol came back to the snapshot.

Not done yet

Snapshots are disk-only. Snapshots with memory, which resume exactly where the VM was, are planned; the checkpoint they need already exists for host restarts.
For VMs with passthrough devices, snapshots cover only the disks keelOS manages. The host cannot roll back disks behind a controller it has given away, and a device's state cannot be saved with memory.
Moving a VM to another machine and scheduled backups with zfs send are planned, not written.

Read the article (Chinese) →

← All features