Memory that is not wasted
What was found
Before choosing a method, we measured. On a host with 17 VMs and 44 GB of guest memory, after a day of running, 49% of guest memory was zero pages, and 35% came in whole 2 MiB chunks of zeros. Kernels clear pages they free, and much of a VM's memory is never touched after boot.
A zero chunk has no content to keep. It needs no compression, no swap and no driver in the guest: to give it back, hand over a fresh zeroed chunk.
How it works
- Every 5 minutes a low-priority thread scans guest memory in 2 MiB chunks. Chunks that are zero in two scans in a row are unmapped from the guest.
- If the guest touches such a chunk, it is mapped back at once.
- Reclaimed chunks become an L2ARC cache device for the host's ZFS pool. When the guest wants a chunk back, the host waits for any cache I/O on it, zeroes it (about 0.1 ms) and maps it back. ZFS checks every L2ARC read against its checksum, so a chunk that was taken back just means one more disk read.
The memory still belongs to the VM. Nothing is lent to other VMs, so there is no overcommit, and there is never a moment when memory cannot be given back.
What was measured
- 7 hours on a development host: 14.9 GB reclaimed, 34% of guest memory; about 5% of reclaimed chunks were asked back.
- Correctness on hardware: guests wrote zeros and random data, waited for two scans and read it back, across a host restart. Every byte matched.
- A random-read benchmark on the host with a 4 GB ARC: 10.7k IOPS without the loaned cache, 16.7k IOPS with 6 GB lent.
Limits
- No overcommit. Using reclaimed memory for more VMs waits for weeks of real data on how often it is asked back.
- VMs with passthrough devices are left out, since devices write to their memory directly.
- The gain is small when the VM disks are on fast NVMe and the ARC already hits most reads; it is meant for pools on hard disks. Disks behind a passed-through controller do not go through the host's cache.
- Chunks that the guest only reads could share one zero page instead of being taken back; not done yet.
- The settings are tunable on an installed system; a web page for them is not done yet.