keelOS next to KVM
Results
| Test | Unit | keelOS | KVM | keelOS / KVM |
|---|---|---|---|---|
| 7-Zip benchmark | MIPS (higher is better) | 23,251 | 24,533 | 95% |
| sysbench CPU | events/s (higher is better) | 4,773 | 4,905 | 97% |
| sysbench memory | MiB/s (higher is better) | 5,203 | 5,182 | 100% |
| Build Redis (make -j4) | s (lower is better) | 41.42 | 39.96 | 96% |
| Process wake-ups (perf sched pipe) | s (lower is better) | 4.598 | 3.579 | 78% |
| 4K random read, QD32 | IOPS (higher is better) | 42,724 | 79,368 | 54% |
| 4K random write, QD32 | IOPS (higher is better) | 12,579 | 36,366 | 35% |
| 4K random read, QD1 | IOPS (higher is better) | 5,461 | 6,456 | 85% |
| 4K random read, QD1, p99 latency | us (lower is better) | 264 | 264 | 100% |
| 1M sequential read, QD8 | MB/s (higher is better) | 1,509 | 1,970 | 77% |
| 1M sequential write, QD8 | MB/s (higher is better) | 1,810 | 1,906 | 95% |
| VM → another machine (1 GbE port) | Mbit/s (higher is better) | 941 | 941 | 100% |
| Another machine → VM (1 GbE port) | Mbit/s (higher is better) | 940 | 941 | 100% |
| VM → VM on the same host, 1 stream | Mbit/s (higher is better) | 7,579 | 31,070 | 24% |
| VM ← VM on the same host, 1 stream | Mbit/s (higher is better) | 7,473 | 30,519 | 24% |
| VM → VM on the same host, 4 streams | Mbit/s (higher is better) | 6,837 | 26,991 | 25% |
| VM ← VM on the same host, 4 streams | Mbit/s (higher is better) | 6,983 | 27,018 | 26% |
Setup
- Server: Supermicro X11SPW, Xeon Gold 5122 (4 cores, 8 threads), 16 GB, two Samsung PM1725a NVMe drives. Why this one and not the server behind the numbers on the home page (2 × Xeon Gold 6144): the comparison needs the disks wiped and Proxmox VE installed, and this is the one of our two servers we can wipe; the other one runs our long tests.
- VM: Debian 13, 4 vCPU, 4 GiB, virtio-blk and virtio-net. The same disk image on the same ZFS volumes (volblocksize 16K, no compression), the same MAC address, the whole test disk written once before.
- keelOS: development build 89cdaaf (FreeBSD 14.5, bhyve). Proxmox VE 9.2: kernel 7.0, QEMU 11.0, ZFS 2.4;
--cpu host, diskscache=none,aio=io_uring, iothread; vhost-net. The same ZFS cache limit on both; no other VM running. - Storage on one NVMe drive on both sides (Linux could not read the second drive's protection information).
- Each test three times; the table shows the median and all three runs are in the raw data.
Why some numbers are behind
- Between VMs on the same host. KVM moves packets in the kernel (vhost-net) and passes large segments between VMs. keelOS's virtio network runs in bhyve's user-space process. The physical 1 GbE port is full on both, so this only shows between VMs on one host. keelOS still moves about 7.5 Gbit/s there, more than a 1 or 2.5 GbE network, so a router VM in front of a NAS VM does not cap a LAN client. Until the faster path is done, give both VMs an SR-IOV virtual function of the same network card and the card switches their traffic in hardware.
- Random disk I/O with many requests in flight. A single request is as fast as on KVM (same p99 latency at QD1); at QD32 bhyve's virtio disk keeps fewer requests in flight than QEMU's iothread with io_uring.
- Process wake-ups. Waking a process on another vCPU costs more under keelOS's hypervisor; it shows in
perf sched pipeand a little in the compile and compression tests.
What we are doing
Multi-queue and asynchronous I/O for the virtio disk and a faster path for virtio network are being worked on now. Every release is measured again on the same server, and this page shows the latest run.