Docs · 1.0

Configuration: Device Passthrough and Hardware

This chapter covers three pages under the Configuration menu: PCI passthrough (give a whole device to one VM), SR-IOV (split a network port or an integrated GPU into several virtual functions for several VMs) and Hardware (what the machine is made of, and how healthy each part is). For the storage and network pages see Storage and Network.

Note: Changing passthrough settings needs host write permission (an admin token). Read-only accounts can only look.

1. How passthrough works

• PCI passthrough: a whole device (NVMe, SATA/SAS controller, network port, graphics card, USB controller...) goes to one VM. The VM uses the device's native driver and the host no longer touches it. It suits a NAS (hand over the controller together with its disks) and graphics or network cards that need native performance.
• SR-IOV: a network port that supports it (and some integrated GPUs) can be split into several VFs (virtual functions). Each VF goes to one VM, and the physical function (PF) stays with the host.
• keelOS identifies a device by the device itself (vendor ID, device ID, serial number, function number), not by slot. Move the card to another slot and it is still found, and the system disk is never handed over by mistake. Devices the host is using (the controller holding the boot disk, the disks of a storage pool, the management port...) cannot be kept for passthrough.

A device can be used by only one running VM at a time. You may list the same device in the settings of several VMs, but the one that starts first gets it, and the others cannot start while it runs.

2. The PCI passthrough page

Open Configuration → PCI passthrough.

At the top is the IOMMU (VT-d) status (on, off, or "no passthrough device has used it yet"). Without an IOMMU there is no passthrough: turn on VT-d (IOMMU on AMD) in the BIOS, see Hardware requirements. Next come two lines, Kept since this boot and Kept from the next boot/reload. When they differ, the page says "keel.json was changed: takes effect at the next root reload or boot".

The device table has one row per PCI function, with the columns Address, Device, Driver / state, Host uses, Guest (the VM that uses or has configured it) and Isolation. Chipset and system devices and VFs are hidden by default; tick Show chipset and system devices or Show VFs to list them.

What Driver / state means:

• Host: driver name: a host driver is using it, and it can be kept. Network cards and storage controllers the host does not recognise show not supported (no host driver), can be passed through to a VM: the host cannot use them, but you can give the whole device to a VM.
• kept (keelppt, since boot): set aside for passthrough at boot, so no host driver can touch it.
• passthrough (ppt): already handed over to passthrough right now.
• kept: from the next boot or reload, given back: from the next boot or reload: the setting is saved but not in effect yet.

The Host uses column tells you why a device cannot be kept, for example: the host's address, a switch uplink, a bridge member, the boot partition, a storage pool, open VFs, the host screen.

The Isolation column warns you that several ports of one card, or devices behind the same PCIe bridge that has no ACS, are not isolated from each other by the IOMMU. By Linux VFIO rules they form one group: give them to the same VM together, or leave them all to the host. keel only warns, it does not enforce.

2.1 Keep a device

1. Click Keep on the device's row.
2. In the dialog, fill in Name (VM settings refer to the device by it; a default is filled in) and Note.
3. Tick Now to hand the device to passthrough right away. The host driver lets go immediately, so every disk behind it must be closed (keel checks). If you leave it unticked, the device is taken over at the next root reload or boot.
4. Save.

The setting is written to keel.json and the boot partition. keelOS applies it on every root reload (upgrade, restart of the base layer) and every firmware reboot.

Note: After you hand the graphics device that drives the host screen (the integrated GPU) to a VM, the host's own monitor no longer shows the host until the whole machine is rebooted. The web page and ssh keep working. For the integrated GPU see §5.

2.2 Give a device back to the host

Click Give back to the host. If a VM's settings still list the device, remove it on that VM's edit page first. With Now ticked, the host driver takes the device back immediately. keel can name the driver itself for NVMe, AHCI and USB controllers; other classes wait for the next boot or reload.

2.3 Add a device to a VM

1. Click Add to a guest… and choose Add it to which guest?.
2. The page jumps to that VM's edit page with the device already added in Devices, in the PCI devices box. After you save, it takes effect at the VM's next cold start.

You can also do this directly on the VM's edit page: Devices → PCI devices, pick a kept device and click Add. If nothing is kept yet, the page says there is no device set aside for passthrough, with a link back to this page.

2.4 Passthrough, host upgrades and reboots

When the host does a root reload (upgrade, restart of keelOS's base layer), a VM with a whole device passed through is suspended and resumed, and its programs keep running. A VM that uses a VF is by default shut down and started again (set At a host reload under Custom per VM), see Virtual machines. The shares of a NAS VM stop for 20-odd seconds during a reload; clients ride it out by retransmitting.

3. The SR-IOV page

Open Configuration → SR-IOV. The host driver manages the physical function (PF); keel creates the virtual functions (VFs) and passes them to VMs, with a fixed MAC address per VF. If no device supports SR-IOV, the page says so.

Each physical port has a row with Network port, Device, Link, Use (what the host uses it for) and the VF count (created N / at most M). Click Set VFs… to change the count. Expand the row to see every VF's MAC, VLAN, Guest and Network.

Set VFs… (dialog "VFs of {port}"):

1. Enter a number. 0 deletes all VFs of this port; you cannot delete VFs that a VM's settings still list.
2. Save. keel rebuilds the port's VFs at once with iovctl. No VM may be using a VF of this port at that moment.
3. If the host uses this port itself (its address, a switch uplink), rebuilding cuts the host off on this port for about 30 seconds, and keel asks you first. When the port renegotiates its link, STP on the switch may also keep the network down for about 30 seconds.
4. Some ports have shown "Failed to add VF", and changing the VF count while running can panic the host. In that case the new count is only saved and takes effect after a reboot through firmware; keel asks whether to reboot now (running VMs are shut down first, and VMs set to autostart come back after the reboot).

The VF count of ix ports (X5xx/X7xx series) is limited by queues. Without a reboot you get at most N. Click Make the setting (after a restart) to reduce the PF queues of all the host's ix ports from 4 to 2, so that more VFs can be created after the reboot; Undo the setting reverts it.

Cards such as the I350 and 82576 can create VFs only when they run under the driver that ships with keelOS. If the page says the driver cannot create VFs, the port is not under that driver right now.

To use a VF in a VM, choose the connection type SR-IOV VF on the VM's network card, or Automatic (uses a VF when the VM is shut down at a reload and the network has a free VF), see Virtual machines. You can also name a VF, or "auto", under PCI devices. To pass a whole network port, choose whole port; the host can then no longer use that port.

4. The Hardware page

Open Configuration → Hardware. This page is mostly read-only and lists what the machine is made of. The data is read once and cached for 30 seconds. Copy as text at the top right copies the whole page as text, which is handy when you report a problem. If anything Needs attention (a disk SMART failure, a sensor out of range, a dead CMOS battery, an NVMe critical warning, memory errors), it is listed first.

From top to bottom:

• System: Manufacturer, Model, Serial, Board, BIOS, UUID.
• Processors: for each socket, the model, core and thread count and maximum frequency.
• Memory: the slots in use, and each module's size, type, speed, whether it is ECC, and its memory errors. Corrected errors mean ECC fixed them; an occasional one is normal, but a constant stream means that module is failing. Uncorrected errors mean the module must be replaced. When errors are recorded you can expand the latest 50.
• Virtualization: CPU vendor, Extensions, IOMMU, Hypervisor layer. If the host is not running keelOS's own hypervisor layer ("native mode"), VMs still run, but every VM must be shut down before a host reboot or upgrade, and there is no memory pool or zero-page reclaim. The reason (for example VT-x or AMD-V is off in the firmware) is shown here; turn it on in the BIOS and reboot.
• Disks: each disk's model, serial, Health (the SMART verdict), Power-on time, Life used, Reallocated/pending sectors and Media errors. Click a disk to see everything it reports. NVMe disks are listed by controller, with their namespaces indented below.
• Check: check a disk before you pass it through, install on it or put data on it. It runs the disk's own short self-test (a few minutes) and a read speed test (about 1 minute, read-only, nothing on the disk is changed). The result is Normal, Attention or Problem, and the history is kept. If the disk is in use, VMs read and write more slowly during the test.
• Format…: on an NVMe disk you can change the LBA format (512-byte or 4Kn sectors). Everything on the disk is erased. The boot disk and disks in use cannot be formatted. On controllers that support namespace management you can also create and delete namespaces here.
• Network ports: each port's MAC, Link state, speed and chip.
• Temperature: sensors the host reads itself (CPU, ACPI thermal zones, PCH). A warning appears when one is within 15 °C of its throttling point. The curves are on the Monitor sub-page of the Overview.
• Memory bandwidth: the current read and write traffic of each memory channel (from the memory controller's performance counters; keel does not touch channels that another tool is using).
• Intel AMT (vPro): on machines that support it, shows AMT's status, version, address and remote functions. In 1.0 keel only reads AMT's status; it cannot turn AMT on for you. To enable it, press Ctrl-P at boot to enter MEBx and set it up by hand; the "how to enable" line on the page has the steps. AMT can control the whole machine from outside the operating system, so use a strong password and never forward AMT's ports to the internet.
• BMC (IPMI): on machines with a BMC, shows its management address, firmware, power draw, Sensors and the BMC event log (the latest 20 entries). Here you can:
• Edit: change the BMC's network (static address or DHCP, netmask, gateway, VLAN, IPv6), its behaviour After power loss (Always power on, Restore the previous state, Stay off) and the Fan mode (Supermicro only for now; the modes listed are the ones this BMC offers). After you change the address or VLAN, the BMC's own web page may be unreachable for a few seconds to tens of seconds. You cannot be locked out, because keel talks to the BMC directly from inside the host and needs no BMC password.
• Users…: manage the BMC's users. If you forgot the BMC password, set a new one here (only the admin token * can).
• Fan thresholds…: after fitting slower, quieter fans, lower the threshold below which the BMC decides a fan is "broken" and runs everything at full speed.
• Take over…: check the BMC settings a previous owner left behind: enabled users, network, SNMP community, alert destinations, event log.
• UID LED: switch the blue identification light on the chassis on or off, to find the machine in a server room.
• Restart BMC…: use it when the BMC does not answer from inside the host. While it restarts (1-2 minutes), the BMC's web page, remote console and fan control pause. The host and VMs are not affected.

Parts a machine does not have (no BMC, no AMT) are not shown. Disks and network ports passed through to a VM are managed by that VM and are not listed here.

5. Integrated graphics

Pass the whole integrated GPU to one VM: keep the integrated GPU (usually 00:02.0) on the PCI passthrough page, then add it to the VM. Windows and Linux VMs can use it for rendering and hardware transcoding with Intel's native driver. If a monitor is plugged into the integrated GPU, the monitor shows that VM's screen, and the host's own screen is no longer shown (the web page and ssh keep working).

• On Intel integrated GPUs of the 12th generation and later (UHD 770/730, Iris Xe and so on), a Linux VM needs xe.enable_dsb=0 in its kernel parameters (i915.enable_dsb=0 with the i915 driver). Otherwise every screen change waits for a 10-second timeout.
• For a VM with the integrated GPU passed through, the web graphical console shows only the GPU's picture, at about 10 frames per second. For a smooth picture use RDP, see Console and remote desktop.
• A SeaBIOS VM (old systems such as Windows 7) with a passthrough device does not start on some hosts. For integrated GPU passthrough, use a UEFI VM.

Split the integrated GPU into VFs for several VMs (12th generation and later, up to 7 VFs): the PF driver of the integrated GPU runs in keelOS's display VM. First keep the integrated GPU for passthrough on the PCI passthrough page and reboot the host through firmware; then set the VF count of the GPU on the SR-IOV page (that row says "PF driver = display VM (xe): changing the VF count restarts it and resets the GPU"). The VF a VM gets has no display output: in Windows it is an Intel graphics card with "no monitor attached", used for rendering and transcoding, and you watch the picture over RDP (run keelagent in the guest). This has been tested only on a few models (12th-generation UHD 770), so try it on an unimportant VM first.

6. What 1.0 does not support yet

• Passed-through devices cannot be hot-plugged: adding or removing one takes effect at the VM's next cold start.
• SR-IOV covers network ports and the integrated GPU only; other kinds of VF are not handled.
• Old systems (Windows XP, Windows 7) use SeaBIOS, which does not start on some hosts when combined with passthrough devices.
• AMT is read-only and cannot be enabled from the web page; BMC fan modes work on Supermicro only.
• A nested-virtualization VM cannot have PCI devices passed through at the same time, see Nested virtualization.