Docs · 1.0

Virtual machines

This chapter covers the day-to-day management of virtual machines: the list page, creating one, the console, power operations, changing settings, snapshots and clones, starting automatically at boot, growing a disk and deleting. At the end it lists what 1.0 does not do yet. If this is your first virtual machine, start with Creating your first virtual machine; this chapter goes into each step in more detail.

1. The virtual machine list

Open Virtual machines. You land on the List page, which has three sub-pages:

• List: all virtual machines, one per row, showing name, state, operating system, CPU, memory, IP, disks and the console buttons.
• Screens (F1–F12): only appears when a monitor is attached to the host and the local screen feature is on. The host's own monitor and keyboard switch between the host and the virtual machines with Ctrl-Alt-F1 … F12; this page assigns a virtual machine to each key, and also has the info screen (clock, photo frame, weather, video). See Console and remote desktop §4.
• Autostart: see §7.

What you can do on the list:

• Search: the search box finds by name, UUID, system, IP, MAC and note.
• Filter by state: All, Running, Stopped, Needs attention (failed, broken configuration, or machines that carry a notice). Each button shows a count.
• Sort: click a column title to sort, click again to reverse.
• Expand a row: click the blank part of a row to see the UUID, each disk, CD drive, network card (which network it is on, MAC and the IP the host sees), passthrough devices, and the Clone button.
• Click the name to open the machine's detail page.
• Each row also says which engine the virtual machine runs on (Runs on: FreeBSD or Linux partition; for old systems see §2.1), whether it starts with the host, what happens to it at a host reload, and any migration in progress (migrating to …).
• If there are virtual machines coming in from other hosts, or ones that have been migrated away but whose copy is still kept, a migration area appears below the list: Coming in shows progress, and Migrated away (…; kept for a rollback), once expanded, has a Roll back button after each machine that brings it back from the target host (the copy over there must be shut down first, and is kept there for 24 hours; changes made over there after the migration are not brought back). Migrations are started by the manager (keelMGR).

Your choices of list, filter and sort are remembered in this browser. kcells (application containers) are listed separately in the kcell group below; bhyve virtual machines that are running on the host but were not created by keel are also listed, marked not managed by keel (read-only).

The detail page at a glance

Click a name to open the detail page. From top to bottom:

• The top: the name, Configuration (shows config.json), Log, Edit, Clone, Move to another pool, and the power buttons (§4). When there is a problem, notice bars appear below: N changes take effect at the next restart (§5.1), Waiting for the console (the template wants the console open before the machine powers on; click Open the console), the network a NIC is on does not exist, a machine migrated from another host whose passthrough devices do not match, configuration problems (with a Fix button).
• The console buttons and a small picture of the guest's screen (§3).
• An information table: UUID, State, System (the system keel recognised and why), CPU and memory, Firmware, TPM, Nested virtualization, Sound card, NUMA, Disks, Ports, Depends on and Needed by (§7), RDP, Pool, dataset, and for each NIC: model, which network it is on, MAC, and the IP the host sees (when there is one), with a row below showing this NIC's Firewall (§5.5).
• Moves between pools (while a move is in progress or just finished).
• USB devices (when USB redirection is on) and Screen recording: see Console and remote desktop.
• Performance, last hour: this machine's CPU use (as a percentage of all its vCPUs), Memory (as the host sees it) (the memory the guest has touched, sampled every 5 minutes), Disk reads and writes, and Network receive and send. Some devices have no counters on the host (file-backed disks, VALE ports, VFs or passed-through devices), and the page says so. Running machines also have a Memory: zero pages chart.
• Snapshots (§6).
• Delete at the bottom (§11).

2. Creating a virtual machine

Click New virtual machine at the top right of the list to start a wizard. The row of steps at the top changes with the template; usually they are System, Resources, Network, Options and Confirm. You can click Back at any time, or Other template to go back to picking a template; if you leave the page before finishing, what you filled in is kept in this browser tab.

2.1 Picking a template

Templates are arranged by category: Linux, Windows, BSD, Router / firewall, NAS and Other. Open a category and choose the specific system (under the Windows and Linux categories there is another level for Desktop, Server and distributions). You can also just use the search box at the top. Next to each system it says tested or untested, and how much memory and disk it needs at least.

A template holds the settings the system needs: firmware (UEFI or BIOS), disk and network card models, whether a TPM is needed, how the display is configured and so on. To configure everything yourself, choose Custom at the end (set everything by hand, or write config.json directly). Custom templates holds template files you imported from another keelOS; you can Export and Delete them.

Note: for old systems such as Windows XP, 2003 and 7, choose their own templates: they use SeaBIOS (the Firmware line of the detail page says BIOS (SeaBIOS)) and the drivers built into the system. Even older systems such as DOS and Windows 9x are not supported in 1.0.

2.2 Where the system comes from

System from has three choices, and the template decides which are offered:

• Installer ISO: pick from the list, Upload from this computer, or paste a download address and Download from URL (the host downloads it itself, so you can leave the page). ISOs are kept in the storage pool and shared by all virtual machines; see Storage.
• Disk image (as the system disk): an already-installed system image (such as router firmware) used as the first disk. .img is used as it is; .qcow2, .vmdk and .dmg are converted to .img automatically by keel when uploaded. Compressed archives must be unpacked first.
• Nothing yet (network boot, or later): the virtual machine just has empty disks.

If you choose an installer ISO but pick no file, the wizard warns that the hard disk is empty and that you must arrange the installation yourself, and continues only after you confirm.

2.3 Resources: name, CPU, memory, disks

• Name: letters, digits, ., _ and -; it is used later in the list, the logs and RDP (for example win11).
• CPU: the number of vCPUs, 1 to 64.
• Memory: in GiB, not below the template's minimum, otherwise the form turns red.
• Disks: the template creates a new disk whose size you can change; + disk adds another. Disks live in a ZFS storage pool; keel creates a dataset for each virtual machine, with the disks, UEFI variables and logs inside.

The wizard has no CPU core type option. If you need one, set it on the edit page after creating the machine; see §5.3.

2.4 Network

For each network card you choose which network (virtual switch) it connects to. The three drop-downs are, in order: which network (or a particular switch), how it is attached, and the card model:

• Automatic: a virtual NIC; but if the virtual machine is shut down at a host reload and the network has a free SR-IOV VF, a VF is used.
• Virtual NIC, SR-IOV VF, Whole port: explicit choices. With a VF or a whole port you do not pick a model; the guest uses the real card's driver.
• Model: virtio-net (fast; Linux has the driver built in, Windows needs the virtio-win driver), e1000 (drivers built into Windows and BSD), pcnet.

If the host has no network at all yet, this step tells you to create one on the Network page; when you come back, what you filled in is still there. The several ports of a router template cannot be attached to the same network. For how to create networks see Network.

2.5 Options

• The template's own questions, for example TPM 2.0 (vTPM) for Windows.
• Screen: the graphical console only exists if VNC is on; Display size (Automatic is the same size as the host's own monitor, or 1024 × 768 when no monitor is attached); wait for the console at the next start (the machine is powered on only after the console is open, so you can see prompts that last only a few seconds, like "Press any key to boot from CD or DVD"; it applies once, and is ticked by default in Windows templates).
• Access password (optional): the RDP password; empty = the admin password, and the user name is the virtual machine's name.
• Starts with the host: starts this virtual machine automatically after the host boots; see §7.
• Nested virtualization: see Nested virtualization; it is greyed out when the host does not support it, with the reason next to it.

2.6 Confirm

The last step lists all your choices. Start it now is ticked by default. To see or change the configuration handed to keel, expand Advanced: config.json, where you can change settings the form does not have. Click Create; the virtual machine is created and the page goes straight to its detail page, while it starts in the background.

3. Console

While a virtual machine is running, each row of the list and the top of the detail page have console buttons:

• Screen (VNC): the virtual machine's screen, as if sitting at a real machine; keyboard and mouse work directly. Only machines with VNC turned on have it.
• Serial console: the virtual machine's serial port (text). Useful for Linux, routers and BSD when the kernel and login terminal are on the serial port.

Each console can be opened in three ways: opens in its own window (click the button itself), in this page, or floating (always on top, and can be dragged outside the browser; needs browser support). When the browser blocks the pop-up window, it automatically opens in this page instead. The buttons in the list are icons with tooltips.

The top left of the detail page also has a small picture of the virtual machine's screen, refreshed every 5 seconds; click it to open the VNC console.

Besides the web page, you can also connect to a virtual machine's screen over RDP: the RDP row of the detail page gives the address, port and user name, and Download .rdp file gives a file you double-click to connect with Windows Remote Desktop (mstsc); you see it from the boot screen onward and nothing needs installing in the guest. keel's own client kRDPclient adds video encoding, USB redirection and sound on top of that. For the serial console, RDP, USB redirection, screen recording and the host's monitor see Console and remote desktop.

4. Starting, stopping and other power operations

Each list row and the top right of the detail page have three buttons:

The result of an operation appears as a notice bar on the page.

The states are: running, stopped, suspended, failed and broken config. For a failed machine the detail page gives the reason, and after you fix the settings you can click Start again.

Suspend: when the host does a reload (an upgrade, a slot switch, a restart of keelOS's lower layer), running virtual machines are by default suspended and resumed, pausing for about 17-35 seconds without the programs inside restarting. Machines using an SR-IOV VF are by default shut down and started again instead; for each virtual machine you can set At a host reload under Custom to suspend and resume (default) or Shut down. See Upgrade and restart for details. For a machine that fails to resume, the Start button becomes Start fresh: it boots from scratch and the suspended checkpoint is set aside and kept.

The 1.0 web page has no "suspend", "resume" or "reset" buttons. The command line does: keel vm suspend ID, keel vm resume ID, keel vm reset ID (reset = force stop immediately and start again; the CD drive is kept, and changes waiting for the next restart are applied). See The keel command line.

5. Changing settings

Click Edit on the detail page to get a form divided into General, CPU and memory, Storage, Network, Display, Devices, Firmware and Remote console. On the right is config.json, in which the lines you changed are marked against the original configuration; for settings the form does not have, click Edit JSON and change them directly. Changes take effect after Save.

5.1 Running virtual machines

While the virtual machine is running, each item has a tag saying when it takes effect:

• now: the name, starts with the host, the RDP password and Console video, switching networks ... the guest does not stop.
• at restart: CPU, memory, CD-ROM, display, firmware, devices and so on. "Restart" means the guest restarting itself, or shutting down and starting again; resuming from suspend does not count.
• partly now: network cards and disks. Switching to another network or switch takes effect immediately and the link stays up (if the subnet changed, renew DHCP inside the guest); the model, adding or removing network cards, and adding or removing disks take effect at the next restart.

After you save changes that wait for a restart, a yellow notice appears on the detail page, N changes take effect at the next restart, listing each one, with two buttons: Discard changes and Restart now (shut down, then start). If settings were changed at the same time from another page, saving is refused and you are told to reopen the edit page.

5.2 Common changes

• Adding or changing an ISO: + ISO under Storage, choosing from the ISO library. After installing the system, remove the ISO (click ✕) so the next start boots from the disk.
• Adding disks and network cards: + disk, + NIC. A new disk appears after the next restart.
• Booting a BIOS virtual machine from CD: Next start: from the ISO first under Firmware, used once.
• Updating the firmware: the Firmware row of the detail page. After keel's UEFI firmware is updated, old virtual machines switch to it at their next start; machines with a vTPM keep the old firmware, because changing firmware makes BitLocker ask for the recovery key. When you want to switch, have the recovery key ready and click Update firmware while the virtual machine is shut down.
• Sound card, USB redirection, nested virtualization, PCI devices: under Devices. For PCI passthrough see Device passthrough.

5.3 CPU core types

Under CPU and memory there is a Cores drop-down: Any (big cores first), Performance (big cores first) and Efficiency (small cores first). It only has an effect on CPUs with big and small cores (P-cores and E-cores): each vCPU is pinned to cores of the chosen kind, and if there are not enough big cores the extra ones go to small cores (and the reverse for efficiency). On other CPUs this setting has no effect for now. It takes effect after the virtual machine is restarted.

The NUMA row of the detail page (only on multi-socket hosts) says which node this machine's vCPUs and memory are on; the Overview → Monitor page has a CPU map of the whole host.

5.4 Sections of the edit page

From top to bottom:

• General: Name; Runs on (as the template says (…), FreeBSD (bhyve) or Linux partition (QEMU): old systems bhyve cannot run, for example Windows NT 4, run in a Linux partition, which is managed under Configuration → Partitions; when a partition is chosen, the network cards attach to the partition's own bridge); Starts with the host; Depends on (§7).
• CPU and memory: CPU (number of vCPUs), Cores (§5.3), Memory, Nested virtualization.
• Storage: disks (+ disk; existing disks can only be enlarged) and CD drives (+ ISO). For a stopped virtual machine there is Write it over the system disk next to the system disk: it writes the selected disk image over the first disk, and everything previously on that disk is erased (only available once it is shut down).
• Network: for each network card choose the network (or switch), how it is attached and the model (§2.4); + NIC; Make or change a network (this form is kept) jumps to the Network page. When the network a card is on no longer exists, that row is marked (does not exist), and the virtual machine cannot start until you switch to an existing one.
• Display: Screen (the VNC switch, Display size, wait for the console at the next start).
• Devices: USB redirection (Devices on the RDP client's side), Sound card (Automatic (keel decides), On, Off), Display device (the Cirrus card is for BIOS virtual machines, virtio-vga for UEFI virtual machines, and standard (bhyve framebuffer) suits both) and pointer, Emulated display hidden from the OS (with a passed-through graphics card, keeps the system from treating the emulated card as a second monitor; normally choose Automatic), PCI devices (see Configuration: device passthrough and hardware). The display device, pointer and sound card take effect at the next cold start.
• Firmware: UEFI or BIOS, and Next start: from the ISO first (works for both BIOS and UEFI, applies to the next start only, and an ISO must be attached first).
• Remote console: Access password (optional) (empty = the admin password, user name is the virtual machine's name) and Console video (frame rate, quality and encoder threads used by kRDPclient; see Console and remote desktop).

5.5 NIC firewall

Under each network card on the detail page is a Firewall row, off (everything passes) by default. Click Edit (Show for read-only accounts) to open the Firewall dialog and set filtering for this card alone:

1. Tick Filter this NIC (inbound denied unless allowed below).
2. Allowed inbound, one per line: protocol[/port][@source], for example tcp/22@lan, tcp/8000-8100@any, [email protected]/24, udp/53. The protocol is tcp, udp, icmp (ping) or any; the source is any, lan (the default, which does not include other virtual machines), guests, or an address/CIDR.
3. Outbound: internet only, internet and LAN, or replies only (only answering what others send). Also allowed outbound, one per line lets you allow specific destinations: [protocol[/port]@]destination, where the destination can be keelauth, frontdoor, app:appname, guests, lan or an address/CIDR; names are resolved to addresses when the virtual machine starts.
4. Save. The change takes effect the next time the virtual machine starts; until then it keeps running with the old rules (the dialog says so).

Reply packets for allowed connections pass in both directions; ARP, DHCP and IPv6 neighbour discovery are always allowed. The bottom of the dialog shows the number of inbound and outbound packets blocked since this start, the current number of connections, and the connections blocked most recently. A card with no firewall has no overhead at all. Passed-through cards and VFs do not go through the host's switch, so no firewall is offered for them.

6. Snapshots and clones

The Snapshots card lower on the detail page:

1. Enter a name in Name (the default is snap-date-time), optionally a Note, and click Take snapshot.
2. Each snapshot in the list shows the time, the state of the virtual machine when it was taken, the space it uses, and three buttons: - Roll back: go back to this snapshot. Everything written to disk after the snapshot is lost, and the settings and UEFI variables go back with it; snapshots newer than it are deleted too. While the machine is running you can tick Shut it down cleanly first (else off at once), and then choose Start it afterwards. - Clone from here: create a new virtual machine from this snapshot. - Delete: delete the snapshot. It cannot be deleted while a clone still uses it.

Snapshots are ZFS snapshots: they take almost no space and are taken very quickly. They contain all disks, the configuration and the UEFI variables at the same moment, but not the memory: a snapshot taken while running is like the state after a sudden power cut (crash-consistent). To guarantee consistency for something like a database, shut down inside the guest or stop the service before taking the snapshot.

Clone can also be clicked directly at the top of the detail page or in the expanded row of the list; it clones the virtual machine's "current state" (keel takes a snapshot first). When cloning you enter the new virtual machine's name and choose:

• Linked: finished instantly and shares data blocks with the snapshot; while the clone exists that snapshot cannot be deleted.
• Full copy: independent of the source virtual machine; it takes as long as copying the disks.

The cloned machine has a new UUID, MAC addresses and console ports, does not autostart, has no RDP password, and by default gets a newly generated vTPM (you can tick Copy the vTPM).

Snapshots are kept in the virtual machine's own dataset and are not a backup: if the storage pool is lost, so are the snapshots. For backups and moving to another host see Snapshots, backup and migration.

7. Autostart at boot

The Virtual machines → Autostart sub-page controls which virtual machines start by themselves after the host boots, and in what order:

1. Virtual machines do not autostart by default. Tick Starts with the host on the edit page (or while creating), or click Autostart in the Not started automatically list here.
2. The Start automatically list is arranged in layers: a virtual machine starts only after the machines it depends on (chosen in the Depends on drop-down in General of its edit page) have started and waited long enough; machines in the same layer start together. It says Layer N and after …. Within a layer the list order decides who goes first: drag a row elsewhere to change the order, or drag it to drop here: last below the list to put it last.
3. On each row, Its dependents wait N s: after this machine starts, how long the machines that depend on it must wait. For example, start the router first, wait 30 seconds, then start the other virtual machines.
4. Edit (at the top of the page) opens Autostart settings: After power-up wait N s before the first (to leave time for the storage pools and network) and Start at most N at the same time (0 = no limit). The wait does not hold up the boot; during it keel and the web page are available as usual.
5. Do not autostart removes a machine from the order.
6. The right of each row shows the result of this machine's last autostart.

Dependencies only matter at autostart; they cannot form a cycle (options that would make a cycle are greyed out and say would make a cycle); when a machine you depend on does not autostart, the option is annotated "autostart is off: does not count at boot". The Depends on and Needed by rows of the detail page list which machines this one depends on or is needed by.

Accounts without host write permission can look but not change.

8. Growing a disk

A disk can only be enlarged, not shrunk.

1. Edit, and increase the GiB of that disk under Storage, then Save.
2. virtio-blk and NVMe disks are enlarged immediately, with no downtime; AHCI and IDE disks take effect after the next restart.
3. Then use the new space inside the guest: on Linux something like growpart and resize2fs; on Windows, in Disk Management first Rescan Disks, then extend the volume.

9. Moving to another storage pool

When the host has two or more storage pools, the detail page has a Move to another pool button: the virtual machine's dataset (disks, snapshots, checkpoints, UEFI variables) is copied to the other pool, verified, and the old one deleted. Choose the pool under To pool and click Move. A running virtual machine can be moved too, with a pause of less than a second at the switch-over; pools without enough space are greyed out with the reason.

10. Logs and configuration

Log at the top of the detail page opens the serial, bhyve, supervisor and event logs, and config.json shows this virtual machine's complete current configuration (the RDP password is shown as asterisks). For troubleshooting see Troubleshooting.

11. Deleting

Deletion is permanent: the disks, snapshots, logs and settings all go, and cannot be recovered.

1. First shut down the virtual machine (the delete button is greyed out while it is running).
2. In the Delete card at the bottom of the detail page, click Delete this guest.
3. Type the virtual machine's name to confirm. If the name does not match, nothing is deleted.

Linked clones made from this virtual machine and other references are not affected; ISOs in the ISO library are not deleted.

12. What 1.0 does not support yet

• The web page has no suspend, resume or reset buttons; the command line does (§4).
• No snapshots with memory: a snapshot has only disks, configuration and UEFI variables, and is crash-consistent (§6).
• Disks cannot be shrunk, and an existing disk cannot be removed in the web page (only one just added and not yet saved).
• CPU core types only work on CPUs with big and small cores (§5.3); other CPU pinning options are planned for 2.0.
• DOS and Windows 9x and similarly old systems are not supported; Windows XP is the oldest target.
• Nested virtualization is only available on Intel hosts and cannot be combined with PCI device passthrough; see Nested virtualization.
• When one passed-through device is assigned to several virtual machines, the one that starts first gets it, and the others cannot start while it is running.