Docs · 1.0

Storage

This chapter covers storage on the host: what you see on the Storage page, creating a storage pool, adding a second pool, where ISOs live, virtual machine disks, replacing disks, regular data checks, looking at disk health, and finally what 1.0 does not do yet. For creating the first pool see First-time setup.

1. How storage is organized

• keelOS storage is ZFS. A storage pool is made of disks, and virtual machine disks, snapshots and ISOs all live in a pool.
• Installation only takes a small piece at the very start of one disk (keelOS's boot partition). The rest of that disk and the other disks are left to you to turn into storage pools in the web page; see §3. The default pool is called zones.
• When there are several pools, one of them is the default storage pool: newly created virtual machines, the ISO library, software packages and keel's own data (database, logs) go on it; see §5.
• keel creates a dataset for each virtual machine, with its disks, UEFI variables, logs and snapshots inside. keel manages datasets itself; you do not create them in the web page.

Note: ISOs and virtual machines go on the storage pool zones, not in the system partition of the boot disk. The boot partition is small and holds only keelOS itself. So without a storage pool you cannot create virtual machines or upload ISOs.

2. The Storage page

Open Configuration → Storage. From top to bottom the page has:

1. Pool cards: one per storage pool. The title is the pool name (the default pool is marked default) and its state (ONLINE is normal; other colours mean a problem). Below are the free space, a usage bar, N% fragmented, and the pool's device tree: mirror, raidz or single disk, with the size and state of each disk. The default pool's card also has a data check (scrub), see §8. keel's pool cards have buttons: Set as default… (non-default pools), Move guests… (when the pool has virtual machines and there are two or more pools), Export…, Delete… (non-default pools), and when the pool is smaller than its disks also Expand…. If the device tree has a row keelcache0, that is memory reclaimed from guests and lent to ZFS as a read cache; when nothing was lent on this boot it says not lent, and hovering shows why (for example this boot is not on hyp). The storage pool works fine without it, and it comes back by itself next time the conditions are met.
2. The Disks table: one row per disk on the host, with columns Disk, Model, Serial number, Size, Partitions (a coloured bar marking the keelOS boot partition, the part used by pools, other partitions and unused space), Sectors, Type (SSD or HDD), Health and Pool (a disk not in a pool says not in a pool).
3. Guests' disks: how much each disk of each virtual machine takes (used / total), with an extra column Pool when there are two or more pools.
4. ISO images: see §6.

Some cards only appear when they are usable: with no storage pool there is Make the pool; when disks hold pools that were not imported (installed earlier, exported by you, or from another system) there is Pools on the disks, not imported, see §9; a host installed on a boot partition has Copy of the boot partition; and when there are empty disks there is New pool (for making a second pool).

Note: the buttons that change storage (making a pool, attaching, detaching, deleting a pool …) need an account with host write permission; read-only accounts can only look.

Disks on storage controllers keelOS does not recognise are not listed here, and the page says Storage controllers keelOS does not support. You can pass the whole controller through to a virtual machine; see Device passthrough.

3. Creating a storage pool

When there is no storage pool, the page shows Make the pool zones.

1. Open Configuration → Storage.
2. Enter a name in Pool name (zones by default). It starts with a lowercase letter, followed by letters, digits, _, . and -.
3. Tick the space to use: - Free space behind the keelOS boot partition (the page says "N free behind the keelOS boot partition"): choose this when you have only one disk. - Whole disks: one row per disk, with its size, model, and "empty disk" or "on it: …". A whole disk is wiped. - When a disk carries labels of another pool, its row is greyed out and cannot be chosen; first Clear labels or import that pool (see §9).
4. When you have ticked two or more disks, Layout appears:

5. Click Make zones…. A confirmation box lists each disk's model, size, serial number, what is On it now, what it Becomes, as well as the layout and usable about N. Rows for disks with data are red, and to continue you must type the names of those disks as the box says: everything on the red disks (including other pools) is destroyed and cannot be recovered.
6. Click Make zones in the confirmation box. After a few seconds it says Pool zones made.

Pools use 4K alignment and have lz4 compression on; when the pool is made autotrim (space freed on an SSD is passed on to the disk) and autoexpand (the pool can grow when a disk gets bigger) are turned on as well. keel fixes these; there are no switches for them in the web page and you need not care.

Note: do not include in the storage pool disks you plan to pass through whole to a NAS virtual machine.

Note: NVMe disks that have T10-PI (protection information) or a per-block metadata format are formatted to a format without these when keelOS uses them to build a pool, because FreeBSD does not write valid checksum tags, and such a disk could not be read after being moved to Linux or passed through to a Linux virtual machine. The sector format (512, 512e, 4Kn, LBA format) is set per disk under Configuration → Hardware. Any is fine for making a pool; systems that only understand 512-byte sectors cannot use a 4Kn disk.

3.1 Putting a second copy of the boot partition

With only one boot disk, if that disk dies the host cannot start. The Copy of the boot partition card can write another copy of the keelOS boot partition on a second disk (the rest of that disk is left unused, and the disk is wiped):

1. Choose a disk under Disks.
2. If the disk has anything on it, type the disk name to confirm.
3. Click Write the copy.

When adding a disk to a mirror pool you can tick this at the same time; see §4.

4. Adding a disk to a mirror

A single-disk pool can take another disk and become a mirror, and a mirror can take one more for another copy:

1. In the pool card's device tree, find the top-level single disk or mirror row and click Attach….
2. Choose a disk in Attach a disk to …. The choices are the space behind the boot partition or a whole disk (which is wiped, and you type its name to confirm). Disks smaller than the disk being mirrored are greyed out with the size they need to be at least, because a mirror member cannot be smaller than it.
3. For a whole disk you can also tick also put a copy of the keelOS boot partition on it.
4. Click Attach. The pool copies all the data to the new disk (resilvering), and the progress shows in the pool card. Virtual machines keep running during the copy, only the disks are a bit slower.

To take a disk out of a mirror: click Detach on that disk and confirm. The data stays on the remaining disks; with only one disk left there is no second copy. The last data disk in a pool cannot be detached. For a disk in a mirror that was pulled out or has died, the device tree shows missing beside its state, and hovering shows the id ZFS uses; for such a disk also click Detach.

Note: after detaching a disk, if the remaining disk is larger than the old size, the pool immediately grows to the size of the remaining disk (autoexpand), and the confirmation box says about N. Once grown it cannot be shrunk again, and a disk smaller than it can no longer be added as a mirror.

5. Several storage pools

User data can be split across several pools, for example a fast NVMe pool for system disks and a big HDD pool for data. (See docs/75 for the design.)

Making another pool: when there are unused disks, the Storage page shows an extra New pool card, which works like §3; the name defaults to the next free one (zones2, zones3 …).

Default storage pool: on each pool card, a pool that is not the default has Set as default…. After you confirm:

• new virtual machines are placed on it;
• keel's own data (/var/db/keel, /var/log) is moved there right away;
• the ISO library and installed software packages are not moved and stay on the old pool. If you later want to delete the old pool, the ISOs must be uploaded again.

The default pool can also be changed under Default storage pool in System → Maintenance.

Choosing a pool for a new virtual machine: with several pools, the wizard for a new virtual machine has a Storage pool field to choose from, with what each pool has left next to it (… · N left).

Deleting a pool: for a pool that is not the default and has no virtual machines on it, Delete… on the card can be clicked. Type the pool name to confirm; the pool is destroyed, its disks become free, and everything left in it is gone. While there are virtual machines on it the button is greyed out; move them away first.

6. ISO images

The ISO images card at the bottom of the Storage page is the library of installation discs. It lives on the default storage pool and is shared by all virtual machines; when you choose an ISO for a new virtual machine, you choose it from here.

• Upload ISO: upload from the browser. The file name must end in .iso or .img; for .qcow2, .vmdk and .dmg, keel converts them to .img automatically after upload. Upload and conversion both show progress, and ✕ cancels. If a file with the same name already exists it says is already there.
• Download from URL: paste an http or https address and click Download from URL; the host downloads it itself, and you can leave this page during the download and come back to look at the progress.
• On the right of each ISO in the list, it recognises which system it is (from the volume label and file name); if it cannot, it says not recognised, and hovering shows the volume label.
• Delete: click the ✕ next to the name. If the ISO is in a virtual machine's CD drive you are asked whether to Delete it anyway; after deletion that virtual machine must change its CD drive before it can start.

In the new virtual machine wizard you can also Upload ISO directly, and click back to the new guest to continue when done. Uploading and deleting need an account with the "upload and delete images" permission.

Note: an already-installed system image (such as router firmware) used as the first disk does not go through the ISO library; see Virtual machines.

7. Virtual machine disks

• When creating a virtual machine, the disk in the wizard is 20 GiB by default (a template may change it); you can change the size and add more with + disk. This is the size in the wizard, not where the ISO goes.
• The disk type is zvol by default: a ZFS block device with snapshots, clones and growing; file is a file inside the dataset, selectable under Custom and on the edit page, and rarely needed.
• The disk interface (virtio-blk, NVMe, AHCI …) is set by the template; to change it use Storage on the edit page.
• Growing: a disk can only be enlarged, on the edit page under Storage; see Virtual machines.
• How much each disk takes: the Guests' disks table on the Storage page.
• Snapshots are taken on each virtual machine's detail page; see Virtual machines. The Storage page has no per-dataset snapshots. Snapshots live in the pool and are not a backup; for backups see Snapshots, backup and migration.

7.1 Moving to another storage pool

With two or more storage pools, a virtual machine can be moved between pools, even while running: first it is copied while running, then a few rounds of increments catch up, and at the final switch-over the virtual machine pauses for less than a second; after verification the old copy is deleted. (See docs/83 for the design.)

• One virtual machine: Move to another pool… on the detail page, choose the pool under To pool (with the free space next to it; pools too small are greyed out with the reason), and click Move. The dialog shows the stage (copying, catching up, switching, checking, finishing) and progress, and for a running machine it says paused N ms when done.
• A batch of virtual machines: Move guests… on the pool card; tick the ones to move and they are moved one by one. A suspended machine is still suspended after the move; failed or broken-config ones are listed and left alone.
• Moves between pools at the bottom of the Storage page lists the moves in progress and just finished.

Note: the target pool's free space must fit what this virtual machine uses now; pools that cannot hold it are greyed out in the drop-down.

8. Data check (scrub)

The default pool's card has Data check (scrub): it reads all the data in the pool, and where a checksum does not match, repairs it automatically if a good copy exists in the mirror or raidz. Virtual machines keep running, but disks are a bit slower during the check.

• Check now starts it, and Stop stops it. During a check it shows a percentage, … to go, a speed curve, and the Read rate per disk: disks in the same mirror should read about equally fast, and a disk that is always much slower than the others may be about to fail, so look at its health.
• Plan…: for Regular check choose Monthly (the first given weekday), Weekly (a given weekday) or Off, then the At time; you can also set One check at a time in the future. Times follow the host's time zone.
• After a check the card keeps the Last check time, duration, speed and number of errors found.

The default is once a month.

9. Importing old pools and clearing labels

keelOS never imports pools automatically by name at boot, to avoid importing someone else's disks by mistake. When a disk holds a storage pool this host has not imported, the Storage page shows a table Pools on the disks, not imported, one row per pool, marked with its origin: keel's (a keelOS installed earlier), exported by you, or another system's.

Importing:

1. Enter the pool name in Import as (by default the original name). The small text under the row explains what this name does: if this host currently has no storage pool and the default name is used, the pool becomes the default storage pool, keel's own data on it takes over and the original virtual machines come back; otherwise it is a guest storage pool besides the default.
2. Choose the import method: - Import for keel: the pool belongs to keel (tagged keel:zones=1), and the virtual machines on it run here. This host's settings, password and tokens are on the boot partition and are not affected. - Import read-only, as a migration source: for disks from an old system, such as FreeBSD with vm-bhyve. Nothing is written to the pool, it is not mounted in the host's directories (it is mounted under /keel/src), and it is not imported again automatically after a restart; import it again if needed.
3. Click Import. It says Pool … imported.

A pool to be passed through to a virtual machine need not be imported. The time of import and the original name are recorded on the pool, and are in the audit log too.

Exporting: Export… on the pool card, and type the pool name to confirm. What is on the disks is left as it is; the host just no longer imports it:

• After exporting the default pool, keel's own data goes back to root and the host is like just after installation, with no storage pool; the virtual machines and virtual switch configuration on the pool come back on the next import.
• After exporting another pool, the virtual machines on it disappear from this host until it is imported again (possibly under another name).

Migration source: copying vm-bhyve virtual machines over: a pool imported read-only is marked migration source (read-only) on its card and has a Find VMs… button. This pool is never written: no labels, no snapshots, no data, so the old system on the disks can still be used.

1. Click Find VMs…, choose in Dataset the dataset where the old system keeps its virtual machines (one is marked suggested), and click Scan. keel looks only one level deep: a subdirectory <name> containing <name>.conf is a vm-bhyve virtual machine.
2. Tick the virtual machines to copy. Each row has Name / new name, Boot (UEFI machines bring their UEFI variable file; bhyveload and grub machines have no UEFI and start with BIOS, and if that does not work you can change to UEFI in the virtual machine's settings), Memory, Disks (untick disks you do not want), NICs and Notes.
3. Disk models and network card models are carried over unchanged, so the guest sees hardware it has drivers for; switch to virtio after installing drivers. ISO images are not copied; upload them to the ISO library and attach them to the virtual machine. Passthrough devices are not carried over; choose them again on this host.
4. Network mapping: each switch name on the old host maps to one keel network; leave unconnected keeps the NIC but connects it nowhere. Your choices are remembered in this browser.
5. Choose the target pool in Into pool; the page shows the space needed (the target pool keeps a 5% margin, and if it does not fit it says too little room), and click Import. They are copied one after another, each showing progress. Closing the window does not stop ones already started, and while the page is open the rest continue; nothing is started automatically after copying.

When you no longer need this pool: Export… on its card releases it; virtual machines already copied over are not affected.

Clearing labels: if you do not want that un-imported pool any more, click Clear labels… on the disk's row in the Disks table and type the pool name to confirm. After clearing, the data in the pool can never be recovered, neither on this host nor any other. Those disks become ordinary disks and can be used to make a pool. keelOS never clears the disks of un-imported pools by itself; when making a pool these disks are greyed out, and you must import or clear their labels first.

10. Disk health and temperature

• The Health column of the Disks table on the Storage page is the disk's own SMART verdict (available for SATA, SAS and NVMe): OK or Failing. Hover to see the current temperature and the power-on hours. A disk that cannot be read shows –.
• In a pool card's device tree each disk has its ZFS state (ONLINE …). Failed or pulled disks change colour here.
• The Overview page warns you when a storage pool is not ONLINE.
• The temperatures of the host as a whole (CPU, motherboard sensors) and the fans are under Configuration → Hardware; there is also a Host check page for disk and host checks and temperature curves, see Troubleshooting.

Note: replace a disk whose health shows Failing, or one that has changed colour in the device tree, as soon as you can; see the next section.

11. Replacing a failed disk

When a disk in a mirror has died or been pulled, its row in the device tree is in a coloured state, and the old disk says missing. What the 1.0 web page can do is "add first, remove after":

1. Install a new disk (not smaller than the disk being mirrored).
2. Click Attach… on the mirror's row (§4), choose the new disk, and wait for the resilver to finish, until the progress in the pool card is done.
3. Click Detach on the dead disk's row. Only the good disks are left in the pool.

A single-disk pool with only one disk cannot have its data saved by replacing the disk; restore from a backup.

raidz and raidz2 pools: in the 1.0 web page you cannot replace disks yet (you can only add a disk to a mirror or single disk, and take a disk out of a mirror). Use the command line, on the host console or over ssh:

zpool status zones                  # see which disk is bad, note its name
zpool replace zones old-disk new-disk     # e.g. zpool replace zones gpt/zones1 nda3

The resilver progress shows in the pool card on the Storage page, or use zpool status zones. After replacing, wait for it to finish; the pool returns to ONLINE.

Note: replacing a disk on the command line does not write a copy of the keelOS boot partition to the new disk; if the replaced disk had a copy, go back to Copy of the boot partition on the Storage page and write it again.

11.1 Expanding a storage pool

After a smaller disk is removed from a mirror, or a disk is replaced by a larger one, the pool may be smaller than its disks. The pool card then shows the notice "the pool can be N larger" and Expand…. After you confirm, the pool grows to the full capacity of the disks; nothing is erased, and after expanding it cannot be shrunk.

12. What 1.0 does not support yet

• The web page cannot replace disks in raidz, nor change a single disk or mirror into raidz, nor add a disk to raidz; for zpool replace on the command line see §11.
• The Storage page has no per-dataset snapshots or dataset browser: snapshots are on each virtual machine's detail page. Datasets are managed by keel and not created by hand.
• No pool-level quotas and reservations, no pool renaming, and no ISO library shared across pools (the ISO library is only on the default pool and is not moved when the default pool changes).
• The ISO library and software packages do not follow the default pool; after changing the default pool, upload the ISOs again (§5).
• The web page has no detailed SMART information, only OK / Failing, temperature and power-on hours; use smartctl for details.
• The Storage page does not do file sharing: to share files, run a NAS (keelNAS) in a virtual machine.