git: 653598764d - main - Handbook - ZFS: Rework
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sat, 18 Jul 2026 13:25:45 UTC
The branch main has been updated by carlavilla:
URL: https://cgit.FreeBSD.org/doc/commit/?id=653598764d3cfc7d0ee9b93a24dfe80756cfe4ca
commit 653598764d3cfc7d0ee9b93a24dfe80756cfe4ca
Author: Sergio Carlavilla Delgado <carlavilla@FreeBSD.org>
AuthorDate: 2026-07-18 13:23:23 +0000
Commit: Sergio Carlavilla Delgado <carlavilla@FreeBSD.org>
CommitDate: 2026-07-18 13:23:23 +0000
Handbook - ZFS: Rework
* Add RAID-Z expansion, dRAID, special vdevs, hot spares with zfsd, TRIM,
pool checkpoints, block cloning, fast dedup, bookmarks, holds,
resumable/corrective send, project quotas, direct I/O, and boot environments
* Add Jails and ZFS crossref target, encryption key loading at boot (zfskeys),
and a Pool Properties section
* Rewrite Tuning to drop removed sysctls (scrub_delay, vdev.cache.size,
arc.meta_limit, …); delete the obsolete ZFS-on-i386 section
* Switch examples to ada device names and GPT partitions
* Fix the inverted reservation terminology entry, zfs scrub: zpool scrub,
stale checksum/compression algorithm lists, and EOL version framing
Approved by: arrowd@, bcr@, michaelo@
Differential Revision: D58175
PR: 287148, 261212, 263321
---
.../content/en/books/handbook/zfs/_index.adoc | 2206 +++++++++++++++-----
1 file changed, 1681 insertions(+), 525 deletions(-)
diff --git a/documentation/content/en/books/handbook/zfs/_index.adoc b/documentation/content/en/books/handbook/zfs/_index.adoc
index 2ee5dd7dbf..86058d1446 100644
--- a/documentation/content/en/books/handbook/zfs/_index.adoc
+++ b/documentation/content/en/books/handbook/zfs/_index.adoc
@@ -50,17 +50,15 @@ endif::[]
ZFS is an advanced file system designed to solve major problems found in previous storage subsystem software.
-Originally developed at Sun(TM), ongoing open source ZFS development has moved to the http://open-zfs.org[OpenZFS Project].
+Originally developed at Sun Microsystems(TM), ongoing open source ZFS development has moved to the https://openzfs.org[OpenZFS Project].
ZFS has three major design goals:
* Data integrity: All data includes a crossref:zfs[zfs-term-checksum,checksum] of the data. ZFS calculates checksums and writes them along with the data. When reading that data later, ZFS recalculates the checksums. If the checksums do not match, meaning detecting one or more data errors, ZFS will attempt to automatically correct errors when ditto-, mirror-, or parity-blocks are available.
* Pooled storage: adding physical storage devices to a pool, and allocating storage space from that shared pool. Space is available to all file systems and volumes, and increases by adding new storage devices to the pool.
* Performance: caching mechanisms provide increased performance.
- crossref:zfs[zfs-term-arc,ARC] is an advanced memory-based read cache. ZFS
- provides a second level disk-based read cache with
- crossref:zfs[zfs-term-l2arc,L2ARC], and a disk-based synchronous write cache
- named crossref:zfs[zfs-term-zil,ZIL].
+crossref:zfs[zfs-term-arc,ARC] is an advanced memory-based read cache.
+ZFS provides a second level disk-based read cache with crossref:zfs[zfs-term-l2arc,L2ARC], and a disk-based synchronous write cache named crossref:zfs[zfs-term-zil,ZIL].
A complete list of features and terminology is in crossref:zfs[zfs-term, ZFS Features and Terminology].
@@ -70,32 +68,32 @@ A complete list of features and terminology is in crossref:zfs[zfs-term, ZFS Fea
More than a file system, ZFS is fundamentally different from traditional file systems.
Combining the traditionally separate roles of volume manager and file system provides ZFS with unique advantages.
The file system is now aware of the underlying structure of the disks.
-Traditional file systems could exist on a single disk alone at a time.
+Traditional file systems could exist on a single disk at a time.
If there were two disks then creating two separate file systems was necessary.
A traditional hardware RAID configuration avoided this problem by presenting the operating system with a single logical disk made up of the space provided by physical disks on top of which the operating system placed a file system.
-Even with software RAID solutions like those provided by GEOM, the UFS file system living on top of the RAID believes it's dealing with a single device.
+Even with software RAID solutions like those provided by GEOM, the UFS file system living on top of the RAID believes it is dealing with a single device.
ZFS' combination of the volume manager and the file system solves this and allows the creation of file systems that all share a pool of available storage.
One big advantage of ZFS' awareness of the physical disk layout is that existing file systems grow automatically when adding extra disks to the pool.
This new space then becomes available to the file systems.
-ZFS can also apply different properties to each file system. This makes it useful to create separate file systems and datasets instead of a single monolithic file system.
+ZFS can also apply different properties to each file system.
+This makes it useful to create separate file systems and datasets instead of a single monolithic file system.
[[zfs-quickstart]]
== Quick Start Guide
+The FreeBSD installer can install the system directly onto a ZFS pool, a configuration known as Root-on-ZFS; see crossref:bsdinstall[bsdinstall-part-zfs,Guided Root-on-ZFS].
+This section shows how to create and manage additional ZFS pools and datasets on a running system.
+
FreeBSD can mount ZFS pools and datasets during system initialization.
To enable it, run:
[source,shell]
....
-# service zfs enable
+# sysrc zfs_enable="YES"
....
-or add this line to [.filename]#/etc/rc.conf#:
-
-[.programlisting]
-....
-zfs_enable="YES"
-....
+This adds `zfs_enable="YES"` to [.filename]#/etc/rc.conf#.
+Running `service zfs enable` makes the same change.
Then start the service:
@@ -104,32 +102,32 @@ Then start the service:
# service zfs start
....
-The examples in this section assume three SCSI disks with the device names [.filename]#da0#, [.filename]#da1#, and [.filename]#da2#.
-Users of SATA hardware should instead use [.filename]#ada# device names.
+The examples in this section assume three SATA disks with the device names [.filename]#ada0#, [.filename]#ada1#, and [.filename]#ada2#.
+Users of SCSI/SAS hardware should instead use [.filename]#da# device names, and users of NVMe storage [.filename]#nda# device names.
[[zfs-quickstart-single-disk-pool]]
=== Single Disk Pool
-To create a simple, non-redundant pool using a single disk device:
+Create a GPT partition on an empty disk first.
+Then add a partition of type `freebsd-zfs` from which a single, non-redundant pool is created:
[source,shell]
....
-# zpool create example /dev/da0
+# gpart create -s gpt ada1
+# gpart add -t freebsd-zfs ada1
+# zpool create example /dev/ada1p1
....
To view the new pool, review the output of `df`:
[source,shell]
....
-# df
-Filesystem 1K-blocks Used Avail Capacity Mounted on
-/dev/ad0s1a 2026030 235230 1628718 13% /
-devfs 1 1 0 100% /dev
-/dev/ad0s1d 54098308 1032846 48737598 2% /usr
-example 17547136 0 17547136 0% /example
+# df /example
+Filesystem 1K-blocks Used Avail Capacity Mounted on
+example 17547136 0 17547136 0% /example
....
-This output shows creating and mounting of the `example` pool, and that is now accessible as a file system.
+This output shows creating and mounting of the `example` pool, and that it is now accessible as a file system.
Create files for users to browse:
[source,shell]
@@ -169,11 +167,8 @@ To unmount a file system, use `zfs umount` and then verify with `df`:
....
# zfs umount example/compressed
# df
-Filesystem 1K-blocks Used Avail Capacity Mounted on
-/dev/ad0s1a 2026030 235232 1628716 13% /
-devfs 1 1 0 100% /dev
-/dev/ad0s1d 54098308 1032864 48737580 2% /usr
-example 17547008 0 17547008 0% /example
+Filesystem 1K-blocks Used Avail Capacity Mounted on
+example 17547008 0 17547008 0% /example
....
To re-mount the file system to make it accessible again, use `zfs mount` and verify with `df`:
@@ -183,9 +178,6 @@ To re-mount the file system to make it accessible again, use `zfs mount` and ver
# zfs mount example/compressed
# df
Filesystem 1K-blocks Used Avail Capacity Mounted on
-/dev/ad0s1a 2026030 235234 1628714 13% /
-devfs 1 1 0 100% /dev
-/dev/ad0s1d 54098308 1032864 48737580 2% /usr
example 17547008 0 17547008 0% /example
example/compressed 17547008 0 17547008 0% /example/compressed
....
@@ -195,9 +187,9 @@ Running `mount` shows the pool and file systems:
[source,shell]
....
# mount
-/dev/ad0s1a on / (ufs, local)
+/dev/ada0p1 on / (ufs, local)
devfs on /dev (devfs, local)
-/dev/ad0s1d on /usr (ufs, local, soft-updates)
+/dev/ada0p2 on /usr (ufs, local, soft-updates)
example on /example (zfs, local)
example/compressed on /example/compressed (zfs, local)
....
@@ -218,13 +210,13 @@ Use `df` to see the data and space usage:
[source,shell]
....
# df
-Filesystem 1K-blocks Used Avail Capacity Mounted on
-/dev/ad0s1a 2026030 235234 1628714 13% /
-devfs 1 1 0 100% /dev
-/dev/ad0s1d 54098308 1032864 48737580 2% /usr
-example 17547008 0 17547008 0% /example
-example/compressed 17547008 0 17547008 0% /example/compressed
-example/data 17547008 0 17547008 0% /example/data
+Filesystem 1K-blocks Used Avail Capacity Mounted on
+/dev/ada0p1 2026030 235234 1628714 13% /
+devfs 1 1 0 100% /dev
+/dev/ada0p2 54098308 1032864 48737580 2% /usr
+example 17547008 0 17547008 0% /example
+example/compressed 17547008 0 17547008 0% /example/compressed
+example/data 17547008 0 17547008 0% /example/data
....
Notice that all file systems in the pool have the same available space.
@@ -248,16 +240,23 @@ One way to avoid data loss from disk failure is to use RAID.
ZFS supports this feature in its pool design.
RAID-Z pools require three or more disks but provide more usable space than mirrored pools.
-This example creates a RAID-Z pool, specifying the disks to add to the pool:
+This example creates a RAID-Z pool.
+First the GPT partitions are created on each disk, then specify these GPT partitions to add to the pool using the `raidz` keyword:
[source,shell]
....
-# zpool create storage raidz da0 da1 da2
+# gpart create -s gpt ada0
+# gpart add -t freebsd-zfs ada0
+# gpart create -s gpt ada1
+# gpart add -t freebsd-zfs ada1
+# gpart create -s gpt ada2
+# gpart add -t freebsd-zfs ada2
+# zpool create storage raidz ada0p1 ada1p1 ada2p1
....
[NOTE]
====
-Sun(TM) recommends that the number of devices used in a RAID-Z configuration be between three and nine.
+Keeping the number of devices used in a RAID-Z configuration between three and nine is recommended.
For environments requiring a single pool consisting of 10 disks or more, consider breaking it up into smaller RAID-Z groups.
If two disks are available, ZFS mirroring provides redundancy if required.
Refer to man:zpool[8] for more details.
@@ -289,24 +288,24 @@ To make this the new home directory for users, copy the user data to this direct
# ln -s /storage/home /usr/home
....
-Users data is now stored on the freshly-created [.filename]#/storage/home#.
+User data is now stored on the freshly-created [.filename]#/storage/home#.
Test by adding a new user and logging in as that user.
Create a file system snapshot to roll back to later:
[source,shell]
....
-# zfs snapshot storage/home@08-30-08
+# zfs snapshot storage/home@2026-07-11
....
ZFS creates snapshots of a dataset, not a single directory or file.
-The `@` character is a delimiter between the file system name or the volume name.
+The `@` character is a delimiter between the file system or volume name and the snapshot name.
Before deleting an important directory, back up the file system, then roll back to an earlier snapshot in which the directory still exists:
[source,shell]
....
-# zfs rollback storage/home@08-30-08
+# zfs rollback storage/home@2026-07-11
....
To list all available snapshots, run `ls` in the file system's [.filename]#.zfs/snapshot# directory.
@@ -323,7 +322,7 @@ Remove the previous snapshot using the command:
[source,shell]
....
-# zfs destroy storage/home@08-30-08
+# zfs destroy storage/home@2026-07-11
....
After testing, make [.filename]#/storage/home# the real
@@ -339,16 +338,16 @@ Run `df` and `mount` to confirm that the system now treats the file system as th
[source,shell]
....
# mount
-/dev/ad0s1a on / (ufs, local)
+/dev/ada0p1 on / (ufs, local)
devfs on /dev (devfs, local)
-/dev/ad0s1d on /usr (ufs, local, soft-updates)
+/dev/ada0p2 on /usr (ufs, local, soft-updates)
storage on /storage (zfs, local)
storage/home on /home (zfs, local)
# df
Filesystem 1K-blocks Used Avail Capacity Mounted on
-/dev/ad0s1a 2026030 235240 1628708 13% /
+/dev/ada0p1 2026030 235240 1628708 13% /
devfs 1 1 0 100% /dev
-/dev/ad0s1d 54098308 1032826 48737618 2% /usr
+/dev/ada0p2 54098308 1032826 48737618 2% /usr
storage 26320512 0 26320512 0% /storage
storage/home 26320512 0 26320512 0% /home
....
@@ -361,6 +360,8 @@ Add daily status updates about the created file systems to the nightly man:perio
daily_status_zfs_enable="YES"
....
+man:periodic[8] can also run scheduled scrubs of the pool; see crossref:zfs[zfs-zpool-scrub,Scrubbing a Pool].
+
[[zfs-quickstart-recovering-raid-z]]
=== Recovering RAID-Z
@@ -387,36 +388,37 @@ crossref:zfs[zfs-term-offline,Offline] state, the pool state will look like this
pool: storage
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
- Sufficient replicas exist for the pool to continue functioning in a
- degraded state.
+ Sufficient replicas exist for the pool to continue functioning in a
+ degraded state.
action: Online the device using 'zpool online' or replace the device with
- 'zpool replace'.
- scrub: none requested
+ 'zpool replace'.
+ scan: none requested
config:
- NAME STATE READ WRITE CKSUM
- storage DEGRADED 0 0 0
- raidz1 DEGRADED 0 0 0
- da0 ONLINE 0 0 0
- da1 OFFLINE 0 0 0
- da2 ONLINE 0 0 0
+ NAME STATE READ WRITE CKSUM
+ storage DEGRADED 0 0 0
+ raidz1-0 DEGRADED 0 0 0
+ ada0p1 ONLINE 0 0 0
+ ada1p1 OFFLINE 0 0 0
+ ada2p1 ONLINE 0 0 0
errors: No known data errors
....
-"OFFLINE" shows the administrator took [.filename]#da1# offline using:
+"OFFLINE" shows the administrator took [.filename]#ada1p1# offline using:
[source,shell]
....
-# zpool offline storage da1
+# zpool offline storage ada1p1
....
-Power down the computer now and replace [.filename]#da1#.
-Power up the computer and return [.filename]#da1# to the pool:
+Power down the computer now and replace [.filename]#ada1p1#.
+Recreate the GPT partition of type `freebsd-zfs` on the new disk.
+Power up the computer and return [.filename]#ada1p1# to the pool:
[source,shell]
....
-# zpool replace storage da1
+# zpool replace storage ada1p1
....
Next, check the status again, this time without `-x` to display all pools:
@@ -424,17 +426,17 @@ Next, check the status again, this time without `-x` to display all pools:
[source,shell]
....
# zpool status storage
- pool: storage
+ pool: storage
state: ONLINE
- scrub: resilver completed with 0 errors on Sat Aug 30 19:44:11 2008
+ scan: resilvered 3.21G in 00:04:36 with 0 errors on Sat Jul 11 10:32:19 2026
config:
- NAME STATE READ WRITE CKSUM
- storage ONLINE 0 0 0
- raidz1 ONLINE 0 0 0
- da0 ONLINE 0 0 0
- da1 ONLINE 0 0 0
- da2 ONLINE 0 0 0
+ NAME STATE READ WRITE CKSUM
+ storage ONLINE 0 0 0
+ raidz1-0 ONLINE 0 0 0
+ ada0p1 ONLINE 0 0 0
+ ada1p1 ONLINE 0 0 0
+ ada2p1 ONLINE 0 0 0
errors: No known data errors
....
@@ -464,23 +466,23 @@ Verifying the data checksums (called _scrubbing_) ensures integrity of the `stor
The duration of a scrub depends on the amount of data stored.
Larger amounts of data will take proportionally longer to verify.
-Since scrubbing is I/O intensive, ZFS allows a single scrub to run at a time.
+Since scrubbing is I/O intensive, ZFS allows a single scrub to run on each pool at a time.
After scrubbing completes, view the status with `zpool status`:
[source,shell]
....
# zpool status storage
- pool: storage
+ pool: storage
state: ONLINE
- scrub: scrub completed with 0 errors on Sat Jan 26 19:57:37 2013
+ scan: scrub repaired 0B in 00:19:16 with 0 errors on Sat Jul 11 10:32:19 2026
config:
- NAME STATE READ WRITE CKSUM
- storage ONLINE 0 0 0
- raidz1 ONLINE 0 0 0
- da0 ONLINE 0 0 0
- da1 ONLINE 0 0 0
- da2 ONLINE 0 0 0
+ NAME STATE READ WRITE CKSUM
+ storage ONLINE 0 0 0
+ raidz1-0 ONLINE 0 0 0
+ ada0p1 ONLINE 0 0 0
+ ada1p1 ONLINE 0 0 0
+ ada2p1 ONLINE 0 0 0
errors: No known data errors
....
@@ -502,30 +504,30 @@ crossref:zfs[zfs-term-volume,volumes].
[[zfs-zpool-create]]
=== Creating and Destroying Storage Pools
-Creating a ZFS storage pool requires permanent decisions, as the pool structure cannot change after creation.
-The most important decision is which types of vdevs to group the physical disks into.
+The most important decision when creating a ZFS storage pool is which types of vdevs to group the physical disks into.
See the list of crossref:zfs[zfs-term-vdev,vdev types] for details about the possible options.
-After creating the pool, most vdev types do not allow adding disks to the vdev.
-The exceptions are mirrors, which allow adding new disks to the vdev, and stripes, which upgrade to mirrors by attaching a new disk to the vdev.
-Although adding new vdevs expands a pool, the pool layout cannot change after pool creation.
-Instead, back up the data, destroy the pool, and recreate it.
+The vdev types determine the redundancy, capacity, and performance characteristics of a pool.
+A pool's layout is not fixed forever at creation time.
+Mirrors allow adding new disks to the vdev, and stripes upgrade to mirrors by attaching a new disk to the vdev.
+RAID-Z vdevs grow one disk at a time with crossref:zfs[zfs-zpool-raidz-expansion,RAID-Z expansion], which requires OpenZFS 2.3 or later, first available in FreeBSD 15.0.
+Adding new vdevs expands a pool at any time, and crossref:zfs[zfs-zpool-attach,removing top-level vdevs] is possible within limits.
+Reshaping a live pool takes time and I/O bandwidth, so choosing suitable vdev types up front remains important.
-Create a simple mirror pool:
+Create a simple mirror pool on disks containing GPT partitions of type `freebsd-zfs`:
[source,shell]
....
-# zpool create mypool mirror /dev/ada1 /dev/ada2
+# zpool create mypool mirror /dev/ada1p1 /dev/ada2p1
# zpool status
pool: mypool
state: ONLINE
- scan: none requested
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
- ada1 ONLINE 0 0 0
- ada2 ONLINE 0 0 0
+ ada1p1 ONLINE 0 0 0
+ ada2p1 ONLINE 0 0 0
errors: No known data errors
....
@@ -534,21 +536,20 @@ To create more than one vdev with a single command, specify groups of disks sepa
[source,shell]
....
-# zpool create mypool mirror /dev/ada1 /dev/ada2 mirror /dev/ada3 /dev/ada4
+# zpool create mypool mirror /dev/ada1p1 /dev/ada2p1 mirror /dev/ada3p1 /dev/ada4p1
# zpool status
pool: mypool
state: ONLINE
- scan: none requested
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
- ada1 ONLINE 0 0 0
- ada2 ONLINE 0 0 0
+ ada1p1 ONLINE 0 0 0
+ ada2p1 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
- ada3 ONLINE 0 0 0
- ada4 ONLINE 0 0 0
+ ada3p1 ONLINE 0 0 0
+ ada4p1 ONLINE 0 0 0
errors: No known data errors
....
@@ -565,37 +566,80 @@ Create a crossref:zfs[zfs-term-vdev-raidz,RAID-Z2] pool using partitions:
[source,shell]
....
-# zpool create mypool raidz2 /dev/ada0p3 /dev/ada1p3 /dev/ada2p3 /dev/ada3p3 /dev/ada4p3 /dev/ada5p3
+# zpool create mypool raidz2 /dev/ada0p1 /dev/ada1p1 /dev/ada2p1 /dev/ada3p1 /dev/ada4p1 /dev/ada5p1
# zpool status
pool: mypool
state: ONLINE
- scan: none requested
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
- ada0p3 ONLINE 0 0 0
- ada1p3 ONLINE 0 0 0
- ada2p3 ONLINE 0 0 0
- ada3p3 ONLINE 0 0 0
- ada4p3 ONLINE 0 0 0
- ada5p3 ONLINE 0 0 0
+ ada0p1 ONLINE 0 0 0
+ ada1p1 ONLINE 0 0 0
+ ada2p1 ONLINE 0 0 0
+ ada3p1 ONLINE 0 0 0
+ ada4p1 ONLINE 0 0 0
+ ada5p1 ONLINE 0 0 0
errors: No known data errors
....
+ZFS aligns and sizes its smallest writes to each vdev based on the vdev's `ashift` value, the base 2 logarithm of the sector size.
+OpenZFS detects the sector size the disks report when creating a vdev and chooses `ashift` accordingly.
+Some drives report 512-byte sectors for compatibility while using 4096-byte sectors internally.
+Create pools on such drives with an explicit `-o ashift=12` to force 4096-byte alignment:
+
+[source,shell]
+....
+# zpool create -o ashift=12 mypool mirror /dev/ada1p1 /dev/ada2p1
+....
+
+Setting the man:sysctl[8] variable `vfs.zfs.vdev.min_auto_ashift` to `12` gives the same result for pool creation and for later `zpool add` and `zpool attach` operations.
+`ashift` is a property of each vdev, not of the pool as a whole.
+It is fixed when creating the vdev and cannot change afterwards, so verify it before committing data to a pool.
+
Destroy a pool that is no longer needed to reuse the disks.
Destroying a pool requires unmounting the file systems in that pool first.
If any dataset is in use, the unmount operation fails without destroying the pool.
Force the pool destruction with `-f`.
This can cause undefined behavior in applications which had open files on those datasets.
+[[zfs-zpool-props]]
+=== Pool Properties
+
+Like datasets, pools have properties that report status and control behavior.
+Display every property of a pool with `zpool get all _mypool_`, or name specific properties:
+
+[source,shell]
+....
+# zpool get health,capacity mypool
+NAME PROPERTY VALUE SOURCE
+mypool health ONLINE -
+mypool capacity 27% -
+....
+
+Change a writable property with `zpool set`:
+
+[source,shell]
+....
+# zpool set comment="Backup pool" mypool
+# zpool get comment mypool
+NAME PROPERTY VALUE SOURCE
+mypool comment Backup pool local
+....
+
+Properties like `size`, `capacity`, `fragmentation`, and `health` are read-only status values.
+Others change how the pool behaves; this chapter uses `autoexpand` to grow a pool automatically (crossref:zfs[zfs-zpool-online,Growing a Pool]), `autoreplace` with hot spares (crossref:zfs[zfs-zpool-spares,Hot Spares and Automatic Replacement with zfsd]), `autotrim` on SSD pools (crossref:zfs[zfs-zpool-trim,TRIM and Initialization]), and `compatibility` for portable pools (crossref:zfs[zfs-zpool-upgrade,Upgrading a Storage Pool]).
+Set properties at pool creation time by passing `-o` to `zpool create`, as shown with `ashift` above.
+man:zpoolprops[7] describes every pool property.
+
[[zfs-zpool-attach]]
=== Adding and Removing Devices
Two ways exist for adding disks to a pool: attaching a disk to an existing vdev with `zpool attach`, or adding vdevs to the pool with `zpool add`.
Some crossref:zfs[zfs-term-vdev,vdev types] allow adding disks to the vdev after creation.
+RAID-Z vdevs accept new disks only through crossref:zfs[zfs-zpool-raidz-expansion,RAID-Z expansion].
A pool created with a single disk lacks redundancy.
It can detect corruption but can not repair it, because there is no other copy of the data.
@@ -613,7 +657,6 @@ Upgrade the single disk (stripe) vdev [.filename]#ada0p3# to a mirror by attachi
# zpool status
pool: mypool
state: ONLINE
- scan: none requested
config:
NAME STATE READ WRITE CKSUM
@@ -622,14 +665,9 @@ config:
errors: No known data errors
# zpool attach mypool ada0p3 ada1p3
-Make sure to wait until resilvering finishes before rebooting.
-
-If you boot from pool 'mypool', you may need to update boot code on newly attached disk _ada1p3_.
-
-Assuming you use GPT partitioning and _da0_ is your new boot disk you may use the following command:
-
- gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
+Make sure to wait until resilver is done before rebooting.
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
+partcode written to ada1p1
bootcode written to ada1
# zpool status
pool: mypool
@@ -637,9 +675,9 @@ bootcode written to ada1
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
- scan: resilver in progress since Fri May 30 08:19:19 2014
- 527M scanned out of 781M at 47.9M/s, 0h0m to go
- 527M resilvered, 67.53% done
+ scan: resilver in progress since Sat Jul 11 11:15:25 2026
+ 738M scanned at 105M/s, 522M issued at 74.6M/s, 781M total
+ 522M resilvered, 66.84% done, 00:00:03 to go
config:
NAME STATE READ WRITE CKSUM
@@ -652,7 +690,7 @@ errors: No known data errors
# zpool status
pool: mypool
state: ONLINE
- scan: resilvered 781M in 0h0m with 0 errors on Fri May 30 08:15:58 2014
+ scan: resilvered 781M in 00:00:16 with 0 errors on Sat Jul 11 11:18:36 2026
config:
NAME STATE READ WRITE CKSUM
@@ -664,7 +702,7 @@ config:
errors: No known data errors
....
-When adding disks to the existing vdev is not an option, as for RAID-Z, an alternative method is to add another vdev to the pool.
+When adding disks to the existing vdev is not an option, an alternative method is to add another vdev to the pool.
Adding vdevs provides higher performance by distributing writes across the vdevs.
Each vdev provides its own redundancy.
Mixing vdev types like `mirror` and `RAID-Z` is possible but discouraged.
@@ -685,7 +723,7 @@ Attach a second mirror group ([.filename]#ada2p3# and [.filename]#ada3p3#) to th
# zpool status
pool: mypool
state: ONLINE
- scan: resilvered 781M in 0h0m with 0 errors on Fri May 30 08:19:35 2014
+ scan: resilvered 781M in 00:00:16 with 0 errors on Sat Jul 11 12:11:56 2026
config:
NAME STATE READ WRITE CKSUM
@@ -697,13 +735,46 @@ config:
errors: No known data errors
# zpool add mypool mirror ada2p3 ada3p3
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2
+partcode written to ada2p1
bootcode written to ada2
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada3
+partcode written to ada3p1
bootcode written to ada3
# zpool status
pool: mypool
state: ONLINE
- scan: scrub repaired 0 in 0h0m with 0 errors on Fri May 30 08:29:51 2014
+ scan: resilvered 781M in 00:00:16 with 0 errors on Sat Jul 11 09:58:28 2026
+config:
+
+ NAME STATE READ WRITE CKSUM
+ mypool ONLINE 0 0 0
+ mirror-0 ONLINE 0 0 0
+ ada0p3 ONLINE 0 0 0
+ ada1p3 ONLINE 0 0 0
+ mirror-1 ONLINE 0 0 0
+ ada2p3 ONLINE 0 0 0
+ ada3p3 ONLINE 0 0 0
+
+errors: No known data errors
+....
+
+`zpool add` also attaches dedicated log and cache devices to an existing pool.
+A crossref:zfs[zfs-term-vdev-log,log] vdev stores the ZFS intent log on separate low-latency storage, accelerating synchronous writes such as those issued by databases and NFS.
+Mirror log devices, because losing an unmirrored log device together with a system crash costs the pool the last few seconds of synchronous writes; see crossref:zfs[zfs-advanced-zil-slog,"Synchronous Writes, the ZIL, and SLOG"] for sizing and tuning guidance.
+A crossref:zfs[zfs-term-vdev-cache,cache] vdev extends the crossref:zfs[zfs-term-arc,ARC] with a second level of read cache on fast storage.
+Cache devices need no redundancy, as ZFS reads any block that fails to read from the cache from the original pool disks instead.
+The contents of the cache, the crossref:zfs[zfs-term-l2arc,L2ARC], survive reboots by default.
+
+Add a mirrored log vdev and an NVMe cache device to an existing pool:
+
+[source,shell]
+....
+# zpool add mypool log mirror ada4p2 ada5p2
+# zpool add mypool cache nda0p2
+# zpool status mypool
+ pool: mypool
+ state: ONLINE
+ scan: resilvered 781M in 00:00:16 with 0 errors on Sat Jul 11 09:58:28 2026
config:
NAME STATE READ WRITE CKSUM
@@ -714,11 +785,17 @@ config:
mirror-1 ONLINE 0 0 0
ada2p3 ONLINE 0 0 0
ada3p3 ONLINE 0 0 0
+ logs
+ mirror-2 ONLINE 0 0 0
+ ada4p2 ONLINE 0 0 0
+ ada5p2 ONLINE 0 0 0
+ cache
+ nda0p2 ONLINE 0 0 0
errors: No known data errors
....
-Removing vdevs from a pool is impossible and removal of disks from a mirror is exclusive if there is enough remaining redundancy.
+`zpool detach` removes single disks from a mirror vdev when enough redundancy remains.
If a single disk remains in a mirror group, that group ceases to be a mirror and becomes a stripe, risking the entire pool if that remaining disk fails.
Remove a disk from a three-way mirror group:
@@ -728,7 +805,7 @@ Remove a disk from a three-way mirror group:
# zpool status
pool: mypool
state: ONLINE
- scan: scrub repaired 0 in 0h0m with 0 errors on Fri May 30 08:29:51 2014
+ scan: scrub repaired 0B in 00:01:11 with 0 errors on Sat Jul 11 03:14:02 2026
config:
NAME STATE READ WRITE CKSUM
@@ -743,7 +820,7 @@ errors: No known data errors
# zpool status
pool: mypool
state: ONLINE
- scan: scrub repaired 0 in 0h0m with 0 errors on Fri May 30 08:29:51 2014
+ scan: scrub repaired 0B in 00:01:11 with 0 errors on Sat Jul 11 03:14:02 2026
config:
NAME STATE READ WRITE CKSUM
@@ -755,6 +832,191 @@ config:
errors: No known data errors
....
+`zpool remove` removes entire top-level vdevs from a pool.
+ZFS evacuates the vdev by copying all of its allocated data to the other vdevs in the pool, then detaches the disks.
+Removal works for hot spares, cache, log, special, and dedup devices, and for data vdevs that are single disks or mirrors.
+Data vdevs are not removable from pools that contain a top-level RAID-Z vdev, and all top-level vdevs in the pool must use the same `ashift`.
+The evacuation runs in the background and the pool stays online throughout; monitor progress with `zpool status`.
+After the removal completes, ZFS keeps an in-memory table mapping the blocks of the removed vdev to their new locations.
+The table is small but permanent; `zpool remove -n` estimates its memory use before starting a removal.
+
+Remove one of the two mirror vdevs from a pool:
+
+[source,shell]
+....
+# zpool remove mypool mirror-1
+# zpool status mypool
+ pool: mypool
+ state: ONLINE
+ scan: scrub repaired 0B in 00:01:11 with 0 errors on Sat Jul 11 03:14:02 2026
+remove: Evacuation of mirror in progress since Sat Jul 11 10:15:31 2026
+ 1.71G copied out of 2.32G at 111M/s, 73.71% done, 0h0m to go
+config:
+
+ NAME STATE READ WRITE CKSUM
+ mypool ONLINE 0 0 0
+ mirror-0 ONLINE 0 0 0
+ ada0p3 ONLINE 0 0 0
+ ada1p3 ONLINE 0 0 0
+ mirror-1 ONLINE 0 0 0
+ ada2p3 ONLINE 0 0 0
+ ada3p3 ONLINE 0 0 0
+
+errors: No known data errors
+# zpool status mypool
+ pool: mypool
+ state: ONLINE
+ scan: scrub repaired 0B in 00:01:11 with 0 errors on Sat Jul 11 03:14:02 2026
+remove: Removal of vdev 1 copied 2.32G in 0h0m, completed on Sat Jul 11 10:15:53 2026
+ 10.9K memory used for removed device mappings
+config:
+
+ NAME STATE READ WRITE CKSUM
+ mypool ONLINE 0 0 0
+ mirror-0 ONLINE 0 0 0
+ ada0p3 ONLINE 0 0 0
+ ada1p3 ONLINE 0 0 0
+ indirect-1 ONLINE 0 0 0
+
+errors: No known data errors
+....
+
+The `indirect-1` entry is a placeholder for the removed vdev's remapped blocks and contains no disks.
+Cancel an in-progress removal with `zpool remove -s`.
+
+[[zfs-zpool-raidz-expansion]]
+=== RAID-Z Expansion
+
+RAID-Z expansion grows an existing RAID-Z vdev by one disk at a time, without changing its parity level.
+
+[NOTE]
+====
+RAID-Z expansion requires OpenZFS 2.3 or later, first available in FreeBSD 15.0.
+Pools created on earlier releases need the `raidz_expansion` feature enabled with `zpool upgrade` before expanding.
+====
+
+To expand a RAID-Z vdev, run `zpool attach` with the name of the RAID-Z vdev as shown by `zpool status` and the new disk:
+
+[source,shell]
+....
+# zpool attach mypool raidz2-0 ada6p3
+....
+
+The expansion reflows the existing data across the enlarged set of disks in the background while the pool remains online and in use.
+Add `-w` to make `zpool attach` wait until the expansion completes.
+`zpool status` reports progress on the `expand:` line:
+
+[source,shell]
+....
+# zpool status mypool
+ pool: mypool
+ state: ONLINE
+ scan: scrub repaired 0B in 06:14:36 with 0 errors on Sat Jul 11 04:31:19 2026
+expand: expansion of raidz2-0 in progress since Sat Jul 11 09:42:12 2026
+ 1.83T / 5.36T copied at 186M/s, 34.14% done, 05:31:43 to go
+config:
+
+ NAME STATE READ WRITE CKSUM
+ mypool ONLINE 0 0 0
+ raidz2-0 ONLINE 0 0 0
+ ada0p3 ONLINE 0 0 0
+ ada1p3 ONLINE 0 0 0
+ ada2p3 ONLINE 0 0 0
+ ada3p3 ONLINE 0 0 0
+ ada4p3 ONLINE 0 0 0
+ ada5p3 ONLINE 0 0 0
+ ada6p3 ONLINE 0 0 0
+
+errors: No known data errors
+....
+
+When the expansion finishes, the capacity of the new disk becomes available:
+
+[source,shell]
+....
+# zpool status mypool | grep expand:
+expand: expanded raidz2-0 copied 5.36T in 08:23:35, on Sat Jul 11 18:05:47 2026
+....
+
+Blocks written before the expansion keep their old data-to-parity ratio and continue to occupy space accordingly.
+The full capacity gain from the new disk therefore applies to data written after the expansion, while old data benefits only when rewritten, for example with crossref:zfs[zfs-zfs-rewrite,`zfs rewrite`].
+Repeat the procedure to grow a vdev by several disks, attaching one at a time.
+
+[[zfs-zpool-draid]]
+=== dRAID Pools
+
+dRAID is a variant of RAID-Z that distributes hot spare capacity across all disks in the vdev.
+A crossref:zfs[zfs-term-vdev-draid,dRAID] vdev is built from internal RAID-Z groups plus optional _distributed spares_, all spread evenly over every disk.
+When a disk fails, ZFS rebuilds into the distributed spare by reading from and writing to all remaining disks in parallel, completing in a fraction of the time a resilver onto a single spare disk takes.
+Replacing the failed disk then rebuilds it from the distributed spare, restoring full protection.
+dRAID targets pools with many disks, where RAID-Z rebuild times grow dangerously long.
+
+The vdev type spells out the complete layout, per man:zpoolconcepts[7]:
+
+[.programlisting]
+....
+draid[parity][:datad][:childrenc][:sparess]
+....
+
+_parity_ is the parity level of the internal groups, 1 to 3, defaulting to 1.
+_data_ is the number of data devices per redundancy group, defaulting to 8.
+_children_ is the total number of disks, serving as a cross-check when listing many devices.
+_spares_ is the number of distributed hot spares, defaulting to 0.
+
+Create a pool from a 24-disk dRAID vdev with double parity, 8 data disks per redundancy group, and 2 distributed spares:
+
+[source,shell]
+....
+# zpool create mypool draid2:8d:24c:2s ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7 ada8 ada9 ada10 ada11 ada12 ada13 ada14 ada15 ada16 ada17 ada18 ada19 ada20 ada21 ada22 ada23
+....
+
+The distributed spares appear under `spares` in `zpool status`, with names like `draid2-0-0` for the first spare of the first vdev.
+
+dRAID trades space efficiency for rebuild speed.
+Unlike RAID-Z, dRAID uses a fixed stripe width, padding smaller writes with zeros.
+With the default of 8 data disks and 4 KB sectors, the minimum allocation is 32 KB, which reduces usable capacity and compression effectiveness for datasets dominated by small blocks.
+The capacity of the distributed spares is also committed up front, whether or not a disk ever fails.
+Pairing a dRAID pool with a crossref:zfs[zfs-zpool-special,special vdev] keeps metadata and small blocks off the wide stripes and recovers much of the lost efficiency.
+For pools with a handful of disks, RAID-Z remains the better choice.
+
+[[zfs-zpool-special]]
+=== Special Allocation Classes
+
+Special allocation classes dedicate vdevs to specific types of pool data.
+A crossref:zfs[zfs-term-vdev-special,special] vdev stores pool metadata such as indirect blocks and dnodes, and optionally the data blocks of small files.
+A `dedup` vdev stores the crossref:zfs[zfs-term-deduplication,deduplication] tables.
+Placing this metadata on fast devices such as NVMe mirrors speeds up metadata-heavy operations like directory traversal, `zfs list`, and deduplication lookups on pools of otherwise slower disks.
+
+Add a mirrored special vdev to an existing pool:
+
+[source,shell]
+....
+# zpool add mypool special mirror nda0 nda1
+....
+
+New metadata allocations then go to the special vdev; existing metadata stays where it is until rewritten.
+
+Setting the `special_small_blocks` dataset property makes the special vdev store data blocks up to the given size as well:
+
+[source,shell]
+....
+# zfs set special_small_blocks=16K mypool/projects
+....
+
+Valid values are zero, which disables storing data blocks in the special class, or a power of two from 512 bytes up to 1 MB on FreeBSD 14.x; FreeBSD 15.0 accepts values up to the maximum block size of 16 MB.
+Setting the property to the dataset's `recordsize` sends all data blocks of that dataset to the special class.
+When a special vdev fills up, new allocations spill back to the normal class instead of failing.
+
+[WARNING]
+====
+A special vdev is not a cache.
+It holds the only copy of the pool metadata allocated to it, and losing it destroys the pool.
+Match the redundancy of special and dedup vdevs to the redundancy of the data vdevs, for example by using a mirror.
+====
+
+Removing a special or dedup vdev with `zpool remove` is possible, subject to the restrictions described in crossref:zfs[zfs-zpool-attach,Adding and Removing Devices].
+In practice, the matching `ashift` requirement often prevents removal when the special vdev uses devices with a different sector size than the data vdevs.
+
[[zfs-zpool-status]]
=== Checking the Status of a Pool
@@ -768,7 +1030,7 @@ Actions to take and details about the last crossref:zfs[zfs-zpool-scrub,`scrub`]
# zpool status
pool: mypool
state: ONLINE
- scan: scrub repaired 0 in 2h25m with 0 errors on Sat Sep 14 04:25:50 2013
+ scan: scrub repaired 0B in 02:25:36 with 0 errors on Sat Jul 11 06:51:26 2026
config:
NAME STATE READ WRITE CKSUM
@@ -784,6 +1046,17 @@ config:
errors: No known data errors
....
+Options adjust the output to the task at hand:
+
+* `-v` displays verbose data error information, printing a complete list of the errors found since the last complete pool scrub, including the names of affected files.
+* `-x` displays the status of pools with errors or pools that are otherwise unavailable, hiding healthy pools.
+* `-s` displays the number of slow I/O operations on each leaf vdev, meaning operations that did not complete within 30 seconds.
+* `-e` displays unhealthy vdevs only, keeping the output short for pools with many devices; this flag requires FreeBSD 14.1 or later.
+* `-p` displays numbers as exact, parseable values instead of rounded human-readable ones.
+* `-t` displays the crossref:zfs[zfs-zpool-trim,TRIM] status of each vdev.
+
+Refer to man:zpool-status[8] for the complete list of options.
+
[[zfs-zpool-clear]]
=== Clearing Errors
@@ -810,7 +1083,6 @@ Replace a functioning device in the pool:
# zpool status
pool: mypool
state: ONLINE
- scan: none requested
config:
NAME STATE READ WRITE CKSUM
@@ -823,11 +1095,11 @@ errors: No known data errors
# zpool replace mypool ada1p3 ada2p3
Make sure to wait until resilvering finishes before rebooting.
-When booting from the pool 'zroot', update the boot code on the newly attached disk 'ada2p3'.
+When booting from the pool 'mypool', update the boot code on the newly attached disk 'ada2p3'.
-Assuming GPT partitioning is used and [.filename]#da0# is the new boot disk, use the following command:
+Assuming GPT partitioning is used and ada0 is the new boot disk, use the following command:
- gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
+ gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2
# zpool status
pool: mypool
@@ -835,9 +1107,9 @@ Assuming GPT partitioning is used and [.filename]#da0# is the new boot disk, use
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
- scan: resilver in progress since Mon Jun 2 14:21:35 2014
- 604M scanned out of 781M at 46.5M/s, 0h0m to go
- 604M resilvered, 77.39% done
+ scan: resilver in progress since Sat Jul 11 14:21:35 2026
+ 781M / 781M scanned, 604M / 781M issued at 101M/s
+ 604M resilvered, 77.39% done, 00:00:01 to go
config:
NAME STATE READ WRITE CKSUM
@@ -852,7 +1124,7 @@ errors: No known data errors
# zpool status
pool: mypool
state: ONLINE
- scan: resilvered 781M in 0h0m with 0 errors on Mon Jun 2 14:21:52 2014
+ scan: resilvered 781M in 00:00:17 with 0 errors on Sat Jul 11 14:21:52 2026
config:
NAME STATE READ WRITE CKSUM
@@ -864,21 +1136,24 @@ config:
errors: No known data errors
....
+On mirrored vdevs, adding `-s` to `zpool replace` or `zpool attach` performs a _sequential resilver_ instead of a healing resilver.
+A sequential resilver copies the data in disk order without verifying each block checksum, restoring redundancy much sooner.
+Since the copy skips checksum verification, ZFS starts a crossref:zfs[zfs-zpool-scrub,scrub] of the pool automatically after the sequential resilver completes.
+Sequential resilvering works on mirror and crossref:zfs[zfs-zpool-draid,dRAID] vdevs, not on RAID-Z.
+Restart an in-progress resilver from the beginning with `zpool resilver _mypool_`.
*** 2385 LINES SKIPPED ***