ZFS: Is 'zpool add' really irreversible?

[Brandon J. Wandersee]

Yuri writes:

> This is very surprising that something as simple as that can't be undone.

`zpool add` adds *virtual* devices to a *pool*, while `zfs attach` adds
physical devices to a mirrored device. So individual disks can be added
to and removed from mirrored virtual devices, but virtual devices cannot
be removed from a pool.

A ZFS pool is striped across all virtual devices included in the pool,
with data and metadata in the pool distributed across the physical
devices that make up those virtual devices, in order to improve
performance, redundancy, or both. Adding a virtual device doesn't merely
make more space available---that virtual device is actually in use from
the moment it is added, beginning with a resilver operation.

>From the perspective of ZFS, removing a virtual device would be similar
in effect to removing a drive from your machine, sawing it in half, and
putting it back in. There's still data on that half-disk, but you'll
never get to it. ZFS tries the same thing every time a pool is imported,
searching for all devices that belong to it and panicking when
devices/data that should exist do not.

So yes, once you've added a virtual device to a pool it is a permanent
part of the pool. You should plan out your storage to accommodate your
needs for the foreseeable future---typically, you would actually make
the pool larger than what you need right now in anticipation of
eventially adding more data. ZFS was essentially designed as a long-term
use, "archival" filesystem; shrinking a pool isn't something that would
ever really happen in situations where ZFS is appropriate.

-- 

::  Brandon J. Wandersee
::  brandon.wandersee at gmail.com
::  --------------------------------------------------
::  'The best design is as little design as possible.'
::  --- Dieter Rams ----------------------------------