ZFS zpool mirror drive replacement confusion

[Freddie Cash]

On Thu, Aug 11, 2011 at 9:03 AM, Martin Cracauer <cracauer at cons.org> wrote:

> My computer keeps doing what I say, not what I want :-)
>
> This zpool with a 2-drive mirror had a dead drive (ad10) which I
> removed.  After I tried to put ad2 as a replacement drive I ended up
> with the drive not in the mirror, it seems to have extended the zpool
> (it's size?).
>
>        NAME        STATE     READ WRITE CKSUM
>        cbackup3    DEGRADED     0     0     0
>          mirror    DEGRADED     0     0     0
>            ad6     ONLINE       0     0     0
>            ad10    REMOVED      0     0     0
>          ad2       ONLINE       0     0    11
>
> Unfortunately I didn't keep note of the specific command that I used.
>
>
Mirror vdevs are slightly special in they way they are handled.  You don't
have to "zpool replace" drives in vdevs.  Instead, you "zpool detach" the
old/dead drive from the pool (this "breaks" the mirror vdev, leaving you
with a single-disk vdev), and then you "zpool attach" the new drive *to the
existing drive*.

For example, the series of commands you should have used would be:
  zpool detach cbackup3 ad10
  zpool attach cbackup3 ad2 ad6

Or, you don't want to lose any redundancy in your pool, you attach the new
drive first, creating a 3-way mirror vdev, and then detach the old drive
after the resilver is complete.
  zpool attach cbackup3 ad2 ad6
  zpool detach cbackup3 ad10

Right now, your pool has 2 vdevs in it:  1 mirror (ad6 + ad10) and one
single-disk (ad2).  You are currently running with a "degraded" pool, as
there is no redundancy in it:  if you lose disk ad2, you lose the entire
pool.

Thankfully, if you have the drive bays, this is easy to fix.  Just attach
another drive to ad2 to create a second mirror vdev.  :)  Then you will have
a pool with 2 mirror vdevs (similar to a RAID10 setup).  Of course, you'll
need to replace the ad10 disk to make it fully redundant.  Something like:
  zpool attach cbackup3 adXX ad2
  zpool detach cbackup3 ad10
  zpool attach cbackup3 adYY ad6


> Two questions:
>
> - Does somebody know offhand what I did and what I should have done
>  instead to get ad2 into the mirror?
>
> You used "zpool add" (which adds new vdevs to the pool) instead of "zpool
attach" (which attaches drives to an existing mirror vdev).  Or, you used
"zpool add" instead of "zpool replace".


> - Would there have been a way to remove ad2 from this set?


Nope.  You cannot remove storage vdevs from a pool.  You can only remove log
and cache vdevs from a pool.

-- 
Freddie Cash
fjwcash at gmail.com