zfs detach/replace

[Freddie Cash]

On Sun, Jan 1, 2012 at 9:09 PM, Michael DeMan <freebsd at deman.com> wrote:
> Sounds realistic to me that the 'replace' command works after a 'detach' is properly done.
>
> In my case, I forgot to do the 'detach' - just did the 'replace' after changing the drive (and after a reboot).  From there, ZFS gets in the state below - where it automagically created label/ada5LABEL/old and appears to be wanting to do the rebuild onto label/ada5LABEL as if it is part of a mirrored pair?

"zpool detach" and "zpool replace" are two very different, and totally
unconnected, things.

"zpool detach" is used to remove drives from mirror vdevs.  The vdev
then has one less drive in it.  The beauty of the way mirror vdevs
work in ZFS is that you can "zpool attach" a drive to an existing
2-way mirror vdev, thus turning it into a 3-way mirror.  Then "zpool
detach" the extra drive.  Makes upgrading drive capacity a cinch, as
you never lose any redundancy in the vdev.

"zpool replace" is used to do an in-place replacement of a drive in a
vdev, and can be used with mirror, raidz1, raidz2, and raidz3 vdevs.

I think the part you may be getting confused with is the "zpool
offline" step that should be done before doing a "zpool replace".
That way, if anything goes wrong, you can always put back the old
drive, do a "zpool online" to it, and then do a "zpool remove" or
"zpool detach" to the "new" drive.  That's saved my bacon a couple of
times.  :)  Learnt the hard way about that, losing a 20 TB pool (only
a couple TB of data in it) due to "insufficient replicas" and "missing
drives" and whatnot.

> I know for a fact, back with FreeBSD 7.x, that this same scenario could occur.  Basically on below - my old notes show there is no way to fix the situation with ada5LABEL and ada5LABEL/old without destroying and rebuilding the pool.  Any attempts to 'detach', 'offline' or anything else on either of those two logical entries fails with a 'no valid replicas'.

If you have the original drive (the /old one), put it back in the
system, "zpool online" it, then you should be able to "zpool remove"
or "zpool detach" the other drive.

-- 
Freddie Cash
fjwcash at gmail.com