ZFS weird issue...

[Will Andrews]

On Sun, Dec 7, 2014 at 3:21 PM, Michelle Sullivan <michelle at sorbs.net> wrote:
> root at colossus:~ # zpool replace sorbs spare-8 mfid8
> root at colossus:~ # zpool replace sorbs spare-8 mfid15
> root at colossus:~ # zpool replace sorbs 933862663 1702922605
> root at colossus:~ # zpool replace sorbs mfid8 mfid8
> root at colossus:~ # zpool replace sorbs mfid15 mfid15
> root at colossus:~ # zpool replace sorbs spare-8 1702922605
> root at colossus:~ # zpool replace sorbs 1702922605 spare-8
> root at colossus:~ # zpool replace sorbs 1702922605 mfid8
[...]

I believe you want to replace 1702922605 (the original member that
used to be mfid8) with mfid15, not mfid8.  According to your 'zpool
status' output (which I assume is still current?), mfid8 is now a
different member of the raidz2 than it was previously.  Of the 16
devices you have, only mfid15 is currently missing, which suggests
that it's the current name of the new drive.

As you said, it's a brand new drive, so "zdb -l /dev/mfid15" should
confirm that it has no ZFS labels, and therefore is the correct drive
to use as the replacement for 1702922605.

> The problem seems to be the guid -> device name transaltion is working
> and failing because there is already a mfid8... and the re-number didn't
> happen when the device was replaced...

The guid -> device name translation isn't meant to be definitive at
all times.  It is just a cached mapping that can become out of date
any time devices disappear and reappear or the system reboots.  This
is why ZFS uses the vdev GUIDs to determine which device is which
regardless of what its current block device name is.

--Will.