zpool remove not working for metadata special devices

[John Delisle]

 I'd really like to use this functionality on my production pool.  I'm
cautious, so I built it out in a lab setting first only to find
removing the special device mirror doesn't work on FreeBSD, yet works fine
with ZFS on Linux.

 Does anyone have any suggestions or insight into what might be going on?


On Sun, Dec 27, 2020 at 9:04 PM John Delisle <jdelisle at gmail.com> wrote:

> To experiment, I've completed this same procedure using Ubuntu 20.10 (on
> the same "hardware", both are identically configured VMs in Azure) .
>
> The steps above work fine with ZFS on Ubuntu, and I can successfully
> remove both data vdev mirrors and the special vdev mirror.  This makes me
> think my syntax is correct at least.. but for whatever reason the exact
> same procedure fails on FreeBSD (both removing a mirror and removing the
> special mirror fail with the same error on FreeBSD).
>
> Thanks
>
> On Sun, Dec 27, 2020 at 6:55 PM John Delisle <jdelisle at gmail.com> wrote:
>
>> I have a pool of mirrors, and added a mirrored special device.  Although
>> documentation suggests it should be removable, I cannot get zpool to do so.
>>
>> All top-level vdevs are mirrors, and all have the same sector size and
>> ashift.  No raidz.
>>
>> ## Current 12.2 p2
>> root at jmdtest:/ # freebsd-version
>> 12.2-RELEASE-p2
>>
>> ## Disk info
>> root at jmdtest:/ # diskinfo /dev/da[2-9] /dev/da[0-9][0-9]
>> /dev/da2        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da3        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da4        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da5        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da6        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da7        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da8        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da9        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da10       512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da11       512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da12       512     68719476736     134217728       4096    0
>> 8354    255     63
>> /dev/da13       512     68719476736     134217728       4096    0
>> 8354    255     63
>>
>>
>> ## Create the pool:
>> zpool create nebula mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror
>> da8 da9 mirror da10 da11
>>
>> ## Add the special mirror
>> zpool add nebula special mirror da12 da13
>>
>> ## zpool status
>> root at jmdtest:/ # zpool list -v nebula
>> NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP
>>  HEALTH  ALTROOT
>> nebula      1.30T  2.57G  1.30T        -         -     0%     0%  1.00x
>>  ONLINE  -
>>   mirror     254G   492M   254G        -         -     0%  0.18%
>>     da2         -      -      -        -         -      -      -
>>     da3         -      -      -        -         -      -      -
>>   mirror     254G   501M   254G        -         -     0%  0.19%
>>     da4         -      -      -        -         -      -      -
>>     da5         -      -      -        -         -      -      -
>>   mirror     254G   596M   253G        -         -     0%  0.22%
>>     da6         -      -      -        -         -      -      -
>>     da7         -      -      -        -         -      -      -
>>   mirror     254G   481M   254G        -         -     0%  0.18%
>>     da8         -      -      -        -         -      -      -
>>     da9         -      -      -        -         -      -      -
>>   mirror     254G   561M   253G        -         -     0%  0.21%
>>     da10        -      -      -        -         -      -      -
>>     da11        -      -      -        -         -      -      -
>> special         -      -      -         -      -      -
>>   mirror    63.5G  2.62M  63.5G        -         -     0%  0.00%
>>     da12        -      -      -        -         -      -      -
>>     da13        -      -      -        -         -      -      -
>> root at jmdtest:/ #
>>
>> ## Remove the special mirror
>> root at jmdtest:/ # zpool remove nebula mirror-5
>> cannot remove mirror-5: invalid config; all top-level vdevs must have the
>> same sector size and not be raidz.
>>
>> ## All mirrors, and all the same ashift:
>> root at jmdtest:/ # zdb -C | grep -e child -e ashift
>>     vdev_children: 6
>>         children[0]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[1]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[2]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[3]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[4]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[5]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>> root at jmdtest:/ #
>>
>> What am I doing wrong?
>>
>>
>>