zpool remove not working for metadata special devices

[Andriy Gapon]

On 2020-12-28 05:04, John Delisle wrote:
> To experiment, I've completed this same procedure using Ubuntu 20.10 (on
> the same "hardware", both are identically configured VMs in Azure) .
> 
> The steps above work fine with ZFS on Ubuntu, and I can successfully remove
> both data vdev mirrors and the special vdev mirror.  This makes me think my
> syntax is correct at least.. but for whatever reason the exact same
> procedure fails on FreeBSD (both removing a mirror and removing the special
> mirror fail with the same error on FreeBSD).


I don't think that you did anything wrong and the syntax is certainly
correct.  I can reproduce the problem as well.
Note that if you insert a reboot between the add special and the remove,
then the latter works fine, at least it does for me.

It looks like there is an issue with recording of spa_min_ashift
internal parameter when a special device is first added.

> On Sun, Dec 27, 2020 at 6:55 PM John Delisle <jdelisle at gmail.com> wrote:
> 
>> I have a pool of mirrors, and added a mirrored special device.  Although
>> documentation suggests it should be removable, I cannot get zpool to do so.
>>
>> All top-level vdevs are mirrors, and all have the same sector size and
>> ashift.  No raidz.
>>
>> ## Current 12.2 p2
>> root at jmdtest:/ # freebsd-version
>> 12.2-RELEASE-p2
>>
>> ## Disk info
>> root at jmdtest:/ # diskinfo /dev/da[2-9] /dev/da[0-9][0-9]
>> /dev/da2        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da3        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da4        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da5        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da6        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da7        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da8        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da9        512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da10       512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da11       512     274877906944    536870912       4096    0
>> 33418   255     63
>> /dev/da12       512     68719476736     134217728       4096    0
>> 8354    255     63
>> /dev/da13       512     68719476736     134217728       4096    0
>> 8354    255     63
>>
>>
>> ## Create the pool:
>> zpool create nebula mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror
>> da8 da9 mirror da10 da11
>>
>> ## Add the special mirror
>> zpool add nebula special mirror da12 da13
>>
>> ## zpool status
>> root at jmdtest:/ # zpool list -v nebula
>> NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP
>>  HEALTH  ALTROOT
>> nebula      1.30T  2.57G  1.30T        -         -     0%     0%  1.00x
>>  ONLINE  -
>>   mirror     254G   492M   254G        -         -     0%  0.18%
>>     da2         -      -      -        -         -      -      -
>>     da3         -      -      -        -         -      -      -
>>   mirror     254G   501M   254G        -         -     0%  0.19%
>>     da4         -      -      -        -         -      -      -
>>     da5         -      -      -        -         -      -      -
>>   mirror     254G   596M   253G        -         -     0%  0.22%
>>     da6         -      -      -        -         -      -      -
>>     da7         -      -      -        -         -      -      -
>>   mirror     254G   481M   254G        -         -     0%  0.18%
>>     da8         -      -      -        -         -      -      -
>>     da9         -      -      -        -         -      -      -
>>   mirror     254G   561M   253G        -         -     0%  0.21%
>>     da10        -      -      -        -         -      -      -
>>     da11        -      -      -        -         -      -      -
>> special         -      -      -         -      -      -
>>   mirror    63.5G  2.62M  63.5G        -         -     0%  0.00%
>>     da12        -      -      -        -         -      -      -
>>     da13        -      -      -        -         -      -      -
>> root at jmdtest:/ #
>>
>> ## Remove the special mirror
>> root at jmdtest:/ # zpool remove nebula mirror-5
>> cannot remove mirror-5: invalid config; all top-level vdevs must have the
>> same sector size and not be raidz.
>>
>> ## All mirrors, and all the same ashift:
>> root at jmdtest:/ # zdb -C | grep -e child -e ashift
>>     vdev_children: 6
>>         children[0]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[1]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[2]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[3]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[4]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>>         children[5]:
>>             ashift: 12
>>             children[0]:
>>             children[1]:
>> root at jmdtest:/ #
>>
>> What am I doing wrong?


-- 
Andriy Gapon