zpool remove not working for metadata special devices

[Andriy Gapon]

On 2021-01-18 14:53, Andriy Gapon wrote:
> On 2020-12-28 05:04, John Delisle wrote:
>> To experiment, I've completed this same procedure using Ubuntu 20.10 (on
>> the same "hardware", both are identically configured VMs in Azure) .
>>
>> The steps above work fine with ZFS on Ubuntu, and I can successfully remove
>> both data vdev mirrors and the special vdev mirror.  This makes me think my
>> syntax is correct at least.. but for whatever reason the exact same
>> procedure fails on FreeBSD (both removing a mirror and removing the special
>> mirror fail with the same error on FreeBSD).
> 
> I don't think that you did anything wrong and the syntax is certainly
> correct.  I can reproduce the problem as well.
> Note that if you insert a reboot between the add special and the remove,
> then the latter works fine, at least it does for me.
> 
> It looks like there is an issue with recording of spa_min_ashift
> internal parameter when a special device is first added.
> 

Actually, just creating a pool can create the min/max ashift discrepancy.

fbsd12# zpool create tank mirror vtbd1 vtbd2
fbsd12# dtrace -n 'spa_close:entry /pid == $target/ { this->spa =
args[0]; printf("min = %d, max = %d\n", this->spa->spa_min_ashift,
this->spa->spa_max_ashift); }' -c 'zpool status tank'
dtrace: description 'spa_close:entry ' matched 1 probe
  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            vtbd1   ONLINE       0     0     0
            vtbd2   ONLINE       0     0     0

errors: No known data errors
dtrace: pid 737 has exited
CPU     ID                    FUNCTION:NAME
  1  66818                  spa_close:entry min = 9, max = 12

  1  66818                  spa_close:entry min = 9, max = 12

Both devices have 4K logical block size, so *the* ashift is 12.
But somehow spa_min_ashift got set to 9.

-- 
Andriy Gapon