FreeBSD-12.1 ZFS boot wierdness

[James B. Byrne]

We experienced the situation last week where one of our hosts failed to reboot
on a warm restart with i/o errors being reported.  This host was configured
with 4 x 8Tb in a raidz2 with root0-on-zfs.

After going down a lot of rabbit holes we established that the zfs pool was
intact.  As a last resort, following replication the pool's content on another
host, we pulled a spared unit with the same hardware configuration, installed
the hdds from the first unit into it.  And that unit booted with problem. 
Problem solved, right?

Not so fast.  Before putting the replacement unit into service I carried out a
series of tests on that host to ensure that any combination of two drives would
actually boot.  And this is were things get a little inexplicable, at least for
me.

Given four HDDs: A, B, C, D; and a host with four hot swap drive bays: 0, 1, 2,
3; I can boot with any combination of two drives EXCEPT when the drive in slot
2 does not have its companion in either slot 1 or 3.  For example 0A,1-,2-,3B
will boot, as will the reverse 0B,1-,2-,3A.  Any two drives in positions 0 and
3 will boot, as will any two drives in 0 and 1, or 0 and 3, or 1 and 3, or 2
and 3. What will not boot is 0X,1-,2Y,3-; for any values of X and Y.

Does anyone have any idea what is going on?

-- 
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3