ZFS stalled after some mirror disks were lost

[Miroslav Lachman]

Adam Vande More wrote on 10/02/2017 20:44:
> On Mon, Oct 2, 2017 at 1:12 PM, Ben RUBSON <ben.rubson at gmail.com> wrote:
>
>> Hi,
>>
>> On a FreeBSD 11 server, the following online/healthy zpool :
>>
>> home
>>    mirror-0
>>      label/local1
>>      label/local2
>>      label/iscsi1
>>      label/iscsi2
>>    mirror-1
>>      label/local3
>>      label/local4
>>      label/iscsi3
>>      label/iscsi4
>> cache
>>    label/local5
>>    label/local6
>>
>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk
>> according to "zpool iostat", nothing on local disks (strange but I
>> noticed that IOs always prefer iscsi disks to local disks).
>> No write IOs.
>>
>> Let's disconnect all iSCSI disks :
>> iscsictl -Ra
>>
>> Expected behavior :
>> IO activity flawlessly continue on local disks.
>>
>
> Perhaps I'm misunderstanding your setup, but my expected behavior would be
> exactly what you see.  I think you'd need something more along the lines of:
>
> home
>    mirror
>      label/local1
>      label/iscsi1
>    mirror
>      label/local2
>      label/iscsi2
> etc...

The OP has four way mirror. It is supposed to work even if 3 devices are 
missing. Just 1 device should be enough.

Miroslav Lachman