ZFS stalled after some mirror disks were lost

[Ben RUBSON]

> On 02 Oct 2017, at 21:47, Steven Hartland <killing at multiplay.co.uk> wrote:
> 
> On 02/10/2017 20:10, Ben RUBSON wrote:
>>> On 02 Oct 2017, at 20:41, Steven Hartland <killing at multiplay.co.uk> wrote:
>>> 
>>> I'm guessing that the devices haven't disconnected cleanly so are just stalling all requests to them and hence the pool.
>> I even tried to ifconfig down the network interface serving the iscsi targets, it did not help.
>> 
>>> I'm not that familiar with iscsi, does it still show under under camcontrol or geom?
>> # geom disk list
>> (...)
>> Geom name: da13
>> Providers:
>> 1. Name: da13
>>    Mediasize: 3999688294912 (3.6T)
>>    Sectorsize: 512
>>    Mode: r1w1e2
>>    wither: (null)
>> 
>> Geom name: da15
>> Providers:
>> 1. Name: da15
>>    Mediasize: 3999688294912 (3.6T)
>>    Sectorsize: 512
>>    Mode: r1w1e2
>>    wither: (null)
>> 
>> Geom name: da16
>> Providers:
>> 1. Name: da16
>>    Mediasize: 3999688294912 (3.6T)
>>    Sectorsize: 512
>>    Mode: r1w1e2
>>    wither: (null)
>> 
>> Geom name: da19
>> Providers:
>> 1. Name: da19
>>    Mediasize: 3999688294912 (3.6T)
>>    Sectorsize: 512
>>    Mode: r1w1e2
>>    wither: (null)
>> 
>> # camcontrol devlist
>> // does not show the above disks
> So these daXX devices represent your iscsi devices?

Yes, and only one is still visible under /dev/,
with its label under /dev/label/.
So I may have one problematic drive among 4.

> If so looks like your problem is at the iscsi layer, as its not disconnected properly, so as far ZFS is concerned its still waiting for them.

Certainly procstat will talk !
I have switched production to another server,
so feel free if any other trace is needed.

Thank you again,

Ben