a strange and terrible saga of the cursed iSCSI ZFS SAN

[Ben RUBSON]

> On 05 Aug 2017, at 19:59, Eugene M. Zheganin <emz at norma.perm.ru> wrote:
> 
> Hi,
> 
> On 05.08.2017 22:50, Eugene M. Zheganin wrote:
>> Hi,
>> 
>> On 05.08.2017 22:22, Ben RUBSON wrote:
>>>> On 05 Aug 2017, at 19:08, Eugene M. Zheganin <emz at norma.perm.ru> wrote:
>>>> 
>>>> (...)
>>>> 
>>>> 1) autumn 2016, SAN is set up, supermicro server, external JBOD, sandisk ssds, several redundant pools, FreeBSD 11.x (probably release, don't really remember - see below).
>>> Hi Eugene,
>>> 
>>> Which SAS adapter ?
>>> SAS 2008 ?
>> Exactly. What's wrong with it ? I'm running latest (or at least not that old) firmware on em.
>> 
> No, wait, my bad,  it's LSISAS3008. mpr(4) one.

Nothing wrong but things such as verifying the firmware version.
Certainly you use the same (IT ?) version on all your 6 HBA.
Not sure about the 3008, but we had a discussion on this list some months ago about 2008 firmwares, some were said to be bad.
You could also enable some HBA debugging, we never know :
sysctl dev.mpr.0.debug_level=0x1B

Ben