Failing ZFS log devices/panic

[M. Casper Lewis]

On Fri, Aug 31, 2018 at 11:35:12AM +1000, Grant Gray wrote:
> I'm going to defer to people with more experience with this hardware
> combination than myself, but I will say I've had compatibility issues with
> SATA SSD's on LSI SAS controllers in the past. In my case this manifested
> as the SSD's disappearing off the bus and not returning. 

And after a reboot they are back?

Interesting.  This is not what we are seeing, rather an accumulation of 
errors until the device is faulted, despite both SMART and the vendor
utility reporting a healthy drive.

> I've currently got an issue between some HGST SATA disks and a SAS3008 HBA
> where mixing SAS and SATA disks on the same port results in intermittent
> I/O errors on the HGST SATA disks, but not other SAS devices.

This sounds like what we are seeing, but we're not mixing SAS and SATA.

> If feasible, it may be useful to move the SSD's onto a plain SATA controller
> as a diagnostic step.

Certainly a step to consider if our controller swap does not improve the
situation.  That said, this is a production fileserver with about half a
petabyte of Important Data(tm) on it, so experimentation is not exactly
the soup of the day.  Which is to say we'd like to take this machine down
as infrequently as possible.  Diagnostic steps sans downtime are
preferable.

-- 
M. Casper Lewis                     |   mclewis at ucdavis.edu
Systems Administrator               |   Voice: (530) 754-7978
Genome Center                       |
University of California, Davis     |