Cannot replace broken hard drive with LSI HBA

[Rick Romero]

  Quoting InterNetX - Juergen Gotteswinter <jg at internetx.com>:

> Am 29.09.2015 um 20:04 schrieb Michael Fuckner:
>> On 9/29/2015 3:51 PM, InterNetX - Juergen Gotteswinter wrote:
>>> From my Experience using SATA Disks on SAS Controllers, no matter if
>>> theres an Expander between or not or mixed, those Setups keep on beeing
>>> flakey / unreliable. I might work under certain conditions, but its
>>> nothing you can bet on.
>>>
>>> Garret Damore (Illumos Project) describes the problem more detailed
here
>>>
>>>
http://garrett.damore.org/2010/08/why-sas-sata-is-not-such-great-idea.html
>>>  
>>
>> come on, the article is 5 years old, some things changed since then!
>>
>> - MUX Boards are unreliable and expensive- long time since I last saw
>> those boards
>
> i am not talking about sata multiplexers, i am talking about sas
> expanders. no one ever should think that sata multiplexers are a
> reliable piece of hardware. they arent.
>
> this problems are design related, since Natacount on disks cant talk
> native with sas controllers. they still exist and occour. like you can
> see here.
>
> a few months ago i had a setup which had exactly what one whould expect
> from sata/sas mix. bus resets, device hangs and so on. in the end, it
> turned out that the indel dcs 3500 ssd wich where connected via a lsi
> sas 2008 where the root cause of all. after they where hooked onto the
> onboard sata ports, everything was fine. on the other side the intel dcs
> 3700 sata ssd works fine in sas/sata mix. you see, its nothing you can
> count on. stay with sas, or with sata and use proper controllers.

I've seen both.  I have two nice Asus boxes with 12 4GB SATA drives in
RAIDZ2, and they work wonderfully.

OTOH, I recently built a new system with a 3 drive RAIDZ (using SSD), and
it constantly blew chunks (timeouts etc). After going through multiple
servers, drives,backplanes (both expanders and just generic hot-swap) and
controllers, it turns out the latest LSI firmware is garbage.  You have to
downgrade the LSI firmware to 'p17' (IIRC) - and preferrably if you're
using ZFS, change it to IT firmware (for 'real' JBOD).

http://lime-technology.com/forum/index.php?topic=12767.630
https://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/page-5

Rick