major pita w/LSI Tyan TA26 combo

[Freddie Cash]

On Friday 03 March 2006 01:50 pm, Ken Gunderson wrote:
> On Fri, 3 Mar 2006 13:43:09 -0800
> Freddie Cash <fcash at ocis.net> wrote:
> > On Friday 03 March 2006 12:38 pm, Ken Gunderson wrote:
> > > Been there and done that w/this before but I'm exasperated and
> > > don't know where to go from here.

> > > Anyways, system details:
> > > FreeBSD 5.4 and 6.0
> > > Tyan TA26 <http://www.tyan.com/products/html/ta26b2882.html>
> > > 2 x Opteron 252
> > > 4 x 1GB DDR400 ECC Registered Ram
> > > LSI 320-2x and LSI 320-1
> > > Fujitsu U320 SCSI drives, both 10K and 15K rpm

> > > Problem:
> > > Create an array, subject it to some moderate I/O and the logical
> > > drive becomes "degraded" w/a "failed" drive.  The logical drive
> > > cannot be rebuilt.  The physical drive is in fact fine, e.g. can
> > > format and consistency check. Same problem if swap in a different
> > > drive, etc., e.g.:

> > Would this controller happen to be plugged into a riser card?  Or is
> > it plugged directly into the motherboard?

> The TA26 uses a riser card.

> > We just returned a whole mess of LSI MegaRAID boards with similar
> > issues (lose a drive on a regular basis, I/O drops down into the KB/s
> > range, CPU spends 75%+ time in iowait state, controller not properly
> > identified during install preventing any OS from being installed). 
> > On Tyan Thunder K8SD-Pro (S2882) motherboards.

> > The 2U rackmount systems that used an approved Tyan riser card for
> > the PCI-X slot had all the problems.  The tower systerms that plugged
> > the RAID controller directly into the PCI-X slot onthe motherboard
> > have been working for months without issues.

> I have long suspected the Tyan PCI riser!  Tyan swears it's golden
> though.  Thanks for the confirmation.

It's a combination of the riser card and the LSI card that doesn't work.  
And LSI refuses to support our cards as they are plugged into riser 
cards.

> > We tested FreeBSD 6.0, 6.1-beta1, and 6.1-beta2, Debian Linux stable
> > (2.6.8 and 2.6.15 kernels), testing (2.6.11, 2.6.12, 2.6.13, 2.6.14
> > kernels), and unstable (2.6.14 and 2.6.15 kernels).  With the
> > included megaraid drivers and the custom drivers direct from LSI.

> > The official word from LSI Support is:  they do not support any RAID
> > controllers connected to riser cards, no matter what the OS.  Go that
> > via e-mail from one of their support techs and one of their RAID
> > engineers.

> I'm in contact w/the LSI support manager. Do you mind if I forward his
> this?

Go ahead.  They probably have a file an inch thick with my name on it (or 
with School District 73 on it).  :)

> > > Does anybody have ideas and/or receommend anything that actually
> > > works?!?!

> > Switch to 3Ware and/or Areca cards.  :)  We replaced all the LSI
> > MegaRAID cards in the 2U rackmounts with 3Ware Escalade 9550SX cards
> > and have been chugging along nicely ever since (FreeBSD 6.1-beta2
> > i386 and amd64, and Debian Linux stable custom install with 2.6.15
> > kernel 64-bit).  Besides, the management tools from LSI are pure crud
> > compared to 3DM2.

> Thanks bunches.  This has been a big help.
> Q:  The 3Ware work in the riser configuration as well??

Works wonderfully plugged into the same riser card and motherboard that 
the LSI cards came out of.  Throughput is high, iowait is low, CPU usage 
is low, the system is rock stable, unplugging the SATA connector on a 
live system doesn't faze it, online rebuild is nice and fast (takes under 
2 hours to rebuild a new install on a 4-drive RAID5, whereas the LSI card 
took just under 10 hours to do the same) even when compiling software 
from the ports tree at the same time.

For systems that use riser cards, LSI MegaRAID cards are almost guaranteed 
to cause problems.  No idea why, and they've told me they won't support 
it.
-- 
Freddie Cash
fcash at ocis.net