PCIe SATA HBA for ZFS on -STABLE

[Jeremy Chadwick]

On Wed, Jun 01, 2011 at 02:34:55PM +0800, TJ Varghese wrote:
> On Tue, May 31, 2011 at 10:31 PM, Freddie Cash <fjwcash at gmail.com> wrote:
> 
> > On Tue, May 31, 2011 at 5:48 AM, Matt Thyer <matt.thyer at gmail.com> wrote:
> >
> > > What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
> > > using ZFS ?
> > >
> > > Not wanting to break the bank.
> > > Not interested in SATA III 6GB at this time... though it could be useful
> > if
> > > I add an SSD for... (is it ZIL ?).
> > > Can this be added at any time ?
> > >
> > > The main issue is I need at least 10 ports total for all existing
> > drives...
> > > ZIL would require 11 so ideally we are talking a 6 port HBA.
> > >
> >
> > SuperMicro AOC-USAS2-L8i works exceptionally well.  These are 8-port HBAs
> > using the LSI1068 chipset, supported by the mpt(4) driver.  Support 3 Gpbs
> > SATA/SAS, using multi-lane cables (2 connectors on the card, each connector
> > supports 4 SATA ports), hot-plug, hot-swap.
> >
> >
> The USAS2 (6Gbps) is supported by the mps driver (on -CURRENT, not sure if
> it's in 8-STABLE yet). Perhaps you're referring to the earlier USAS which
> does 3Gbps and is supported by the mpt driver.

Folks considering use of mps(4), which was committed to RELENG_8 roughly
around 2011/02/18 (thus is not in 8.2-RELEASE), should read the below
threads just in case.  Always good to be educated.  Of course, the
mailing lists are usually filled with complaints rather than success
stories, so the tone of my mail here will therefore sound negative; I
don't mean it that way, I just ask that people "be aware".

* 2011/04/29 -- mps driver instability under stable/8
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/thread.html#62507
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-May/thread.html#62518

* 2011/04/27 -- MPS driver: force bus rescan after remove SAS cable
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/thread.html#62438
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/thread.html#62443

* 2011/03/10 -- LSI SAS2008 performance with mps(4) driver
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-March/thread.html#61862

To the OP (Matt Thyer):

Sadly I don't have a recommendation for you, since you effectively want
a 6-port SATA300 controller that's reliable, you're almost certainly
going to be paying Big Bucks(tm) given the number of ports and your
requirement that it be PCIe-based.  You state quite boldly "not wanting
to break the bank", but what you're asking for almost certainly WILL
break the bank.

For example, an "affordable" controller might be one driven by Silicon
Image's SiI3124 chip -- four (4) SATA300 ports, but it's only hooked to
PCI or PCI-X, not PCIe, which means you're susceptible to a much more
severe bus bottleneck than with PCIe:

http://www.siliconimage.com/products/family.aspx?id=3

FreeBSD does have support for many Silicon Image chips via the siis(4)
driver, and support is quite good since SI provides mav@ with technical
documentation and support.  I commend SI for that; it's good to see
companies supporting developers, regardless of OS.

I tend to avoid consumer-grade Marvell and JMicron SATA chipsets like
the plague, however.  That's based on my experiences with them under
Windows, where I would expect (truly) the drivers to be rock solid given
the marketing demographic of the chips in question.

Be aware that SATA port multipliers (if someone recommends them to you
as a way of providing expansion) will also limit your I/O bottleneck,
especially when multiple drives are used over a single multiplier port.
E.g. 4 drives operating at 100MByte/sec (common read speed with
consumer-grade Caviar Black drives!) will saturate a SATA300 connection
easily.  One port per drive solves this dilemma of course, putting the
focus back on the PCI/PCI-X/PCIe bus as a bottleneck.

Anyway, you need to ask yourself what your requirements really are, or
what sort of monetary limitations you have, then make a decision based
on that.  Remember: a good, solid controller will probably be a one-time
purchase.  But also think about the future and if in 2-3 years you want
to go about buying another controller (likely costing more than whatever
it is you buy now).

Good luck, and please let us know what controller you *do* end up going
with and your experience with it!  Positives are as important as
negatives.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |