Recommended HBA for ZFS, contemporary

[Karl Denninger]

On 8/22/2016 11:30, Dmitry Morozovsky wrote:
> Karl,
>
> On Mon, 22 Aug 2016, Karl Denninger wrote:
>
>> I'm still a fan of the LSI SAS-9211 cards along with (for higher
>> density) a SAS port expander.  I've been extremely happy with this
>> combination with one caveat -- most systems must boot from the actual
>> card and not a port on the expander(s) you connect to it due to BIOS
>> constraints.  They're also crazily cost-effective on-balance.
>>
>> In practice this means you have 4 ports on the base card available (2
>> used for mirrored boot drives, 2 for other things) and the rest of the
>> disks go on the expander, or you run more than one card.
>>
>> Whether you can actually saturate this combination depends on what you
>> attach.  In large configurations, especially those stuffed with SSDs,
>> you can -- at which point something faster such as the 93xx series which
> which immediately leads me to the question: what is the driver supporting 9300 
> in HBA mode?  Quick googling does not reveal interoperability, and 9300 is not 
> in hardware list for FreeBSD yet.
So much for LSI's claims eh? :-)
>> are materially faster (but still in "host" mode as opposed to any sort
>> of "smart", "raid" or "buffered" mode) should be considered.
>>
>> The "smarter" the card the dumber the results tend to be when ZFS is in
>> use, in my experience :)
> Yes, I'm well aware of it ;)
>
> Thanks!

Incidentally actually managing to saturate a 9211 (which comes up as a
"SAS-2008") on the mps driver is non-trivial.  If you THINK you are,
make sure you really are.  If you really are and have a second PCIe slot
with the requisite lanes available that's probably the next thing to do.

Yes, it's possible (especially with SSDs) to saturate these but not
easy.  You can get the 2xxx series cards in 16-port versions (4 x 4)
which leaves you three ports that can each be attached to an expander. 
The limiting factor usually winds up being the configuration of said
plugs and the shelves for the drives themselves (2i/2e configurations
work great if you're using expansion chassis with an expander in them,
by the way.)

I have a number of the Intel 6-port expanders in production servers and
have never have had any trouble with them; they just plain work.  The
only note I'll make on the expanders (and these host cards) is to make
sure you have sufficient airflow in the chassis; they're passive
heat-sink cooled so airflow matters.  In any sort of proper server
chassis this is probably not going to be an issue but if you're going
for something that doesn't sound like an old 707 jet on take-off you
need to pay attention to make sure there's sufficient cooling airflow.

I tend to avoid using the motherboard SATA ports; I've had my share of
"fun" with them over time, especially if I need to hot-swap a drive
(e.g. a failed member of the root mirror.)  You only need to have a
machine crash on you unnecessarily in this fashion once to be disabused
of that sort of insanity in the future.

-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2996 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20160822/a75f37ef/attachment.bin>