Supermicro HBA

Thu Jan 17 18:59:20 UTC 2019

I have two servers using the AOC-S3008L-L8i (one 400TB+, the other 140TB
(SSD)). They both work great, no real issues performance wise. The firmware
update is a bit of a pain with the supermicro card (make sure you write
down the SAS ID of the card before you do the update), but they work well.

shawn

On Thu, Jan 17, 2019, 8:03 AM Valeri Galtsev <galtsev at kicp.uchicago.edu
wrote:

>
>
> On 1/17/19 8:28 AM, Tenzin Lhakhang wrote:
> > If cost is negligible between Supermicro HBA using 3008 controller versus
> > Broadcom/Avago/LSI card.  I'd say opt for the Broadcom card, their
> > support + drivers + firmware downloads is much better than Supermicro's
> FTP
> > site.  Broadcom keeps whitepapers, archive of all their past released
> > firmwares.
> >
> > Must dos:
> > - Make sure to do hot-add hot-remove disks (with workload).
> > - Make sure to do disk led blink light tests.  You will have to use the
> > sas3ircu utility, unless you have OS specific utilities to find them.
> > - Perform an fio disk benchmark to get baseline system performance.
> > https://github.com/axboe/fio
> >
> > Issues I've seen:
> > - SuperMicro's FTP site for their firmware doesn't include archive
> copies,
> > only latest.  Sometimes latest FW doesn't work nicely and hopefully you
> > have an old FW downloaded prior.
> > - SuperMicro FTP site:  ftp://ftp.supermicro.com/Driver/SAS/LSI/
> > - Sometimes SM HBA's FW and LSI HBA's FW can be used interchangeable and
> > sometimes not.  If you flashed a LSI HBA with SMC firmware, then you have
> > to add -nossid when flashing using the sas3flash when going back to LSI
> > firmware.  Sometimes it flashes fine and everything looks ok, you see all
> > the disks, but when you blink disk 4, it blinks disk 1.  When you
> > physically remove disk 2, ZFS says disk 3 is gone.  It's best to
> thoroughly
> > qualify the system after FW flash.
>
> Thanks, Tenzin, for nice write up!
>
> I only would udd one general thing about Sipermicro. If you get
> Supermicro system board ("motherboard" is common jargon name for system
> board for over couple of decades), then make sure you go with Intel
> CPUs. If you need/prefer AMD CPUs, stay away from Supermicro. They are
> notoriously poorly designing system boards for AMD CPUs. I've seen
> several bad/flaky due to poor design myself (that is for AMD CPUs, not
> Intel); you also can search reports of that on the web. In my book one
> bad is one too many if it is you who got that. I tend to get Tyan ones
> whenever I can, and definitely when I want AMD CPUs. Tyan is in server
> board business since forever, and they never had flops I would hear about.
>
> Sorry about small rant; I hope this may help someone.
>
> Valeri
>
> >
> > Broadcom page for 9300-8i
> > -
> >
> https://www.broadcom.com/products/storage/host-bus-adapters/sas-9300-8i#overview
> > - It's a bit hard to understand all the different downloads, but they're
> > helpful when you understand the workflow to flash FW and use the sas3ircu
> > utility.  In the latest release, they've bundled all the required
> downloads
> > into the Firmware download.  In the past you would have to individually
> > find the different ROM and BIN files.
> >
> > All said, Supermicro HBA when it's working, it works pretty well.  I've
> > seen HBA failures, but the systems were usually 5+ years old.
> > - If you have multiple HBAs in a system, record which physical controller
> > is on PCIe slot is the logical controller reported by OS.  Otherwise
> when a
> > single HBA fails, it's a bit hard to trace which software HBA is which
> > physical HBA.
> >
> > Thanks,
> > Tenzin
> >
> > On Thu, Jan 17, 2019 at 7:51 AM Paul Pathiakis via freebsd-fs <
> > freebsd-fs at freebsd.org> wrote:
> >
> >>   I've used the LSI 3008.  I haven't had any issues with it.  It is
> >> supported by that driver.  (I haven't built a server with one lately)
> >> Yes, avoid all the hardware RAID cards they are unnecessary and a JBOD
> >> controller with ZFS is a good choice.  Make sure it supports the SAS3
> spec
> >> of 12 Gb/s.  That's where the speed is.  I found the following as an
> FYI.
> >> Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews
> >>
> >>
> >> |
> >> |
> >> |
> >> |  |  |
> >>
> >>   |
> >>
> >>   |
> >> |
> >> |  |
> >> Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews
> >>
> >> The Supermicro LSI SAS3008 HBAs (which share the same controller as the
> >> LSI 9300-8i HBAs) are engineered to deli...
> >>   |
> >>
> >>   |
> >>
> >>   |
> >>
> >>
> >> I looked at it as more of a specs education.  It looks solid.  I've used
> >> Supermicro without issue in the past.  (I just decommissioned my home
> >> server which was Supermicro and has that very card in it.)
> >> There is a lot to consider when using ZFS beyond just the hardware.
> Don't
> >> get me wrong.... I want to have ZFS' baby.  :D  Just be sure of all the
> >> nuances of HDD, SSD, Hybrids, how much memory you have to dedicate to
> ZFS
> >> and CPU cores you have.  You have to take into consideration all the ZFS
> >> features you're planning to make use of now and in the future.
> >> Also, are you ever planning on expanding the storage to have an
> additional
> >> JBOD shelf?  If so, you may want a card with some external connectors.
> >> Some of the people at http://www.ixsystems.com have done some serious
> >> research on application specific throughput of ZFS and I believe they
> also
> >> spec out SuperMicro servers too.  It comes down to IOPS, raw throughput,
> >> etc.  (I'm actually talking to them right now about some very large
> backup
> >> servers that can handle 0.75 PB.... The consideration I have is space
> and
> >> using RAIDZ2 and multiple streams from 10Gb interfaces and serious
> >> compression and deduplication.  SO,  IOPS not so much, but heavy raw I/O
> >> and RAID checksum computation and dedup.
> >> There's also things like dedicated SSDs as ZIL and cache to be thought
> >> about.
> >> So:  go up the theoretical OSI layer model and optimize each layer right
> >> through the application layer. :D  (I actually find it fun)
> >> I hope this all helps.
> >> P
> >>
> >>      On Thursday, January 17, 2019, 4:33:38 AM CST, Julien Cigar
> >> <julien at perdition.city> wrote:
> >>
> >>   Hello,
> >>
> >> We are planning to replace some (web) applications servers (currently
> >> running HPE) with Supermicro and the vendor offers the following
> >> choices for the Hardware Raid Controller/HBA 4P part:
> >>
> >> 1) Supermicro AOC-S3008L-L8E, LSI 3008 8 x SATA/SAS III JBOD
> controller, up
> >> to 122 hard drives via expander backplane, PCI-E, ideal for Nexenta/ZFS
> >> + € 205,6
> >>
> >> 2) LSI 9300-4I, SATA/SAS III JBOD controller, tot 256 harde schijven via
> >> expander backplanes, PCI-E, ideal for Nexenta/ZFS
> >> + € 214,5
> >>
> >> 3) LSI MegaRAID 9341-4i bulk, 4 x SATA/SAS 12Gbs internal entry level
> >> hardware RAID, no cache/BBU possible, PCI-e
> >> + € 177,97
> >>
> >> 4) LSI MegaRAID 9361-4i 1GB cache, 4 x SATA/SAS 12Gbs internal
> >> hardware RAID, max. 240 hdd using expander backplanes
> >> + € 401,7
> >>
> >> 5) LSI 9300-4i4e, SATA/SAS III JBOD controller, 4 x internal, 4 x
> external,
> >> up to 256 hard drives via expander backplane, Nexenta Certified, ideal
> >> for ZFS, PCI-E
> >> + € 273,95
> >>
> >> 6) LSI MegaRAID 9380-4i4e bulk, 8 x SATA/SAS 12Gbs, 4 x external and 4 x
> >> internal hardware RAID, 1024MB cache, up to 128 hard drives via expander
> >> backplane, support for SSD CacheCade 2.0 write and read caching,
> >> CacheVault support (advised), ideal for high en
> >> + € 676
> >>
> >> As the plan is to use ZFS, I was planning to choose the
> >> "Supermicro AOC-S3008L-L8E, LSI 3008" (it looks like it is supported by
> >> mpr) and I was wondering if anyone has any feedback on it ? Would
> >> another option be a better choice ?
> >>
> >> Thank you!
> >> Julien
> >>
> >> --
> >> Julien Cigar
> >> Belgian Biodiversity Platform (http://www.biodiversity.be)
> >> PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
> >> No trees were killed in the creation of this message.
> >> However, many electrons were terribly inconvenienced.
> >> _______________________________________________
> >> freebsd-fs at freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> >>
> > _______________________________________________
> > freebsd-questions at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to "
> freebsd-questions-unsubscribe at freebsd.org"
> >
>
> --
> ++++++++++++++++++++++++++++++++++++++++
> Valeri Galtsev
> Sr System Administrator
> Department of Astronomy and Astrophysics
> Kavli Institute for Cosmological Physics
> University of Chicago
> Phone: 773-702-4247
> ++++++++++++++++++++++++++++++++++++++++
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscribe at freebsd.org"
>