Supermicro HBA

Fri Jan 18 13:51:39 UTC 2019

On Thu, Jan 17, 2019 at 10:59:06AM -0800, Shawn Wallbridge wrote:
> I have two servers using the AOC-S3008L-L8i (one 400TB+, the other 140TB
> (SSD)). They both work great, no real issues performance wise. The firmware
> update is a bit of a pain with the supermicro card (make sure you write
> down the SAS ID of the card before you do the update), but they work well.
> 
> shawn

Thank you all for your feedback :)

> 
> On Thu, Jan 17, 2019, 8:03 AM Valeri Galtsev <galtsev at kicp.uchicago.edu
> wrote:
> 
> >
> >
> > On 1/17/19 8:28 AM, Tenzin Lhakhang wrote:
> > > If cost is negligible between Supermicro HBA using 3008 controller versus
> > > Broadcom/Avago/LSI card.  I'd say opt for the Broadcom card, their
> > > support + drivers + firmware downloads is much better than Supermicro's
> > FTP
> > > site.  Broadcom keeps whitepapers, archive of all their past released
> > > firmwares.
> > >
> > > Must dos:
> > > - Make sure to do hot-add hot-remove disks (with workload).
> > > - Make sure to do disk led blink light tests.  You will have to use the
> > > sas3ircu utility, unless you have OS specific utilities to find them.
> > > - Perform an fio disk benchmark to get baseline system performance.
> > > https://github.com/axboe/fio
> > >
> > > Issues I've seen:
> > > - SuperMicro's FTP site for their firmware doesn't include archive
> > copies,
> > > only latest.  Sometimes latest FW doesn't work nicely and hopefully you
> > > have an old FW downloaded prior.
> > > - SuperMicro FTP site:  ftp://ftp.supermicro.com/Driver/SAS/LSI/
> > > - Sometimes SM HBA's FW and LSI HBA's FW can be used interchangeable and
> > > sometimes not.  If you flashed a LSI HBA with SMC firmware, then you have
> > > to add -nossid when flashing using the sas3flash when going back to LSI
> > > firmware.  Sometimes it flashes fine and everything looks ok, you see all
> > > the disks, but when you blink disk 4, it blinks disk 1.  When you
> > > physically remove disk 2, ZFS says disk 3 is gone.  It's best to
> > thoroughly
> > > qualify the system after FW flash.
> >
> > Thanks, Tenzin, for nice write up!
> >
> > I only would udd one general thing about Sipermicro. If you get
> > Supermicro system board ("motherboard" is common jargon name for system
> > board for over couple of decades), then make sure you go with Intel
> > CPUs. If you need/prefer AMD CPUs, stay away from Supermicro. They are
> > notoriously poorly designing system boards for AMD CPUs. I've seen
> > several bad/flaky due to poor design myself (that is for AMD CPUs, not
> > Intel); you also can search reports of that on the web. In my book one
> > bad is one too many if it is you who got that. I tend to get Tyan ones
> > whenever I can, and definitely when I want AMD CPUs. Tyan is in server
> > board business since forever, and they never had flops I would hear about.
> >
> > Sorry about small rant; I hope this may help someone.
> >
> > Valeri
> >
> > >
> > > Broadcom page for 9300-8i
> > > -
> > >
> > https://www.broadcom.com/products/storage/host-bus-adapters/sas-9300-8i#overview
> > > - It's a bit hard to understand all the different downloads, but they're
> > > helpful when you understand the workflow to flash FW and use the sas3ircu
> > > utility.  In the latest release, they've bundled all the required
> > downloads
> > > into the Firmware download.  In the past you would have to individually
> > > find the different ROM and BIN files.
> > >
> > > All said, Supermicro HBA when it's working, it works pretty well.  I've
> > > seen HBA failures, but the systems were usually 5+ years old.
> > > - If you have multiple HBAs in a system, record which physical controller
> > > is on PCIe slot is the logical controller reported by OS.  Otherwise
> > when a
> > > single HBA fails, it's a bit hard to trace which software HBA is which
> > > physical HBA.
> > >
> > > Thanks,
> > > Tenzin
> > >
> > > On Thu, Jan 17, 2019 at 7:51 AM Paul Pathiakis via freebsd-fs <
> > > freebsd-fs at freebsd.org> wrote:
> > >
> > >>   I've used the LSI 3008.  I haven't had any issues with it.  It is
> > >> supported by that driver.  (I haven't built a server with one lately)
> > >> Yes, avoid all the hardware RAID cards they are unnecessary and a JBOD
> > >> controller with ZFS is a good choice.  Make sure it supports the SAS3
> > spec
> > >> of 12 Gb/s.  That's where the speed is.  I found the following as an
> > FYI.
> > >> Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews
> > >>
> > >>
> > >> |
> > >> |
> > >> |
> > >> |  |  |
> > >>
> > >>   |
> > >>
> > >>   |
> > >> |
> > >> |  |
> > >> Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews
> > >>
> > >> The Supermicro LSI SAS3008 HBAs (which share the same controller as the
> > >> LSI 9300-8i HBAs) are engineered to deli...
> > >>   |
> > >>
> > >>   |
> > >>
> > >>   |
> > >>
> > >>
> > >> I looked at it as more of a specs education.  It looks solid.  I've used
> > >> Supermicro without issue in the past.  (I just decommissioned my home
> > >> server which was Supermicro and has that very card in it.)
> > >> There is a lot to consider when using ZFS beyond just the hardware.
> > Don't
> > >> get me wrong.... I want to have ZFS' baby.  :D  Just be sure of all the
> > >> nuances of HDD, SSD, Hybrids, how much memory you have to dedicate to
> > ZFS
> > >> and CPU cores you have.  You have to take into consideration all the ZFS
> > >> features you're planning to make use of now and in the future.
> > >> Also, are you ever planning on expanding the storage to have an
> > additional
> > >> JBOD shelf?  If so, you may want a card with some external connectors.
> > >> Some of the people at http://www.ixsystems.com have done some serious
> > >> research on application specific throughput of ZFS and I believe they
> > also
> > >> spec out SuperMicro servers too.  It comes down to IOPS, raw throughput,
> > >> etc.  (I'm actually talking to them right now about some very large
> > backup
> > >> servers that can handle 0.75 PB.... The consideration I have is space
> > and
> > >> using RAIDZ2 and multiple streams from 10Gb interfaces and serious
> > >> compression and deduplication.  SO,  IOPS not so much, but heavy raw I/O
> > >> and RAID checksum computation and dedup.
> > >> There's also things like dedicated SSDs as ZIL and cache to be thought
> > >> about.
> > >> So:  go up the theoretical OSI layer model and optimize each layer right
> > >> through the application layer. :D  (I actually find it fun)
> > >> I hope this all helps.
> > >> P
> > >>
> > >>      On Thursday, January 17, 2019, 4:33:38 AM CST, Julien Cigar
> > >> <julien at perdition.city> wrote:
> > >>
> > >>   Hello,
> > >>
> > >> We are planning to replace some (web) applications servers (currently
> > >> running HPE) with Supermicro and the vendor offers the following
> > >> choices for the Hardware Raid Controller/HBA 4P part:
> > >>
> > >> 1) Supermicro AOC-S3008L-L8E, LSI 3008 8 x SATA/SAS III JBOD
> > controller, up
> > >> to 122 hard drives via expander backplane, PCI-E, ideal for Nexenta/ZFS
> > >> + € 205,6
> > >>
> > >> 2) LSI 9300-4I, SATA/SAS III JBOD controller, tot 256 harde schijven via
> > >> expander backplanes, PCI-E, ideal for Nexenta/ZFS
> > >> + € 214,5
> > >>
> > >> 3) LSI MegaRAID 9341-4i bulk, 4 x SATA/SAS 12Gbs internal entry level
> > >> hardware RAID, no cache/BBU possible, PCI-e
> > >> + € 177,97
> > >>
> > >> 4) LSI MegaRAID 9361-4i 1GB cache, 4 x SATA/SAS 12Gbs internal
> > >> hardware RAID, max. 240 hdd using expander backplanes
> > >> + € 401,7
> > >>
> > >> 5) LSI 9300-4i4e, SATA/SAS III JBOD controller, 4 x internal, 4 x
> > external,
> > >> up to 256 hard drives via expander backplane, Nexenta Certified, ideal
> > >> for ZFS, PCI-E
> > >> + € 273,95
> > >>
> > >> 6) LSI MegaRAID 9380-4i4e bulk, 8 x SATA/SAS 12Gbs, 4 x external and 4 x
> > >> internal hardware RAID, 1024MB cache, up to 128 hard drives via expander
> > >> backplane, support for SSD CacheCade 2.0 write and read caching,
> > >> CacheVault support (advised), ideal for high en
> > >> + € 676
> > >>
> > >> As the plan is to use ZFS, I was planning to choose the
> > >> "Supermicro AOC-S3008L-L8E, LSI 3008" (it looks like it is supported by
> > >> mpr) and I was wondering if anyone has any feedback on it ? Would
> > >> another option be a better choice ?
> > >>
> > >> Thank you!
> > >> Julien
> > >>
> > >> --
> > >> Julien Cigar
> > >> Belgian Biodiversity Platform (http://www.biodiversity.be)
> > >> PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
> > >> No trees were killed in the creation of this message.
> > >> However, many electrons were terribly inconvenienced.
> > >> _______________________________________________
> > >> freebsd-fs at freebsd.org mailing list
> > >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> > >>
> > > _______________________________________________
> > > freebsd-questions at freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > > To unsubscribe, send any mail to "
> > freebsd-questions-unsubscribe at freebsd.org"
> > >
> >
> > --
> > ++++++++++++++++++++++++++++++++++++++++
> > Valeri Galtsev
> > Sr System Administrator
> > Department of Astronomy and Astrophysics
> > Kavli Institute for Cosmological Physics
> > University of Chicago
> > Phone: 773-702-4247
> > ++++++++++++++++++++++++++++++++++++++++
> > _______________________________________________
> > freebsd-questions at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to "
> > freebsd-questions-unsubscribe at freebsd.org"
> >
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"

-- 
Julien Cigar
Belgian Biodiversity Platform (http://www.biodiversity.be)
PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20190118/c92d0030/attachment.sig>