Supermicro HBA

Thu Jan 17 14:28:35 UTC 2019

If cost is negligible between Supermicro HBA using 3008 controller versus
Broadcom/Avago/LSI card.  I'd say opt for the Broadcom card, their
support + drivers + firmware downloads is much better than Supermicro's FTP
site.  Broadcom keeps whitepapers, archive of all their past released
firmwares.

Must dos:
- Make sure to do hot-add hot-remove disks (with workload).
- Make sure to do disk led blink light tests.  You will have to use the
sas3ircu utility, unless you have OS specific utilities to find them.
- Perform an fio disk benchmark to get baseline system performance.
https://github.com/axboe/fio

Issues I've seen:
- SuperMicro's FTP site for their firmware doesn't include archive copies,
only latest.  Sometimes latest FW doesn't work nicely and hopefully you
have an old FW downloaded prior.
- SuperMicro FTP site:  ftp://ftp.supermicro.com/Driver/SAS/LSI/
- Sometimes SM HBA's FW and LSI HBA's FW can be used interchangeable and
sometimes not.  If you flashed a LSI HBA with SMC firmware, then you have
to add -nossid when flashing using the sas3flash when going back to LSI
firmware.  Sometimes it flashes fine and everything looks ok, you see all
the disks, but when you blink disk 4, it blinks disk 1.  When you
physically remove disk 2, ZFS says disk 3 is gone.  It's best to thoroughly
qualify the system after FW flash.

Broadcom page for 9300-8i
-
https://www.broadcom.com/products/storage/host-bus-adapters/sas-9300-8i#overview
- It's a bit hard to understand all the different downloads, but they're
helpful when you understand the workflow to flash FW and use the sas3ircu
utility.  In the latest release, they've bundled all the required downloads
into the Firmware download.  In the past you would have to individually
find the different ROM and BIN files.

All said, Supermicro HBA when it's working, it works pretty well.  I've
seen HBA failures, but the systems were usually 5+ years old.
- If you have multiple HBAs in a system, record which physical controller
is on PCIe slot is the logical controller reported by OS.  Otherwise when a
single HBA fails, it's a bit hard to trace which software HBA is which
physical HBA.

Thanks,
Tenzin

On Thu, Jan 17, 2019 at 7:51 AM Paul Pathiakis via freebsd-fs <
freebsd-fs at freebsd.org> wrote:

>  I've used the LSI 3008.  I haven't had any issues with it.  It is
> supported by that driver.  (I haven't built a server with one lately)
> Yes, avoid all the hardware RAID cards they are unnecessary and a JBOD
> controller with ZFS is a good choice.  Make sure it supports the SAS3 spec
> of 12 Gb/s.  That's where the speed is.  I found the following as an FYI.
> Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews
>
>
> |
> |
> |
> |  |  |
>
>  |
>
>  |
> |
> |  |
> Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews
>
> The Supermicro LSI SAS3008 HBAs (which share the same controller as the
> LSI 9300-8i HBAs) are engineered to deli...
>  |
>
>  |
>
>  |
>
>
> I looked at it as more of a specs education.  It looks solid.  I've used
> Supermicro without issue in the past.  (I just decommissioned my home
> server which was Supermicro and has that very card in it.)
> There is a lot to consider when using ZFS beyond just the hardware.  Don't
> get me wrong.... I want to have ZFS' baby.  :D  Just be sure of all the
> nuances of HDD, SSD, Hybrids, how much memory you have to dedicate to ZFS
> and CPU cores you have.  You have to take into consideration all the ZFS
> features you're planning to make use of now and in the future.
> Also, are you ever planning on expanding the storage to have an additional
> JBOD shelf?  If so, you may want a card with some external connectors.
> Some of the people at http://www.ixsystems.com have done some serious
> research on application specific throughput of ZFS and I believe they also
> spec out SuperMicro servers too.  It comes down to IOPS, raw throughput,
> etc.  (I'm actually talking to them right now about some very large backup
> servers that can handle 0.75 PB.... The consideration I have is space and
> using RAIDZ2 and multiple streams from 10Gb interfaces and serious
> compression and deduplication.  SO,  IOPS not so much, but heavy raw I/O
> and RAID checksum computation and dedup.
> There's also things like dedicated SSDs as ZIL and cache to be thought
> about.
> So:  go up the theoretical OSI layer model and optimize each layer right
> through the application layer. :D  (I actually find it fun)
> I hope this all helps.
> P
>
>     On Thursday, January 17, 2019, 4:33:38 AM CST, Julien Cigar
> <julien at perdition.city> wrote:
>
>  Hello,
>
> We are planning to replace some (web) applications servers (currently
> running HPE) with Supermicro and the vendor offers the following
> choices for the Hardware Raid Controller/HBA 4P part:
>
> 1) Supermicro AOC-S3008L-L8E, LSI 3008 8 x SATA/SAS III JBOD controller, up
> to 122 hard drives via expander backplane, PCI-E, ideal for Nexenta/ZFS
> + € 205,6
>
> 2) LSI 9300-4I, SATA/SAS III JBOD controller, tot 256 harde schijven via
> expander backplanes, PCI-E, ideal for Nexenta/ZFS
> + € 214,5
>
> 3) LSI MegaRAID 9341-4i bulk, 4 x SATA/SAS 12Gbs internal entry level
> hardware RAID, no cache/BBU possible, PCI-e
> + € 177,97
>
> 4) LSI MegaRAID 9361-4i 1GB cache, 4 x SATA/SAS 12Gbs internal
> hardware RAID, max. 240 hdd using expander backplanes
> + € 401,7
>
> 5) LSI 9300-4i4e, SATA/SAS III JBOD controller, 4 x internal, 4 x external,
> up to 256 hard drives via expander backplane, Nexenta Certified, ideal
> for ZFS, PCI-E
> + € 273,95
>
> 6) LSI MegaRAID 9380-4i4e bulk, 8 x SATA/SAS 12Gbs, 4 x external and 4 x
> internal hardware RAID, 1024MB cache, up to 128 hard drives via expander
> backplane, support for SSD CacheCade 2.0 write and read caching,
> CacheVault support (advised), ideal for high en
> + € 676
>
> As the plan is to use ZFS, I was planning to choose the
> "Supermicro AOC-S3008L-L8E, LSI 3008" (it looks like it is supported by
> mpr) and I was wondering if anyone has any feedback on it ? Would
> another option be a better choice ?
>
> Thank you!
> Julien
>
> --
> Julien Cigar
> Belgian Biodiversity Platform (http://www.biodiversity.be)
> PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
> No trees were killed in the creation of this message.
> However, many electrons were terribly inconvenienced.
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>