ZFS obn FreeBSD hardware model for 48 or 96 sata3 paths...

[Joshua Boyd]

On Fri, Sep 16, 2011 at 9:22 PM, Jason Usher <jusher71 at yahoo.com> wrote:

>
> Hello,
>
> I am building my first FreeBSD based ZFS system and am deciding on a
> hardware model.  The overriding requirement is:
>
> 1) immediately support 48 internal sata3 drives at full bandwidth - every
> drive has independent path to CPU
>
> 2) future expansion to support another 48 drives on an attached JBOD, all
> of which ALSO have their own independent path to CPU
>
> The first question is:  how many pcie 2.0 lanes does a motherboard need to
> run 96 independent sata3 connections ?  Am I correct that this is extremely
> important ?
>
> Next, I see a lot of implementations done with LSI adaptors - is this as
> simple as choosing (3) LSI SAS 9201-16i for the 48 internal drives and (3)
> LSI SAS 9201-16e for the external drives ?  I remember that these cards are
> supported with mps(4) in FreeBSD, but only in 9.x (?) - is that still the
> case, or is that support in 8.2 or later in 8.3 ?
>
> So I will boot of a pair of mirrored SSDs formatted UFS2 - easy.  But I
> would also like to spec and use a ZIL+L2ARC and am not sure where to go ...
> the system will be VERY write-biased and use a LOT of inodes - so lots of
> scanning of large dirs with lots of inodes and writing data.  Something like
> 400 million inodes on a filesystem with an average file size of 150 KB.
>
> - can I just skip the l2arc and just add more RAM ?  Wouldn't the RAM
> always be faster/better ?  Or do folks build such large L2arcs (4x200 GB SSD
> ?) that it outweighs an extra 32 GB of RAM ?
>
> - provided I maintain the free pcie slot(s) and/or free 2.5" drive slots,
> can I always just add a ZIL after the fact ?  I'd prefer to skip it for now
> and save that complexity for later...
>
> Thanks very much for any comments/suggestions.
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>

I've built something similar, using 3 Supermicro SC933 chassis, 2 HP SAS
expanders, 2 AOC-USAS-L8i cards, and 1 card with 2 external ports (I can't
remember the exact name, but it's an LSI chipset card). This is a 45 drive
capable setup, so smaller than what you're wanting.

I'd recommend you get two of these:

http://www.supermicro.com/products/chassis/4U/847/SC847E26-RJBOD1.cfm

That gives you 90 drives in 8U. They each have dual port expanders
integrated to the backplanes. Then build a separate 1 or 2u box that holds
your boot drives/cache drives. In this box put in 2 6Gb cards with external
SAS connectors. Something like the 9750-8E, which are 6Gbit/s cards and
support drives bigger than 2TB. You'll need to run 8-STABLE, as these cards
use the mptsas driver, which isn't in 8-RELEASE last I checked.

I don't have any experience with separate cache/log devices, so I can't
offer much advice there.

-- 
Joshua Boyd

E-mail: boydjd at jbip.net
http://www.jbip.net