ZFS obn FreeBSD hardware model for 48 or 96 sata3 paths...

[Jason Usher]

Hello,

I am building my first FreeBSD based ZFS system and am deciding on a hardware model.  The overriding requirement is:

1) immediately support 48 internal sata3 drives at full bandwidth - every drive has independent path to CPU

2) future expansion to support another 48 drives on an attached JBOD, all of which ALSO have their own independent path to CPU

The first question is:  how many pcie 2.0 lanes does a motherboard need to run 96 independent sata3 connections ?  Am I correct that this is extremely important ?

Next, I see a lot of implementations done with LSI adaptors - is this as simple as choosing (3) LSI SAS 9201-16i for the 48 internal drives and (3) LSI SAS 9201-16e for the external drives ?  I remember that these cards are supported with mps(4) in FreeBSD, but only in 9.x (?) - is that still the case, or is that support in 8.2 or later in 8.3 ?

So I will boot of a pair of mirrored SSDs formatted UFS2 - easy.  But I would also like to spec and use a ZIL+L2ARC and am not sure where to go ... the system will be VERY write-biased and use a LOT of inodes - so lots of scanning of large dirs with lots of inodes and writing data.  Something like 400 million inodes on a filesystem with an average file size of 150 KB.

- can I just skip the l2arc and just add more RAM ?  Wouldn't the RAM always be faster/better ?  Or do folks build such large L2arcs (4x200 GB SSD ?) that it outweighs an extra 32 GB of RAM ?

- provided I maintain the free pcie slot(s) and/or free 2.5" drive slots, can I always just add a ZIL after the fact ?  I'd prefer to skip it for now and save that complexity for later...

Thanks very much for any comments/suggestions.