ZFS obn FreeBSD hardware model for 48 or 96 sata3 paths...

[Daniel Kalchev]

On Sep 17, 2011, at 08:45 , Rich wrote:

> The 9201-16e or similar would do fine for 6 Gbit SAS rates if you
> don't want hardware RAID.
> 
> To get full bandwidth SATA 3 from 48/96 drives, that's 750 MB/s * 8/10
> (8 data bytes per 10 bytes transmitted raw - SATA 3 does an 8b10b
> encoding) ~ 600 MB/s * 48/96 = 28800/57600 MB/s
> 

There is not single magnetic drive on the market that can saturate SATA2 (300 Mbps), yet. Most can't match even SATA1 (150 MBps). You don't need that much dedicated bandwidth for drives.
If you intend to have 48/96 SSDs, then that is another story, but then I am doubtful a "PC" architecture can handle that much data either.

If you are looking gat IOPS rather than raw throughput, then by all means consider (more) SSDs. You may also consider using 2.5" SAS drives that will be much more compact and less power hungry. The LSI2008 controllers will manage both SATA and SAS drives (at the same time, in the same zpool).

Memory is much more expensive than SSDs for L2ARC and if your workload permits it (lots of repeated small reads), larger L2ARC will help a lot. It will also help if you have huge spool or if you enable dedup etc. Just populate as much RAM as the server can handle and then add L2ARC (read-optimized).

Daniel