ZFS obn FreeBSD hardware model for 48 or 96 sata3 paths...
Alexander at Leidinger.net
Mon Sep 19 20:37:17 UTC 2011
On Mon, 19 Sep 2011 12:00:11 -0700 (PDT) Jason Usher
<jusher71 at yahoo.com> wrote:
> --- On Sat, 9/17/11, Daniel Kalchev <daniel at digsys.bg> wrote:
> > There is not single magnetic drive on the market that can
> > saturate SATA2 (300 Mbps), yet. Most can't match even SATA1
> > (150 MBps). You don't need that much dedicated bandwidth for
> > drives.
> > If you intend to have 48/96 SSDs, then that is another
> > story, but then I am doubtful a "PC" architecture can handle
> > that much data either.
> Hmmm... I understand this, but is there not any data that might
> transfer from multiple magnetic disks, simultaneously, at 6GB, that
> could periodically max out the card bandwidth ? As in, all drives in
> a 12 drive array perform an operation on their built-in cache
> simultaneously ?
A pragmatic advise: Do not put all drives into the the same vdev. Have
a look at the ZFS best practices guide for some words about how much
drives shall be in the same vdev. Concatenate several RAIDZx vdevs
instead. Play a little bit around on paper what works best for you.
With 8 controllers (assuming 6 ports each) you could do 6 raidz1
vdevs (one drive from each controller in the same raidz1) which are
concatenated to give you a pool of
ports*num_controller_minus_one*drivesize amount of storage. From each of
those 6 vdevs one drive (6 in total = number of raidz1 vdevs) or
one controller (number of controllers which can fail = the X in raidzX)
If you need speed, rely on RAM or L2ARC (assuming the data is read
often enough to be cached). If you need more speed, go with SSDs
instead of harddisks for the pool-drives (a L2ARC does not make
much sense then, except you invest in something significant faster
like a Fusion-board as already mentioned in the thread).
Optimizing for the theoretical case that all drives deliver everything
from the HD-cache is a waste of money because you are either in the
unlikely case that this really happens (go play in the lottery instead,
you may have more luck). If the access pattern is really that strange
that it happens often enough for you that such an optimization would
give a nice speed increase, invest the money in more RAM to have the
data in the ARC instead.
> > Memory is much more expensive than SSDs for L2ARC and if
> > your workload permits it (lots of repeated small reads),
> > larger L2ARC will help a lot. It will also help if you have
> > huge spool or if you enable dedup etc. Just populate as much
> > RAM as the server can handle and then add L2ARC
> > (read-optimized).
> That's interesting (the part about dedup being assisted by L2ARC) ...
> what about snapshots ? If we run 14 or 21 snapshots, what component
> is that stressing, and what structures would speed that up ?
A snapshot is a short write to the disks. I do not know if it is a sync-
or async-write. If you do not do a lot of snapshots per second or
minute (I hope the 14/21 values mean to take a snapshot per
(working-)hour), I would not worry about this.
http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137
More information about the freebsd-fs