hardware for home use large storage
dillon at apollo.backplane.com
Tue Feb 9 19:49:56 UTC 2010
The Silicon Image 3124A chipsets (the PCI-e version of the 3124. The
original 3124 was PCI-x). The 3124A's are starting to make their way
into distribution channels. This is probably the best 'cheap' solution
which offers fully concurrent multi-target NCQ operation through a port
multiplier enclosure with more than the PCIe 1x bus the ultra-cheap
3132 offers. I think the 3124A uses an 8x bus (not quite sure, but it
is more than 1x).
AHCI on-motherboard with equivalent capabilities do not appear to be
in wide distribution yet. Most AHCI chips can do NCQ to a single
target (even a single target behind a PM), but not concurrently to
multiple targets behind a port multiplier. Even though SATA bandwidth
constraints might seem to make this a reasonable alternative it
actually isn't because any seek heavy activity to multiple drives
will be serialized and perform EXTREMELY poorly. Linear performance
will be fine. Random performance will be horrible.
It should be noted that while hotswap is supported with silicon image
chipsets and port multiplier enclosures (which also use Sili chips in
the enclosure), the hot-swap capability is not anywhere near as robust
as you would find with a more costly commercial SAS setup. SI chips
are very poorly made (this is the same company that went bust under
another name a few years back due to shoddy chipsets), and have a lot
of on-chip hardware bugs, but fortunately OSS driver writers (linux
guys) have been able to work around most of them. So even though the
chipset is a bit shoddy actual operation is quite good. However,
this does mean you generally want to idle all activity on the enclosure
to safely hot swap anything, not just the drive you are pulling out.
I've done a lot of testing and hot-swapping an idle disk while other
drives in the same enclosure are hot is not reliable (for a cheap port
multiplier enclosure using a Sili chip inside, which nearly all do).
Also, a disk failure within the enclosure can create major command
sequencing issues for other targets in the enclosure because error
processing has to be serialized. Fine for home use but don't expect
miracles if you have a drive failure.
The Sili chips and port multiplier enclosures are definitely the
cheapest multi-disk solution. You lose on aggregate bandwidth and
you lose on some robustness but you get the hot-swap basically for free.
Multi-HD setups for home use are usually a lose. I've found over
the years that it is better to just buy a big whopping drive and
then another one or two for backups and not try to gang them together
in a RAID. And yes, at one time in the past I was running three
separate RAID-5 using 3ware controllers. I don't anymore and I'm
a lot happier.
If you have more than 2TB worth of critical data you don't have much
of a choice, but I'd go with as few physical drives as possible
regardless. The 2TB Maxtor green or black drives are nice. I
strongly recommend getting the highest-capacity drives you can
afford if you don't want your power bill to blow out your budget.
The bigger problem is always having an independent backup of the data.
Depending on a single-instanced filesystem, even one like ZFS, for a
lifetime's worth of data is not a good idea. Fire, theft... there are
a lot of ways the data can be lost. So when designing the main
system you have to take care to also design the backup regimen
including something off-site (or swapping the physical drive once
a month, etc). i.e. multiple backup regimens.
If single-drive throughput is an issue then using ZFS's caching
solution with a small SSD is the way to go (and yes, DFly has a SSD
caching solution now too but that's not pertainant to this thread).
The Intel SSDs are really nice, but I am singularly unimpressed with
the OCZ Colossus's which don't even negotiate NCQ. I don't know much
re: other vendors.
A little $100 Intel 40G SSD has around a 40TB write endurance and can
last 10 years as a disk meta-data caching environment with a little care,
particularly if you only cache meta-data. A very small incremental
cost gives you 120-200MB/sec of seek-agnostic bandwidth which is
perfect for network serving, backup, remote filesystems, etc. Unless
the box has 10GigE or multiple 1xGigE network links there's no real
need to try to push HD throughput beyond what the network can do
so it really comes down to avoiding thrashing the HDs with random seeks.
That is what the small SSD cache gives you. It can be like night and
More information about the freebsd-stable