Anyone using freebsd ZFS for large storage servers?
wojtek at wojtek.tensor.gdynia.pl
Fri Jun 1 09:00:03 UTC 2012
> 48TB each, roughly. There would be a couple of units. The pizza
> boxes would be used for computational tasks, and nominally would have
> 8 cores and 96G+ RAM.
> Obvious questions are hardware compatibility and stability. I've set
> up small FreeBSD 9 machines with ZFS roots and simple mirrors for
> other tasks here, and those have been successful so far.
> Observations would be appreciated.
you idea of using disks in JBOD style (no "hardware" RAID) is good, but of
using ZFS is bad.
i would recommend you to do some real performance testing of ZFS on any
config under real load (workload doesn't fit cache, there are many
different things done by many users/programs) and compare it to
PROPERLY done UFS config on such config (with the help of gmirror/gstripe)
if you will have better result you certainly didn't configure the latter
case (UFS,Gmirror,gstripe) properly :)
in spite of large scale hype and promotion of this free software (which by
itself should be red alert for you), i strongly recommend to stay away from it.
and definitely do not use it if you will not have regular backups of all
data, as in case of failures (yes they do happen) you will just have no
chance to repair it.
There is NO fsck_zfs! And ZFS is promoted as it "doesn't need" it.
Assuming that filesystem doesn't need offline filesystem check utility
because it "never crash" is funny.
In the other hand i never ever heard of UFS failsystem failure that was
not a result of physical disk failure and resulted in bad damage.
in worst case some files or one/few subdirectory landed in lost+found, and
some recently (minutes at most) done things wasn't here.
if you still like to use it, do not forget it uses many times more CPU
power than UFS in handling filesystem, leaving much to computation
you want to do.
As of memory you may limit it's memory (ab)usage by adding proper
statements to loader.conf but still it uses enormous amount of it.
with 96GB it may not be a problem for you, or it may depends how much
memory you need for computation.
if you need help in properly configuring large storage with UFS and
gmirror/gstripe tools then feel free to ask
More information about the freebsd-questions