Suggestion for hardware for ZFS fileserver

[Karl Denninger]

On 12/21/2018 10:51, Sami Halabi wrote:
> Hi,
> First thanks for everyone who responded, very helpful.
>
> One 2 more questions/clarifications:
> 1. As many mentioned 2008/3008 LSI HBA I searched (for lsi sas hba 24
> ports) and found many but with lsi i/o controller other than 2008/3008, for
> example 3224)..it seems 2008/3008 are more popular in 8port hba cards.
> 2. I got That as much ram is better however I saw almost no docs /
> recommendations and how to estimate CPU.. So I need more clarification on
> how to size CPU with the following configurations ZFS for raidz2, all
> without dedup:
> A) lz4 compression.
> B) encryption (I must say I still not quiet know what are my options)
> C) A+B
>
> Thanks,
> Sami
> בתאריך יום ו׳, 21 בדצמ׳ 2018, 12:33, מאת
IMHO --

1. RAM *must* be ECC.  No wiggle room here.  Undetected RAM corruption
on a ZFS-heavy system is EXTREMELY likely to get onto the disk with a
correct checksum, which means permanent, undetectable corruption of the
data and possibly a pool that, when certain operations are performed on
it, immediately panics the system.  In other words it's entirely
possible to get into a situation where the only "remedy" is to destroy
the pool impacted and re-create it.  Incidentally this means that
/backups are not optional; you must have them and develop something that
results in a PROVABLE good copy that can be restored in the event of
disaster because of not only this risk but also human error that the
operating system and hardware executes exactly as you requested, either
of which results in unrecoverable data loss./

2. More RAM is better, up to a point, in that cache is faster than disk
I/O in all cases as operations are avoided.  HOWEVER, there are
pathologies in both the FreeBSD VFS and the ARC when considered as a
group.  I and others have tried to eliminate the pathological behavior
under certain workloads (and some of us have had long-running debates on
same.)  Therefore, your workload must be considered -- simply saying
"more is better" may not be correct for your particular circumstances.

3. LZ4 is good for compressible data.  It is worthless for
non-compressible (or already-compressed) data, such as for example MP3s,
MP4s (video), etc. and in fact makes performance worse since the ZFS
code has to detect that the compression is futile and that takes cycles.

4. On FreeBSD I prefer GELI on the base partition to which ZFS is then
pointed as a pool member for encryption at the present time.  It's
proven, uses AES hardware acceleration on modern processors and works. 
Read the documentation carefully and understand your options for keying
(e.g. password only, password + key file, etc) and how you will manage
the security of the key component(s).


-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4897 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20181221/a2957ad5/attachment.bin>