zfs performance degradation

[Dmitrijs]

2015.09.24. 17:40, Paul Kraus wrote:
> Do you have compression turned on ? I have only seen ZFS limited by 
> CPU (assuming relatively modern CPU) when using compression. If you 
> are using compression, make sure it is lz4 and not just “on". RAM 
> effects performance in that pending (async) writes are cached in the 
> ARC. The ARC also caches both demand read data as well as prefetched 
> read data. There are a number of utilities out there to give you 
> visibility into the ARC. `sysctl -a | grep arcstats` will get you the 
> raw data :-) When you benchmark you _must_ use a test set of data that 
> is larger than your RAM you you will not be testing all the way to / 
> from the drives :-) That or artificially reduce the size of the ARC 
> (set vfs.zfs.arc_max=“<bytes>” in /boot/loader.conf). 

Nope, no compression, no deduplication, only pure zfs. Even no prefetch, 
as it is not recommended for machines 4Gb RAM and below.
I've tested performance with 40Gb file on 4Gb ram machine, so cache 
should not count so much.
I really hoped that I could get from 2HDD MIRROR at least 1.5x read 
performance of a single HDD, but it's more tricky as you explained.
Now I'm not sure what configuration will make better performance for 4 
HDD - raid10 or raid-z2? Or two separate mirrors? Need directions for 
scale things up in the future.


>> Still haven't found at least approximate specification\recommendation as simple as "if you need zfs mirror 2 drives, take at least core i3 or e3 processor, 10 drives - go for e5 xeon, etc". I did not notice cpu impact on windows machine, still i've got " load averages:  1.60, 1.43,  1.24 " on write on zfs.
> How many cores / threads ? As long as you have more cores / threads than the load value you are NOT out of CPU resources, but you may be saturating ONE CPU with compression or other function.
>
> I have been using HP Proliant MicroServer N36L, N40L, and N54L for small file servers and I am only occasionally CPU limited. But my work load on these boxes is very different from yours.
>
> My backup server is a SuperMicro with dual Xeon E5520 (16 total threads) and 12 GB RAM. I can handily saturate my single 1 Gbps network. I have compression (lz4) enabled on all datasets.

I've got 
http://ark.intel.com/products/78867/Intel-Celeron-Processor-J1900-2M-Cache-up-to-2_42-GHz
And 4Gb RAM. Thought it would be sufficient, but now I'm in doubt. I can 
live with reduced performance for my 1st NAS, but would be nice to have 
clear performance requirements in mind for planing future storage boxes.

I see QNAPs and Synology NAS, they use like 1Ghz CPU and 1Gb of RAM for 
4 HDD, so either I'm doing it wrong, either those NASes don't have 
performance (or safety?) at all.
HP Proliant MicroServer is nice, but i've made my diskless system 2-3 
times cheaper (200euro vs 530/650euro), so I need a reason or 
recomendation to spend x2x3 money on the thing, which specification 
looks the same.

best regards,
Dmitriy