zfs performance degradation

[Dmitrijs]

2015.09.24. 16:17, Paul Kraus wrote:
> So if I plan to work with data a lot, get decent performance and still be sure I'm on the safe side with mirror-raid1, should I choose another filesystem? Especially, if i do not really need snapshots, clones, etc.
> What is your definition of “decent” performance ? What does your _real_ workload look like ?
>
> Did you have performance issues doing real work which caused you to try to find the cause -or- were you benchmarking before trying to use the system for real work ?

As we use our file server mostly to store users' backup and hdd images 
for old and new computers. File sizes mostly go 10-100Gb+. In the 
daytime the usually have 1 "writer" and several "readers". And datetime 
performance is very important to us. In the nighttime several "writers" 
and no "readers". Most file operations are sequential. Working 
environment mostly ms windows 7. We used to have software mirror raid on 
one windows 7 computer, shared to our gigabit network. Now we decided to 
have dedicate box for file storage. In my terms "decent" where 
performance is limited mostly by gigabit network, not the file 
operations. So if zfs raid mirror can give me total 100Mbs+ output at 
least on 1-2 readers - it's ok. When I have total 70-80Mbs and less in 
perspective - I worry that I'm doing something wrong.

And as you see, I'm just beginning to dig into freebsd file systems etc, 
so my question and answers may be inadequate.

>> Or is it not possible at all, and I should put something like raid0 for work and tolerate slow backup on raid1 at nights?
> There are many places in ZFS where you can run into performance bottlenecks. Remember, ZFS was designed for data integrity (end to end checksums), data reliability (lots of ways to get redundancy), and scalability. Performance was secondary from the very beginning. There are lots of other filesystems with much better performance, there are few (if any) with more protection for your data. Do not get me wrong, the performance of ZFS _can_ be very good, but you need to understand your workload and layout the zpool to accommodate that workload.

Thank you for clarification. I made some research and found out that zfs 
is really very good system for the points you mentioned. That's why we 
choose freebsd and zfs. And that's exactly where I need help - to 
understand how big is price for zfs reliability and options. If we need 
to buy better CPU, add more RAM, more disks to get better performance - 
it must be considered, that's one thing.  And if there are tweaks, that 
can be adjusted via settings - that's another thing.

> For example, one of my critical workloads is NFS with sync writes. My zpool layout is many vdevs of 3-way mirrors with a separate ZIL device (SLOG). I have not been able to go production with this server yet because I am waiting on backordered SSDs for the SLOG. The original SSDs I used just did not have the small block write performance I needed.
>
> Another example is one of my _backup_ servers, which has a 6 drive RAIDz2 zpool layout. In this case I am not terribly concerned about performance as I am limited by the 1 Gbps network connection.
>
> Also note that in general, the _best_ performance you can expect of any zpool layout is equivalent to _1_ drives worth of I/O per _vdev_. So my 6 drive RAIDz2 has performance equivalent to _one_ of the drives that make up that vdev. Which is fine for _my_ workload. The rule of thumb for performance that I received over on the OpenZFS mailing list a while back was to assume you can get 100 MB/sec and 100 random I/Ops from a consumer SATA hard disk drive. I have seen nothing, even using “enterprise” grade HDDs, to convince me that is a bad rule of thumb. If your workload is strictly sequential you _may_ get more.
>
> So a zpool made up of one single vdev, no matter how many drives, will average the performance of one of those drives. It does not really matter if it is a 2-way mirror vdev, a 3-way mirror vdev, a RAIDz2 vdev, a RAIDz3 vdev, etc. This is more true for write operations that read (mirrors can achieve higher performance by reading from multiple copies at once).
Thanks! Now I understand. Although it is strange, that you did not 
mention how RAM and\or CPU matters. Or do they? I start observing that 
my 4core Celeron J1900 is throttling writes. Still haven't found at 
least approximate specification\recommendation as simple as "if you need 
zfs mirror 2 drives, take at least core i3 or e3 processor, 10 drives - 
go for e5 xeon, etc". I did not notice cpu impact on windows machine, 
still i've got " load averages:  1.60, 1.43,  1.24 " on write on zfs.

best regards,
Dmitriy