Re: ZFS pool balance and performance

On 24/08/2025 06:46, Chris Ross wrote:
> I have been having some performance problems with a large ZFS pool, which
> mostly contains large files that are accessed (read) via NFS.<
<snip>
> Anyway, sorry for long text, but would appreciate any thoughts.
> Thank you.
>
> - Chris

I'm glad you were asking for thoughts rather than answers. I had a few 
thoughts, which may have occurred already.

It looks like one of the drives has been replaced. I've bought 
replacement drives of the same model only to discover they've changed to 
SMR - the array ran very badly until removed.

I can't explain the uneven use, as (I believe) ZFS normally favours 
writes to the vdev with the most free space. Has it been skewed by 
compression? It is also said that ZFS favours the fastest vdevs, which 
is why I mention the SMR drive. Could it be its decided to use raidz1-1 
in favour of a very slow raidz1-0?

However, there's a lot said about how clever ZFS is and it's hard to 
tell the aspiration from the actuality.

Fragmentation is CoW's dirty secret and the answer, IME, is actually RAM 
- and lots of it. If the working data is in a cache the underlying head 
movements don't matter as much.

You also mentioned you were using NFS, but not what for other than 
reads. You might want to take a look at this if there are any 
synchronous writes going on:

https://blog.frankleonhardt.com/2017/esxi-nfs-zfs-and-vfs-nfsd-async/

In the end I hacked and recompiled the NFS code.

And when it comes to fragmentation, in my experiments (nearly ten years 
ago), it was pretty clear that ZFS pool performance degraded on 
fragmentation and the amount of free space. Up to 50% full it was fine; 
beyond 80% full the performance went off a cliff. I'm pretty sure the 
fragmentation is to blame, which gets much worse when the pool is at 
80%. Logically this is going to depend on job mix, but after being paid 
to spend six months finding out what ZFS configuration worked best for a 
couple of use cases I now never let a zpool go beyond 80% if I can help 
it. The roll off was dramatic.

Regards, Frank.