Re: ZFS pool balance and performance

On 8/24/2025 01:46, Chris Ross wrote:
> I have been having some performance problems with a large ZFS pool, which
> mostly contains large files that are accessed (read) via NFS.  The pool
> was built with two 3-disk raidz vdevs, together.  I have replaced the
> drives in those vdevs a couple of times over the years to increase capacity.
> As of today I have a 75T pool that is 89% capacity and 41% fragmented.  I
> know that performance can become an issue as a pool becomes nearly full,
> and I plan to move this whole system to a newer system with larger faster
> drives, etc etc.  But that is taking longer than I'd planned.
>
> So, noticing particular problems tonight I looked more deeply at it.  And,
> I found that the two vdevs are unequally full.  By a good amount!
>
> NAME                         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
> tank                        75.8T  67.8T  8.01T        -         -    41%    89%  1.00x    ONLINE  -
>    raidz1-0                  37.6T  30.8T  6.87T        -         -    27%  81.7%      -    ONLINE
>      da1p4                   12.6T      -      -        -         -      -      -      -    ONLINE
>      diskid/DISK-QGH0S3UTp1  12.7T      -      -        -         -      -      -      -    ONLINE
>      diskid/DISK-QGH0Y5ATp1  12.7T      -      -        -         -      -      -      -    ONLINE
>    raidz1-1                  38.2T  37.0T  1.14T        -         -    55%  97.0%      -    ONLINE
>      diskid/DISK-9JG7REXTp1  12.7T      -      -        -         -      -      -      -    ONLINE
>      diskid/DISK-9JG3M05Tp1  12.7T      -      -        -         -      -      -      -    ONLINE
>      diskid/DISK-9JG7RRNTp1  12.7T      -      -        -         -      -      -      -    ONLINE
>
> So one of the vdevs is 97% full.  I fear that is causing my occasional
> read performance issues on my NFS exported filesystem.
>
> So, questions.  (1) Can I rebalance.  Searching tells me no, ZFS can’t
> do that.  Nor defrag.  (2) Is there any way I can identify files that
> are more on raidz1-1, and delete them?

I generally use mirror sets rather than raidzs and on the "large" pools 
I run I haven't run into this sort of imbalance, despite doing 
incremental pool expansions (e.g. replace all the vdev elements in one 
with larger disks, thus expanding the storage) several times.

As someone else noted you might want to see if one of those elements in 
radidz1-0 has some sort of problem (e.g. being an SMR disk where the 
others are not, etc.)  zpool status -s might be useful (shows any vdev 
elements that are slow but do complete the I/O) which could cause zfs to 
disfavor that vdev.

The only place I don't get hinky with fragmentation is on SSDs which of 
course have no rotational or head movement latency; on any rotating 
media it hurts you with only caching (e.g. ARC buffering) being of use 
against it.  I'm unaware of any good means to rebalance allocation 
"in-place."

-- 
Karl Denninger
karl@denninger.net
/The Market Ticker/
/[S/MIME encrypted email preferred]/