Re: Unusual ZFS behaviour

On 11/22/23 00:04, Eugene Grosbein wrote:
> 22.11.2023 13:49, Jonathan Chen wrote:
>> Hi,
>>
>> I'm running a somewhat recent version of STABLE-13/amd64: stable/13-n256681-0b7939d725ba: Fri Nov 10 08:48:36 NZDT 2023, and I'm seeing some unusual behaviour with ZFS.
>>
>> To reproduce:
>>   1. one big empty disk, GPT scheme, 1 freebsd-zfs partition.
>>   2. create a zpool, eg: tank
>>   3. create 2 sub-filesystems, eg: tank/one, tank/two
>>   4. fill each sub-filesystem with large files until the pool is ~80% full. In my case I had 200 10Gb files in each.
>>   5. in one session run 'md5 tank/one/*'
>>   6. in another session run 'md5 tank/two/*'
>>
>> For most of my runs, one of the sessions against a sub-filesystem will be starved of I/O, while the other one is performant.
>>
>> Is anyone else seeing this?

More details of the disk, disk controller, and FreeBSD version may be 
helpful. If it is SATA, maybe there is impact form its own organizing of 
when to handle the queue of tasks it is given in addition to what 
FreeBSD+ZFS have assigned and OS (+ZFS if not its default) version could 
say what IO balancing is currently present/available.

> 
> Please try repeating the test with atime updates disabled:
> 
> zfs set atime=off tank/one
> zfs set atime=off tank/two

atime's impact is a write and writes get priority so if anything there 
would be 'little' breaks in the reads to write such data. I doubt the 
scenario+hardware in discussion is bottlenecking on writing atime data 
for the access of these 10GB files but it would be interesting. On the 
other hand, I think it is atime that trashes a smooth disk of freshly 
created file structure with many files after default cronjobs pass over 
it due to it + COW fragmenting the data structure to do basic things 
like list disk contents; I have not tested properly what the source of 
that repeatable issue was yet. Accessing data within the file doesn't 
seem impacted the same way as the directory listing though.

I was thinking maybe an impact with sysctl settings involving prefetch 
may impact the sequence. vfs.zfs.prefetch_disable=1 is probably the one 
I am thinking of but I normally don't tweak zfs and related settings if 
I don't have to as I usually later find the tweaks are problems themselves.

I keep my system running smoother when I put it under excessive load 
with idprio and nice being used on the heavier noninteractive load.

> Does it make any difference?
> Does it make any difference, if you import the pool with readonly=on instead?
> 
> Writing to ~80% pool is almost always slow for ZFS.

Been there, done that. It is painful but I think it is more complicated 
than just a total free space counter before that issue shows up. Other 
performance issues also exist as I've had horrible I/O on disks that 
never exceeded 20% used since being formatted.