Re: FYI; 14.3: A discord report of Wired Memory growing to 17 GiBytes over something like 60 days; ARC shrinks to, say, 1942 MiBytes

On 8/11/25 18:24, Sulev-Madis Silber wrote:
> damn if this is the same thing that i experience on 13. battled for long time now
> 
> the worst version of this is when wired grows to entire ram size. then entire system becomes unusable, unless you don't need userland ever
> 
> things that use mmap seem to be very good at it, as well as having storage device write speeds low "helps" as well
> 
> i'm not the only one who reports it
> 
> the problem is also hard to debug it seems, otherwise it would have been fixed? i've seen number of zfs related wtf's over a decade but they do get eventually fixed
> 
> in my case, i don't think it leaks. it does give it back on memory pressure. to a point, still super high wired and i can't see where it goes to. it's not in anywhere in that lines that various stats utils give

If wired is taken up by ZFS ARC it should give it up reasonably quickly 
to within ARC's bounds. Unfortunately ZFS ARC is not the only thing to 
use memory as wired and wired memory is not supposed to be able to be 
swapped out. That has been confusing for ZFS having came from UFS as UFS 
memory was more clearly marked as 'temporarily using RAM while we can' 
for tools like top.

> i think that it rightfully assumes that free ram is wasted ram and just caches something somewhere and doesn't ever give it back or at least does it very slowly

And without caching, ZFS performance can be quite horrible as 
copy-on-write should cause noticeable fragmentation to all but single 
write files while metadata similarly never seems to get regrouped 
efficiently after edits to it without a full copy rewriting it.

> i for example observe it being ok for a while after a reboot but after you start to actually using zfs, running scrubs and so on, it gets into weird state where it's slower. nothing fails, it just stays like this

   I hadn't tracked it down specifically but see ZFS get slower too. My 
'guess' is it has trouble reasonably tracking what file metadata to keep 
in memory once heavy memory pressure has been applied on the system 
(definitely firefox or 'sometimes' a large poudriere job seem easy 
enough to trigger it). Performance stays down if memory pressure causing 
processes have not been closed and it doesn't always seem to work well 
again after closing them.
   I see biggest performance hits to fragmented ZFS filesystem metadata 
which reads very slowly from magnetic disk.
   I think atime updates were one cause of it which is likely why 
installs have disabled that on all but 1 zfs pool; haven't tested if 
relatime is still much of a source of it. I saw very high performance on 
a pool that received a backup of my full system (either 20MB/s or 
200MB/s doing directory listings after transfer clean reboot but can't 
remember) and found very bad performance checking it a short time later 
(<2MB/s not uncommon after cron jobs ran so likely that less database, 
permissions, pkg checks, etc. ran around 'accessing' the content). I had 
disabled atime for the backup pool and set it to read only trying to 
mitigate such unnecessary performance drops but never did a lot of 
testing around it.
  Other disk modifications/deletions cause it too. Git runs slower over 
time doing updates, checking its in a clean state or not, and even 'git 
log' after doing a series of updates. Running `git gc` seems to help but 
seemed less effective when ran every time instead of waiting until a 
number of pull requests and maybe other activities caused slowdown 
first. Cache from ccache is another candidate for horrible performance. 
I have a ccache4 61GB cache that shows it has been through 301 cleanups 
that seems to be getting slower but I've had <20GB cache running more 
than 15x slower that this one after a good # of cleanups. Not permitting 
cleanups until willing to make a copy seems the best choice for 
performance. Running `ccache -X 13` or # greater than original 
compression setting seems to help reorder it somewhat but requires 
ccache4 (more efficient but misses some compiler substitutions in 
poudriere builds so less can be cache accelerated).
   I normally 'fix' the performance degradation by moving the impacted 
area and copying it back (ex: `mv /var/cache/ccache 
/var/cache/ccache.orig&&cp -a /var/cache/ccache2 /var/cache/ccache` and 
then rm -rf the original if the copy creates without error but that 
sometimes becomes more troublesome if a folder is a dataset instead of 
just a folder. It gets silly looking for and managing candidates as even 
things like a Firefox or Thunderbird profile get a lot slower with 
metadata fragmentation. I haven't tried tools to rewrite files in place 
to see if that is enough to cause metadata to be rewritten well without 
full copies being temporarily stored.
   I haven't tracked how much impact snapshots vs no snapshots have 
regarding slowing down datasets that have been through a lot of 
modification/deletion. Making copies this way will increase disk size if 
dedupe and block cloning are not in use and last I heard block cloning 
copies expand out to their own after ZFS replication.
   I find that unmounting and remounting swap devices after memory 
conditions are reasonable again helps but don't know if its truly 
related. If you don't have swap, try adding some and note if there is 
any impact too.

> anyone else with such issues?
> 
> 
> 
> On August 11, 2025 11:18:47 PM GMT+03:00, Mark Millard <marklmi@yahoo.com> wrote:
>> Context reported by notafet :
>>
>> 14.3-RELEASE-p2 GENERIC on amd64 with ZFS in use
>>
>> RAM:  looks to be 24 GiBytes, not explicitly mentioned
>> SWAP: 8192 MiBytes
>> (From using the image of top's figures.)
>>
>> Wired:       17 GiBytes
>> ARC Total: 1942 MiBytes
>>
>> SWAP used: 1102 MiBytes
>>
>> The link to the storage channel's message is:
>>
>> https://discord.com/channels/727023752348434432/757305697527398481/1404367777904463914
>>
>>
>> ===
>> Mark Millard
>> marklmi at yahoo.com
>>
>>
>