13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task

From: Edward Sanford Sutton, III <mirror176_at_hotmail.com>
Date: Tue, 27 Feb 2024 00:24:54 UTC
   Currently trying to do port builds with 
poudriere-devel-3.4.99.20240122 on 13.3-STABLE FreeBSD 13.3-STABLE 
stable/13-n257396-134580c103b4 GENERIC amd64 and have had poor system 
responsiveness on this and a -STABLE that was likely at least 2 months 
before it with `/usr/bin/nice -n 18 /usr/sbin/idprio 31 poudriere bulk 
-J2:12 -j local -p local -f /root/installed-port-list -f 
/root/prime-origins` and also launching it with no nice change and just 
starting with 'idprio 31' (in case it would react different with the 
builtin idprio. It used to be that under just idprio 31 I could continue 
to use the machine during builds with minor impact on responsiveness 
with MAKE_JOBS_NUMBER=4 on a 2 core + hyperthreading i7 which lead to an 
oversaturation of 8 build processes across the 4 virtual cores yet 
stayed mostly smooth.
   Responsiveness seems to have strange effects of laggy mouse/keyboard 
input and thought I recall logs saying system couldn't keep up with the 
mouse communication, programs intermittently freezing/unfreezing, and 
even building becomes unstable with electron28 failing as runaway 
process after over an hour when killed as runaway process during extract 
phase. During the lag, programs seem to take many times as long to 
respond while also getting high cpu use during that time such as tmux 
switching between windows; ctrl b+ctrl+p getting almost 100% cpu core 
time for about 5 seconds added to it under top. blacklistd has take 
about 24 minutes of CPU time for < 3 days uptime per top while managing 
a usual slow bot break-in attempt on ssh.
   More recently looked and see top showing threads+system processes 
shows I have one core getting 100% cpu for kernel{arc_prune} which has 
21.2 hours over a 2 hour 23 minute uptime. Previously I know I have seen 
higher system % time than I'd expected but not always sure when it is 
justified or not. I started looking to see if 
https://www.freebsd.org/security/advisories/FreeBSD-EN-23:18.openzfs.asc 
was available as a fix for 13 but it is not (and doesn't quite sound 
like it was supposed to apply to this issue). Would a kernel thread time 
at 100% cpu for only 1 core explain the system becoming unusually 
unresponsive? cron was stopped after last boot so shouldn't be throwing 
up any unexpected background work.
   System has 32GB ddr3 RAM with i7-3820 processor with only 2 enabled 
cores + hyperthreading in BIOS. Issue appears specifically when CPU load 
is high and idle% in top is 0.0% but it is only sometimes present under 
that condition. I usually use ccache and WTIH_META_MODE to try to speed 
up compiling base and ports and have zstd at best compression for 
packages in hopes of faster extraction at the tradeoff of more disk space.
   I haven't tried yet but considered trying OpenZFS from ports. Any 
suggestions of what else to look at or watch for?
Thanks,
Edward Sanford Sutton, III