Re: pkg upgrade, vfs.zfs.arc.free_target, vfs.zfs.arc.sys_free, vm.pageout_oom_seq
Date: Sat, 30 Aug 2025 13:50:45 UTC
On Aug 30, 2025, at 02:19, Graham Perrin <grahamperrin@gmail.com> wrote: > On 30/08/2025 03:46, Graham Perrin wrote: > >> ā¦ >> >> pkg upgrade -fUy >> >> a) failed at least once with 12 GB memory without tuning >> >> b) succeeded at least once with 13 GB without tuning >> >> c) succeeded at least twice with 2 GB, ARC tuned. >> >> ā¦ >> >> Temporarily set before (c): >> >> sysctl vfs.zfs.arc.free_target=256000 > > Maybe preferable, an alternative to tuning the ZFS ARC: > > vm.pageout_oom_seq=120 Such assignments of larger than default (12) values should help avoid failures, for sure. (It is not biased to controlling other performance characteristics to be better for the context.) > First test: 2 GB given to the VirtualBox guest, /boot/loader.conf used for the tuning. Success. > > Second test: 1 GB given, temporarily tuned in single user mode before an exit to multi-user. Success. > > Ref: <https://lists.freebsd.org/archives/freebsd-ports/2022-August/002459.html> ā again, thanks to Mark Millard. > > > The physical storage that I use for the virtual disks is slow, even when connected to super speed USB (5Gb/s). An old StoreJet Transcend mobile HDD. The varying slowness might explain why upgrades fails for me (pkg killed, failed to reclaim memory) more often than for other testers. > > During and after a simple ten-step upgrade from FreeBSD-ports: > > grahamperrin@mowa219-gjp4 ~> zpool iostat Transcend 5 > capacity operations bandwidth > pool alloc free read write read write > ---------- ----- ----- ----- ----- ----- ----- > Transcend 777G 151G 36 65 4.01M 5.00M Hmm. 151G/(777G+151G) approx= 0.163 for the fraction free. That is well outside the recommended free fraction range for efficient ZFS use. The Design and Implementation of the FreeBSD operation System, second edition, page 548, 2nd bullet: "Like all non-overwriting filesystems, ZFS operates best when a least a quarter of its disk pool is free. Write throughput becomes poor when the pool gets too full. By contrast, UFS can run well to 95 percent full and acceptably to 99 percent full." Also, page 549: "ZFS was designed to manage and operate enormous filesystems easily, which it does well. Its design assumed that it would have many fast 64-bit CPUs with large amounts of memory to support these enormous filesystems. When these resources are available, it works extremely well. However, it is not designed for or well suited to run on resource-constrained systems using 32-bit CPUs with less than 8 Gbytes of memory and one small, nearly full disk, which is typical of many embedded systems." Note: I take that last to mean that even one of: ) 32-bit CPUs ) few 64-bit CPUs ) slow 64-bit CPUs ) less than 8 Gbytes of memory (or insufficient memory more generally) ) nearly full disk (slow storage would make that worse) can lead to the "not well suited" status for ZFS use. Use of ZFS inside virtual machines with any such characteristics would not be exempted from that potential status. Also on Page 548: "ZFS caches its data in its ARC that is not part of the unified-memory cache managed by the virtual memory. The result is that when mmap is used on a ZFS file, . . . This approach provides coherency between memory-mapped and I/O access at the expense of wasted memory due to having 2 copies of the file in memory and extra overhead caused by the need to copy the contents between the two copies." (sendfile also mentioned as having a similar issue.) > Transcend 777G 151G 34 120 5.28M 9.44M > Transcend 777G 151G 79 80 8.04M 5.97M > Transcend 777G 151G 82 42 1.72M 3.99M > Transcend 777G 151G 166 18 3.23M 2.11M > Transcend 777G 151G 157 74 3.89M 5.59M > Transcend 777G 151G 64 91 4.48M 9.01M > Transcend 777G 151G 8 27 904K 2.27M > Transcend 777G 151G 0 18 16.0K 1.36M > Transcend 777G 151G 0 0 0 0 > Transcend 777G 151G 0 0 0 0 > Transcend 777G 151G 0 12 819 218K > Transcend 777G 151G 0 0 0 0 > Transcend 777G 151G 0 0 0 0 > Transcend 777G 151G 0 0 15.2K 0 > Transcend 777G 151G 0 13 0 369K > Transcend 777G 151G 0 11 0 139K > Transcend 777G 151G 0 0 0 0 > ^Cā > grahamperrin@mowa219-gjp4 ~ [SIGINT]> zfs version > zfs-2.3.1-1ubuntu2 > zfs-kmod-2.3.1-1ubuntu2 > grahamperrin@mowa219-gjp4 ~> lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 25.04 > Release: 25.04 > Codename: plucky > grahamperrin@mowa219-gjp4 ~> === Mark Millard marklmi at yahoo.com