From nobody Wed Mar 04 10:03:42 2026 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fQpDf2Vv3z6TGxk for ; Wed, 04 Mar 2026 10:04:34 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (prime256v1) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "E7" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fQpDd6Tj2z4BsQ; Wed, 04 Mar 2026 10:04:33 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1772618661; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7iUhZWpN0zM2PomGosxXw/Cawd8vSoJBFvCZEsUqQoI=; b=A2IlxUA41Id1rPW9Ekuy5Vic8ZDX4dGOEY5QKxjFRY0eAdWT16jgVeSKIzHT3xxZpBZ9Pd Nd9RYYSG+FsO8shBwsxSSmkT74aSxWB8AsQ0eubZcINkEf7nD6fFpj0prjingMyLPziFMt N6/SxuKyqx1isbn/T0kS/S2BDUVNwHim5hGup0NaMqeJVMXZ/BlgV/OYDtIkKsbT6/YrQa 04iSReKvb+6852OKeFLzWFaTAXYFBTFXSLtsVOeCeHcZLAyttYjw15zOqzaXMG/5BeXOwz YhY+1B6ee7SZjpw6DxesgePovnLkmuDSA6xAC39516/0PIHNbUlPabn6KHTHbA== Date: Wed, 04 Mar 2026 11:03:42 +0100 From: Alexander Leidinger To: Doug Ambrisko , Mark Johnston Cc: Rick Macklem , Peter Eriksson , FreeBSD CURRENT , Garrett Wollman , Alexander Motin Subject: Re: RFC: How ZFS handles arc memory use In-Reply-To: References: <22b478c6bad8212c61ca19a983a8e2e4@Leidinger.net> Message-ID: <0d466ee1739ff7ddc967d725453dda35@Leidinger.net> Organization: No organization, this is a private message. Content-Type: multipart/signed; protocol="application/pgp-signature"; boundary="=_57870c04fc020f8149b341cf93edbefe"; micalg=pgp-sha256 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE] X-Rspamd-Queue-Id: 4fQpDd6Tj2z4BsQ X-Spamd-Bar: ---- This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_57870c04fc020f8149b341cf93edbefe Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8; format=flowed Am 2026-03-03 23:45, schrieb Doug Ambrisko: > On Tue, Mar 03, 2026 at 02:25:11PM -0800, Rick Macklem wrote: > | On Tue, Mar 3, 2026 at 12:33 PM Doug Ambrisko > wrote: > | > > | > On Sun, Nov 02, 2025 at 11:48:06AM +0100, Alexander Leidinger > wrote: > | > | Am 2025-10-29 22:06, schrieb Doug Ambrisko: > | > | > It seems around the switch to OpenZFS I would have arc clean > task > | > | > running > | > | > 100% on a core. I use nullfs on my laptop to map my shared ZFS > /data > | > | > partiton into a few vnet instances. Over night or so I would > get into > | > | > this issue. I found that I had a bunch of vnodes being held by > other > | > | > layers. My solution was to reduce kern.maxvnodes and > vfs.zfs.arc.max so > | > | > the ARC cache stayed reasonable without killing other > applications. > | > | > > | > | > That is why a while back I added the vnode count to mount -v so > that > | > | > I could see the usage of vnodes for each mount point. I made a > script > | > | > to report on things: > | > | > | > | Do you see this also with the nullfs mount option "nocache"? > | > > | > I seems to have run into this issue with nocache > | > /data/jail/current/usr/local/etc/cups > /data/jail/current-other/usr/local/etc/cups nullfs rw,nocache 0 0 > | > /data/jail/current/usr/local/etc/sane.d > /data/jail/current-other/usr/local/etc/sane.d nullfs rw,nocache 0 0 > | > /data/jail/current/usr/local/www > /data/jail/current-other/usr/local/www nullfs rw,nocache 0 0 > | > /data/jail/current/usr/local/etc/nginx > /data/jail/current-other/usr/local/etc/nginx nullfs rw,nocache 0 0 > | > /data/jail/current/tftpboot > /data/jail/current-other/tftpboot nullfs rw,nocache 0 0 > | > /data/jail/current/usr/local/lib/grub > /data/jail/current-other/usr/local/lib/grub nullfs rw,nocache 0 0 > | > /data/jail > /data/jail/current-other/data/jail nullfs rw,nocache 0 0 > | > /data/jail > /data/jail/current/data/jail nullfs rw,nocache 0 0 > | > > | > After a while (a couple of months or more). My laptop was running > slow > | > with a high load. The perodic find was running slow. arc_prunee > was > | > spinning. When I reduced the number of vnodes then things got > better. > | > My vfs.zfs.arc_max is 1073741824 so that I have memory for other > things. > | > > | > nocache does help taking longer to get into this situation. > | Have any of you guys tried increasing vfs.zfs.arc.free_target? > | > | If I understand the code correctly, when freemem < > vfs.zfs.arc.free_target > | the reaper thread (the one that does uma_zone_reclaim() to return > pages > | to the system from the uma keg that the arc uses) should be > activated. > > I haven't tried that. I set: > kern.maxvnodes > vfs.zfs.arc.min > vfs.zfs.arc.max > vfs.zfs.prefetch.disable=1 > > I need to make sure kern.maxvnodes is small enough so it doesn't thrash > when vfs.zfs.arc.max set to 1G. The issues tend to take a while to > happen. On the plus side I can adjust these when I hit them mostly by > reducing kern.maxvnodes without having to do a reboot. There was this commit recently_ https://cgit.freebsd.org/src/commit/sys/fs/nullfs?id=8b64d46fab87af3ae062901312187f3a04ad2d67 I have not checked if this race condition could result in anything related to what we see. From the commit message I can not deduct if this could for example lead to a (even temporary) resource leak which may explain this behavior. Mark, what is the high-level result of this race condition you fixed in nullfs? At first look at the commit log I would rather assume vnodes of the lower FS could rather be freed more early and not at all because of the race condition. Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_57870c04fc020f8149b341cf93edbefe Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc; size=833 Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmmoA44ACgkQEg2wmwP4 2IYDKg/9FTVs+hfPa5Tn2eQ2BxhvsUSEELIivdJP/FfF160CAossxcsVZiRU8Y4d rNSHxaDB58r6SewYQ4ky5yo3zRnJA0+hrgdhow+65u/dY95+op+dELXiSdG72kLU 22+aMGuB5hGXs35Sykhc0hoBAzQRnrynaIg2z4S2L/fO2qI1QObYOTkzRrq98znX JCY4HY3v1eBVNoHQWzx7fm9ONfSre9K5LDjaGXo/ww65luWiYOWe6o13TMnW1vD5 ioOjM9S7GvXMOFWi6vA05/JyAEb107oy80JDOijR0U/xn2KI1mRHeGskRLE8yaYp ZfFgTYskcDxfVkzK2wGIu1ue8pUIaVTCWtYjrdGPEtD6R6vdbZgC0JPY4orsfFoU MWOEx9LgOVqjJEOOGQJzltrQi2f5fS0D7gt3Nz3xNpO7ypmeo/2SDypEvyZVNddw gkU3AZ+wiH2MXWLtzKHsWUuVXEIjK7G3I7O5CzvoGNs8i9B9N3+uycWQ7PJgo+I6 PRKxaTHqUXqvL9agUIu9NAHfHV8x3hHmYXhnXJG/boGnM66Wcb+jplANt2dqJ/oH PwpwJEwZFg/tjL0Zeg1rjpgAM+3Kw8ftTUGSBxP819/9ywyB+qtM9Lasp8B2kKqR 9QBho3iqWdgK90owxEACFaNkCk0cjzckWV0zgy89Pe3j1k0UaSM= =TVeK -----END PGP SIGNATURE----- --=_57870c04fc020f8149b341cf93edbefe--