From nobody Wed Oct 22 14:34:39 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4csBX23GMgz6FMyB for ; Wed, 22 Oct 2025 14:34:58 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4csBX164vZz476j for ; Wed, 22 Oct 2025 14:34:57 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=NaPmRmjz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2a00:1450:4864:20::52c as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-ed1-x52c.google.com with SMTP id 4fb4d7f45d1cf-63c1a0d6315so12582686a12.1 for ; Wed, 22 Oct 2025 07:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1761143692; x=1761748492; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=WOMQVKfdVsWV4nO1TV7WgN9kSQVVnqLtMTtOLAFEgzw=; b=NaPmRmjzE1MA3kpyP7TRq8VyVSVTutr4iywGHWiVykR2Oynk0t8zHtR6xBc3E1wH9q AiWV0/Jr9ykHYr+3IeInOxhMn3sk4CFOmdbCYZziUpJn8sr9NT0fD2B5K73NNHhWm6iI zAEQXK4FDlIvIBe/F/awT7u/LXhbSB+WiNvSAYWdyGNWPfczPJLyQHRNUU56YgjNsbLa k8V6cu7T1bwm/1P7NiypDiQHghVMptKsCmS1YLFvGT9mCJJwhNvdeF7IWCLFm3yL9FU5 gPVYg/PnvImA8l2wKTMKuxLMFwW5nYHTDMVB7LgnJUwXH67oJpb/nYE9vOf2RpS41x/m Rjbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761143692; x=1761748492; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=WOMQVKfdVsWV4nO1TV7WgN9kSQVVnqLtMTtOLAFEgzw=; b=SEKb1cxcsoqsVDi+G2Ew0Pl4sCSu9QQm2z8K5Qs64rkc3Vh9P9eeue5gJRxEm9MEpv vM5gm5YqQbUIkbdhkGwtW2RAPQqZKGE3N8w7LzYS84tkzA9D47YtTuOnWOHAeCvJXfZ3 evs6p/y1ebgxNfamTwug7WleWkZhxwem8TtRZa+lOdsbTrST8vow8AIoyvyhY+F8gWXg 3xvehWnYJYX9bm4e1uyoGzL9nvGG7y7Qwoca1NxJgZLMzGy8lnTfSiSz1uLUhIJC42uO frsy/vzPxqAEkDkmm+mHkLTlQ8la5VhA9r0oAPL6o8HE9DsKD7hJoBvlFp59IIUnnejm 5gag== X-Gm-Message-State: AOJu0YzmOwKsWPlXN92HsyNWbZSBH9cJtqBkhxRDgSBSrkEdwdi7atFl I+04WJE/Rvn19ljIe+dqfNqXzwn2klw6exU/1PZrcTgcOZrbT3sf4STpR8O8ygnRx+nXyEdBYfX CXefjjlVaLX6bBWPS7WitcOkjgwP2mCqadxc= X-Gm-Gg: ASbGncvqDbRftFCTkJl+L3mnDUwpB5uZNbkMx6UujPzPxU6va5VmsucYTKpLDSNhcUJ DqEGlXtICkQhDrhGuxwlK0N9YwTjsf7MzNhkwWU1+EAZ2VS9Ehtt/9FHoJF+ViEMhT6/sSgq8U7 QZijyLWeFFEYWgKNz7D/j4x0+EJ0dK6H3xTvh13Y5H0z83Mw6Y9tDYQuGMJc4S2fr8B5rx5ZDYB nFGeA+vjil/bBrwOcBvl/RY1VUlC5mRwKnAIKV5p4YHVj2kYl3FWBcgP+EvmUnBpxzTZTcIxtye F/L2+AfOZx2VjWphr5YJZhhnTO787jucbgzThA== X-Google-Smtp-Source: AGHT+IH0qLqiJ2quScudGaRnk31R/ZT/BLKZW+HOtAn9ZFSyDFkyIz2iDv30Iqrl9WzuBvkJfI4DaTeKuKrG57Atapg= X-Received: by 2002:a05:6402:4444:b0:63e:b49:c9c3 with SMTP id 4fb4d7f45d1cf-63e0b49cd06mr5587789a12.31.1761143691451; Wed, 22 Oct 2025 07:34:51 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 From: Rick Macklem Date: Wed, 22 Oct 2025 07:34:39 -0700 X-Gm-Features: AS18NWBcx_E_UcUksGqTzzWAazhYaHEbbqw6eDuSGFl4a_lPP1q2n4guU8fzbkA Message-ID: Subject: RFC: How ZFS handles arc memory use To: FreeBSD CURRENT , Garrett Wollman , Peter Eriksson , Alexander Motin Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TAGGED_FROM(0.00)[]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52c:from] X-Rspamd-Queue-Id: 4csBX164vZz476j Hi, A couple of people have reported problems with NFS servers, where essentially all of the system's memory gets exhausted. They see the problem on 14.n FreeBSD servers (which use the newer ZFS code) but not on 13.n servers. I am trying to learn how ZFS handles arc memory use to try and figure out what can be done about this problem. I know nothing about ZFS internals or UMA(9) internals, so I could be way off, but here is what I think is happening. (Please correct me on this.) The L1ARC uses uma_zalloc_arg()/uma_zfree_arg() to allocate the arc memory. The zones are created using uma_zcreate(), so they are regular zones. This means the pages are coming from a slab in a keg, which are wired pages. The only time the size of the slab/keg will be reduced by ZFS is when it calls uma_zone_reclaim(.., UMA_RECLAIM_DRAIN), which is called by arc_reap_cb(), triggered by arc_reap_cb_check(). arc_reap_cb_check() uses arc_available_memory() and triggers arc_reap_cb() when arc_available_memory() returns a negative value. arc_available_memory() returns a negative value when zfs_arc_free_target (vfs.zfs.arc.free_target) is greater than freemem. (By default, zfs_arc_free_target is set to vm_cnt.v_free_taget.) Does all of the above sound about right? This leads me to... - zfs_arc_free_target (vfs.zfs.arc.free_target) needs to be larger or - Most of the wired pages in the slab are per-cpu, so the uma_zone_reclaim() needs to UMA_RECLAIM_DRAIN_CPU on some systems. (Not the small test systems I have, where I cannot reproduce the problem.) or - uma_zone_reclaim() needs to be called under other circumstances. or - ??? How can you tell if a keg/slab is per-cpu? (For my simple test system, I only see "UMA Slabs 0:" and "UMA Slabs 1:". It looks like UMA Slabs 0: is being used for ZFS arc allocation for this simple test system.) Hopefully folk who understand ZFS arc allocation or UMA can jump in and help out, rick