From nobody Tue Aug 03 13:59:34 2021 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8265012DB905; Tue, 3 Aug 2021 13:59:40 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward105p.mail.yandex.net (forward105p.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GfGhX157Kz4Xb5; Tue, 3 Aug 2021 13:59:40 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from iva1-6891ef6bb416.qloud-c.yandex.net (iva1-6891ef6bb416.qloud-c.yandex.net [IPv6:2a02:6b8:c0c:929a:0:640:6891:ef6b]) by forward105p.mail.yandex.net (Yandex) with ESMTP id A4A024D421D8; Tue, 3 Aug 2021 16:59:35 +0300 (MSK) Received: from iva5-057a0d1fbbd8.qloud-c.yandex.net (iva5-057a0d1fbbd8.qloud-c.yandex.net [2a02:6b8:c0c:7f1c:0:640:57a:d1f]) by iva1-6891ef6bb416.qloud-c.yandex.net (mxback/Yandex) with ESMTP id 0gunElhaBe-xZICEbxc; Tue, 03 Aug 2021 16:59:35 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1627999175; bh=QZnMxJlWPbGQzgWWmYzAleC3/Kbcq1Mtsn9vTDWdZOY=; h=In-Reply-To:Message-ID:Subject:From:Date:References:To:Cc; b=ed/YVQ/gm0TmCbLKR6l11fH0nj12xeR7l093ayNuIAtLIme2cgcn3x5Lf3mJ7o1mk 58uIgCNFF23qqi3Es+vdEN5qBCcI0hJBa3QTbjzo+RgzBtSx+V+DHfg24tFCxGmNsx KeFZCUyGZfyacqmTaj85TIJV1ZJN1ua9ddytikwE= Received: by iva5-057a0d1fbbd8.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id 44Vvc2SS4r-xY3Km1aH; Tue, 03 Aug 2021 16:59:35 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) To: Mark Johnston Cc: =?UTF-8?Q?=c3=96zkan_KIRIK?= , FreeBSD Net , freebsd-stable References: <5dc957ec-9483-0a80-b29e-be4b71c1b9d9@yandex.ru> From: "Andrey V. Elsukov" Subject: Re: Wired Memory Increasing about 500MBytes per day Message-ID: <35e9249e-37f7-3ff8-23c5-a22d8b909f09@yandex.ru> Date: Tue, 3 Aug 2021 16:59:34 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="AyljISCoKMiz299w3FED5YVd1ZGjCksFl" X-Rspamd-Queue-Id: 4GfGhX157Kz4Xb5 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_RCPT(0.00)[]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --AyljISCoKMiz299w3FED5YVd1ZGjCksFl Content-Type: multipart/mixed; boundary="ONABrLSmCq904Ths9Cl9WpnnbYBsOT961"; protected-headers="v1" From: "Andrey V. Elsukov" To: Mark Johnston Cc: =?UTF-8?Q?=c3=96zkan_KIRIK?= , FreeBSD Net , freebsd-stable Message-ID: <35e9249e-37f7-3ff8-23c5-a22d8b909f09@yandex.ru> Subject: Re: Wired Memory Increasing about 500MBytes per day References: <5dc957ec-9483-0a80-b29e-be4b71c1b9d9@yandex.ru> In-Reply-To: --ONABrLSmCq904Ths9Cl9WpnnbYBsOT961 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable 03.08.2021 16:47, Mark Johnston =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >> We noticed the same problem, I'm not sure the exact version, but you c= an >> check the output: >> # vmstat -z | egrep "ITEM|pgcache" >> >> The page cache grows until lowmem is not reached. Then it automaticall= y >> cleans and begins to grow again. >=20 > The pgcache zones simply provide a per-CPU cache and allocator for > physical page frames. The sizes of the caches are bounded. The number= s > of "used" items from the pgcache zones do not really tell you anything > since those pages may be allocated for any number of purposes, includin= g > for other UMA zones. For instance, if ZFS allocates a buffer page from= > its ABD UMA zone, and that zone's caches are empty, UMA may allocate a > new slab using uma_small_alloc() -> vm_page_alloc() -> pgcache zone. >=20 > So if there is some wired page leak, the pgcache zones are probably not= > directly responsible. We don't see any leaks, but our monitoring shows that "free" memory migrates to "wired" and only these zones are grow. So, we have on the graphs linear growing of wired memory over 7 days. When free memory reaches ~4% all returns to normal, and then again linear growing for 7 days. And pgcache zones reset their number of USED items to low value. This is on the server with 256G RAM. E.g. This is when 9% of free memory left: $ vmstat -z | egrep "ITEM|pgcache" ITEM SIZE LIMIT USED FREE REQ FAILSLEEP XDOMAIN vm pgcache: 4096, 0, 5225, 139, 412976, 0, 0, 0 vm pgcache: 4096, 0,28381269, 77,190108006, 24, 0, 0 vm pgcache: 4096, 0, 166358, 11523,1684567513,3054, 0, 0 vm pgcache: 4096, 0,29548679, 576,780034183,1730, 0, 0 $ bc >>> 5225+28381269+166358+29548679 58101531 >>> 58101531*4096/1024/1024/1024 221 >>> This is when lowmem triggered: % vmstat -z | egrep "ITEM|pgcache" ITEM SIZE LIMIT USED FREE REQ FAILSLEEP XDOMAIN vm pgcache: 4096, 0, 5336, 337, 410052, 0, 0, 0 vm pgcache: 4096, 0, 3126129, 117,56689945, 24, 0, 0 vm pgcache: 4096, 0, 49771, 3910,413657845,1828, 0, 0 vm pgcache: 4096, 0, 4249924, 706,224519238, 562, 0, 0 % bc >>> 5336+3126129+49771+4249924 7431160 >>> 7431160*4096/1024/1024/1024 28 >>> Look at the graph: https://imgur.com/yhqK1p8.png --=20 WBR, Andrey V. Elsukov --ONABrLSmCq904Ths9Cl9WpnnbYBsOT961-- --AyljISCoKMiz299w3FED5YVd1ZGjCksFl Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAmEJS8YFAwAAAAAACgkQAcXqBBDIoXrY Bgf+Ix+x4QzmAJ7KGDeB7icUWNfbXHlnklFTchXMaqAbr0QngrXm2BGhLx8bPX/r8WryvgJGv+xk gk2u6hHJdvjiYoF6pd3GOj2CxL/GM6qdweHI0G1iDz/2EbJnmEbH0gEdDNawqEXUx6+IzogzES+a wLwVweVYeQGMLBxg/5SjzCl/GlSCeaKwVZRnlybhhr2gMRQh6S40bNsSxi6sl1Bbgzhb/XF4eZUG EHlcbKT0/4+m3MW2QC697I7z9Kf96R7YG0K0zjJGkgrjds0fMQsvV1TKgwtAZ9B2dvXK9qyAfLBx RW8RimbrxoUpGq4pM0ceW33iKjroUlnr246zJCm6gQ== =2Ei0 -----END PGP SIGNATURE----- --AyljISCoKMiz299w3FED5YVd1ZGjCksFl--