From nobody Sat Jan 15 20:27:47 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A50AE195409B for ; Sat, 15 Jan 2022 20:28:00 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-25.consmr.mail.gq1.yahoo.com (sonic304-25.consmr.mail.gq1.yahoo.com [98.137.68.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4JbqVS2Vsdz3Gl0 for ; Sat, 15 Jan 2022 20:28:00 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1642278473; bh=umaCkl+5ZFZ7k+ZFxQULnwV0LT5cso5hqhMBLPRAikc=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Gr8rfOakzwA+f9bIZSeVkXiav3NRxSVLw6hlz68iOvQJRn5HZ07jIGFOkrWxMTb9ZKZS9KepfYDIiRdZ3o8crxpvjUor9QgobM1RCq6OlPLVMH73e8SGG7f6NN2u+bKd6m3vgutxonutVMHptnkG8bdtuStJpsC69coUL4FucYY4B8Kcvb2ru6mZtKDa2TCzW3gKoqEL68ForNlI2d3MFdtaM/YQn8r/GBnkwZgGMLvcBuCgTdJWESj1/BaVyUEY70XnqQYtB8Y5yCWzdLeITvFo8YJbQF8PHNR+Agvt9u31drFPxMWbUH9wBYmy3v4GgQr07ske/s4qCQZRXvbxrA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1642278473; bh=2+0DF/3UhWH/+CRWCrIioCivRP29udTcIp6lPta6stF=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=UiQDPGvdndc4yNHwth+XjgOyJZA3Sdn8JlgQntoNMzpck2J/XWgocuTC+buDoh4CHtvl+g3SgO/GVbaNRz/6wn5hJJqYTLS3omNnBFYGs/jQ/A/Qa142+1yHR7kitth+kfezl986TqjRmRhjUd1u3e5hXPqszl+dsl4ffZ60iefjIftEcSFnjBKd1E/IxZffL8Ub0i5JYl/yoDK2Pq6fhY4E2KIhOGcVq1IoGetKLz2JGir6H1Hk+W6q3HsdzaCXnYLlvcrrU+ScoeVYJJ7YHdYNO65Iljfg0+XkoAmKnv3gVhlVshR61E7WDiYeqo+e5ubCY0xMthySpHWWy2njXw== X-YMail-OSG: uhCKjcUVM1mcjLrz2rBSyBNXqNZObehRcxHDKqujNveG0ZtlHxjEufvEXdMpMLi QxSOYGxeSCKy5nU9cQNkJ267Uwl8SUnHMgiH1eLpP2ssGDIfFMxOa2jG32okxeROUpzPj1nlsgix vKMGDzOOIDOzQqhdj61ZIdcatJo_G75Y8cPGJnmwwrDTu1aTiA1amOBN3RWJWthcIRL7vGy_rwQi KSZVsvMEm.UldWMGSeKdZf2hxmJJYBTJJwZX7DM.qF95J7d3NNJFaS5p.zPPqci6wmkZ9r58EJIo MIqS_dLvLeYrtREDgnN_NrazAXc18yqrguenhSpPYDVwKI23PqaRrmWO6DrcLYYienVxWgjBpr.i lIbsdvt.1ORirtNcAEiuNzfj.fHxDumiOEomIqKBSB3vTXfSfG5Bk5ALP.SsLX12CKTG0EU0hbgu nDGJRYbrzQG7SOdNkR5IXlDsO45wqXqGW8WfWzBIe7UBT9D_0ZXk4dBUBNaVfRNy.sdMn_ezd_RU GbZUtn2s2i07Uv1BGbgKFSbPScyQb0QZ0HQV2AknGcoM3ZLEB69VYoAsWkTBPqdHLSjRWi1h3hbQ K50CZ182e39jBqvuCNfQG8Pl795ueGPyAET4up_TW71Zi7nUlCtB9eSM6UVWgjsYJP7tp8wmLxWY BahwmuwYsJVhqbaEbAe3G467BfTl1RsCFqiqYjdmVsjXMfOgqq0S8DAGNxTT6MCjuDzdTPxXh8NQ 96toL6_BLBIK2gkoqmz1apDlVdxgbGZB89BkUI5wjFvSVLMT2sv8tHG1.h0dZhGpvrfHT9Rbvs4L 3fhtEOSyDFjguZlymPhQRVyq.FWE2SE3KbU.lkoXKMszDvILWErkhDWHlZjc4WwBM7GbaKXAUVQB Fw2uUfvsw6x1K.SgoAhLXJ4HisGcQ0JdcRDpf02DMxoP7hyBwXs_pH4HhiLWFpqSkXYH03gKUkzk sYRVRnTTWGMg4a0fw4aTBnG9.KfcyH57PRM18lzGIYwsTHTqF8zh7zFRvXpogBpDrmGJDXvBAbF0 0Nx.QnhH70iKeYhFcAbX6g9CtS4SSHoFaZ7SLfs71.PjWEQ0bfLFm9Q9DwYuD73i2bMnXc9Ky95E iFcnHuVwCG7p9DmXDwzLnBW9EzEqDTbX0iqxE0I5pUf6hUfBpfmEEynrAp59xylZffW_QKR8hdGD BYwyhWGPKSmLr5_seP5gRFz.BJNuMiXcyRuaJMXiPoNUpgE7ApsROZzXCEW3SU14yA1ryTsSWIr8 r_A3ypuQdWtf4TZ5frRbXMwo.yGWeXN6g.aVrVzayz1I4tJCCfQuwM0ecNR1r4tRMBoNn2cw5OWA NhCBag6MFM7nxE7eCTrQfxUlIGHlX4ausLQjUoJtZW1tM18whloFupPOF5YZfbswEz2COSGlH3v5 W_MQRgLJm_NEe32sMGJKoCuEhKhR64lzUEfQoLM23B5tykaMR16JzfuSfUMJFpBzdhg5VGaj5wKp xNDFbr_rvxyu6bXu1yN1XTWThj1rfPdOcHlKz5zos67y0cRZwcVn2AnKAIXCU3m3niMqhAvgswZG vg4dSlO6vT56bU7EIMvBfmgt6w68KFGoSbC8iOv47azPIPLe33aklw_yhSTyvhix8WsrzMaFzNhV RcNiw9q225ve1DMvo3J9h28XrRvo3due05zHskQHIth1fWbiUQ5drFtLQwohfoGzmpwi56H5YYqW 9uHopFznz_Vb2zSJY5iKSqhGoR2.F2N05DxYPoQnL.tcwkHwV4PIaOJehe6oo3yU0vpH.hk3_BRY sFnPYspSlkxst_a34qAzlA45J0y95PXhLSnZu8r7ENWjMCabDnTUfcy573f2vysOPo0f1FIaoJxt TuxwVMS1cD3cydSMaWWpQOQ9.xj0THKTh9iVyEueQppnqs4PdsPvEtzYTuZXEji3JS8mZA2U.Exu 1DYaHVzPiQoN9_CK5uzjchDxwBU2rwjoMJsViE.6hxkBaqt07hF_YgJ63nqgkOYkf1IWWXSm1918 98iDaeHHzYcSeQcmCqHRAex2P_Htp8y1p.gRwkza9Ru.M3Ox50DMQmCKhj9VL.Xdl3hHij4ySUBJ GDYuQLJoRl8WwIh2SyaoDsm5fdaSqTmfbMtihBTXkAPJsB3MgfqP5y3yg_myLJp4_07CJLfE.PXo JzRx38yQvTk9BpBqVa0f8ypOCL1GYPMXP9LsOjzFFqh8QI4nfrCKfzP4Sx_38G.jIYWiyvV6mewi NxNKVAEQAxULhUyqccCpFCk2b12W7_C2I077VbA-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Sat, 15 Jan 2022 20:27:53 +0000 Received: by kubenode536.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID cdb6161b041431185fd21d6d405d5f92; Sat, 15 Jan 2022 20:27:47 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: git: 4a864f624a70 - main - vm_pageout: Print a more accurate message to the console before an OOM kill From: Mark Millard In-Reply-To: Date: Sat, 15 Jan 2022 12:27:47 -0800 Cc: freebsd-current , dev-commits-src-main@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <5171EBFB-EB6B-49F9-AC7B-D3867D40CFA6@yahoo.com> References: <1EF55D96-F7E3-4AA6-A331-782362A70878.ref@yahoo.com> <1EF55D96-F7E3-4AA6-A331-782362A70878@yahoo.com> To: Mark Johnston X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4JbqVS2Vsdz3Gl0 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On 2022-Jan-15, at 07:55, Mark Johnston wrote: > On Fri, Jan 14, 2022 at 09:38:56PM -0800, Mark Millard wrote: >> Thanks. This will allow me to remove part of my personal additions >> in this area --and my having to explain the misnomer when trying >> to help someone analyze why they end up with OOM activity so they >> can figure out what to do about it. >>=20 >> There seem to be two separate sources of VM_OOM_SWAPZ. Showing >> my personal additions for them (just making them explicit in the >> sequence of messages generated): >>=20 >> diff --git a/sys/vm/swap_pager.c b/sys/vm/swap_pager.c >> index 01cf9233329f..280621ca51be 100644 >> --- a/sys/vm/swap_pager.c >> +++ b/sys/vm/swap_pager.c >> @@ -2091,6 +2091,7 @@ swp_pager_meta_build(vm_object_t object, = vm_pindex_t pindex, daddr_t swapblk) >> 0, 1)) >> printf("swap blk zone = exhausted, " >> "increase = kern.maxswzone\n"); >> + printf("swp_pager_meta_build: swap = blk uma zone exhausted\n"); >> vm_pageout_oom(VM_OOM_SWAPZ); >> pause("swzonxb", 10); >> } else >> @@ -2121,6 +2122,7 @@ swp_pager_meta_build(vm_object_t object, = vm_pindex_t pindex, daddr_t swapblk) >> 0, 1)) >> printf("swap pctrie zone = exhausted, " >> "increase = kern.maxswzone\n"); >> + printf("swp_pager_meta_build: swap = pctrie uma zone exhausted\n"); >> vm_pageout_oom(VM_OOM_SWAPZ); >> pause("swzonxp", 10); >> } else >>=20 >> Care to comment on the distinctions and why there are two >> contexts classified as "out of swap space"? Would either >> one show the swap space as (nearly?) all used in, say, top? >> Or might one of them still end up looking like a misnomer >> from just a top (or whatever) display? >=20 > Hmm, those cases should likely be changed from "out of swap space" to > "failed to allocate swap metadata" or something like that. Based on your description (later below), I agree. > Running out > of swap space is not itself a reason to trigger an OOM kill; if the = page > daemon can continue to reclaim clean pages while swap is full, then > it'll do so without killing anything. If the swap devices are full = and > the only way to reclaim memory is by laundering dirty pages, then > "failed to reclaim memory" is the message you'd likely see after this > commit. >=20 > The two cases which call vm_pageout_oom(VM_OOM_SWAPSZ) arise when the > swap pager fails to allocate structures used to map physical pages to > their location on a swap device. swap_pager_swap_init() pre-allocates > these structures during boot, and the size of the reserves is based on > the amount of physical memory. In particular, each VM object = maintains > a trie of "swap blocks", each of which maps a run of SWAP_META_PAGES > pages contiguous within an object to individual blocks on a swap = device. > One zone provides internal nodes for the trie, while the other = provides > these swap blocks. Assuming perfect efficiency, the reserves provide > enough memory to allow all of physical memory to be swapped out, I > believe. The swap space can be bigger than the RAM space and something approaching more like RAM+SWAP "memory space" can be in use overall. The system complains of mistuning when (approximately) the ratio of SWAPSPACE to RAMSIZE gets too large. Relative to the mistuning notices, as I remember, armv7 (so: 32-bit), for example, has a noticably smaller multiplier of the RAM size for how big a SWAP can be compared to, say, arm64/aarch64 (so: 64-bit). But, using aarch64 as an example, the complaints start at somewhat under 4*RAMSIZE, where the as-if-no-page-index-space-fragmentation figure would be somewhat under 8*RAMSIZE. (8*RAMSIZE matches some documentation but that documentation is appearently incorrect for the likes of armv7.) In other words, I think the wording above is misstated in its detail but fairly accurate wording is probably a much more complicated thing to provide and read. > In practice there can be external fragmentation of the page > index space which leads to less than perfect utilization of these > metadata structures, in which case it's possible to exhaust the > reserves. This seems to be a fairly rare scenario though. Side note on an example context that is possibly related to the "fairly rare scenario": At least one person has had the system consistently hang up for a build activity when avoiding the configuration generating the swap mistuning messages that I reference above. This was when a monitoring top did not show swap as nearly fully used --but definitely in significant use. But all they had to do to avoid the hangup was have an even larger swap space (so the mistuning message ended up being generated). As I remember, the used swap space did not get to the original swap space size in the monitoring via top. May be the fragmentation of the swap space itself contributes so the bigger swap space was easier to handle? (I've no actual clue.) Anyway, it was not a result I had expected. I still avoid causing the swap space mistuning notice by keeping my swap spaces somewhat under the sizes where the messages are generated. (While fairly rare, I sometimes do experiments where such a RAM+SWAP total size is important.) =3D=3D=3D Mark Millard marklmi at yahoo.com