From nobody Fri Apr 15 20:02:47 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D92972DD779 for ; Fri, 15 Apr 2022 20:03:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kg6h61Nrfz4nWC for ; Fri, 15 Apr 2022 20:03:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1650052974; bh=TYxIQiplUTuMvr6pTp3DRdz1ZkZmhKRHf5cgNgz+eX4=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=NQoaXpOOiqxTIBY72Eu7gBb05OyzwsC1Wb1XrXMo5mj7xD1WXzfn4t7VbhBx3qoJ0F8KdMABlJd1i5lq03jUf/7rUTjpMFuoQcpw5K7o9e43tBwUAMYGpgfW8y9LFWxPX9HRx2s9eLYc6JnlJ5+sKfTDP9ZK/10Jj6WNNAtin17ymqFjpB0aCB2Pte8QmRw74AOi09ff5eQV39SCu+pXQX/8IzSSfV8q4Tlas6uBR/mlaa2bljdYr/OEJ6X5D1UjGyqF0sLPyqdGCuBqJ4yZe2lvs5jC4uDOSDkmmbvP6N9RRKWCTULyz5IUsYqwuVOQbQdhj8+BwdBEvgdwNW1aJg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1650052974; bh=VtZ/jqJWAPLuRFTm8g7hSiZ8W/ISqVwKptRE5xJkAnn=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=E2rhwbzAF0W0BeaCMoqMjxJL+raohSpRwOjWHSalPON4h5E4C2IYKBGkcLWkzPucnAvwPptZddn1f89U6YJkIGzKVvQ+Vjve8RqU7xwee44aqPZs1ftCjCaNqiIgR2jCByngHmFL4SGaULz5zRVHVYcHBmFqReMRSoNBnwC+hJgCufmCE7GBhKY1J/k++bjg3qMA9rCuzs0phNyk25eI2YKcoWX+xcvxhlxYh6AHA9+1iZWF+XQzhoaYTcqvDkMQEJuEwMXeNyhBHDhbsjFPsU/PGDdOWrYTnoz2KUdZGLYhPAR7yXOxOi4Jbpz88wfCn1EEiUxcTaigh6WQ8PsKhQ== X-YMail-OSG: 1lZe2pAVM1mCmNLhzkdtA9h9mwv267P72yR3wriBlpp6vxYf6dQwNEEr7iAWIGw 4CHp2WAd5WoqE8hxLE1Id8Nesn3rvXAXR8FTSYnPc16ajMHBdKkolvPpF4cSTJa99SCCupvJKNnO mvxaydSCFKryZtVDlaTqnNAWxbnOvcdAJXFmTqWrqaIsKxpqr3xZy5lK5jAz_2luA_aZa6RRKq.O z_B6ervsJrZTtsamSqCj_9rUgavYxuiLWjw.qh8_k_4.G1HDnXrKh5_RvewIZkqTDHrrkKwGrqL2 rZq22JvYnNlRcAl5.sZeV899aeE2SPaaJnrz.fOrHoe2Jy7bT1uCW7MV7lOY03uXsaC3LjNmXBxB depXrv_HA64yWd5tXPTuai85WOsTDYUMBWAYxI9CovdigdspKdZMiWIG0NmowkzDi3jQO6MMr20Y w0iSj4sf2TGinRhtBRz3htToAVv8db08D1a10HOEmmmk2b2n5AyYVzf90DpTGfmvSi8ocNge2ihw qKLhwzhUeC1NUoy_pLOYYkpo9a8EdxwCcQWhB07pBtgLlnqVFtkKWH1wfFkTEsWwfgCKqqjThQB2 N61.uq95Z_I0Nt19fyxbgLivmOcdN8KIEOS_jp2PptwHgNVdMWPa0veGejOvM8EaP5tyH7m_4bVB z1fSFhSfVJb69FfyPwpxE1L0Dng3noG6d0q.SCeENhDmHJtuuMbGsemtHJTEBRWo94GugrFBlMQg B4Z6UvSnvVpjlre24Y24VarOVNts4j.SlQSL3ru0tRzlByRiCmi7gzeJve2mW82l0ZyW7j_QKJQT l4dMmk3EQIbrc3dDvGi3JhQQAU5an8VbiD78SmGsQ6fkGUFs3GFP4o3DHSu38qMnsteLEdPX6g2A 9W5gsxc_7oo4_IxlEQ5zl1_2LWvdIuIJVwn6c_cdcpOdMaPpEPkY5OPYjc0CKFqSTADpY8OdtQIy mAD62hclWr6Kh_ia4d5nU6LcNDgb4e6m47T0h5m9dPID1XgISGGeOrpvXRqJFfzWis_hdBXp1lCS PWmklHcd53p_HBqfOPGK4AVpkMwLc2vQkzpAWZaQCfWYFsNfx2GnD6GKClfpyRONKfWwO1_0WZME 4ECMIHc.Lmb8TTEOPHI1VyGNYrO3HmYWYc88xTvfGMiuv7lv.NPJoWkCrVWoLQ8A7Fvjz_BnpSiw UC2i8iSY1AlUpuNI6ika1ivjp7HrSpbuaD5E0z9ggbAece4u.fabkSp3NQSymR2VCH56tJhnFOhD 3XbsRxR6inLRHvJA6NEYlQw1kcB2ASr6SB5kIEvnrFmC_YfCgawfrLJElUZZkv20BBwWw6ek1CGZ PH.Jd0GSPB_SqYfwkEFUP9pIyomt_GgndmN7x4Yad52mXr1ukAd0BEST1ZnoOGOrklPaJNGWmCW5 z_u41iJc5BORpQkvMtfWAkxYuiSypzj94TW0tbIrSzl6V0yM9bYY11m7P6hqcFp.e9UgRX0BE5DV jwhHAnRvlozS6TKwVpghAvX.ZixTUpSoXVn7FRg2gT3foPiDNQG3WmyX.lmRGV9ETdfVYOaq03lq 4q6cOOuPKzQmiKonWKkq2NMN8.Ui3TLORJH17ZUtq.mg2XFzClREZibAsFtxR8Hjt9vJztFFfh60 NrCHYj.g7mYWKg7qblzOhfsYZkAgS4Uqxq9h_uzH_9Ba4B8xq4nCATJkat6N3RjysdDOaPiMidiq 6KU.sWBC6l0QNHuUmIejZpoyw9fE9Ut8EmnmNoOPTe_wdgM2n98Nlw_KRHwJfdBxCKAhnL_e.BNK OZ7di_CapEtN6pQlfZkQXbCqfzxaG418pv9nJiQ0_IAfEGKPSwDQl.ZbZJOJEGq53vIPdihHzka5 7anJOPrOEusVhWxlxzwwg.DzE2nnO69cCTuXa6AQgjNRhjM2SdyXLHaE42iDYaAREPV77CvN0hsS qKVHBsn3n.vv8Z7LNSh.CIe0stKkze.wQzS4aavKJS9GLIcznTJrgN_Eywh121uL4KPWNvxSrNIx rp_GqzbdDjEYEQHy4QgfB_8EUVzJJBYHqVWrpxTCjVM2zcDTOd26QAukGF6Rp5M8jzqWD0J6Z5.4 3yyyytyWnKILHBLRh1RmfXwmOv7slTuuWqG4aiRt2MviQzFAwqbq5JRQk1ZVhUUsRiC_3bk1NaRp .QoHDN5wiwY6pW_MSZOVCkAiLiTO2wNGOcLBdsx1xxKiIgCjczjab1x5n3bMsyN1phDvVvxA7X.m c1iFWg_c9F38AoJ1AVHE9KMR_i2Y25_Si84gcvub1TuAa2VYo1aGXvJScOcZE4rHe4OtbyddDvNJ eEmsicSY- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Fri, 15 Apr 2022 20:02:54 +0000 Received: by hermes--canary-production-ne1-c7c4f6977-rzpnl (VZM Hermes SMTP Server) with ESMTPA ID 6f334ef75a52fca1bb4d65b0dedb51a6; Fri, 15 Apr 2022 20:02:49 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Can not build kernel on 1GB VM Date: Fri, 15 Apr 2022 13:02:47 -0700 References: To: freebsd07@wayne47.com, FreeBSD Hackers In-Reply-To: Message-Id: <4A0FAE3A-BA58-450E-B8DF-F47F44B62A50@yahoo.com> X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Kg6h61Nrfz4nWC X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=NQoaXpOO; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.49 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.99)[-0.989]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.83:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Apr-15, at 11:40, Mark Millard wrote: > From: Michael Wayne =20 > Date: Fri, 15 Apr 2022 13:49:53 -0400 : >=20 >> I have a VM with 1GB RAM running FreeBSD 12.1-RELEASE-p3 >>=20 >> I'm trying to upgrade the machine to 12.3 and having swap failures. >>=20 >> This machine runs bird to advertise BGP, ssh and not much else so >> the small amount of RAM is (usually) fine. >>=20 >> For a long time, there was a 1 GB swap file which handled the >> occasional time when excess memory got used. >>=20 >> Machine needs a custom kernel for BGP, the conf file consists of: >> include GENERIC >> ident ROUTING >> options TCP_SIGNATURE >>=20 >>=20 >> Today, while building the 12.3 kernel with: >> cd /usr/src >> sudo make toolchain >> sudo make buildkernel KERNCONF=3DROUTING >> the machine ran out of swap. with a bunch of messages like: >> Apr 15 12:11:26 g1 kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 240593, size: 4096 >> Apr 15 12:11:35 g1 kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 236224, size: 16384 >> Apr 15 12:11:37 g1 kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 245, size: 12288 >> Apr 15 12:11:46 g1 kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 240593, size: 4096 >> Apr 15 12:11:55 g1 kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 236224, size: 16384 >> Apr 15 12:11:57 g1 kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 245, size: 12288 >>=20 >> Thinking it was a sawp space issue, I increased the swap to 4 GB and >> tried again with the same results. Boot gave the kern.maxswzone = message, >> I ignored it as I had planned to change as soon as I completed the = build. >>=20 >> So I pulled up top in a console window and watched swap during the >> build. About 400 MB of RAM was free and about 3 MB of swap was >> used when the machine started linking the kernel: >> ctfmerge -L VERSION -g -o kernel.full ... >> While this command was running, I saw swap usage go to ~5MB (so >> just over 1%), then started seeing processes being killed due to >> out of swap space. >=20 > The "out of swap space" message is usually a misnomer I should have been explicit that the misnomer messages are when it is part of a OOM kill notification message. There is a separate message about "out of swap space" that is just a notification of that status. This message is not a misnomer and need not imply that I OOM kill will or has happened. > and has > been replaced in main [so: 14], stable/13 , and releng/13.1 : >=20 > case VM_OOM_MEM: > reason =3D "failed to reclaim memory"; > break; > case VM_OOM_MEM_PF: > reason =3D "a thread waited too long to = allocate a page"; > break; >=20 > (There is one more case that still has the misnomer but > case VM_OOM_SWAPZ seems unlikely to actually happen.) >=20 > Given that you are getting the swap_pager: indefinite wait buffer > notices I can not tell which of the two above is happening. >=20 >> So, how to proceed? >=20 > My /boot/loader/conf has the likes of: >=20 > # Delay when persistent low free RAM leads to > # Out Of Memory killing of processes: > vm.pageout_oom_seq=3D120 > # > # For plunty of swap/paging space (will not > # run out), avoid pageout delays leading to > # Out Of Memory killing of processes: > vm.pfault_oom_attempts=3D-1 > # > # For possibly insufficient swap/paging space > # (might run out), increase the pageout delay > # that leads to Out Of Memory killing of > # processes (showing defaults at the time): > #vm.pfault_oom_attempts=3D 3 > #vm.pfault_oom_wait=3D 10 > # (The multiplication is the total but there > # are other potential tradoffs in the factors > # multiplied, even for nearly the same total.) >=20 > The vm.pageout_oom_seq=3D120 delays VM_OOM_MEM. > The vm.pfault_oom_attempts=3D-1 avoids VM_OOM_MEM_PF. >=20 > Note: vm.pfault_oom_attempts=3D-1 can lead to deadlock > if you actually run out of swap as I understand. >=20 > You could try setting both vm.pfault_oom_attempts and > vm.pfault_oom_wait but I've no specific suggested > values for your context. >=20 >=20 > Note: I do not recommend having so much swap that > you get the the kern.maxswzone message. I do not > recommend adjusting kern.maxswzone as it competes > with other kernel resources --unless you understand > the tradeoffs in fair detail. (I do not understand > them in much detail.) >=20 FYI: "swap_pager: indefinite wait buffer" is for a swap read taking over 20 seconds (at least in main [so: 14]): /* * Wait for the pages we want to complete. VPO_SWAPINPROG is = always * cleared on completion. If an I/O error occurs, SWAPBLK_NONE * is set in the metadata for each page in the request. */ VM_OBJECT_WLOCK(object); /* This could be implemented more efficiently with aflags */ while ((ma[0]->oflags & VPO_SWAPINPROG) !=3D 0) { ma[0]->oflags |=3D VPO_SWAPSLEEP; VM_CNT_INC(v_intrans); if (VM_OBJECT_SLEEP(object, &object->handle, PSWP, "swread", hz * 20)) { printf( "swap_pager: indefinite wait buffer: bufobj: %p, blkno: %jd, size: = %ld\n", bp->b_bufobj, (intmax_t)bp->b_blkno, = bp->b_bcount); } } VM_OBJECT_WUNLOCK(object); Also, for reference: # sysctl -d vm.pageout_oom_seq vm.pfault_oom_attempts vm.pfault_oom_wait vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM vm.pfault_oom_attempts: Number of page allocation attempts in page fault = handler before it triggers OOM handling vm.pfault_oom_wait: Number of seconds to wait for free pages before = retrying the page fault handler The default for vm.pageout_oom_seq was 12 last I checked. =3D=3D=3D Mark Millard marklmi at yahoo.com