From nobody Mon May 16 19:07:18 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3692F1AEECF3 for ; Mon, 16 May 2022 19:07:32 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4L27zl0rckz4pwm for ; Mon, 16 May 2022 19:07:31 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1652728043; bh=oJ3e0q80t23pIqoMh+cjeu3M8iRCF+Lua27pt+e5w78=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=mTOItqBTjfwCbG4IQefZhlnt5kMMZ2GWYiYaJO/u3vQWPTEx88p9PMUXa2O/5/nuzaUxbXIDadJ4qEA2KOESAshpLL3hg/i8xrVukXS0xH9DEHxHOVjMM8x/yFY+4KrcrjsCrTm0uK+wbmKtqAEL9EirnO/xIqFka2rkSezvpwFypkBpGpZopEJPRf7bCahkHXhNgEbiXlnTSMR3jvlLwsCY/xxuSuA8/IY908B8p3rWEBin0mThTx2OHA6Aj3WbNY4nrF5IH5B8u7o5Xxkfgj/TtJmlr8UycmIeT5cl/RVX0YsxwQk1T55fBCc5cTPS6RIyN6bRxUQzVIGdeHKzEA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1652728043; bh=RLAt3Us1OLbA6UqnWX3XxiqZbNs252Wo0Nba6C47Xro=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=OlAsblIeix2P/2WHQWuuaUkUaQnuIfkgGu6LSmB2hMdMbZmKnyrPVyVGxQ6SRA/ffYkpYZlY9xO2pkTdoL/jTTBEA6SQ8PkVSqe42Q5AxE2X+vNkJ3F/oxClTbTC2AatxrIE8E6V2PwiLc7iONxSO7xpFZfnAUQn9zjzlSerGWYRYo9s6fU84fRQmJmUvZAuZTVxukmPDeqSJvB6yjLJ8Q5GpxivSGOzeMWFtgt1dZP2lEuhq/iKSUdeVXk+LWLUUzIBqmYXvmxIqry8W+/HlZ/Lost5m2nQDO5qlmzLiQHZ9c6x5Q4DqPOgiPrPPZ5Pp81e9SQbvoEwEFI6bBTaNA== X-YMail-OSG: RbKv4cwVM1l4FIqW1Pc_syMt_2sCa_ZFCgDl1nPclp6kcb4xoelxNx3QxFLnlcU w6CPtLGOG24zVMgYpTZXDIpXQ2_IXYwrPKZ7HoD8LBZdBUScq4OOJF7kyq9R3Mr7dWD4dpCfDXUV 0DzUdTI2jqKek.mYNNur262WhJUc6ePKM6N5f0TrAaEkY6Mvt.rSKcEt9Lv3QPTETP.inYTRBq0e fxvEdw7vaVZOJHnBZWVfFqAkOZQVnRTuMqhpQ.IalfBa0kaiqbUwnOSqQYU1oB3iyQXZjAErvka2 Azu675ObCoOwWsePCP63Ttd8U0NmA3t.uyUeYrYxDFSXoGI7eKdwS_jOx1MBzkAFQzcLyA9NHqxg _M1.bMDaRF_uOREFFi9Aoh4FNYTeDCpswaMQlFzTJGUY2nI8nypJcuEsN7SRQArfmMHOgNyiW6yL si3iiTWXo4GY1iVEbOuBEfFwjtLdppBo1Szlrc3xNRodFof2zU3McaqvlI0D2lEuUwHeHpLsSDf3 F_067L8wpIDn0WmvITrX_5Cbxe8utGdvYnqznGTBGulFB.PsF_pM.TV4.kem7EOaX_ARbmUDkmaw Z1injhz5rrQS9o.f0nkjVrTCpgBIeb1JEejYYC7WSt20lMBASGziOEw3QgTCEAYoJI2KfQaaWmc4 xoLGtvCP_aCJQyEUDowzn.ph_3w0fI.H74G55sxqxWxUBWNJYT9W8dyGPjjaI1E3xevKQS.jDnjc 3RvV5p2LDksvyvQDr8p0NoRgepHWAYtQn3rdwRVbW3duO2y9Rjn6z84gwxON4QmlYQA391lNYN6z 2hotkj.2YtBxIasG5r9Jhb_k4JX_z77.Pp7sUWyH_rh9mDw7fyWnUwGAbExGTR9Yna1YtYWz3mad FO5yEfv1zrY1zkAMpxN_ixf4X7rPwfdxjg3RDT2GR8KUewJg5G380vhulGcz5G484LAgwh4F7A27 sxwrWfRlAwSqxAuhzZ_yWGgTuJWJoiLT_lWiYnU0AOMkF9zOgrnp5awmNnBGvBymM9Pdea0K5318 .s9lHVi57nh.SDXssr5tpEswXgLTSqMXfRHl7Nx33a1N5j6_jxlDrk8KhdJRTzKNnUKmo6SDSkLI yqzP.eU8uzQvtXyZCfXYwhFPs_iNQEleO_pCQYgRu3XHbFmD4Ncs18xaE2CVq41e4TKEeOD2rJ4g N5c.KifXVRihCmHGJv0sDXfKAzEUCMugEusiQppN3.yKEp3mEX0hfIMrTZuKiOMKYyb4CSJ1_cD0 jqkAvtBxd7YNacGbUyd9bBf5tS3Oq_8xUrXTfyWaG3pFqaQIXaaS66KVyiiMQ3AqgLS7fFngci0w H.PXpcuaDbXdKluX2aIMhhmyEO4cAY8j89xKdQfUlvEHRNMI90hgNTx2yDavOOX4VUK72zrtdXi0 Et8e5cTbc9ca3DVB4PWaf2Qfk5511O3o4L3eSdIDZX5rRTFnJxJuAJVVLUbcyMZ5fhn.Uzn.OK2F uJQ4VkO9K5OlNDwevbppucSG3orOZhvJiZTamQCmL9AvmUQB0KWyS0kHZng4XdEgwe4tFgQXvXgo .5BiOkMT7evohG9ZDEk5ard20lD.CDXrzoyd7XpTM3cXkj_wg2KSqHYAp7f_7wA7kdgG1G5NP24n jBvvzcpDpoG2skbvf32BqLQbHqy7tOJBbcRSyAd08mOYJtYSW3iuJjhi3ZqR.wn19TZZBs_C_tn. btSwbj3YIX9Qk0AFpmrMIg8tq2m7oaKzOP4kerxFAzZ0yAojEktfgYz1zaJoTOdlCrb0WAFoYBwl aln796L.x0lTwlui_BYUncNlZuWDr.C1lhVedjVKyCeLMXFSA0ur_ogcfPO0nxTt8yQxRiKMhdbT sbeWPvE5SOPMPOlwTzsr7eNVZc_g1ReebdFN9r_0iM4lZS6Ov.O2PpLrbm5ZNt1sHRROwHkKbuD_ X6vCayJyFX3CUTOccEUSSjBJwm1slAWRiAEjFcl0OGqfO74X70c7fCggneLOb5TSTsnZ9W1gbVkK 6DPok36Qv5DnCrpdPwC094N11N6g7CR.rKYkUxWg3jB8tS2vg2y0WSl9JiC.O05p0Yf2jX0wf0BF Leu5JBaoNFJ1Jtbvhi69yFpUSdWvQYlvqw_EVjj69fjjE90kA5iDaWLmcHrtoMYyxrkMdW.uQviA eNDe07lJ103dPk2X1rpTIjMlyNm3WwRanocm.0wEnSGiAicjGKfPBxIL8vlOu.2o8rM416_mTQJd Y5TEQ1fMy83o1I9LmxWLozv5RwKMIYWdKxYLmZUW5rr9kVMrkical9B_YHMjdVKInnWJtPWXgOHO H3Tg_gfGpM9Si X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Mon, 16 May 2022 19:07:23 +0000 Received: by hermes--canary-production-ne1-8676f67b88-z8zfq (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 1dbc3656ab8f8fa531c43a2a50d4a9ec; Mon, 16 May 2022 19:07:22 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Can not build kernel on 1GB VM From: Mark Millard In-Reply-To: <20220516143753.GY72471@post.wayne47.com> Date: Mon, 16 May 2022 12:07:18 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <934C3159-4B1A-4A2F-9C21-93DC7CC90A72@yahoo.com> References: <27171A11-13B1-48A8-AF46-605091E1093F.ref@yahoo.com> <27171A11-13B1-48A8-AF46-605091E1093F@yahoo.com> <20220516143753.GY72471@post.wayne47.com> To: Michael Wayne X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4L27zl0rckz4pwm X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=mTOItqBT; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.44 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-0.94)[-0.944]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.205:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MLMMJ_DEST(0.00)[freebsd-hackers]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-ThisMailContainsUnwantedMimeParts: N On 2022-May-16, at 07:37, Michael Wayne wrote: > More info. I am running UFS so the ZFS should not be an issue >=20 > % pstat -s > Device 1K-blocks Used Avail Capacity > /dev/md99 1048576 0 1048576 0% That may well explain some (or all?) of what is going on: file-system/vnode backed swap spaces are subject to deadlocks. On 2017-Feb-13, at 7:20 PM, Konstantin Belousov = wrote on the freebsd-arm list: QUOTE . . . swapfile write requires the write request to come through the filesystem write path, which might require the filesystem to allocate more memory and read some data. E.g. it is known that any ZFS write request allocates memory, and that write request on large UFS file might require allocating and reading an indirect block buffer to find the block number of the written block, if the indirect block was not yet read. As result, swapfile swapping is more prone to the trivial and = unavoidable deadlocks where the pagedaemon thread, which produces free memory, needs more free memory to make a progress. Swap write on the raw partition = over simple partitioning scheme directly over HBA are usually safe, while = e.g. zfs over geli over umass is the worst construction. END QUOTE I only use swap partitions for swap space --in order to avoid the deadlocks in parts of the kernel operation that I ran into otherwise. > % df > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/ufs/rootfs 25388500 17630840 5726580 75% / > devfs 1 1 0 100% /dev >=20 > On Sat, May 14, 2022 at 08:02:29AM -0700, Mark Millard wrote: >> The way to know if out of swap might actually be involved >> in the context are some other messages that do not of >> themselves announce kills: >>=20 >> swap_pager: out of swap space >> swp_pager_getswapspace(. . .): failed >>=20 >> If you are getting either of those 2, then you are actually >> running out of swap space. Otherwise you are not. >=20 > I am not getting either of those. Watching swap as ctfmerge runs up=20 > confirms that I am likely not running out of swap. >=20 > Note also that I continue to run a kernel built with the suggested = patch to=20 > /sys/vm/vm_pageout.c (The oldest code I cover for extra messaging relative to OOM kills and and the like is releng/13.0 code these days. As I remember, the code has been moved around between files some since 12.x . So the below may read odd for your context.) Which suggested patch(s)? Any patches for . . . sys/vm/swap_pager.c ? swp_pager_meta_build: swap blk uma zone exhausted swp_pager_meta_build: swap pctrie uma zone exhausted sys/vm/vm_fault.c ? vm_fault_allocate_oom: proc %d (%s) failed to alloc page on fault, = starting OOM sys/vm/vm_page.c ? thread %d waiting for memory For sys/vm/vm_pageout.c I have messages: vm_pageout_mightbe_oom: kill context: v_free_count: %u, v_inactive_count: %u, v_laundry_count: %u, v_active_count: %u (The 2 forming one line of output.) failed to reclaim memory a thread waited too long to allocate a page (The 2 above are now the standard messages in releng/13.1 , stable/13 , and main .) swblk or swpctrie zone exhausted (In releng/13.1 , stable/13 , and main this is still shown as the mismoner "out of swap space".) >> If you are not the real reason is one of 4: >=20 > Likely the swap device isn't responding within a "reasonable" time. = (from another mail in this chain) >=20 >> FYI: >>=20 >> kernel: swap_pager: indefinite wait buffer: bufobj: . . ., blkno: . . = ., size: . . . >>=20 >> is for a swap read taking over 20 seconds (including >> time when queued but waiting in the queue to start >> the transfer). >=20 > I got this message on 12.1. I am not getting it on 12.3 >=20 >=20 >> /boot/loader.conf can use the likes of: >>=20 >> vm.pageout_oom_seq=3D120 >> vm.pfault_oom_attempts=3D-1 >> #vm.pfault_oom_attempts=3D 3 >> #vm.pfault_oom_wait=3D 10 >>=20 >> I do not know if you have tried any of these. >=20 > Quoting one of my old emails: > On 12.1 I used sysctl to set >> vm.pageout_oom_seq=3D120 >> vm.pfault_oom_attempts=3D-1 >>=20 >> There was no improvement. I still see processes getting killed due >> to no swap space despite only 7-8 MB being reported used. It sorta >> feels like it's not really able to use swap at all. >>=20 >> Note that everything worked fine on 11.x, this is a new issue on 12. >=20 > I have not tried this on 12.3, nor have I tried the other two. Will = do. >=20 Can you switch to using a swap partition instead of file-system/vnode backed swap space? =3D=3D=3D Mark Millard marklmi at yahoo.com