From nobody Sat Jun 11 04:27:06 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6234E840C5B for ; Sat, 11 Jun 2022 04:27:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-20.consmr.mail.gq1.yahoo.com (sonic302-20.consmr.mail.gq1.yahoo.com [98.137.68.146]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LKlD62ZrBz4tGc for ; Sat, 11 Jun 2022 04:27:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654921630; bh=mfCZKSWNc+bjNEvnUJSdVyeshl1538IXlEgA4/voheQ=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=PgH6hZlJGb/VxIhdgMIah/p8PZWfYOlE/wd2ONtpqK0cSuWQhtkyBQu+vEf3YEJ6L4Owl/fH4H6cUfTfQLwf+SVEm3+04Eqd2G7FY1XdftSJV5iw7RQrvGWk/dzVY06Xmc0ups1bEWwXDLzF6AnYve5mL0CeF0Imo5Ol3DVGiU5dC1b2e1Hx4l+QJyv9XI/aXVM+sRzfrxc5Erk+vRR7c22vYdVHdqyAdkSDGWY6LoGZT9iYD1aFIQaKEbH41RrlFkqgcglt7dO07xFzvdC8rs5ZytUgTxah9MoF+YDDErq2jf4ce5AEphDpI4W882ym2BKcJ66RbrFfp6qRD5z4OQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654921630; bh=5bPLXkNCQXSokxRF13Wf6Zl9K8lYxkQZ0yVzPcUoJV+=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=tjSDC0VXw/9LIlAu5aDL7Fd+bIlwoh8DjzqnV480vY+/bvhYcSSVZ2shg1a55JRMG+C1wHsAtEbne3uNGyHALtPABRspoz/zCFtTRJ5/99G9OK/FhFrlEXcBj1QOLVOgEgIuhcJmrhRPEgh7efe4/gBJpM/Zv9djaJktrn6ndPiQIEkV+g7ajSiM6VSXekNEZla0lj97cOmiyw4SH3EO+ApTXa8tQ1SnZ4Bo1Hidw9Gi13+Do62Ox2JuD1UPU/8reaqxUTgQjfNoG535YF5XSxWJj+CCiYCcYlEWRLuKpGHM0wqy49YcjTHkBlhcFBEz8ZMfvuyHzw3JpWNwWrnJSg== X-YMail-OSG: 657RueoVM1msEygxQudmUNGSoKgYW.9cic43ffcg8g2sMDBNZFLjnkDPNbN..vi qD6oNKBr0B6lZblNNrUQxeqL_cCTTm_R5xOsCBwib07qNBK63sfn3pgrW5T01e.lsszfQb6H2Evp _OrfPsofy3aSxXsOoyKfCgbCJ1oP0KO0GvBDVRsJFcRhbPSYWor4tFzA.WHYema3FiKMz13DE7Ft GALC6laFfcbrLF6Zw4aG.luOtH6sZ4No_EoyOQgIJIL9h.p64KvTPZmEEU24EQgYWRValE0L5F8p KCgENQMVuH7hgyY_duk2OpmYgHpZsQm4Dx0M53YYz0B4esgvJCyKY6cHl6LARPI2W4GAbPGv5MGT IMWpCL4GJcpYtHWqPcXVD95LDpi4az1B6UHC_p6inkQ5KHMf8MQHY9qXjkrxW8Ji7xk20byfhFLO jHzA9YVSlB0uSRgsQ9Ha3QfON5vl8UYtqp9cdm7v5.fsIhEHbChnfB9TpTW.F09e1z2najahGGdo xwyw3nWIfYYf.JhErchDXIKavfYIU5FneiJK.CoTdvwj_e0Wfu7zAoofOz4O5GKS7klKpudq.uAp BgTOKBVxFub_QBwe0WM43Z4080z9.EjUuCuF2Ij7diGoEqJt7r4gHx4GQm3saCUO2h6jUe.e8PT2 VlfwQaNCU0PrbtYmHz2xESypZKjVBN5.O5MiXnDwz90T5MRncXT31orp2jr4NsNkqbo3Vh7DcXFq vxDiyb6Ao0GWBPYu_cEUzONmH1N1coA2.y.CbVIzYq7oeZsCVTgDhkm2sp3Tq0bnXdi4Z5yKh2wx fzcckJxrpjPN3FA0tETa.D6yKobV0i4uDVCpTjGvfmAJ6jpubDuRhlIwzNlDJ3R9vXrXP25ha5YV ragdhdYvnE1iB7z0gIsuxsKfH9WZXqyGG3tfcfPDfKyvcpK5KOWhqw0Z.crMXEetVvO4nmG5Tl6Z z9pYV.0fYWG7CelC9wMvJz4yCmDBwPuflWi6OlpFhzT_1_UIhMtHGNVAcsZRJbh6vauP61ef7VFI Wnwnf.Cw0pOWehyIXWH7sAT_ouVaRQnkypjSyj6DzmK7fAOIP.oC48HIGjPgJHuE7oXpvv9fh7Mt yvPoF651yyF4pVxmAajpL3PYwL6kxJyraW4b7EWP1im_j8LDy43tgP_KVuM_EmT53BHhw_PEGLA. KMJLa7BmbjjKPRQQEFLTsDsqQe5Gfq2mhu_baUCTojTMAzqYEWXrDWFUqwjn_jbqKicyWirgLJwm Z69qMNvvUdZnrwuSHFLNvCIBnCJe51Go_.rz0_KqB22HO4iXyQfLVIpkGgmdcBu6XTClQgGH7fcb Pf697.Nny7r2IErmZc6jU7TRD1H7DWgHVZlRQPjNtN3RdFAxtOIVqfdal5fDwZKvb4oxGkwLZiDg 0WJr30.c46TdR9vmNmmudsSYpadU0eEpIdys0Vy_.LQ_eqiVBVYG06iK_wnlUCfTJjgccmWmiBtZ QenKm88XI8sJOB6YHDDcqvWdaJulAmiz7Kkl9bDhjuRvxYN9vbV6ykFD1nl6J6yYRvEhIn8Ycnz3 8mF36UW9XvcsKTMoOARVhnvbkKOmP_SwiVdW0dPINhqm6vzEBLJgTaedjV9I6rSptRhkerfeZ8M9 ZQypeXROxCMZ3X3sOdZbh_5I9Q6djhyYIwB3h2zZHg4xLoo9d7qebmYYpXSP037RlAVog1yQEqUn Os.P72gTFdw4X9Kfg4qJ6KFfWFj2JtaL.QooZQLhQB8KROMaWiHJ7wLIEvK9RGDIiBiW1CnF84ba oGKeZwgGi1XHLdCw2ZjCXuSraxIeaLQZl.3t.d1.WEmQ2969wUwTV.55TcRqTQyWs1TC_HsxCbj3 5j._xDWM.a9VuVll_bbZr2oB5uLDbu1oc314Ze3qizWdlgyhh4GVUtOiRiJQWPyAhX43YeVkQ3Hf IPS0PbjqnAS2htqzVZd_1gSzxwHzRf95LPktblg2So3orz6oBsKK.BPOlN6H6ZI_GpWMScGc_XO1 1ATBkSLbj1nDq3VVwGv7g4pUNwb_cGlua_k1OQFEtgAxoYf0SCsNZJcc5nmO._KLWRPD014i2w2m UN3ZX16t2OOm1DOxpgJJ89uPPlP47ODtVGZGrU3.XW7LNjNgMBv.1yTU5QUmpibm.2lpwlDxmT2A 64as_08bGbrxANIyPRmx31iZAl4n4CpHRdp1QKqZle8fOX.rzihW.gMKbTUfjAQa3GY6fNRu7zCo h8P_gmKVs8fNue6og_3RhKcAxRFCEk13ASR6OalZHiHP_Q6k9bs2gVbpr3jEY3ge6wMMQsE0JAYH k9NwXBRbKbYcpDA-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.gq1.yahoo.com with HTTP; Sat, 11 Jun 2022 04:27:10 +0000 Received: by hermes--canary-production-gq1-54945cc758-mg5mc (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID ca8ecba5a9699aca1b3259997d56ec3c; Sat, 11 Jun 2022 04:27:07 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.) From: Mark Millard In-Reply-To: Date: Fri, 10 Jun 2022 21:27:06 -0700 Cc: Daniel Ebdrup Jensen , David Cross , Robert Clausecker Content-Transfer-Encoding: quoted-printable Message-Id: <573B8B0C-5209-459D-98AD-EE92DDA4DF83@yahoo.com> References: To: FreeBSD Hackers X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4LKlD62ZrBz4tGc X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=PgH6hZlJ; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.146 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCPT_COUNT_THREE(0.00)[4]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.146:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Jun-5, at 15:04, Mark Millard wrote: > On 2022-Jun-5, at 12:42, Mark Millard wrote: >=20 >> I have a poudriere bulk -a -c going on a 8 Gibyte >> aarch64 system. top has been showing an occasionally >> increasing swap usage but never any sizable decreases. >> Over 5800 ports have built so far. The context is UFS >> only. The system is running a non-debug build of main. >>=20 >> Part of the context is ( in /etc/sysctl.conf ): >>=20 >> vm.swap_enabled=3D0 >> vm.swap_idle_enabled=3D0 >>=20 >> Also ( in /usr/local/etc/poudriere.conf ): >>=20 >> USE_TMPFS=3D"data" >>=20 >> poudriere's TMPFS reports normally total under 128 >> KiBytes across the 4 builders. >>=20 >> For reference, example figures . . . >>=20 >> A top variant shows: >>=20 >> Swap: 30720Mi Total, 306816Ki Used >>=20 >> vmstat -s shows: >>=20 >> 78152 swap pager pages paged out >>=20 >> Note: (78152*4096)/1024 =3D=3D 312608Ki >>=20 >> So nearly all of the "swap pager pages paged out" >> pages are still sitting out in the used swap/paging >> space. Thus, the usage is not held by user processes >> or is held via very long running processes or is >> not directly tied to user processes --or some mix. >>=20 >> The variant of top reports never having observed >> more than: 6658Mi MaxObs(Act+Wir+Lndry). >> ("MaxObs" is short for "Maximum Observed".) >> Such high usage is for a bounded time, long past >> at this point. (Until some combination of port >> builds ends up active that uses such.) >>=20 >> So I'm curious: >>=20 >> What can I learn about the data that is staying >> paged out (and is gradually growing)? How can I >> learn it? >>=20 >>=20 >> Other notes: >>=20 >> The poudriere jail being built is: >>=20 >> # poudriere jail -jmain-CA7-bulk_a -i >> Jail name: main-CA7-bulk_a >> Jail version: 14.0-CURRENT >> Jail arch: arm.armv7 >> Jail method: null >> Jail mount: /usr/obj/DESTDIRs/main-CA7-poud-bulk_a >> Jail fs: =20 >> Jail updated: 2022-05-23 02:21:24 >> Jail pkgbase: disabled >>=20 >> (Just in case the armv7 jail usage or the null method >> or such is important to the issue.) >=20 > Hmm. systat -swap reports a toal for the Devices/Paths Used > that is somewhat less than the total for what reports for the > Pid . . . Total figures (not the Pid Swap figures!): >=20 > # systat -swap > /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 = /10 > Load Average |||||||| =20 >=20 > Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ = 90\ 100| > gpt/CA72USBswp14 14G 150M > gpt/CA72USBswp16 16G 150M > Total 30G 300M >=20 > Pid Username Command Swap/Total Per-Process Per-System > 1453 root nfsd 1M / 15M 9% 0% > 1451 root mountd 1M / 15M 7% 0% > 1481 root sshd 912K / 20M 4% 0% > 1406 root ntpd 740K / 27M 2% 0% > 1513 root login 724K / 14M 5% 0% > 1514 root sh 656K / 13M 4% 0% > 342 _dhcp dhclient 516K / 13M 3% 0% > 1363 root rpcbind 448K / 13M 3% 0% > 1454 root nfsd 400K / 12M 3% 0% > 341 root dhclient 380K / 13M 2% 0% > 1341 root syslogd 324K / 12M 2% 0% > 1505 root getty 292K / 12M 2% 0% > 1510 root getty 292K / 12M 2% 0% > 1511 root getty 292K / 12M 2% 0% > 1512 root getty 292K / 12M 2% 0% > 1509 root getty 292K / 12M 2% 0% > 1508 root getty 292K / 12M 2% 0% > 1507 root getty 292K / 12M 2% 0% > 1506 root getty 288K / 12M 2% 0% > 1135 root devd 272K / 11M 2% 0% > 338 root dhclient 264K / 13M 2% 0% > 1 root init 244K / 11M 2% 0% > 1486 root cron 188K / 13M 1% 0% >=20 > I'm, Still looking for a clear indication of what > most of the 300 MiBytes or so of swap/paging space > is in use for. I finally gave up and checked if a swapoff would actually bring in all the pages from swap space that were needed (if any) and then un-configure the swap space. It did. (The bulk -a was still ongoing. It was not doing memory-hog builder activity at the time.) So such an activity may be a workaround for long running things like bulk -a to avoid a swap space accumulation that seems to be happening. I do not know how much was brought in to RAM vs. simply deallocated from swap space (pages not changed and still in RAM). If I do such a test again, it would be good to figure out how to monitor what the swapoff does for bringing in pages vs. just discarding them --if possible. After a while 12136Ki Used showed up after the swapon that reconfigured the swap space, which is about the size of the increments that I'd observed for its sustained increases. =3D=3D=3D Mark Millard marklmi at yahoo.com