From nobody Sun Jun 05 22:04:22 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9CEB91BE9830 for ; Sun, 5 Jun 2022 22:04:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LGVyq2mwbz4WcW for ; Sun, 5 Jun 2022 22:04:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654466668; bh=5M7GVTBRX89tW/6d5rIaxLDlb2Yt4pPmAYemEz+Y03s=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=ChlTWnFw1oy/4U1ArH7i5Uq4ynj3xyYMIxDdL4WwD3crhZIdx9Y8CaVx7TRGO3tmPLvlBKeo6Sfr8uFAK7GM2+O3+KJjTThdI5kihuBG7J2pEXXNyH7a0ruYGKiJo8cWLo2rlz26cthT0xerzTiud7IdnMLjBZ1XSPZVd9wnu0h3M0G3cJ9aZ4PayyoDbqjK8gc802pyMh5XsdNO6FRbXb9a2H+wPvdRZlgGFKqH5IYVNwsOnLSBRYM4Dhc0zAGiiEC+LYWIBnKk8UT8VQcZblSonrQJTuS3tWjjpFi5qlW/gH9QSQGgkhNYKqu4O6JvkxqcRmpNZwbOICENc5w9Cw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654466668; bh=lISBFzY0mQg4zZvkU+rMiT7a4eoZvV875m1UODoZI14=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=jAX4ZnO8qk1T5n6bv2I0qvD+GqDMmYGTD7H0OLnM0YZ0TlxAngTEdXOrdVAlzQVvUZmDungKOAdJI9vrUDTt1yvtHwkDvK6M/e+EfdJEeHnEi1wEBW7A2L03JjqQ1h/N9Qt96/SgfPYqonzYQgP1q967l8Hf5EZbyGajf7orQt253MwnnFR8evmoCeq16je+fHTmnPE3Dg+Ux6vBxrN06CJMaRad3pSgyboDmi5FYoFyE1KTnnccc81x7/9cKOjy44jhrpi8qlLxBpRGbfzvqqyoAwL+U2PnIadbsYkQRBXeYWi7pIVWBprkrPxWqM21iJxKUjnSS4FRbeNL90/Ffw== X-YMail-OSG: vNDOuv0VM1lHuLIBi5f30gsvSXu3EHTIexvGKPJHfCQ._iJUu5xpSoR14imjrsE OMYyfLKD3hPjp06MqakVkDQel37dCB2l4.Ziyc7WtEYthQKYB48hytjjKsj2fomMb_ZWjACJ12DC qDJYk1hgd9X7TIIFHaRWtnadKApRJDtnejoD1kCVeTRaQSWH3OJ4l7FemjogbNgdizCqk4LnWrrL x8VSTlh5K2yC3ZUunP9ty3gWHbGV3Oqs6CA8b7pKyfKDTInpyijMimDr_GrXwI07a9WIirIEqPuL Y9gVE2sdG0MZCCeJDwfDNZnKtrc52vTm3CwFeEgwYv7NixLh95ZK5cUcaaIKbWt_6j8OpRZBbHRT PpI5wcCYcq239ue.S1Qw2w.u4r2i41NfuCIRo4ow3RRAQxny9hg3cuZr5AuYpN0AlLK5NS6pq4so CsgBDhHPEPBJze5NfyX14Q3.YSOYAg4.ECJDSaLfTJNs28bUGJPcPSIuqvhn9ayK9j1Yh7slQJA7 FN4EJJCvElNQpPnlEf1v3V.hrZRaaS6wZxInANHYF3spsT7cu5Hw0I8PpXZ4NWNiWOpI4Henasa0 d98RJOTI1_h_LAw4rAKOl8ypBgkWib.jE_3Mryg3WGkwojPDN6uPeQ7hN8swssqGKaPBKC_Sd_lv J3r44MINHo5OZnz0avrFtMFC5V1NNNKaS58wtjg3csR5UntcADlH3uX60nfNentT9vID29MDcgmA poc0ncb9mbVOd1F9t8nHGyEpEinWgfUSNDJCMoKpS9ym._VFI076MPd.thfuGrpc3qEyYdxvMlYZ _GRJ.LtMGrGtf37BjkUt_m0ebfjeOHblKqahhNrzyan1.R.JutduxlQSUTcJx3IePzJNyaBnmKRm HgRnYsZsywjTnfgdXi4gEm3Iu7FCG2tjyzNJFVgV18NOqTVeRKJVsJt.PtbNNN.HH0ncwHwHZWZB ROUh6.Nzr6r_USrG.NJ9zM7Rtz4eW4_u4Z2_xsxOiddUVpUrIsPbKBXHjoz.mma7.Mp2IcbZk6mu E0LQeMgpKG9vq0PunI6GepKiMTupf5k7UGfIsmoNT52cOz0Ivep8YIzQlOoA1nZBiZQ_gn_dpAev 1.9wxWrSqrSuIDzwzlMxtVH56rvKp22xiuT7.5a94nKZTA202cdKZbzYR.a0_gXcmZBQfnKSf1by filE9oQEZxnJXvIBe51..GLUp1QEKWMQ5vYc6f6ioJQhSClVLh7LkpxM0LYoTP4i0Wswaxmxw0ES 1SLlBoLs._K3cLmOKmZIgFfdZggBzugPFlzsESJudW_7pQWB4jEkYx2BINxuicNKlliguLcLMl0O iGjB_yZGwAIcNQPdgrM3RAVAjvk1vB8n0s9yq4RZCVcDEHvSTshQcAs3TP9nscTJB0dFMQZfv.gt 6Jp9wplX9eg8DRyO4M2z.bamNMH5MqYhVSOgVd1AGTBrR4ZuwUPGQWBYgG3m2g4o2KDFf_5iY4l4 jOBLf.wCX31M7bkrp68yET.tDjZrEj9qmI5RGRhiBS7n0_x282UsMxmVXS2Uiky02GhkXXDfXpmL 9826hIYqbewS0rQuGlLIYVvelZsvGJ6HEQZTGJhdAFfe7YGS1jz3I5QdXamT80lbR7iz0WLQR6q. KPArwU.xEvs5ayPhuNqGb_s8sGe_Dmm1zhVXMpMpvVrsCOxswNnVilVdkF.zXLYxvlvj34T044sB y84sfiSYpNn2sNAH69qVdFTe40e0agfcs_U0DHdXIxt9G2iapx.qHqVuHHSm8Q8FK.skb6y3gkxj 4kGx3a0EfWN5CyAPO4p8H33x2eeookoF9STPDqSmVoTDXnjfmtxDbjAAhrP7v6xGsAzx3EHH_3K3 YD9xVlEnzttgZJT_EqANWQDM801wMYcEL7B1Jd1fYXwhudL6Uv7Y4ZRy_q8V1ACq5eIwMKpwgJUa 6jTbE6BXNSffTmMbPmzN7C_QK24HMJFTqxJpvFs0nWXbob.6xH5XZwDWmAWZXBvfZAzfZ_ISEuvH uVkPCE7e82.JHplYZ3r1gjDVyJ1J6QMOlPHqTQt6Dh0CjhCj0zXiWWVNjpryLs1a5gzX_naxo33E 46qqQbUehKc.M8PtYquEpeuAcOL9ItVYO2bZpF0XXpE.XvxCUJkHoLjL2JJ7R8lbYxYf4PUDedq6 JviNUmkbJLJ013EC.nR4HIKudfcfo4VMZnnKl6VTgNUeX7WXGrqVp4Ekq7s_nYEWhjPmTwV54Zkh fg9LVtCyznk6mvIUmCYeDUTjYp5jRuBRVmyhIjDpxbv29yBpSpBM7w9G0TmDpOqhD30D3poC8 X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Sun, 5 Jun 2022 22:04:28 +0000 Received: by hermes--canary-production-ne1-799d7bd497-7lvgk (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 53d7fcdb84f2d78da636cabe054bfa0a; Sun, 05 Jun 2022 22:04:24 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.) Date: Sun, 5 Jun 2022 15:04:22 -0700 References: To: FreeBSD Hackers In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4LGVyq2mwbz4WcW X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ChlTWnFw; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-1.53 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.19)[-0.186]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.991]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_MEDIUM(0.15)[0.150]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.148:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Jun-5, at 12:42, Mark Millard wrote: > I have a poudriere bulk -a -c going on a 8 Gibyte > aarch64 system. top has been showing an occasionally > increasing swap usage but never any sizable decreases. > Over 5800 ports have built so far. The context is UFS > only. The system is running a non-debug build of main. >=20 > Part of the context is ( in /etc/sysctl.conf ): >=20 > vm.swap_enabled=3D0 > vm.swap_idle_enabled=3D0 >=20 > Also ( in /usr/local/etc/poudriere.conf ): >=20 > USE_TMPFS=3D"data" >=20 > poudriere's TMPFS reports normally total under 128 > KiBytes across the 4 builders. >=20 > For reference, example figures . . . >=20 > A top variant shows: >=20 > Swap: 30720Mi Total, 306816Ki Used >=20 > vmstat -s shows: >=20 > 78152 swap pager pages paged out >=20 > Note: (78152*4096)/1024 =3D=3D 312608Ki >=20 > So nearly all of the "swap pager pages paged out" > pages are still sitting out in the used swap/paging > space. Thus, the usage is not held by user processes > or is held via very long running processes or is > not directly tied to user processes --or some mix. >=20 > The variant of top reports never having observed > more than: 6658Mi MaxObs(Act+Wir+Lndry). > ("MaxObs" is short for "Maximum Observed".) > Such high usage is for a bounded time, long past > at this point. (Until some combination of port > builds ends up active that uses such.) >=20 > So I'm curious: >=20 > What can I learn about the data that is staying > paged out (and is gradually growing)? How can I > learn it? >=20 >=20 > Other notes: >=20 > The poudriere jail being built is: >=20 > # poudriere jail -jmain-CA7-bulk_a -i > Jail name: main-CA7-bulk_a > Jail version: 14.0-CURRENT > Jail arch: arm.armv7 > Jail method: null > Jail mount: /usr/obj/DESTDIRs/main-CA7-poud-bulk_a > Jail fs: =20 > Jail updated: 2022-05-23 02:21:24 > Jail pkgbase: disabled >=20 > (Just in case the armv7 jail usage or the null method > or such is important to the issue.) Hmm. systat -swap reports a toal for the Devices/Paths Used that is somewhat less than the total for what reports for the Pid . . . Total figures (not the Pid Swap figures!): # systat -swap /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 = /10 Load Average |||||||| =20 Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ = 90\ 100| gpt/CA72USBswp14 14G 150M gpt/CA72USBswp16 16G 150M Total 30G 300M Pid Username Command Swap/Total Per-Process Per-System 1453 root nfsd 1M / 15M 9% 0% 1451 root mountd 1M / 15M 7% 0% 1481 root sshd 912K / 20M 4% 0% 1406 root ntpd 740K / 27M 2% 0% 1513 root login 724K / 14M 5% 0% 1514 root sh 656K / 13M 4% 0% 342 _dhcp dhclient 516K / 13M 3% 0% 1363 root rpcbind 448K / 13M 3% 0% 1454 root nfsd 400K / 12M 3% 0% 341 root dhclient 380K / 13M 2% 0% 1341 root syslogd 324K / 12M 2% 0% 1505 root getty 292K / 12M 2% 0% 1510 root getty 292K / 12M 2% 0% 1511 root getty 292K / 12M 2% 0% 1512 root getty 292K / 12M 2% 0% 1509 root getty 292K / 12M 2% 0% 1508 root getty 292K / 12M 2% 0% 1507 root getty 292K / 12M 2% 0% 1506 root getty 288K / 12M 2% 0% 1135 root devd 272K / 11M 2% 0% 338 root dhclient 264K / 13M 2% 0% 1 root init 244K / 11M 2% 0% 1486 root cron 188K / 13M 1% 0% I'm, Still looking for a clear indication of what most of the 300 MiBytes or so of swap/paging space is in use for. =3D=3D=3D Mark Millard marklmi at yahoo.com