From nobody Fri Sep 13 00:19:18 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4ZgV567mz5V2q1 for ; Fri, 13 Sep 2024 00:19:34 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4ZgT5CrMz4YD2 for ; Fri, 13 Sep 2024 00:19:33 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=P7sM6sac; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1726186771; bh=+hBrdLMwgyVqmje4+Io/0/6PhVpWgEYLT47kheprSIg=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=P7sM6sacAPldF0SFcS0ldl1Wj8eH9ggSIbgd+kzE/Nnr8E7T0qLBwNgyr278Ozmczilal/i6UaZWv68gSY9bSIxf0HApynAjN9XKWMbpc5nA/yuu6EfBFYaSA+QKk5g8st5gdRGM+8X/8gnqyrBnF1KOnSF8h8BwSIT+ez1RFKI8W4jrPpko/BfpVXidq4HeNLr8rcWZjjzivJhOr/Aintk8RmIHeJvRW+aLni+LiEpIamuGv05CsuESxKwgW5MsR+dmPsyakYoXm+Q9mALdW3FbcVZsL+FEZ0EW9C6wWY823+XxSRr9rx6PN87o2PE8AejC5GejV6EYr1agEFVe4Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1726186771; bh=NjfpAiw48QF9FnglYY/bTWUKGmxK69AZsExHinI/C5Y=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=ZhPM/eXb+xbWDVCDt5c2R6ZtK7J66ud0CXJTz6fyi5QLkyvHS0I9fJNeuHNb+ZTFxVRbsXPRyKf72yUTDlGjhtCiyMj4y0HHd+GcjLNhX5kgvWpLg3vCcHT5ux3MBRaTbVN6eb1oau+AWro+rGO8poQy0vZpFBSsHyy49pxVCdsK32/Z93efpArquQ0CSlqbZPtG3cjs1tiKmzEsjAuR4IYTHfr7WfxGXt9MO+LnOBusXn8yBsWY1NGdgNZHRLDKJEDmCS9cC8VZQPrvIbCjbFlTtoyuLefkjJKpkmNYqeoLrbPXy7QImu2tOQuOsvdqYVehOySN3IoWWVoPMFANBw== X-YMail-OSG: N08BqT8VM1lAxfXfgtO8MyHclfD3vUatotjntMrilNGWc2cUmW5ruuGYCcRwzNP BiAy5GNOlp1kg1eo1gxPB.8Shhcjj.wT_vCc7zI2ONhaBf0afVTuiHDdu6j2d91Xx0a3.oIhr.PF s40nxRve5JPTNQNoK38Zd6vEshcyIcTezST0M_ErtUdbfqEyQl3gjKl821IZ1H_B14MQl9ndjjTF wclbJ7s3tKWe7axxaPs0_9v8sRfx3tVcQ9iak3ht8sgQ9AVPBFfZMA_1BnbDbAnHyYquNZA3ELv_ 4SJbwrnDv63tEtxcMOZcPDiw_zVX1xbi88CejzCrqiv9pJ6PVtrogTKncATUujnwDJVwaVgMhKHj 4sKXlB.X.IsaPq8CPhtZFzV3sge0xdRBypEtkLbWXTtp17DHatzbDGINIadmCe9eHB9GFhSE6h3O muOlIxaHdvFfar3DbnZnVttrRobfysM9yxagmbkETbQigkd53hPhFzktFEc4I5BlVOnA9kGaqj9x 7sZBp6T820dAoIeo6SfzMoEZR8ArRcJ4Q6Rx.J5oM9tt8Sfzdyl9XEww_acaEVK3pyQphniSQ_Uo XtH.gnSkb_lNkCRHJnz2eMjLSo6htM.HKgyfM4fK8MaTySMSNu3n6cbgOxSfY6_2MheJ21vMuoGV 1aSxKVW.BKo2jOjbh2XLAqTrU6JrdDEhL0PMq7D29zGBYEqjpP_H7zrVEzebA.auDWcQqCt98Ba9 w1XjbHGbCy.5hc4rYljgi3C_EhJHS747ZiVtaL15r_wKllPPbM9hkb.3kj3cvfRNXZXzPT_AfVBO CuT89H.WoQ01er6XJ8V2DO6vyg8N5kWXIOiffRQZUP.51dYOWEl1lmgyYPKJG5Va.8a7qth0XPb3 mTh1I7j4.nUemn3n8fNXheuTdPWaiFt1qV7BKNxk.WPr3ZLs0sLhsY20B6T7gFNtRgQpTfAK9zlY ZC3UHgc2pc6U5_SeTKfVyR2T_KG5b6NMRG0Jp5HMqjdipz6OZtPuJGh3mTpceFHNlt0dChfFql3S SxcOV.u3vnngNoPVofmJKFAsgWsOD4FPk7Wu6.dX3cfRND8zia.VHV72_FFu90WaWCy0SrZ5QjGr u.FhwmRKtd8d1WN0Setd3DUx2ALeAiOGKUDdU7P4U28ENpruGe_X9fCmLOnTiASh8l1AzaMX9RUs mdA91IrJ9YnVBwvotJnoauDFUpYGGMh78B.lTGvgbrhXnr.IcAK3MX_lPE6oBytQUB6XPdtx8Qcf dqhLWFCCc1wsiRjxrZxCJVQP55AUVtk8M5bHYCPVKLuiIk01rgRiIN7vVLOT7pqpGJofI0oxLnN. ow7b.UP12OgCl22xh3TKdXzt1rsEpy_yD4MgL0Iw.biDESXFNBIVWLdsJo.RroeX8Lkf42aoqZfs JXYpK5O12VvFiurY9w1TtoFqp4mCrOd6Pf92N.IjA5KwVwz3d9nxNo6VCpcAzBqiPjBUiZkNlnXj XQHRF0tyccTChN7APJShzCrzHKfIuXXb3kFCN63yIPk1YGn4TW_65T3Zv5ufqv3lW8_7SzOJ.K8G YCUpy59BwLvBHZ9RGQkME0RDjmRjJEu.NH.5isWq20bxa8eyFh5IaoDtySYh4micA2Ez_f5HZ9Hz F5r_zqPnLca6F1UWDd7e8d.IgBBstB9uPFH255JQo0ueaQCkYTKwPhCEU9WeiZZFEewzWsVMoERg zC8bd9VTutmEU.LC0m7dKEh4lEld.F61eJuvWV7qFrVAi9DRXJKy3COjDn5AbfdZxh3UvBgYGZz4 CTQSa0fKHzlL_7bjbaXTu_O_jWtpB9Zm.BbNAKmPkOEQkR2RgWFmAYpR7I.F62nZhj.Olt83bT0f 41XAb3ZNPAouNE4RGMY1DGPEOTpyuySLfp83BQHfxA.d8vi7zFmMB9Z2SrPfR8uOWQ_T9z8r_EZy PH.ThIXQoWekQufeER_E_.VKan555NnwQa8PpT3tDzbHl6wdPTzliYHb.9M1VeZMrlns_7CZAT4v D2zmudrKY6f_2pun3TyB_sggahQ4T4.cowU90OqVhfLq5cdM3XXJJbBIK6rn8OnXvJ7IfJr.fIZJ 8gdoeHZK2cfeIFSZkvzxBZMWQFqkHgGBdctQAZta2mSaLS3g0CInAOCETX.CLQBZN62S9_taACyk H.Z1MT7rtrM7YmzxD5TQG3wQ2TkE3FkzXzOcNID.TbAJ54TYPwLhN2IJbwkle6jotDxNRiH6pFfd CUjNIzqYTZ5sSoQ2p9akQnrnghvUmBDLJIc7jNoKKazmCpOy4cjPbNGvY8m3GMpimj7x57xXuSbI .ykFkr5VcKPGjFo.2rA-- X-Sonic-MF: X-Sonic-ID: aebbc545-c955-487f-ad9b-2c4cc6593ddc Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Fri, 13 Sep 2024 00:19:31 +0000 Received: by hermes--production-gq1-5d95dc458-5j27b (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID b8adfa242ede0db0e2226e841d39fba6; Fri, 13 Sep 2024 00:19:29 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: RE: How to explain high memory consumption of a jail after all large processed in it have finished? Message-Id: Date: Thu, 12 Sep 2024 17:19:18 -0700 To: Yuri , freebsd-hackers X-Mailer: Apple Mail (2.3776.700.51) References: X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.980]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_FROM(0.00)[yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; APPLE_MAILER_COMMON(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.84:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.84:from] X-Rspamd-Queue-Id: 4X4ZgT5CrMz4YD2 Yuri wrote on Date: Thu, 12 Sep 2024 18:45:16 UTC : > I noticed that when the port lang/rust is building in the poudriere jail > the memory consumption of the host system remains high all the way into > the packaging phase when the pkg-static process is the only active > process and it consumes a very little memory. > > > During build a lot of memory is consumed, which is understandable. The > system remains at ~500MB of free memory through the build process, > according to top(1). > > > But once the build is finished, poudriere goes into the "packaging" > phase which only runs a small pkg-static process that compresses the > built files. pkg-static is the only active process in the poudriere jail. > > > What looks strange to me is that the host system's memory consumption > remains high through the "packaging" phase which itself is low in > memory, and only goes down when the jail is destroyed. > > > How to explain the high memory consumption of a jail after all large > presses have finished? You do not give much information about configuration properties that contribute to memory usage patterns and how to interpret them. How many FreeBSD cpus does the system have? How much RAM? What do the likes of the boot output (dmesg -a) show for: real memory = ??? (??? MB) avail memory = ??? (??? MB) What does the likes of: # swapinfo -m show for "1M-blocks" (Total) once SWAP is fully set up? ZFS in use? (Any tuning/configuration of note?) Only UFS in use (no ZFS anyway)? /usr/local/etc/poudriere.conf (or analogous command line settings): How many poudriere bulk builders are there that have been or are active prior to or during things? (The likes of PARALLEL_JOBS= . . . can contribute to the answer.) What is USE_TMPFS= . . . set to? (all? yes? wrkdir? data? localbase? no? some list of such?) What is TMPFS_BLACKLIST= . . . set to (if anything)? (What is TMPFS_BLACKLIST_TMPDIR= . . . set to [if anything]?) What is ALLOW_MAKE_JOBS= . . . set to? What is ALLOW_MAKE_JOBS_PACKAGES= . . . set to (if anything)? /usr/local/etc/poudriere.d/*make.conf (or analogous command line settings): What is MAKE_JOBS_NUMBER_LIMIT= . . .set to (if anything)? (Or analogous settings: there are several related ones.) What is TMPFS_LIMIT= . . . set to (if anything)? What is MAX_MEMORY= . . . set to (if anything)? What is CCACHE_DIR= . . . set up (if anything)? Outside such configuration files: There are other things that do not stand by themselves well. For example: "system remains at ~500MB of free memory". Is the amount of SWAP space usage varying? For "Mem:" Active varying? Inact varying? Laundry varying? (Is top even showing it? If not: zero.) Wired varying? (You are indicating Free is roughly constant at ~500MB for a type of context.) (Example figures for comparisons/contrasts could prove interesting.) ZFS's ARC Total varying? (if zfs ARC is in use; contributes mostly to Wired) Other things for your context: When rust is building, is it its builder the only active builder? What does the likes of: # df -mt tmpfs | sort -k6,6 show during, say, the packaging stage? Note: tmpfs RAM usage can stick around longer than one might expect, including while a builder is idle between jobs or after its last job. One thing for rust builds is that they have a large file system usage with materials that stick around even during the package phase. Thus, there are consequences for RAM+SWAP space competition/usage if such end up being handled via tmpfs for rust builds. There are contexts for which (Active+Inact+Laundry+Wired+Free) shrinks over time compared to "avail memory" --even to the point of failure/hangup. I've no clue what is at issue if significant such shrinkage is happening for your context. (No claim such is happening but it is something that data might show.) Note: top's "Buf" overlaps with data in the the other categories and would cause a double-counting of some RAM usage if included. === Mark Millard marklmi at yahoo.com