From nobody Fri May 23 23:39:10 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4b41pP06HPz5wm23 for ; Fri, 23 May 2025 23:39:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-25.consmr.mail.gq1.yahoo.com (sonic311-25.consmr.mail.gq1.yahoo.com [98.137.65.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4b41pN27hHz3yYj for ; Fri, 23 May 2025 23:39:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1748043562; bh=3LzUmuAUorc78/9rjYnLl7FXE0DHnla4mGRjXJT9Ff4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Y1mlQFl/CihaBAN4I9L5FajKRk5fSbGEQbK0HXH5C3nVHwUaoA2F0YTVJdIJkxIz5x1S1tuKhsLjyu/kExeisVvP7Dq9tkpgVV6uT7rn2KMASgRhzEQ6E0na+csYrkXGDijRfOAw/7rFS7udpLSJKdGt5TrDZvx3QwvtMCfmgAjPC/QXTqcoTgZbylMJis/0W89VJn2j9XnWhybGJI5yTIm60VvVZpheHRL3uSiysCW75beqewuDY7euSRSvzmyD3QHl9Hv19DU7aKaI7SVZHObtwCcc23MaqR4Wfn0U3Q1YVgKFi7wzpt9ikJRNkkt7+S9rr6LlMPm7xWF5Lu25Pw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1748043562; bh=+6XOW2aa5ki5ck7uc6Fds5vC2sfY/ZUG3TdTins3zj5=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=U9uYJHmz78Ob/L1EAUz620tm1ZxielZQD4Z22iesbdKrxjJMzYjwge+Z5vbwSt8TUO+1IHliG5059eqCosgDgOjgkxsrRX5ChHfZjLbjkhOmtjNj358u/NSbjhVF/NDFyNs9h+GHncVf8h5FPs/0lZ7COPZDula+k3tBtuPv4ElA5k/5eWrkO1bYqXPtuTOx2Cj0l8EahbfyWAl6YANAwLmn590lO6HSpJjkpNddwCEM+1bGKYAGxrQPqsmHgSshABU4V34Sri2bAayTKI8BLdQDOT9zG6rEVKVyMDgDOG7CC2fkH+iD/T+gcJdiD7KyA6Dy8dRpGhCKU/DT8PNk1Q== X-YMail-OSG: xHWvmWsVM1kX0TsDQwztnDKVrYsp3a5dxPR42yLuvICUW39vNQV8wqQute2vT_T 8RlATP.7_1.diES1oz.zQ2q9j.wkurQcdX1auyD3XwJ5_6.CKNJp9YP6OqObgeiFMsZ9_r9hQY4u mWj6JEcq.AwciAGzvut8NmkoR1GoPgf6Yibwj_Ct84P_J3N_uni3K_gnhavauTjnQf12uFxqjVb. Xq44VQ2jvr0HA2mdzt8Ckp1R.DwWEO7Satbuzdw8w7GxGHtKdMa3p2fKC5QnVMHVRdnAYxguj093 YF2hhoUxYV4Oa_ApTUmLxuB5F3FX1_CNHcGcYsH_s.u2GFsy0YgvShmaLUl7I.sx29AiM.onVTLC yU1nIocmyv9nCkCGHpNm9fhhSozBBXBiVvJSb7LSEbczLlWqTNUWF.kP0SVvni3LxbTZDKQ.xwnv yBuKJTW9DYLonDUURXCuvloP55aDe.I_l1IsoW8DTaHPGxefyJg_UX.cmGQDRECtbHzWtz7sd50S rfiO8h196shK30knMX4Dmb99aSiY7lxE..ZZQUEnDJgw2G24CeMupP_E7sQ_Qn.HjEepEkzm8sdq LA4dnzusRQogoMZlPfLce88o3V7KL4dg8FBRNP9ED4ysPPaIsqeP6KW4UvmsJ_9IWKZJM3fdO5.I VtDdAmdGrKoheYQ7Gketdcxn8OxU9_mEBDJg9VH.daYh_0HWGOdEFxju8skNpSqhXSfRFzAccW8K ViSyhcvssYp9yS4UlqoFfU4ejmtYiGuRWOkRiVu1RDor3HakNHMRSHXhqutoGe7Eby1fGK8HUYja cvVRMITcj4azi847u6gwpq13JefALGyZXSG4ld3.MR_dvA3B2ii6jZPoBLxTkSUq3VcFtYF5bvF_ 6OCf5QE4dlNw7kX.GbDeZexEzyUBybLIzvBSUuq.dGQHtYI96DAB3nvzoxHR5jl.qamEAUNA3XQz SASwJYrmMB72FBx3JmB56O6xLLgE_qLs08ExhsnSr.FX5NaYjqwddnOUQvMZbzzy7c2IT_f5lQeN 0tUU7U43tgSfR2R69YWJ1scEIoMNqv4eYOmaVeKRiPRAcQiNWiyYmrox7ReYJ6JCL_RS9Gh87v03 6e4o_oA5.QK6u_QAfmLk9FOxuAyMyUxLxhpt_pcJx29B1rq0xmbzcfQYmxufiRc2iDPGeMDKMVv9 1QcTYozm4HWG09dxFjJzkXKDcYA1l5ZwcMrfTlbHMrwLvEbEm5.xV1FSNC6cbfeYNs98WdSJkC9o UWrAyB0PQ_t657tfGnXTO1l0jnj3Mbn.QMtofACFx0GEJU2WQwCjU4jCt9jLSSWbJ7OQXxP94m_a SsT1KKvddeGYeXERhM_8a0_gSCCMm.5ku4VDkwx9Nw4mTO6s1XcoQ5ivcmkDLL.RNM_edjkjxS3R 2VLSUXkQ6XF6r0TcGjwaMqTDr2lLglf1BF7DojILiD5.uB3j174pcGYeYHHEm6QZZ4fg6NPnlJRB 12nop5nQQ8TaiEMlOfFGj4TwayVqzNGstr9geoPqMw7nhKoqikg45wBLB_iYlpCkTGzSbqD00UpJ Wdtl0jgZ5hBtWsxiEyXZSf_O8Zmpncu4I.62U3cYQjJE0Mu2fIQXizCYUVR28XU11IxZqcsiKxVQ Dd1pSJ4jQGZmX43..nvzCXy0UmYnLiZCMx9CejIQR5OHkAQvHkxt2YUXI6lQYXriPOE2fEsM6kFw m7P2DxZGt4Fy2hRtLGxNr4P1CKFDwURNEU.9pmv00RnrEs7.ZlogryZCCvGUW5MvO7YXjUj5dmum g7NIPp83LkaZIVOtLn.zzF27kynIrYzDbl_xWy1YI1KtdC3H2zkG5hiubXFnlaPvT5narN7T2TZH x2a49cR9Cisfd5L5g7wfGPXx4ASWtevdxSpcNQ3Wo9lesUrJ.iD4rBHeXjkBg5WniK.DZWJ47DAu cScL03cQAMsVgMMq7xEPVkcZi65pJbAJc6fmcOQZQCdAr87g1URfrvzuZ8aVo4vwcdi6LzkLiU0I RXNwd8wuUjw5Rg8b43mphnZ4_FPdMT6ZuuV_Kqn7O5AvogsCB085PscH4QYWeYMAccOcppCr4ALO uBV.72u.Zw402MNK4m9RC7CyUtQA2Oukzfed7diJruy1RrGGWty9ASdS3YEx7.33_hnraT9kF3_J W25Q.x7dE11WB1qGGUKS9nmxEuOXQWmN2fqhMW_r._G47wRuTfPXrvG2Ui_.gppRkazdGVO_XOLW d_TvaPJzUoxhBM6trjVy.bJF960JaXqF.T_x_grgyZrT0jOr7pQHmyNTQEGQGEhpvpekvolqUZVK OBLBwE7Y- X-Sonic-MF: X-Sonic-ID: e6d10402-c6cf-4081-a53d-2cf316d826b3 Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Fri, 23 May 2025 23:39:22 +0000 Received: by hermes--production-gq1-74d64bb7d7-tqd77 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 1de00a6d2894e1a9cec4dd2db81e871d; Fri, 23 May 2025 23:39:21 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.600.51.1.1\)) Subject: Re: Is there a way to tell poudriere to allocate more memory to a pkg build? From: Mark Millard In-Reply-To: Date: Fri, 23 May 2025 16:39:10 -0700 Cc: FreeBSD Current Content-Transfer-Encoding: quoted-printable Message-Id: <46F6ED96-CA4C-4BFE-812F-86FFA4E3B758@yahoo.com> References: <30CF8464-32D1-4752-865C-1EB1CA9DB4E2.ref@yahoo.com> <30CF8464-32D1-4752-865C-1EB1CA9DB4E2@yahoo.com> To: Dennis Clarke X-Mailer: Apple Mail (2.3826.600.51.1.1) X-Rspamd-Queue-Id: 4b41pN27hHz3yYj X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Spamd-Bar: ---- On May 23, 2025, at 12:13, Dennis Clarke wrote: > On 5/23/25 15:00, Mark Millard wrote: >> Dennis Clarke wrote on >> Date: Fri, 23 May 2025 17:45:17 UTC : >>> I have been watching qt6-webengine-6.8.3 fail over and over and over >>> for some days now and it takes with it a pile of other stuff. >>>=20 >>> In the log I see this unscripted trash of a message : >>>=20 >>> [00:05:03] FAILED: v8_context_snapshot.bin >>> [00:05:03] /usr/local/bin/python3.11 >>> = ../../../../../qtwebengine-everywhere-src-6.8.3/src/3rdparty/chromium/buil= d/gn_run_binary.p >>> y ./v8_context_snapshot_generator = --output_file=3Dv8_context_snapshot.bin >>> [00:05:03] >>> [00:05:03] >>> [00:05:03] # >>> [00:05:03] # Fatal error in , line 0 >>> [00:05:03] # Oilpan: Out of memory >>> [00:05:03] # >>> [00:05:03] # >> Way to little context so all I can do is basically form >> questions at this point. >=20 > Sorry ... I just realized that other people replied to me OFF-LIST and > that is not helpful to others. >=20 > So the machine titan is fairly beefy : >=20 > titan# > titan# uname -apKU > FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1 = main-n277353-19419d36cf2a: Mon May 19 20:40:28 UTC 2025 = root@titan:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 amd64 1500043 = 1500043 > titan# > titan# sysctl hw.model > hw.model: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > titan# > titan# sysctl hw.ncpu > hw.ncpu: 64 > titan# > titan# sysctl hw.physmem > hw.physmem: 549598998528 > titan# > titan# sysctl hw.freemem > sysctl: unknown oid 'hw.freemem' > titan# > titan# sysctl kstat.zfs.misc.arcstats.memory_free_bytes > kstat.zfs.misc.arcstats.memory_free_bytes: 404796436480 > titan# sysctl vm.kmem_map_free > vm.kmem_map_free: 431405096960 > titan# >=20 > Also plenty of storage and local NVMe stuff etc etc and dual > NVidia GPU's that do nothing at all. For now. >=20 > We ( myself and others ) have already found that the problem was > me. No big surprise. >=20 > USE_TMPFS=3Dyes > TMPFS_LIMIT=3D32 > MAX_MEMORY=3D32 > # MAX_FILES=3D1024 > MAX_EXECUTION_TIME=3D172800 > PARALLEL_JOBS=3D64 > PREPARE_PARALLEL_JOBS=3D64 >=20 > That was the problem in the poudriere config. >=20 > I commented out the MAX_MEMORY and TMPFS_LIMIT and then watched > as www/qt6-webengine built just fine. Guess the jail needed more > than 32G eh? >=20 >> I assume that you have not explicitly restricted the memory >> space for any processes, so that RAM+SWAP is fully available >> to everything. If not, you need to report on the details. >>=20 >=20 > Yup .. I had restrictions in place. Those very very few packages > are hogs. Just massive running pigs for memory it seems. >=20 >=20 >> How much RAM? How much SWAP space? (So: how much RAM+SWAP?) >> (RAM+SWAP does not vary per process tree or per builder, >> presuming no deliberate restrictions have been placed.) >=20 > 512G mem and 32G swap which never gets touched. So RAM+SWAP =3D=3D 544 GiBytes 544 GiBytes / 64 jobs =3D=3D 8.5 GiBytes/job [mean] Some builders will exceed that, for sure. But you are really managing the total across the 64 jobs (adjusting the jobs count would be a possibility). I do things like: For 32 GiBytes of RAM: RAM+SWAP =3D=3D 150 GiBytes (so: RAM+SWAP =3D=3D 4.6875*RAM) So for 8 hw threads (and, so, 8 jobs): 150 GiBy=E2=80=A0es / 8 jobs =3D=3D 18.75 GiBytes/job [mean] For 64 GiBytes of RAM: RAM+SWAP =3D=3D 300 GiBytes (so: RAM+SWAP =3D=3D 4.6875*RAM) So for 12 performance hw threads (and, so, 12 jobs): 300 GiBy=E2=80=A0es / 12 jobs =3D=3D 25 GiBytes / jobs [mean] But for 16 hw threads (and, so, 16 jobs): 300 GiBy=E2=80=A0es / 16 jobs =3D=3D 18.75 GiBytes / job [mean] For 192 GiBytes of RAM: RAM+SWAP =3D=3D 704 GiBytes (so: RAM+SWAP =3D=3D 3.66...*RAM, I had other reasons to constraint this context more) So for 32 hw threads (and, so, 32 jobs): 704 GiBytes / 32 jobs =3D=3D 22 GiBytes / job [mean] But I also use TMPFS_BLACKLIST extensively with USE_TMPFS=3Dall to avoid having much probability of running out of resources. (This actually uses less than All but does not eliminate tmpfs use from being involved for the blacklisted package builds.) This is for use of ALLOW_MAKE_JOBS=3Dyes (so a high load average context much of the time). I do end up with examples of SWAP being used for paging. In a couple of those hw contexts, I, on very rare occasion, do a 'bulk -c -a' run (as a test), also with ALLOW_MAKE_JOBS=3Dyes . (I generally do not use MAKE_JOBS_NUMBER_LIMIT or the like.) A bias is to keep the hw threads busy with already-available useful work to do without having to systematically wait for the next unit of work. (Of course, other tradeoffs are also managed.) Note: My original questions should have asked about ALLOW_MAKE_JOBS and the likes of MAKE_JOBS_NUMBER_LIMIT use as well. (Not that such matters now.) >> Do you even have "whatever it seems to want" configured >> for the RAM+SWAP? (I'm guessing that you do not know that >> the "128G" figure is in fact involved.) >>=20 >=20 > I commented out those restrictions. Makes me worry that some other > packages will come along and fail because they need 384G of mem or > something silly like that. I'd be more worried about multiple jobs that each use more than 8.5 GiBytes that happen to lead to the total being more than (RAM+SWAP) [544 GiBytes]. Side note: I'll note that at one point in a past 'bulk -c -a' run test, a process got a run-way error that rapidly wrote to the log file without bound. Not good if the log file is in tmpfs. But such has happened only with one vintage of the ports tree. It is the only context I've seen that would reach or pass 384 GiBytes for one builder. > I have been advised ( in the last hour ) > that chromium ports generate +40000 source files and such. That is > just abusive but the way of the future I am sure. rust with USE_TMPFS=3Dall and not in TMPFS_BLACKLIST can use 27+ GiBytes of RAM+SWAP. And it is not the largest example around. >> How many parallel builders are active in the bulk run >> as the bulk build approaches the failure? >=20 > I think 64 max. >=20 >> How much RAM+SWAP in use by other builders or other things >> on the system as the system progresses to that failure (not >> after the failure)? >=20 > .... >=20 > *sigh* >=20 > The problem was me. >=20 >> ZFS (and its ARC)? UFS? If ZFS: Any tuning? >>=20 >=20 > No tuning. It just works(tm) and that is ZFS. Most of my use is UFS based. Only the machine with the most hw threads and RAM is ZFS based these days. (Part of that is to be a UFS test case since its use for such activities is less common: At one time, I noticed and reported that 'bulk -c -a' was at the time broken under UFS because of reaching a UFS limitation that was later adjusted.) >> Basically: all the significant sources of competing for >> RAM+SWAP? >> .... =3D=3D=3D >> Mark Millard >> marklmi at yahoo.com >=20 >=20 > It feels like the correct approach is just give everything to the > poudriere bulk situation and then watch for flames. >=20 > No flames? No smoke? Great .. it is working. I did experimentation with 'bulk -c -a' to arrive at configuration choices that would survive such and left sizable margins for handling growth without regular adjustments. So my flame tests were more up front. =3D=3D=3D Mark Millard marklmi at yahoo.com