From nobody Fri Jan 28 06:35:51 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9438B197AD17 for ; Fri, 28 Jan 2022 06:36:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-20.consmr.mail.gq1.yahoo.com (sonic301-20.consmr.mail.gq1.yahoo.com [98.137.64.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4JlSQZ34xbz4Zfs for ; Fri, 28 Jan 2022 06:36:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643351758; bh=T9GlqcB0Kbu7spTWb5t/067cnmcShKQygVfcW6SjCvA=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=rH33nHmNQFGYFOt+QYdyz1kCguiLknzpYqRDUCuvoa6HlUP0mZKwVjxUfcMmEOYBpWWs7AQKeijIKJm0XRDNAlJxX7XTikxxA41/0mfDicgALw/E8cY2SMNHLYHIqFSWcoomXPhfjZd7NQ2z0ctr67w0PAhdjqQQZ9JfHS9P6QpxEKTfv2qELzZ6KbV7t8S6S+6Xtn1+9yE8S6zsi0DyOFIqhtih4Kr+nQo2CFZbRL/z6gbNX6spoWgPbShIN+hkWtLyomx9YjS3nYKWRo+g3pAOJllOkg67GrqY30d5XtfXNKoQ5YyLFdJXj7qLs7NPfAbGvpkn/kTxW1iwQScViQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643351758; bh=qY5m1zPBY/8Y55Fh9aJuJphQ30pm42365whlsKYaUMg=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=G4FbbSHUsdWXobi27zzLIQVuXwlUz8yIc2aSMTlUrwHWFiTa/fiBryAJNonNTCZtfhJIAP2Mg3c+dXqizV0ufE12O1kSr2B+VaVbpoduTx+YmVCGhpgLB909tqvGuki2v6jiQmPTlxHmN2JthHA6IOlP7ZiGfWvx2jfTzYC34CHjx09BfRVqSZmQ4GMQoFIZ3Q7lDdDlJabXgrbMqM70BVCv9EXOgBERk41Qde20iGcbpFA2Qjf1QhYGWGegVgPTQ2JxouaST6/IhAWyIUlkGmZKn5QT8oqRe2l4IUZlhiMpNWG8VEvELokb9ycZEqDP/2EcskY+gqbsvmsrns5TgQ== X-YMail-OSG: tOeUBD8VM1kjraXX1YpQQjRAGHbq0Q9kUcjLC7k.u0MpcDMGY2SZcjXmvfInhUz Ut2HT_8BT1idjxcXXThhlJrmSDuWdiMfLwiLHLTrGEhCUvtFJKXzqi3a6x7iYmSDMn4kFUskQ8Lm j7ZlFevDiBDBKMb3pndZEkc0iEj6pKCWdsGT_w6xiHhpk73wA.Hz3l9LNozlimJzcSEYKgToddhK gZZssGaxENWVLTI14628yykFcSnggh9W6ZhIM5Sz5vNWY.ENjN7eXJPTSTCbQp_xtmRjp.2NhVYg 2veISsrwtiNrBan2ccx90W55FYqKyMzwXN_HLpNpWdgi1fyrwbSgixhk3iicdXdqAgvljyyOOp.. .pw08amGBwRBBrpJaVKnExS22V_kru3FI1aO.t94JDR5bQMuzFRWwvehJvCmQ4vcp2gn.LlCWAE4 OGC7csjAi4EKVwNiqIfHtjc2WCOczPQni16ryX22o.d.1qN9sj4UkV5WieIo.mxpw7TvUrAhPNVm BLBzAkIzsXAdHKine2Ou.9VKDyKpMANAWwuUvWAthAQFUyEaEOZ8HNHVVTnRllA4xqdoBmsgAZAA M06o4Zm6gHT_ds47DEUvbW2naK7yS3y0tOE2SpP4HFcxdjs.Art4GkKNwWwOGOYVNKccaCns7kaF toshpDve6CYyaYkdWSYWIbrLv85g5cb0MBzM8o3BTZHlwEnurmwfChHlLBw_OAbqWQI.CHK6lC.x btJHAB2jnDGC_plg3yxjYfcry5vAo6jQSdPqo5B8I5c1mmiy3HU8.S5JEYEoUwHBtIfn0LxmuYc1 TH4gMogigyEB.ZxTLPYQ3XfHMv8chI9os5F2qUTmr4JKGiTfG8YJ6cMmTjEicRNOB3QRAaFKyvBx W7RAbZm5U9YC4j9jv0tw.0ubXbApgSk7cAKxVCZvJorvtgXGF0FxIQY3Y.0mIPiTonk8QsO6oK4t oQ9GVCM4sYvrkHZGrecWSt2OVINL26Fbl1pmT3aIixYvNXj5jnf1.AblIlHkQzVxqi4CFXr2d.S. KUq.zhFHmfzDJceTptTE_NkyR2y0pwHJx7fIor2Mcb_MXXuFv3n8r0MBMoU0XxYs6mprrW2C2d1. O0V_vs3lhwNPcFgGtktzx4v8JSZ53DIMBc0NOMQqkq.HeN9.zMgxYZyyS8ivcAS9p6GZDPtlm2CQ JjuMq8QmVtCjZVcZlt7JyOK9X3klz9TLd2HAsDW0gzYXDF2SStUycRJKlcpYEC81.MUPgDX671Wq _4jemME3acznfsyIXeWVR.7jD07PqsUXJMOSR.rrWQlnLmejND20i_KeLhP17FA8BHpZ7eQdDUxx zkxKjNMPftB.bIKM_aem41ocpIrCP8RLRZ.Ilomf4B72E0gS.5gyo62kZZDHzT1cmWQAaLOvDucD s3S122KMRqvgn_kLuFdkcort9lbcblu4yJ9C7u1xx5dnrLuMFYXxRYttm2o6koITsDIr1RGGzTBs Rn_NvzdQeU9qRi6..XW30TbDOEmhW5uWOEI_rNkpf7PtHTyZhRlReGDPUlAOYrTZCYpeEob.z9.g zwX7_6jQ5oDlafnTGSo8vTMsJEPPlSd38WzqgFTJ.RRcKzhNktuL7PzRxW6dWTmunfsyjhDTY9uJ 1mnAOpP3Lt6lMEvW2gOiwN1vNdwv0aiR_NRrKJy547OMUnZoz7joVCOHVm6tJ_tlFnSJcr7.u8F1 hRTb5qcpY4tRkxYSV6Tgg9Cm0iZMP_ocBnrVQpBTd6x0KgMD9poRZrzykl7IAM7iLfASr5KqWwsv Vcy3GWzlhtMBRUp_NeWz6N7eoS5a9eIDM0UwDyc3iMzU4UQVpIKKTWMC8eLbsF3LvW51sS52i1fU QsEr78bqpWkqrCZGSki3eZTeTk.02QxCSwsVqtH3zMnXRa5Xt0Vy5Pn58yns2Wa6j_eXTjp9qa6_ nYC_FMib.FNhf6ypTN3Rur5Wfrw9oqimhAqkLIB0ydfN6Z.y.nPNc1vVQfq.lMrIA4KGo9qk7spy V3Lxa21FQhP2Pj_hJGGbC5Qmex_qoTCWORQxd4849qVrNwvGesXeaCvJ2U2YUW912Hw6f7r.YyGs mhM0Ux882vG5TDp2k7zulSWnPA_BHnZLwVkyRKvFm01ZiK82hKFRdF68_BUBwaafs43D10.yugsk D6NTuhnVeaTJkosoaBkCdEp6or_jOwi__XxkUmP6MC4t68KIQerS09dJB3Fh06XjQa9SCiUk5WXK chLm0UeIHBfamLx1zGeGvXzNfmmEA4xxHcaCwdlrmzhgi0F2rgJ75layYyKThMW38rZjciUU4sl8 QbvdHZGAS6MRnn_l1 X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Fri, 28 Jan 2022 06:35:58 +0000 Received: by kubenode503.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID e8c3f1cea59068fa4c9a039ee8d39f5b; Fri, 28 Jan 2022 06:35:53 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: devel/llvm13 failed to reclaim memory on 8 GB Pi4 running -current [ZFS context: used the whole swap space] From: Mark Millard In-Reply-To: <54CD0806-3902-4B9C-AA30-5ED003DE4D41@yahoo.com> Date: Thu, 27 Jan 2022 22:35:51 -0800 Cc: Free BSD Content-Transfer-Encoding: quoted-printable Message-Id: References: <20220127164512.GA51200@www.zefox.net> <2C7E741F-4703-4E41-93FE-72E1F16B60E2@yahoo.com> <20220127214801.GA51710@www.zefox.net> <5E861D46-128A-4E09-A3CF-736195163B17@yahoo.com> <20220127233048.GA51951@www.zefox.net> <6528ED25-A3C6-4277-B951-1F58ADA2D803@yahoo.com> <10B4E2F0-6219-4674-875F-A7B01CA6671C@yahoo.com> <54CD0806-3902-4B9C-AA30-5ED003DE4D41@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4JlSQZ34xbz4Zfs X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=rH33nHmN; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.146 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.146:from]; MLMMJ_DEST(0.00)[freebsd-arm]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.146:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N [Back to omitting Mark Johnston.] On 2022-Jan-27, at 22:00, Mark Millard wrote: > On 2022-Jan-27, at 21:55, Mark Millard wrote: >=20 >> On 2022-Jan-27, at 17:43, Mark Millard wrote: >>=20 >>> On 2022-Jan-27, at 15:30, bob prohaska wrote: >>>=20 >>>> On Thu, Jan 27, 2022 at 02:21:44PM -0800, Mark Millard wrote: >>>>>=20 >>>>> Okay. I just started a poudriere bulk devel/llvm13 build >>>>> in a ZFS context: >>>>>=20 >>>>> . . . >>>>> [00:00:37] Pkg: +BE_AMDGPU -BE_FREEBSD +BE_NATIVE -BE_STANDARD = +BE_WASM +CLANG +DOCS +EXTRAS -FLANG +LIT +LLD +LLDB +MLIR -OPENMP = -PYCLANG >>>>> [00:00:37] New: +BE_AMDGPU -BE_FREEBSD -BE_NATIVE +BE_STANDARD = +BE_WASM +CLANG +DOCS +EXTRAS +FLANG +LIT +LLD +LLDB +MLIR +OPENMP = +PYCLANG >>>>> . . . >>>>> [00:01:27] [01] [00:00:00] Building devel/llvm13 | llvm13-13.0.0_3 >>>>>=20 >>>>=20 >>>> Is this ARM hardware, or an emulator? >>>=20 >>> 8 GiByte RPi4B, USB3 NVMe media with a ZFS partition. The content >>> is a slightly modified copy of the HoneyComb's PCIe slot Optane >>> media. >>>=20 >>> The UFS-based 8 GiByte RPi4B is also based on copying from the >>> same Optane media, both for the system materials and various >>> ports/packages/pouriere related materials. (Not, necessarily, >>> other things.) >>>=20 >>>> I've been using plain old make in /usr/ports/devel,=20 >>>> might it be informative to try a poudriere build as well? >>>=20 >>> The Pkg:, New:, and llvm13 lines I listed are poudriere(-devel) >>> output. I am doing my builds via poudriere. ALLOW_PARALLEL_JOBS=3D >>> and USE_TMPFS=3D"data" in use. >>>=20 >>> I have a context in which almost all prerequisites had already >>> been built. (The change in options lead to 2 very small ports >>> to build before devel/llvm13's started in a builder.) >>>=20 >>> (You might not have a jail that already has the prerequisites.) >>>=20 >>>> One would expect the added overhead to increase memory use. >>>>=20 >>>=20 >>> Well, from the context I started in, only devel/llvm13 is being >>> built once it starts. Once it gets to the build phase (after >>> dependencies and such are set up), there is not much overhead >>> because the only activity is the one builder and it is only >>> building llvm13 --via make in the builder. At the end there >>> would be extra activity as poudriere finishes up. During the >>> build phase, I only expect minor overhead from poudriere >>> monitoring the build logs and such. >>>=20 >>> I expect that the mere fact that a poudriere jail is in use >>> for the builder to execute in does not contribute to >>> significantly increasing the system's memory use or changing >>> the system's memory use pattern. >>>=20 >>>=20 >>> There are some other differences my context. The instances of >>> main [so: 14] are non-debug builds (but with symbols). The >>> builds are optimized for the RPi4B (and others) via use of >>> -mcpu=3Dcortex-a72 usage. My /usr/main-src/ does have some >>> personal changes in it. (Some messaging about the kills is >>> part of that.) >>>=20 >>> The RPi4B's are using: >>>=20 >>> over_voltage=3D6=20 >>> arm_freq=3D2000=20 >>> sdram_freq_min=3D3200=20 >>> force_turbo=3D1=20 >>>=20 >>> (There are heat-sinks, fans, and good power supplies.) >>>=20 >>> The media in use are USB3 1 TB Samsung Portable SSD T7 >>> Touch's. I'm unlikely to see "swap_pager: indefinite >>> wait buffer:" notices if the cause was based on the >>> media performance. (You have spinning rust, if I >>> remember right.) >>>=20 >>> I do not have a monitoring script making a huge log file >>> during the build. So less is competing for media access >>> or leading to other overheads. (But, as I remember, >>> you have gotten the problem without having such a script >>> running.) >>=20 >>=20 >> ZFS context: >>=20 >> Well, the ZFS example used up all the swap space, according >> to my patched top. This means that my setting of >> vm.pfault_oom_attempts is not appropriate for this context: >>=20 >> # Delay when persistent low free RAM leads to >> # Out Of Memory killing of processes: >> vm.pageout_oom_seq=3D120 >> # >> # For plunty of swap/paging space (will not >> # run out), avoid pageout delays leading to >> # Out Of Memory killing of processes: >> vm.pfault_oom_attempts=3D-1 >> # >> # For possibly insufficient swap/paging space >> # (might run out), increase the pageout delay >> # that leads to Out Of Memory killing of >> # processes (showing defaults at the time): >> #vm.pfault_oom_attempts=3D 3 >> #vm.pfault_oom_wait=3D 10 >> # (The multiplication is the total but there >> # are other potential tradoffs in the factors >> # multiplied, even for nearly the same total.) >>=20 >> I'll need to retest with something more like the >> commented out vm.pfault_oom_attempts and >> vm.pfault_oom_wait figures in order to see the >> intended handling of the test case. >>=20 >> What are you using for each of: >> vm.pageout_oom_seq ? >> vm.pfault_oom_attempts ? >> vm.pfault_oom_wait ? >>=20 >>=20 >> For reference, for ZFS: >>=20 >> last pid: 380; load averages: 1.50, 3.07, 3.93 MaxObs: = 5.71, 4.92, 4.76 = up 0+07:23:14 21:23:43 >> 68 threads: 1 running, 65 sleeping, 2 waiting, 19 MaxObsRunning >> CPU: 13.3% user, 0.0% nice, 4.9% system, 0.9% interrupt, 80.8% = idle >> Mem: 4912Mi Active, 167936B Inact, 1193Mi Laundry, 1536Mi Wired, = 40960B Buf, 33860Ki Free, 6179Mi MaxObsActive, 6476Mi MaxObsWired, = 7820Mi MaxObs(Act+Wir+Lndry) >> ARC: 777086Ki Total, 132156Ki MFU, 181164Ki MRU, 147456B Anon, 5994Ki = Header, 457626Ki Other >> 59308Ki Compressed, 254381Ki Uncompressed, 4.29:1 Ratio >> Swap: 8192Mi Total, 8192Mi Used, K Free, 100% Inuse, 19572Ki In, = 3436Ki Out, 8192Mi MaxObsUsed, 14458Mi MaxObs(Act+Lndry+SwapUsed), = 15993Mi MaxObs(Act+Wir+Lndry+SwapUsed) >>=20 >> Console: >> (Looks like I misremembered adjusting the "out of swap space" >> wording for the misnomer message.) >>=20 >> swap_pager: out of swap space >> swp_pager_getswapspace(18): failed >> swap_pager: out of swap space >> swp_pager_getswapspace(1): failed >> swp_pager_getswapspace(1): failed >> swap_pager: out of swap space >> swp_pager_getswapspace(1): failed >> swp_pager_getswapspace(7): failed >> swp_pager_getswapspace(24): failed >> swp_pager_getswapspace(3): failed >> swp_pager_getswapspace(18): failed >> swp_pager_getswapspace(17): failed >> swp_pager_getswapspace(1): failed >> swp_pager_getswapspace(12): failed >> swp_pager_getswapspace(23): failed >> swp_pager_getswapspace(30): failed >> swp_pager_getswapspace(3): failed >> swp_pager_getswapspace(2): failed >>=20 >> . . . Then a bunch of time with no messages . . . >>=20 >> swp_pager_getswapspace(5): failed >> swp_pager_getswapspace(28): failed >>=20 >> . . . Then a bunch of time with no messages . . . >>=20 >>=20 >> Top again: >>=20 >> last pid: 382; load averages: 0.73, 1.00, 2.40 MaxObs: = 5.71, 4.92, 4.76 = up 0+07:31:26 21:31:55 >> 70 threads: 1 running, 65 sleeping, 4 waiting, 19 MaxObsRunning >> CPU: 0.1% user, 0.0% nice, 5.6% system, 0.0% interrupt, 94.3% = idle >> Mem: 3499Mi Active, 4096B Inact, 2612Mi Laundry, 1457Mi Wired, 40960B = Buf, 34676Ki Free, 6179Mi MaxObsActive, 6476Mi MaxObsWired, 7820Mi = MaxObs(Act+Wir+Lndry) >> ARC: 777154Ki Total, 135196Ki MFU, 178330Ki MRU, 5995Ki Header, = 457631Ki Other >> 59520Ki Compressed, 254231Ki Uncompressed, 4.27:1 Ratio >> Swap: 8192Mi Total, 8192Mi Used, K Free, 100% Inuse, 409600B In, = 4096B Out, 8192Mi MaxObsUsed, 14458Mi MaxObs(Act+Lndry+SwapUsed), = 15993Mi MaxObs(Act+Wir+Lndry+SwapUsed) >>=20 >>=20 >> I then used top to kill ninja and the 4 large compiles >> that were going on. I'll change: >>=20 >> vm.pfault_oom_attempts >> vm.pfault_oom_wait >>=20 >> and reboot and start over. >>=20 >>=20 >> I expect that the ongoing UFS test will likely end up >> similarly and that similar adjustments and restarts >> will be needed because of actually running out of >> swap space. >>=20 >=20 > I forgot to report: >=20 > [00:01:27] [01] [00:00:00] Building devel/llvm13 | llvm13-13.0.0_3 > [07:49:17] [01] [07:47:50] Finished devel/llvm13 | llvm13-13.0.0_3: = Failed: build >=20 > So the swap space filling happened somewhat before > that much time had passed. ZFS context: I will not start the next bulk until just before bed. I do not want it to fail while I'm not monitoring it. The last 4 reported compile starts reported in the log are: [ 64% 4725/7265] . . . flang/lib/Evaluate/fold.cpp [ 65% 4726/7265] . . . flang/lib/Evaluate/fold-character.cpp [ 65% 4727/7265] . . . flang/lib/Evaluate/check-expression.cpp [ 65% 4728/7265] . . . flang/lib/Evaluate/fold-designator.cpp But it is possible one or more of these completed and some earlier one(s) was(were) still running. So, if you do not need the Fortran compiler, you can probably avoid the problem by setting the options for devel/llvm13 to not build flang. =3D=3D=3D Mark Millard marklmi at yahoo.com