From nobody Sun Mar 27 15:52:08 2022 X-Original-To: toolchain@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9D07D1A57050 for ; Sun, 27 Mar 2022 15:52:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-20.consmr.mail.gq1.yahoo.com (sonic317-20.consmr.mail.gq1.yahoo.com [98.137.66.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KRL1b4Hc3z3FM7 for ; Sun, 27 Mar 2022 15:52:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1648396332; bh=pfZYbPgwO0gGTpPQ1z5OTjHxFh9UqFnyh2LkBkOgNT8=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=XFCVVzfCf3NkBAkVCWKVxbDelV2FFd918kIr/zum6ySLywmsNeVphVVjsqi0cqGoYmOyeuOJrLYwAspzPoVxU6QYG75JIKp2tL4EFaJcKgLzUK6la//slqQf7WrAgISocof+u2t+ejCGxEzrSN6g/EduGfq80BtxUh75ITiJ7fQsvlnQysewb5GDkPGTDqndRguipMZ3hPCtQGe7xWOCW87dks54A0hqJJPuFtxMJG+uDPEwCN6oI7vWZTVilaYC8n1Hd2UhTRc5Lql5XXLQ5L2tBwaj81MNe/Qc9YKf/hyqN/M73mftwbTJbBwSW6f7HWEGSrTQtiHhrqV0+PjwXw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1648396332; bh=jZ4TKvl4H4mvgj9W5CvRIxCSl+S3ik+L08IhMMtlYfs=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=SkKpv6/5XYHHPt/XNWERYGkJQwZaUAfTzLpYVkFiJlkX+Tu14Cp1N19TDGExg0eC6+++EG4pxIGPOrx6lOAeONN8FIGTeZHAWdNC8p/nhmzHyFyFotpKmsB9dEhMK1hP1OwC7E6t8wtJlw5PCTPrH3BuP5heUBNQ/f3qyoEBOLH/r2imoFX10uCIF1erj7vqoHowLmAsszarRwD+/9sPS359fCAu1h7vc28nJ0rPT3WamTyApIXaji0aGWNG5k6bYLCBC14lVUnTDj8qXP3+wWd2IkmWjzOE0VAw63Qn2/RzP2QJR7rNh7sgCGHOlbeYLmtuS0rwnzSltLt4NR0DCQ== X-YMail-OSG: 2Uwpkk8VM1mUDL9qdgdCIlwbLBrWYradoc3A2rLpKzHcIzfX8V0blLRz1wGOFHD Q2ijn3WKVmnsLQiRBAVnwqfToDQmFNSKGuGvE4ogg44ITuBrZhuYe.ZqSmmyMfAxte_M4YGGMX8o NQNhPtiQBAo8uHCALpPOiKZQrswBGAGNsNWsT.6tFmDKkvYaA_2xuZxNvaYCx7.sgS5uCdQaoE6v FAaJSRMQD8ueZ4q9JjKdUI6CbJhleGQ8VKiMBEM.yKurTWnKCII9OjuuITjghbYtvuBCqCkgo1oy zl1KwHxxdW5ox.z1FBShtI6eFE_D1sdJTy.omaOr_cpK7LZBjTEdrilPPMahvL5TUyMd7jfEsp.H bPpZ6oMPw.M46bemJ.nmrz_umQirKmy0wz1dvdlAqJALGHJRmm9R7Y.t4zb1VZmDKL5FYwyIJPCK B8XAEQMiGyDAhdPMGsoHg944McqmdyM0VDvfKDQf8ngR4qjL0vz5Kv9HVGLI2P6N68iw9yAPudUp Zje7CLxvcVdzTk9hjkn1fuSHLpSTAdahs657_hxRS_pvwCP3ox.6B1Rp4P4ncR__ndj0trbUwYa5 j3W1oea7CqiGQj1kgM6jZi2WS1eLkI4gS4A854nf7VRHKTOjiwxfBE2GymBN2afexdVIoqrmqq53 9FbIy3e_asR0127WKBdTsdLHm.pjjzcQK9SsAM5yDDf9_SZ0i2yMu8SbmixGoQYdO2jVuZ6Z3g9Z tw.F6JSavFYkzds11foMIBnbOuHCAe_ysPy2F7vGafnvUi.E5UUdoMi6qv6S0LcY_6RCDhZYJnXs xxTVt_nL9Zq2XWKxZFyLiUm1WpTBHx1d_AGyIc1iDENR46gmpTjp1tIdiMST_h6wSdxQ99Vgzb.C 5YYPbsNCUl3ziDg5nViwsEMSjn4PUh2BSb5HA9wn.T4NjZsW_nb75lfpoYF.hvY0fH8veNn4zwn3 31Pm46_e7IimOIqc9T.su8Vf6om96eLl6ndhomVb1zML0KKoqumAADr7bgOKaY8kjZTwAyoz.qdX 9gqRQAspMLMHnhAgVpkihlU_HV7esyWT95CuWJ82ZCWGE1_XuLnLIcl6V7lw1HadQeoAFQdXljvD 7MtKW_6cpYpbQNh_9vuNY2XP6hAGWYV.zxPR4I46YNqY6wEIbOAfh9jTc0C.PnSD_9h3uoMkanE2 lsqfsFocZ97gFHjOpq92HJwAdvz_eylmGyky1ia6a9OacTUA.pbDpSZnwT8WA2x4i3hENP49H8dl bN0dVXKltHUlSmhd2Mr0OZ0Un1m9m5o3StsS6znljmyQnm7e6wmsT6vzVkgGreo0tHeCpKuQxKje bfJzp0KEoZYVGCZRzd8SUfK4K52iR.ono5yVOIo3XYE.16Rfi2zI.suXrvnWAuEidiGH4jYjrkPc _GgMA64wUuphP7XUA9aSaBvchsakoZkp3WI_7oXoeu8G3Xt_Ui9OiwJ1cFcjzwHgooYsh8UdG6Tc sczFMA0dYgG3xqKMVKpuHYhb6Kn8YPmPo34jUdBVMr1pFnpM9Z0izri2dKrHPm1vepe1UVcbdQOh eprb5DLoEpBgM0EmW3NnAtXpXukijkkEaaJZkbokom6Ht9FHiydo5hgtcmpr6yTaVAbNLYKMnYC4 wmT6E7kdKVz_vp2.GROnPDlJu2F55N1Sp1ku.kJZxKfhucKmS02F43gE7hc0XdMW4VZnryb0C3qI CZHuPAhMR_zklbKjQiUe82Vi4KPPxl3jwvNT_RCK4zFd_Wmt3zN8TdDzcA51IP_.gOkGK2ZcCECW adsajlWRS6f5MxUDXu1AtyC8K7J_I5NxNjBZr50uzh4rwMMxTcfm0kUVQgiZB7uClWkbHbJ7Q4C8 ozyhR2eyY8qslqL.Kw8mpu6sGiRaq3xsJusTVdOOAKd5Bi0X1TqYn7ooZx_F80A2tMKA5.HGEt51 k8_tZvY8adkrxGWX34WhTOXiE03vrLmWoH5jwUyc3tEQhenYbxYMSGQhcJz_uHDirNZhWbiUNKQ6 B_2AdPQZklZcLripEEUfOg4cn1OovF_O98I1jQK0vmNDz8AHt82VFfBQdCJcmvUps4qO7tGs4ZYJ sVW7enaeCg3ITiOWl90WRUPfhQlNTPkwuskF2zJ3Xq6VLYuojoDxktK1H8gKoy1Veb6OHeWaFFr0 P5O5V9E35G2cG29hp0C5tuKc32HrTrrhfyJ.k_mA3CejHoKwcxnqT2I77mNOPCsze9zek3wCfEvG TFM.I5thfKfyDMTc347hbOAmHBem2xRXF7BfKUu2tpbyfC2d8eL2QNAVdnsIdhCBUOdY8EIjfaom 9s85Bh8XOowW.m.EIzJ3y1OYzEs1bHs4sK_8gT8SykZLMWI_0eUBzkYWfVGzsckmW6JorqnQPZFb NBVJGcYUY5eKY_VGft_K6Tz5QmmUbNIquAHeHm.o- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Sun, 27 Mar 2022 15:52:12 +0000 Received: by kubenode551.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 0e18a8c186855271fb2662464b27f4a1; Sun, 27 Mar 2022 15:52:09 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Maintenance of FreeBSD s integrated toolchain List-Archive: https://lists.freebsd.org/archives/freebsd-toolchain List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: [package - 130arm64-default][lang/gcc12-devel] Failed for gcc12-devel-12.0.1.s20220306_2 in build/runaway From: Mark Millard In-Reply-To: <17CAD266-C7C0-4CD7-B255-3DC07F422EB5@yahoo.com> Date: Sun, 27 Mar 2022 08:52:08 -0700 Cc: "toolchain@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <2D081409-B3E7-422D-98C4-AC7394915F72@yahoo.com> References: <202203261416.22QEGtRR065106@ampere3.nyi.freebsd.org> <21D1C2BF-151E-4252-936C-B5B22C9C8071@yahoo.com> <75A61EB5-70D1-4E1F-89D2-524407854D6F@yahoo.com> <17CAD266-C7C0-4CD7-B255-3DC07F422EB5@yahoo.com> To: Dimitry Andric X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4KRL1b4Hc3z3FM7 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=XFCVVzfC; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.146 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.146:from]; MLMMJ_DEST(0.00)[toolchain]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.146:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Mar-26, at 16:15, Mark Millard wrote: > On 2022-Mar-26, at 14:37, Mark Millard wrote: >=20 >> On 2022-Mar-26, at 13:16, Mark Millard wrote: >>=20 >>> On 2022-Mar-26, at 12:35, Mark Millard wrote: >>>=20 >>>> On 2022-Mar-26, at 07:26, Dimitry Andric wrote: >>>>=20 >>>>> On 26 Mar 2022, at 15:16, pkg-fallout@freebsd.org = wrote: >>>>>>=20 >>>>>> You are receiving this mail as a port that you maintain >>>>>> is failing to build on the FreeBSD package build server. >>>>>> Please investigate the failure and submit a PR to fix >>>>>> build. >>>>>>=20 >>>>>> Maintainer: toolchain@FreeBSD.org >>>>>> Log URL: = http://ampere3.nyi.freebsd.org/data/130arm64-default/60ab72786154/logs/gcc= 12-devel-12.0.1.s20220306_2.log >>>>>> Build URL: = http://ampere3.nyi.freebsd.org/build.html?mastername=3D130arm64-default&bu= ild=3D60ab72786154 >>>>>=20 >>>>> So there isn't any actual error message in this log, except at the = end: >>>>>=20 >>>>> ... >>>>> =3D>> Cleaning up wrkdir >>>>> =3D=3D=3D> Cleaning for gcc12-devel-12.0.1.s20220306_2 >>>>> Killed >>>>> build of lang/gcc12-devel | gcc12-devel-12.0.1.s20220306_2 ended = at Sat Mar 26 14:16:58 UTC 2022 >>>>> build time: 12:31:35 >>>>> !!! build failure encountered !!! >>>>>=20 >>>>> It looks like the last command being run before "Killed" is the = cc1plus >>>>> executable being linked with LTO, so I am assuming the build is = killed >>>>> due to an out-of-memory condition? >>>>>=20 >>>>> But this is only visible to people that have access to the machine = the >>>>> poudriere instance is running on. Can somebody with access please = check? >>>>>=20 >>>>=20 >>>> I do not have access but I've started a poudriere build >>>> of my own on a HoneyComb. I've a patched top that monitors >>>> and reports various Maximum Observed (MaxObs????) figures, >>>> 64 GiBytes of RAM and slightly over 246 GiBytes of swap. >>>> So hopefully it will report on about how big the memory use >>>> gets. But it is allowed to use all 16 cores and there will >>>> be no competing bulk builds using resources. So not a match >>>> to the build server context. >>>>=20 >>>> Note: It is a ZFS context, so MaxObsWired is normally large >>>> and shrinks over times where memory needs to be used for >>>> other things. So the primary memory figures would be: >>>>=20 >>>> MaxObsSwapUsed (if any) >>>> MaxObsActive >>>> MaxObs(Act+Lndry+SwapUsed) >>>>=20 >>>>=20 >>>> Side Note: >>>>=20 >>>> = http://ampere3.nyi.freebsd.org/build.html?mastername=3D130arm64-default&bu= ild=3D60ab72786154 >>>>=20 >>>> reports a Time of 11:48:41 but the log reports "build time: = 12:31:35". >>>> My guess is that processing the log file for extracting the type of >>>> error makes some (much?) of the difference. (Type being = runaway_process >>>> in this case.) >>>>=20 >>>>=20 >>>=20 >>> I did just observe a cc1plus take somewhat over 30min >>> of CPU time before completing and the lto1 related activity >>> starting. It was under 5 GiBytes MaxObs(Act+Lndry+SwapUsed) >>> [No swap use observed] before the lto1 related activity >>> started. >>>=20 >>> For the lto1 related activity MaxObs(Act+Lndry+SwapUsed) >>> has, so far, gotten up to around 12 GiBytes, still >>> no swap use observed: >>>=20 >>> 12079Mi MaxObsActive >>> 12278Mi MaxObs(Act+Lndry+SwapUsed) >>>=20 >>> I'll note that: >>>=20 >>> last pid: . . .; load averages: . . . MaxObs: 28.02, 16.88, = 15.82 >>> . . . threads: . . . running, . . . sleeping, 77 MaxObsRunning >>>=20 >>> So, on the timescale of the first load average, it does >>> not always stay limited to the hardware threads available. >>>=20 >>> No process with sustained CPU activity sticks around across >>> the lto1 activity. So I'll not be able to observe much about >>> cpu time. >>>=20 >>> The elasped time doing lto1 activity has been going for a >>> while but I'm unlikely to be able to observe its end happen. >>> So I'll likely not have a good clue about that. >>>=20 >>=20 >> Looks it spent about 1.5 or so hours on the particular block of >> lto1 related activity. For reference, somewhat after that: >>=20 >> last pid: . . .; load averages: . . . MaxObs: 28.02, 17.04, = 16.87 >>=20 >> The 16 core are Cortex-A72's. >>=20 >> The following did not change (so far): >>=20 >> 12079Mi MaxObsActive >> 12278Mi MaxObs(Act+Lndry+SwapUsed) >>=20 >> Still no observed swap use reported.=20 >>=20 >>=20 >> The build is continuing. The build phase has been a little >> over 2.5 hr so far. >>=20 >=20 > I'll note that: >=20 > = http://ampere1.nyi.freebsd.org/build.html?mastername=3D130arm64-default&bu= ild=3Dd09582c49c48 >=20 > reports for lang/gcc12-devel a build time of 21:53:01 . >=20 > (So lang/gcc12-devel has built after LTO was enabled.) >=20 >=20 > Back to my lang/gcc12-devel build . . . >=20 > The earlier large block of lto1 activity was from 7 .usr/.local/bin/ld = 's > going in parallel. (I had a ps -auxd output in a file to look at.) = Another > block of 7 is going and taking significant time. (Seen via top.) >=20 > [Waiting for it to get past that block of lto1 related activity. . . = .] >=20 > Hmm. I was not there to observe the end of the lto1 activity and what > immediately followed. It is now in package: >=20 > [00:26:46] [01] [00:00:00] Building lang/gcc12-devel | = gcc12-devel-12.0.1.s20220306_1 > . . . > load: 1.36 cmd: sh 10124 [nanslp] 15151.01r 21.72u 70.93s 0% 4096k > mi_switch+0x104 sleepq_catch_signals+0x41c sleepq_timedwait_sig+0x14 = _sleep+0x1c0 kern_clock_nanosleep+0x1ac sys_nanosleep+0x3c = do_el0_sync+0x5cc handle_el0_sync+0x40=20 > [13_0R-CA72-default] [2022-03-26_11h31m18s] [parallel_build:] Queued: = 19 Built: 18 Failed: 0 Skipped: 0 Ignored: 0 Fetched: 0 Tobuild: 1 = Time: 04:12:31 > ID TOTAL ORIGIN PKGNAME PHASE = PHASE TMPFS CPU% MEM% > [01] 03:45:51 lang/gcc12-devel | gcc12-devel-12.0.1.s20220306_1 = package 00:08:12 32.00 KiB 100% 0.2% > [04:12:37] Logs: = /usr/local/poudriere/data/logs/bulk/13_0R-CA72-default/2022-03-26_11h31m18= s >=20 > So somewhat over 3 hrs to reach the package stage. >=20 > The following have not changed since the prior reported figures: >=20 > last pid: . . .; load averages: . . . MaxObs: 28.02, 17.04, 16.87 > . . . threads: . . . running, . . . sleeping, 77 MaxObsRunning > and: > 12079Mi MaxObsActive > 12278Mi MaxObs(Act+Lndry+SwapUsed) > (No report of observed swap usage.) >=20 > [Waiting for the "Finished" message from poudriere. . . .] >=20 > [04:33:13] [01] [04:06:27] Finished lang/gcc12-devel | = gcc12-devel-12.0.1.s20220306_1: Success >=20 Just for reference: I tested the build time for reverting the code in my context: # git -C /usr/ports/ diff lang/gcc12-devel/Makefile diff --git a/lang/gcc12-devel/Makefile b/lang/gcc12-devel/Makefile index 644abf2cbb86..fab28c952f80 100644 --- a/lang/gcc12-devel/Makefile +++ b/lang/gcc12-devel/Makefile @@ -83,7 +83,7 @@ CONFIGURE_OUTSOURCE=3D yes .if empty(PORT_OPTIONS:MBOOTSTRAP) CONFIGURE_ARGS+=3D--disable-bootstrap .else -CONFIGURE_ARGS+=3D--with-build-config=3Dbootstrap-lto-noplugin +CONFIGURE_ARGS+=3D--with-build-config=3Dbootstrap-debug ALL_TARGET=3D bootstrap-lean .endif INSTALL_TARGET=3D install-strip The result was 01:38:45 for using bootstrap-debug: [01:39:00] [01] [01:38:45] Finished lang/gcc12-devel | = gcc12-devel-12.0.1.s20220306_1: Success instead of the earlier 04:06:27 for using bootstrap-lto-noplugin : [04:33:13] [01] [04:06:27] Finished lang/gcc12-devel | = gcc12-devel-12.0.1.s20220306_1: Success =3D=3D=3D Mark Millard marklmi at yahoo.com