From nobody Sat Apr 02 18:27:51 2022 X-Original-To: toolchain@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 608731A4BC03 for ; Sat, 2 Apr 2022 18:28:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-20.consmr.mail.gq1.yahoo.com (sonic306-20.consmr.mail.gq1.yahoo.com [98.137.68.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KW5BZ2mgnz4YYV for ; Sat, 2 Apr 2022 18:28:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1648924078; bh=hH67VmqgB/Foyf/Ae6SOxzrwmcvHKaXsGUA5E/oDOMU=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=JDxBTUA8EOlEAULv4fsrA9GT15z3Jj9/Y0KpbaO30cQtjOgKneyHAqeu7dAkdEcZEOrjqKezpBveNh37GYy24uvqF6aCg9Ig38YgTX1Y8/lGCEjHxd9LEsxlDpEAwQ9x7huf/zgwtLBZiKabTKVqodAH70WiyZY7Qqw/mwrx8nkhvCFDP+psIa2cjReo4pRiAFb+oT2fhpSsbEjnDJKUHawbuGGqfsfyvHAORuLcPkxxElr7233fbFCJQv/hvOTT0jYPE8VZ+x98BTTge0+FLcMPmTqIso7wauS4olYgFAdCVn59FCTp4oMyINFqPGRIMh6OGBFD2EYZdMUgrXcPog== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1648924078; bh=JQfH28v4TDTr0voSsia7yCdX4oDKp6nVluG17U6j9wY=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=PFpyaiTWnrrKXFa4YiZL4lJrlkrHD36HMD5yByDzCKUiMUNFk5H99j5d64inBhZGcl8NGMEmHNQzjRSc8/oLZMyRor0ai5ODfvwHDGEXMybuKHpYhcdFvdubEuzO3VDO0WMqucoMleqN5aQM1+RmJ2q2PaxLW4lvB48ZzSUI72D1Hm/HAJAxRSOdbfR1eIwk2P4VLTPhP1aT9tX8kIP5Vu+QbP6zFlM1rxyaSJPj6AzE2lT9CoGohr0z/NaOyHZn2VlxWkLny/3MWas2RN5VOFrbnHprj4hTRG6bb7SeB16IsDd01DnQKu0w8/LfWoB3RkkRXtFbOVkb9XcKWFPtLg== X-YMail-OSG: npcUFXIVM1mwRafDBJmtRPhi5dk5MxY74tbsnfcVscpECizKBXhF9f4lX3JB2Ij ER5Z4NjwI0DKoDCisX2UX_nHbot74Uu8G6orgz4Xp_HxUY3PWLHwrRrnu1TfI8dGxGbDgRQXnuXE 13BSm6ryJfeDcTGQpzunT98sPt2aNmcqBkLWJiiNZGhVhC8uVgMa.hMgemXppOe0DGvCZtDb_PK2 nKDVXXIKvNbgpiNVmcjzARH58pjsPsO_WZAghz.SlDKWv2gkJAjWrf_eldamiidwg1Sav4MNnWFt BqHkpwNA5TrCsx9Q5cxtgp52NsgeduIW6etoKEbVqLyWfMaKRw1W59b0AQUMrq4KVWXe57rKF3Y8 0ruCz3Jpe2Z3CZikaCvyCR7QdaM3IezbbFwzNvyvRdZ6oOiuGaVewp07WrEzMpCVQNHRJmp8uRl1 Rp6CTNT_Z0g5ODN0iE4ZkT9LkY_F0Lye4ddwtSdff8zbeFAK7ZcemOozqstM62qTUYCrNQ_sNX3R 1s1VPKAtb8og3kLGOUU0Y1hykRqyef72teClSJ3ffhkxiLL6_0mxzPeMiOUjh1aDbTtj0EqU5S94 3JSamsPJawcc2M4xCSfj24x_JbNllsnLarKIsOr9AifyFpmIbNSns25QUWn9Pyj7VfcYX1TZOjeF y32BJOKzlFPSBb8.TfG3uBhaLusuE0A9Nc8KbZud7yLH0PZTyyCZBWhRSy3sbYGk9rFa3QWdMCTp IIb.6LDO3earHqMcVgzAqRhwd7Eo8yhQubWahTEE8XfMy7Mt0ZZ0WT4N73NNM8GpATXKLWZLnvyI k2SaucjGMXR.5RrezAJFjfrnsrV1bu6i.JoKlxCfrZfkgVZdQNlAYb5mG1Jn9KOZXppUDHR4tO12 SQv5YFQf1.0pjocjtPOY2BuoHx9ACjKvS7KhNYiYGP9H4f6YrJ_2HTRMhm0Hlix2OUq3ArZngXgg GhzNaaPqBBtO4r_mmkQZgW31Pd9crekA3LGxe6Khvf0kBtGXtg0rum9yJVRWjb7ML5OWTNcMTO3z uqDms646w42GKzr5YRWkeKDeL4AitegTfytQktSObTQqFiFhFob5qOCwdyB_B6PNsZlTWuKMPS.I BHHbouDYyvzf6r.6Go1izVARIxZjF6mR0qlf0vlP73Zs5Tw6voaAvGlVaS4d6okO8T0fjeGfVGBy 0ar38z9Aiy3unlotTeX.7ssUjmIjpl_B.ZRD4NQEV_sYt9K0oxK1NviTVtu.zG6Uph.r1whUdkOK CYukKE2RkMUMy27Aqx2uZUr2vWXbHnPIgqIdN6rN5Cp4sMVlUAwGRQl_qowmfUlwJEuQ3x1b.Ntc U35pUCRbYGRTTdPPyzCWbB_UhuHAdM0T5kyTGkZw.k7PpQsOH.ULgeTN3nSb6B17mdToxnLJIEgc emCWvTFQJ5BuqKMpW_rQ.YIeLBcUYliCu8q5IZLJXkOXZTgCHBxFxrWkMNslFZa4lIHReFF5rc9D bOUT7sKjc2EoEGvEEogxdKu2QhYGyOvi417XTdDu0JXYcRp9wMOyWkLJt9Zh3i6hksf0B9xDyYuV wc92KekbmdyWDPJM5W9XBYigNUZjuEmf25AqKjQ1m035jgR956mY1IzQOg4.2JzF6pUWEJksLymR lGlsLFeuOsUmlb.IpNNA657uiv2GZjPUJVKjspEcrIdu69ZfgFQYWknd4lzvnhspOeXQ2uZvBEBJ w9Kr9ABPyZGlISmgBwyGrE1mHsZ4VG7v5F426KUuv.EIUKM1Jqkmbl8anlglKivHGGUQLstCEZrN M53pUBW.lQbUukdWGBCsnOJB.p7bl0D7PZAZVKDuwoKDvyEARqjLBhSEqG6UnJSfaSYFXiYxwCCg pmdoJzr8WolEo6oOfOBD740Z9tFrfNmrIc2_xidn7wZxv7Fq8UR1lUkZIyZmt1jeH7GwgrCilhMJ wBRF1biGufKrzDKdF5qi0gMOWCx3hK9jBlU.8uME4z3b2Q7_tr0DaT.1tlutEXbw3ll4YOfoIpAv 5zEPnEaMPAN6p9xootPx.bpdUSJ4AfUMeEXp4iSv7D54PzcLcegZnr.jfr_JxU6IC3ukA6WnjM.G y1WKXHEEigx4abd_DebQ.1czQZxoxodxx2ASFW6Uk2Ekm.lorGJFCOJnW1IVL4YHw5uMYacn1peY 20JdnYcL5yBZ0sC9bI.lDxP7OfoFpoqiiGIEOJ_O8_C5zXWkF1atHH8WdwuotIhsVdvakwp8wdwx sjfEfmb9shAujAjOtAADJmOrTRQ705qNj9cfZjanIEAOjIn8sldO.HGL4IuvSCL2Bb6_XqxsrQoa DZJmuegt8rY_9QuKh51etVefz9pHFb.wt5F9mthA4QQEPaiwkjdSF5gWmxhwLyrgMGRCsgTU2rDV RmT71eq9AAPrifwpjBglkJw-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Sat, 2 Apr 2022 18:27:58 +0000 Received: by hermes--canary-production-bf1-665cdb9985-l8dtt (VZM Hermes SMTP Server) with ESMTPA ID 95693b3d82f7a35af82c60f1be5d7fcf; Sat, 02 Apr 2022 18:27:54 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Maintenance of FreeBSD s integrated toolchain List-Archive: https://lists.freebsd.org/archives/freebsd-toolchain List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: [package - 130arm64-default][lang/gcc12-devel] Failed for gcc12-devel-12.0.1.s20220306_2 in build/runaway From: Mark Millard In-Reply-To: <17DDA9F6-CFD0-479B-B3B5-51B570893863@yahoo.com> Date: Sat, 2 Apr 2022 11:27:51 -0700 Cc: "toolchain@freebsd.org" , Piotr Kubaj , Glen Barber Content-Transfer-Encoding: quoted-printable Message-Id: <5EEA824B-1EC5-48AB-B40D-A3D18E73B739@yahoo.com> References: <202203261416.22QEGtRR065106@ampere3.nyi.freebsd.org> <21D1C2BF-151E-4252-936C-B5B22C9C8071@yahoo.com> <75A61EB5-70D1-4E1F-89D2-524407854D6F@yahoo.com> <17CAD266-C7C0-4CD7-B255-3DC07F422EB5@yahoo.com> <2D081409-B3E7-422D-98C4-AC7394915F72@yahoo.com> <17DDA9F6-CFD0-479B-B3B5-51B570893863@yahoo.com> To: Dimitry Andric X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4KW5BZ2mgnz4YYV X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=JDxBTUA8; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.22 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.73)[-0.727]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.83:from]; MLMMJ_DEST(0.00)[toolchain]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.83:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Mar-27, at 09:02, Mark Millard wrote: . . . > On 26 Mar 2022, at 15:16, pkg-fallout@freebsd.org = wrote: >>=20 >> . . . >> Log URL: = http://ampere3.nyi.freebsd.org/data/130arm64-default/60ab72786154/logs/gcc= 12-devel-12.0.1.s20220306_2.log >> . . . Turns out that log (and other examples of lang/gcc12-devel runaway kills) does have a hint about what timeouts to change: QUOTE =3D>> Killing runaway build after 7200 seconds with no output END QUOTE Quarterly's build for 12.3 also got a kill, with the same message. See: = https://lists.freebsd.org/archives/freebsd-toolchain/2022-April/000478.htm= l I'll note that the after the message, the kill can be hours later in the build's activity, depending on the size of the log file: the log file is evaluated before the kill is done and the involved scans (plural!) of huge log files can be on that kind of time scale. The message is from: # grep -r "seconds with no output" /usr/local/share/poudriere/ | more /usr/local/share/poudriere/common.sh: = msg "Killing runaway build after ${NOHANG_TIME} seconds with no output" So, at least NOHANG_TIME needs to increase as long as = bootstrap-lto-noplugin is in use. (There may be more.) Note that NOHANG_TIME is not specific to the individual port. As I remember, the kills tend to happen between 11 and 12 hours into the aarch64 build but the successful builds take 20..24 hours. It is not = great evidence, but it might suggest more than doubling NOHANG_TIME (for = aarch64 jails?). Looking at it differently, since it does sometimes build a = smaller increase might avoid most of the kills that are now happening. For poudriere, there are: NOHANG_TIME MAX_EXECUTION_TIME MAX_EXECUTION_TIME_EXTRACT MAX_EXECUTION_TIME_INSTALL MAX_EXECUTION_TIME_PACKAGE MAX_EXECUTION_TIME_DEINSTALL QEMU_MAX_EXECUTION_TIME QEMU_NOHANG_TIME These are not independent, however. Setting a larger MAX_EXECUTION_TIME* value can be ineffective with a small NOHANG_TIME, for example, if the activity that takes the extra time happens to not output periodically. (I've run into that before when I tried a bulk -a for WITH_DEBUG=3D in use.) One of the issues with poudriere's timeouts is that they do not = auto-scale to match machine performance. Figures for slower environments may be time/power wasters on faster hardware when a build process really does runaway. Another point is that there is no scaling based on expected/historical = time frames. So runaways of port builds that should not take much time = instead run for a long time before being killed --in order to allow ports that are expected to take a long time to build instead of being killed. Another issue is that, for multiple builders doing a build over the same = time frame, the other activity can lead to longer times of "seconds with no = output". Part of the issue for lang/gcc* is that part of the = bootstrap-lto-noplugin processing does not respect the limits on parallel activity. Having = multiple bootstrap-lto-noplugin going because of multiple lang/gcc* building at = the same time apparently can lead to very high load averages for a time. In fact, = even just one lang/gcc* doing bootstrap-lto-noplugin can have a load average = for a time that is something like 1.5 * (# hardware threads) when the build = indicated to use the # hardware threads as the limitation. (Cores in this = context.) When changes are made to how things build, who is supposed to determine = how to adjust poudriere's settings to match on the various build architectures? = Is this something "exp run" sort of experiments are appropriate for determining = for each build architecture --or at least for tier 1 architectures? (Just me pondering, given that what *TIME* settings to use is not obvious.) =3D=3D=3D Mark Millard marklmi at yahoo.com