From nobody Sat May 20 18:59:51 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QNtM00stTz4BQDl for ; Sat, 20 May 2023 19:00:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-55.consmr.mail.gq1.yahoo.com (sonic307-55.consmr.mail.gq1.yahoo.com [98.137.64.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QNtLy3b5Xz3sWD for ; Sat, 20 May 2023 19:00:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=TQpnBD8N; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684609208; bh=GFYPLGSKwQRLhD6Ie70ub61dyM+OdnDSESugTWCCAK4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=TQpnBD8NuyCYV4nrk4Le13bQ5w3g9febGrJGVnahDHw94ggv55aHEEJwdYAcXO4PhaZpnfCgrp8mKBvBE6IZDX8clVn+j5jrlJ5ROnJEIQilyutxjuuFmlEhvjAlqjhXZlkXpXqEFy0VazNaRbb3Soc1WHpvTFnXJyow/DGD1qWSEUGpv4GDrPg/T1RZTYCq3SZTGqRXxt3m7lOIdpaGLCRhvkgWGPpzOsvE8g0QDZ8WGG/3j+QZC9UOUDZLkLaLGdphatZkrgIFW63GGDchaulzWKiJXDR9rrRjnOp5Hy9V7RQnaOjmaV0E8kg5NVQ+3XmhGVz8NhHypcys8n3szA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684609208; bh=E/Ug/QJH9CaYhMqnOBrZu/e3RtDxu75mWO6id0icz2S=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=d9+LTBuEBsB8uewXRHY3fCLTr8vWPa9yuVEosN9CUTf9GWEXuTxLJuzkHwsR52jyd9CXTemnv2CRpaJrj3FPgdsik/SlMAyezZJeWvepJEMQukNPVYPxTawxBcxWdim6nMV1YxbWsTjg+VFV0bzqcgQYlRsCuNo6f4IoJNegdGITNjVfzPbj/E7sUCBWzYaQ8W0YT01GL1GQ7E2QEyGSwiO1JyHOmBnt+wCsGluTNhjufblZTlsXlq0JVZ1eZFgA60dJ0bT2in1vow3ohTbkmFj9eBhVQp8keNI3POe3303OptkBX+eKLtVgJjefHw01QXzTaP+9BKbBAtXoLl/eDw== X-YMail-OSG: BfAkGN0VM1llp9x6fUHl2fdSpVIIqncra_odTFLhsnxoI1rihC1NIIwNtUAKGK4 E2.AO7sDS5IngX8NReRzeXr9wBDeCz0ivD8LurM0vMWC5rypz9SSHhZ_sFW0Gnvnj1QcjxHfdEZI 6OfcDDrQvoC_qmUq3V0YJ9Gr62l2oXHIA9MVdWI7skeAN65DrNAgb72ncQRdYiTFthJVrx5hd4x. UfDpOYSLsH3LkK1NCWCfH2d3McbzvBPi7sNnPa3mO4a1hm2PLmcW57yAdt.Bwmtum0_dQvqIuOir XunDbjHIpTyBSz8dpUUIRlICCE4e55p_Eq5I7A85G7XHxTRQznV03mkuWs7uzqOBBd8_kyIbE2hW KQWJDsIIJFWxXX5QR3ARuAgPtF_.XotohlvJj5ZfLcHrDJoggM64oCk2IYE3E.V3XOT6IiRXex20 fJn44bX3QU1S7wUNmB1FF4Li3UzEdAPUqizHtAu82n0CaUsXZlzIru_uuZcoyB85_hkXizp.5LEz 4BWj.GgXZlxhapyG9pn65mQivclbwA1zFNj53A57.1zHJeNfWwq.3UQdbh642V7bRShqYkHELU6f 1G3TUbhWaJMq4rutNFIA7ufjpMJH_nbh0lh3fLtsB22RJPTX5TYSV_v_iiZlqPhOsGJNOBNqDA0I 6hKFqyZul9uUa4HMa5v579awDjQUn21oMhVPZSpZOCwNvS5hTYjRsZ45Q4g7FaRTpWQIsmlm7K9B .oQ1d1Q0aR_LXLuIuhLixITZb0jBymk1E5BWqjEdY8yfjt8i6jMEAVrNmzlZ1BMcVGczTd18dayn wRysXj1rq1_Hzrb3KtiHHxekbP0dWnd3vbUHe7xVyKfXhsWDrtGNlF4JYS0O5y5OtoiBgAwPHNYI 5D5YZ5jdqALq_DNE8u_U9GUAft6t56sV.JieKwGZScIQpwZ8jCiri.BJv24I8HJZ1rqRP53ignGw apyEQfKmkR539xaqMYyo7Jx5GLrtG3un2M4k5lNwx7Ip1BxLdXbo.cYvzc7DWhT3yT2WxHpuBlAc 76Nq5x7ja3x_kcZrbgnjBXjjZ7aUNCi.BFKkbejt4mWhDSjvwNt6gGO9WnH7QBWhqgUpa6xThh6C TRgCN.zLRriRUvAkLW5KcEkPWnTxxs54tCMeWefNab0YH2bffaZIAHHldM1bgJYzk6IiNtA7Zw0l dPf4TFgLO8epnexKF.qY3U_TISnGBv6ZkmA1BjCHf8q_0JUPmRfEZhzc.TZxJiUgbG1DRCQCJfXd iVv6sfrX2WIdl_jgOFEyiKT9CnaFk0ebe0KN21Tcelx2fPobFgDscx8xjE5BKE5D.aAAoIjTGQU_ E95S_UivdvtlTM2dF5Oq_3iNhrqULETWXrvz23vXvQtG8SOygxh9HFGUr1nXhObOdAZWoFfcW5cb tFI.ean2K7H8r75XtoltEP6bhHO0zX44SIrt8GFfALT9LK494BJscDvO0iAkY5yCIskJxis2VEto M5UX6vDPwegXlsvovpc3WlFWY7HKNVF_Ia0mDw72ai9km73M3L6bnlFXEH3_LLcePW8GgePWXXUI iugrpbdrYBB7L2AgFBSLikh6QkQhTDPGo6JkRuhiYg_WDVsvDms4bimKH8_rH17t.UfA1543pZv7 ykcwKxt9FSn5xF5jDJzqfbQLiQ6Z4OemMbt0ZxdM.v2Zhan29muw6D_pxw_OAm67oMXEn9k7MeQT 7RdeBvcWJPHxr_2p1nhjhdvcQQgOWU_fGQaGXTktjFM2RExB0YjyFnzVWKoUHW7tVRQv.uvN1IEa fp_PuNHc3h8WeXP0mEMEy8OydCaxBw7dQA8lQK3qOtxfxFYh0bJlRxSLtS.TRB4hnCTwxdRuttnN e5peiaDdGijrnwK.otvfIuKzCe.8q8KF9H33GS5yiJu0MigfAahZEzISJj1ithTXRb6TZGB92SEd XciJ01ISar7M3l2vCaFIoNjdQNhNmsT_FlBotItmEpEtJAhHd1AmUOpCRtU5wI9CHPIirwyeljyh fO4CvpUfs13AlWwCDa8nraISH3DWrwn6yU.S46C.pUBaoPY4GeqpSgFZluQlDn4fyEzv2DvwPyKK Hx2x6jLSNmFnue3QXcRdYtYvJpPnQWwl.bJpQu6z0cWoQq2j6FJzWa3nCq9ysGbi5ZhkPeZVholT A1m7qABZTOMDQwBVBXTeG2FlmsvcAB1VTiLQHjETbYdkO5WQG7zVinWWT8AQxBlSJ1N12R3zLCh6 TA4lz45A2YtMqSD2MFbCVEwAXsMc_JSq.QDNnlI8gQFEhbYlblc61fxokOS5ziUZAcGMtCmVrHNU oAw-- X-Sonic-MF: X-Sonic-ID: fe8a122f-0cc0-4c92-b717-75a68e6418af Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Sat, 20 May 2023 19:00:08 +0000 Received: by hermes--production-ne1-574d4b7954-bq277 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 40fb482f13db21b8a093bb27b262092f; Sat, 20 May 2023 19:00:03 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Armv7 (rpi2) getting stuck in buildworld for -current From: Mark Millard In-Reply-To: <03F330A1-35E4-40D4-B9C6-407041BBEC58@yahoo.com> Date: Sat, 20 May 2023 11:59:51 -0700 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: References: <7758885B-3115-47F0-A453-1C010313D4B8@yahoo.com> <03F330A1-35E4-40D4-B9C6-407041BBEC58@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.42 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.92)[-0.920]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.31:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org] X-Rspamd-Queue-Id: 4QNtLy3b5Xz3sWD X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N I set up the RPi2B v1.1 and started a -j4 buildworld buildkernel from-scratch rebuild on/of: # uname -apKU # long output line split for readability FreeBSD OPiP2E_RPi2v1p1 14.0-CURRENT FreeBSD 14.0-CURRENT #74 main-n262658-b347c2284603-dirty: Fri Apr 28 23:07:41 PDT 2023 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA7-nodbg-clang/usr/main-src/arm.a= rmv7/sys/GENERIC-NODBG-CA7 arm armv7 1400088 1400088 (The original build was done on another machine.) At somewhat under 18 hrs it finished with the large swap use during the overlapping time frame for these 4 builds: clang/libclang/CodeGen/CodeGenAction.o clang/libclang/CodeGen/CodeGenFunction.o clang/libclang/CodeGen/CodeGenModule.o clang/libclang/CodeGen/CodeGenPGO.o These are my notes from the information for somewhat after the swap use dropped off. My armv7 builds disable targeting other architectures but I also have WITH_CLANG_EXTRAS=3D . (Not that the build has gotten that far yet.) I show the controlling file content later below. I used no assignments for: #vm.pfault_oom_attempts=3D-1 #vm.pfault_oom_attempts=3D 10 #vm.pfault_oom_wait=3D ??? but did/do have: vm.pageout_oom_seq=3D120 vm.swap_enabled=3D0 vm.swap_idle_enabled=3D0 in use for this experiment. FYI: make[1]: "/usr/main-src/Makefile.inc1" line 326: SYSTEM_COMPILER: = Determined that CC=3Dcc matches the source tree. Not bootstrapping a = cross-compiler. make[1]: "/usr/main-src/Makefile.inc1" line 331: SYSTEM_LINKER: = Determined that LD=3Dld matches the source tree. Not bootstrapping a = cross-linker. (Those 2 have significant time implications for the overall build.) Based on (my modified) top, sampling every 3 seconds, Mem: . . ., 754020Ki MaxObsActive, 186756Ki MaxObsWired, 923356Ki MaxObs(Act+Wir+Lndry) Swap: 1740Mi Total, . . ., 756828Ki MaxObsUsed, 1442Mi MaxObs(Act+Lndry+SwapUsed), 1615Mi MaxObs(Act+Wir+Lndry+SwapUsed) So: slightly over 739 MiBytes of swap observed to have been in use at one time. As for the overlapping time's duration: file creation and modification times, in time order were: (via extraction from ls -TldU output:) 09:37:28 creation of clang/libclang/CodeGen/CodeGenAction.o 09:38:44 creation of clang/libclang/CodeGen/CodeGenFunction.o 09:40:19 creation of clang/libclang/CodeGen/CodeGenModule.o 09:41:28 creation of clang/libclang/CodeGen/CodeGenPGO.o (via extraction from ls -Tld output:) 09:47:15 modification of clang/libclang/CodeGen/CodeGenFunction.o 09:49:53 modification of clang/libclang/CodeGen/CodeGenAction.o 09:50:10 modification of clang/libclang/CodeGen/CodeGenPGO.o 09:54:49 modification of clang/libclang/CodeGen/CodeModule.o So: 09:41:28 . . . 09:47:15 (under 6 min) for the overlapping time frame and the highest swap space use happened inside that interval. During this, there were times mixes of CPUn and "swread" STATE for the compiles. But at no point were all observed to be blocked waiting, at no point was only 1 observed to show a CPUn with a large WCPU. This is largely attributable to the USB media having tiny latencies compared to spinning rust and having reasonable transfer rates for the type of I/O: NMVe USB3 media (that is also USB2 compatible for USB powered usage). My use of: # # Delay when persistent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=3D120 and: # # Together this pair avoids swapping out the process kernel stacks. # This avoids processes for interacting with the system from being # hung-up. vm.swap_enabled=3D0 vm.swap_idle_enabled=3D0 did not lead to any problems so far. For reference: Via systat -vmstat I monitored . . . VN PAGER SWAP PAGER in out in out count =20 pages ioflt . . . . . . intrn . . . Both VN in and SWAP in can contribute to ioflt, faults that required I/O. The ioflt number would be before the "ioflt" text. There is a later line that lists intrn (somewhat below ioflt): "in-transit blocking page faults". The intrn number would be before the "intrn" text. The figures varied under 600 for ioflt and intrn for what I saw during the large swap space use, with matching SWAP activity, no significant VN activity. (The figures are for an about 5 second update interval, as I remember.) (I watched the on screen updates. I did not try to capture the material in a file.) I expect that these figures would be large for a sustained period in your context. I also monitored with "gstat -spod". I assume that stat is more familiar. (I use -spod even when "d" happens to not be going to show any activity.) [I do not recommend leaving "systat -swap" running: it accumulates a large set of memory leaks and so can mess up tracking swap space use by being a signficant contributor. I did not put it to significant use other than discovering that problem.] Configuration points . . . /boot/efi/config.txt has: enable_uart=3D1 dtoverlay=3Dmmc # # Local addition that avoids (at least) USB3 SSD boot failures that look = like: # uhub_reattach_port: port ? reset failed, error=3DUSB_ERR_TIMEOUT # uhub_reattach_port: device problem (USB_ERR_TIMEOUT), disabling port = ? initial_turbo=3D60 # # Local additions: uart_2ndstage=3D1 dtdebug=3D1 kernel=3Du-boot.bin.2023.01.armv7 kernel7=3Du-boot.bin.2023.01.armv7 dtoverlay=3Ddisable-bt # force_turbo=3D1 ( /etc/rc.conf has powerd commented out. ) (I build u-boot with a couple of settings added.) (Leaving initial_turbo in place allows disabling force_turbo independently --but still allowing the USB booting to work during the temporary turbo status. intial_turbo is not required when force_turbo is enabled --but does not hurt.) /boot/loader.conf has : # # Delay when persistent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=3D120 # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: #vm.pfault_oom_attempts=3D-1 # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes: #vm.pfault_oom_attempts=3D 10 #vm.pfault_oom_wait=3D ??? # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) (As I understand you are now using defaults for vm.pfault_oom_attempts and vm.pfault_oom_wait . So I did as well for those 2 for this experiment.) /etc/sysctl.conf has: # # Together this pair avoids swapping out the process kernel stacks. # This avoids processes for interacting with the system from being # hung-up. vm.swap_enabled=3D0 vm.swap_idle_enabled=3D0 # more ~/src.configs/src.conf.CA7-nodbg-clang-alt.aarch64-host=20 TO_TYPE=3Darmv7 # KERNCONF=3DGENERIC-NODBG-CA7 TARGET=3Darm .if ${.MAKE.LEVEL} =3D=3D 0 TARGET_ARCH=3D${TO_TYPE} .export TARGET_ARCH .endif # WITH_SYSTEM_COMPILER=3D WITH_SYSTEM_LINKER=3D # WITH_ELFTOOLCHAIN_BOOTSTRAP=3D #Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=3D WITHOUT_LLVM_TARGET_AARCH64=3D WITH_LLVM_TARGET_ARM=3D WITHOUT_LLVM_TARGET_MIPS=3D WITHOUT_LLVM_TARGET_POWERPC=3D WITHOUT_LLVM_TARGET_RISCV=3D WITHOUT_LLVM_TARGET_X86=3D WITH_CLANG=3D WITH_CLANG_IS_CC=3D WITH_CLANG_FULL=3D WITH_CLANG_EXTRAS=3D WITH_LLD=3D WITH_LLD_IS_LD=3D # WITH_LLDB=3D # WITH_BOOT=3D # WITHOUT_WERROR=3D MALLOC_PRODUCTION=3D WITH_MALLOC_PRODUCTION=3D WITHOUT_ASSERT_DEBUG=3D WITHOUT_LLVM_ASSERTIONS=3D # # Avoid stripping but do not control host -g status as well: DEBUG_FLAGS+=3D # WITH_REPRODUCIBLE_BUILD=3D WITH_DEBUG_FILES=3D # XCFLAGS+=3D -mcpu=3Dcortex-a7 XCXXFLAGS+=3D -mcpu=3Dcortex-a7 # There is no XCPPFLAGS but XCPP gets XCFLAGS content. (An armv7 host does not need differing content than an aarch64 host, thus the use of the *.aarch64-host file.) However long the overall build ends up taking, the above is part of why the details end up as they will end up. /etc/crontab notes: I do not know if you leave the following enabled during the long builds or not ( from /etc/crontab ): # Perform daily/weekly/monthly maintenance. 1 3 * * * root periodic daily 15 4 * * 6 root periodic weekly 30 5 1 * * root periodic monthly that can run things like "/usr/local/sbin/pkg check -qsa" (daily example) that would compete for resources. I left them active, so daily competed with the build for a while, but it did not happen to overlap with the high swapspace use time frame. I commonly disable these for builds that will span into the hours it indicates, at least when I'm monitoring builds for comparisons and such. =3D=3D=3D Mark Millard marklmi at yahoo.com