From nobody Sun Jul 11 11:03:12 2021 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 24B538D1D4C for ; Sun, 11 Jul 2021 11:03:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-24.consmr.mail.gq1.yahoo.com (sonic304-24.consmr.mail.gq1.yahoo.com [98.137.68.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GN3sk702qz4cP3 for ; Sun, 11 Jul 2021 11:03:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1626001401; bh=rJWtwjnmwBCeNQ2vVHNJtE9XIBpWA+XB3xpeaWSsEz4=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=QjRtpDgmeFU6U8EkIjIAxgQpcT/akjObFrEdMC7MXmN227wpQvXo1AZaoNAPnsSqFuLFHd1ZpUsQ2kKsKS8vjiZwPdP+/nliQNBu/+/ux7xCRpWhcGoYftILVTM5FSHM/S00lBRKc5MxGnwf2mOj1CXkEE9PY6w6j/1gpmNNmZXoPhtQfQQcHTKOHgm9o3fUApO9cx6z+hB1Cj9QL2jsXZK/ORi954KFGDW+0GCon0lowDUS/CBtMOdqyqdRO5A9XVxyFgx29wdmpx4P/NjewXgfFEwHwENbexOhIaQHu24MxhpDVav2JPiMAKa3Oo6jMMfYRwe3XYCbQPj+PBPQKA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1626001401; bh=2d3hwn5kMOudculH2XegtlgkZTJoqWIiYY4Zj8YaMuw=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=AY91PCKPfhNNS9ihB7Ar23nrwZtTufab2HWlNboAJS5aHib2hEYkzpXflavG9Ur+ElCwF8xOSEJ4UOfq1FHUpg/FcfN0b2VInzqyA/N+BquLfdsxVbWL5d3OR297XiXuyZVmGNrT/BdublobNnUNVkYuuLSExnQF7Dvi2/lQp3BvaIUxsQy5mj06gx8uDhZZ+55FMvF2QOMRtdU2qeYZypbi0gX32WC4HhhdHvhsOWYz6FVeC/wrsZk1skOvt8N+rGJkjG4jJ25qkq50IHQL6Ii09WIAP4nWKaRV06ACWE0Pdn9ap6jNrvAq8YFwV/CemqpyULngoNw3LGpPem1N/A== X-YMail-OSG: UPHd71kVM1kL7MVA94L.7ajZ9OAkyGYVVwyB3LfNnkLlSmCizwuchCGD4_0RzdS gJxaqgwj2qguVCTlONVt34JwVFPbD4aY0j4zAo9aYU1rGdNDmnXD0j5x_zDCNi5LUqI3Dy769di9 Hnr4vQNkxy4OWBkXMwalcJPJsJKrWbeO.TmgALPWm7dVIy6fhXDNxl_9l5OeBxc8N_0fwOe5Yqip jUIAJ5SdChqoTfeM0sa3uGsJnH1cgWk7fpoO3wDrG.bEhsEK2vLQC584LYSD0ieYZ6iN.XdxR9Ql MrfFYSwPLk6Khgzq8mRS_BbfVB4UXvoOuFGlOUZZqvcD1GFJkzR3s3pnMrFPWqs9MgbuQ.C8.AlV zy8OOVkp3QhYs2PEil1tzOStDHxTGVq16KUSYa1JCj9aieE_yuBGX6jIgIOQhm5VbTHXK.kYwZ.b _xxZNdpoH_kWUluvENDNReKCIrgvHkhtoyK3zFm9wjVbWLpASOc07indp.lpDCg_ryO6wGa5j00C nclHAbO8HDfssx4bwLOdDwh5a5FlfL6lRD5HDseFVrQ3wk8zivlNTp2WLAmWTtzElcb3fXs8wW_J 2aYEsUwt96u_5oM.ydct4taKHcYO6zTyZYhoc23nDJzV7sF0tYH43paPnbtoK8kvfFD74pQ8VM6i FZG25EKV8qHAPLVr2YN6.NlKOE.FUhLM7pnMycSS0AgKiMp7.WzlhJoKwaad8Kjq7Aextc0GrS6w tGAmGpv88d3w9ruigXh9iFJ7j69eJ2oOppFCe8uwSAlGvq9.C8y42UejVwSjwQ6nYnDIOhQn04fQ QYdIh7sUvXYYgIZ0FjIDZnr9SxArRqoHFLgc3jWeP2Trvm0A.mbjYRmn2b2A0YpZMNUgSTj1VJVi 4YbMptiYTOloIoUw86cQ19STyEA1_heTY6wPYDPTv5H1RWFa0wd1Al44Occ8V.4NgdwQWqwSP7jD C_d3T1SCuvbKTtdKFzcHTyTOyDqkC.FURBxnzKRs5.ihNaE0Y_Gahz9vPFUB2.6WboK8ybkyOGd8 pZFqtpAj6yRKEkrSjawHiZdSfmIbfYGkKlB3Pr6NxvZDyDYKUI3KZE4YqGOi_RSse.SIt_leth5r CSAJpw8e9H57qm.uBObczR7XmJMxs5sAiTgO9LUODbECCg0aHLXcS2xr6KAyXY_4nD.Dh1rRMTNT ArSn9T.Knp18XBuTaErYOLaz_qAMNfbUuAlEGD9Rf4haSJTd9ERxGzTtIVveY8H1QohJWIEt0_7q HDJOj6A7CKbxR69X_wTFzi6LkdDHAssLPiNX9bXbcspQjUz69MfoEfXRVpG4IiSXDoKsb2TNvBjH 8RHtqCwQL28RzQMqb4lB_4HC9D4p92dCMMrSSU18Tpb2G1CiIov1P2Iud3UVX8OqsTQxX9MV3PIj y.oFeYLY723YNiq.9osXW9A8W27nNRVS8pI.2n2uQ_Pgb5Ac0S6Gu4O91mGgo4Pl35r45jUS0_Ca qnK4wC1fZSEiWRHF7MB9_F9MVL_WV.NzQ1nruXDbQTjRMuPHB7HXpLbHjESYqViIACNIF4iwrjai EnmqpFDOG2y6SBgcVwoQ1now94AuH286t8i3JSzRyqKAfOpS6X0Zswn1OUKVxFnVWblh9fXYlaX2 9nD.RYHbK4pflbNtlfFUOPJoQxHTyvLIr73AnIPdaMju6nf3AOMTWavFsplwANEdsyLus8LI1sNG m8edwPOQnbBhqx5CN5hyErBU7.ljsfYnZXJJqxrVzFZapBpwFl_ORAndR7jYTBdHb2YtkM3qqCCz g6v9zOcyWzVGLBObMpP_DbyNW3tMGAUbQzYXBDtj3tzitUoc.4zwnW.OgztXKlzv5oV_KISnAKGv Kqe5kY_R.kl_u_b9RI8pCC9TnN5DopJ_3Dh9CiSpcKmOPB5jwuFn8FNxMPCLrO4yE4oGJc9QZOay nOcTooaiAGv7Lb1zl96DDY84804jMkzRScz5ffS7NaQGgqrccN.rtqCqz8UCk.oMWlYeNy6T.tPp aX2hbLlcrZYEppJFy.bGSAhMrtBntTEULWUfiLbiF8Gq_WNK3dKs9oXv53sG.Hsd2npS8UMY9UyF EfZg5Mp6hgnjEWFjfHoDXtn5_N_VWKPs9jmg1BSC6EVw5S6oTUGDlPQIfOFmRDuU9Nb_QN0I8HTG CBVkwiMdFbsUx9ufpILDj7xS5M9i074558wy_Du7J2RTiy6HmG9F6SA5Q7g95LJN8GFOgTWaQkHz k0R1GhpXyqSdupc6hUaQQ4XvEycOFAw0gAtpAfQfO4sHwOHSxMPmvjaeK8wn9vXQOmrV5bSFr5zj fZMBp7CyxQofhg46U.idPvBxHAMTXI1VpEwU2ISgslRs_lOsZ4OulaNPqxypj5RuBERZn0zk8JzH 1X2RrbirgBSChzQIbZb926Kz6b5rPBu.ZtbxERWh9Vb3potygV3N39NUdz.vHsshpCdyZP7GNVhN mX8Akztn3o2hPRCHw1is4jpyGQQV5qREH9klCobn4qNb3p53RaKjizw-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Sun, 11 Jul 2021 11:03:21 +0000 Received: by kubenode523.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 7cdaf083024ea63a7dfce66b079fec97; Sun, 11 Jul 2021 11:03:14 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: HoneyComb first-boot notes [a L3/L2/L1/RAM performance oddity] Date: Sun, 11 Jul 2021 04:03:12 -0700 References: <8A6C415F-A57B-4F2F-861F-052B487166D6.ref@yahoo.com> <8A6C415F-A57B-4F2F-861F-052B487166D6@yahoo.com> <40AE6447-77AF-4D0E-864F-AD52D9F3346F@yahoo.com> <12A4EDD1-A2AB-4CE3-AB0E-A4B5D6FB4674@yahoo.com> <5B1B5E1A-8AE4-4889-ABE6-50C206F896FB@yahoo.com> <7DBDC8AB-C80B-4E26-B58F-251A3D29CE41@yahoo.com> <5BBF1B55-F02C-4817-B805-677EDDC5B809@yahoo.com> <0B577668-97AB-44B6-B1A7-C68F6CC299E5@yahoo.com> To: freebsd-arm In-Reply-To: <0B577668-97AB-44B6-B1A7-C68F6CC299E5@yahoo.com> Message-Id: X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GN3sk702qz4cP3 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=QjRtpDgm; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-2.97 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.68.205:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-0.67)[-0.672]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.80)[-0.800]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[98.137.68.205:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.205:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.205:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-arm X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jul-10, at 22:09, Mark Millard wrote: > On 2021-Jun-24, at 16:25, Mark Millard wrote: >=20 >> On 2021-Jun-24, at 16:00, Mark Millard wrote: >>=20 >>> On 2021-Jun-24, at 13:39, Mark Millard wrote: >>>=20 >>>> Repeating here what I've reported on teh solidrun discord: >>>>=20 >>>> I decided to experiment with monitoring the temperatures reported >>>> as things are. For the default heat-sink/fan and the 2 other fans >>>> in the case, buildworld with load average 16.? for some time has >>>> stayed with tz0 through tz6 reporting between 61.0degC and = 66.0degC, >>>> say about 20degC for ambiant. (tz7 and tz8 report 0.1C.) During >>>> stages with lower load averages, the tz0..tz6 tempuratures back off >>>> some. So it looks like my default context keeps the system >>>> sufficiently cool for such use. >>>>=20 >>>> I'll note that the default heat-sink's fan is not operating at = rates >>>> that I hear it upstairs. I've heard the noisy mode from there = during >>>> early parts of booting for Fedora 34 server, for example. >>>=20 >>> So I updated my stable/13 source and built and installed >>> the update, then did a rm -fr of the build directory >>> tree context and started a from-scratch build. The >>> build had: >>>=20 >>> SYSTEM_COMPILER: Determined that CC=3Dcc matches the source tree. = Not bootstrapping a cross-compiler. >>> and: >>> SYSTEM_LINKER: Determined that LD=3Dld matches the source tree. Not = bootstrapping a cross-linker. >>>=20 >>> as is my standard context for doing such "how long does >>> it take" buildworld buildkernel testing. >>>=20 >>> On aarch64 I do not build for targeting non-arm architectures. >>> This does save some time on the builds. >>=20 >> I should have mentioned that my builds are based on tuning >> for the cortex-a72 via -mcpu=3Dcortex-a72 being used. This >> was also true of the live system that was running, kernel >> and world. >>=20 >>> The results for the HoneyComb configuration I'm using: >>>=20 >>> World build completed on Thu Jun 24 15:30:11 PDT 2021 >>> World built in 3173 seconds, ncpu: 16, make -j16 >>> Kernel build for GENERIC-NODBG-CA72 completed on Thu Jun 24 15:34:45 = PDT 2021 >>> Kernel(s) GENERIC-NODBG-CA72 built in 274 seconds, ncpu: 16, make = -j16 >>>=20 >>> So World+Kernel took a a little under 1 hr to build (-j16). >>>=20 >>>=20 >>>=20 >>> Comparison/contrast to prior aarch64 systems that I've used >>> for buildworld buildkernel . . . >>>=20 >>>=20 >>> By contrast, the (now failed) OverDrive 1000's last timing >>> was (building releng/13 instead of stable/13): >>>=20 >>> World build completed on Tue Apr 27 02:50:52 PDT 2021 >>> World built in 12402 seconds, ncpu: 4, make -j4 >>> Kernel build for GENERIC-NODBG-CA72 completed on Tue Apr 27 03:08:04 = PDT 2021 >>> Kernel(s) GENERIC-NODBG-CA72 built in 1033 seconds, ncpu: 4, make = -j4 >>>=20 >>> So World+Kernel took a a little under 3.75 hrs to build (-j4). >>>=20 >>>=20 >>> The MACCHIATObin Double Shot's last timing was >>> (building a 13-CURRENT): >>>=20 >>> World build completed on Tue Jan 19 03:44:59 PST 2021 >>> World built in 14902 seconds, ncpu: 4, make -j4 >>> Kernel build for GENERIC-NODBG completed on Tue Jan 19 04:04:25 PST = 2021 >>> Kernel(s) GENERIC-NODBG built in 1166 seconds, ncpu: 4, make -j4 >>>=20 >>> So World+Kernel took a little under 4.5 hrs to build (-j4). >>>=20 >>>=20 >>> The RPi4B 8GiByte's last timing was >>> ( arm_freq=3D2000, sdram_freq_min=3D3200, force_turbo=3D1, USB3 SSD >>> building releng/13 ): >>>=20 >>> World build completed on Tue Apr 20 14:34:38 PDT 2021 >>> World built in 22104 seconds, ncpu: 4, make -j4 >>> Kernel build for GENERIC-NODBG completed on Tue Apr 20 15:03:24 PDT = 2021 >>> Kernel(s) GENERIC-NODBG built in 1726 seconds, ncpu: 4, make -j4 >>>=20 >>> So World+Kernel took somewhat under 6 hrs 40 min to build. >>=20 >> The -mcpu=3Dcortex-a72 use note also applies to the OverDrive 1000, >> MACCHIATObin Double Shot, and RPi4B 8 GiByte contexts. >>=20 >=20 > I've run into an issue where what FreeBSD calls cpu 0 has > significantly different L3/L2/L1/RAM subsystem performance > than all the other cores (cpu 0 being worse). Similarly for > compared/contrasted to all 4 MACCHIATObin Double Shot cores. >=20 > A plot with curves showing the issue is at: >=20 > = https://github.com/markmi/acpphint/blob/master/acpphint_example_data/Honey= CombFreeBSDcpu0RAMAccessPerformanceIsOdd.png >=20 > The dark red curves in the plot show the expected general > shape for such and are for cpu 0. The lighter colored > curves are the MACCHIATObin curves. The darker ones are > the HoneyComb curves, where the L3/L2/L1 is relatively > effective (other than cpu 0). >=20 > My notes on Discord (so far) are . . . >=20 > The curves are from my C++ variant of the old Hierarchical > INTegration benchmark (historically abbreviated HINT). You > can read the approximate size of a level of cache from=20 > the x-axis for where the curve drops faster. So, right > (most obvious) to left (least obvious): L3 8 MiByte, L2 1 > MiByte (per core pair, as it turns out), L1 32 KiByte. >=20 > The curves here are for single thread benchmark > configurations with cpuset used to control which CPU is > used. I first noticed this via odd performance variations > in multithreading with more cores allowed than in use (so > migrations to a variety of cpus over time). >=20 > I explored all the CPUs (cores), not just what I plotted. > Only the one gets the odd performing memory access > structure in its curve. >=20 > FYI: The FreeBSD boot is UEFI/ACPI based for both systems, > not U-Boot based. >=20 Jon Nettleton has replicated the memory access performance issue on the one cpu via a different HoneyComb, running some Linux kernel, using tinymembench as the benchmark. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)