From nobody Thu Jul 15 20:48:11 2021 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B686A1275E0E for ; Thu, 15 Jul 2021 20:48:16 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GQmfl37cpz3wPj for ; Thu, 15 Jul 2021 20:48:15 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1626382093; bh=VdWIVLrJ17QlNtc/C3DlP3OWmO/G9LxjEUV2/WYuSaI=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=rIQSUsue+8s0CYlCWJN4YyzpHrfkOwomE7AOXxJeTEbwrXV2BR2MmRTsCn/1qmAYODs/2pLJQbdmLJnKnZA2cI3eXdbUrtA9SgjTi5JhuiGeGrA6Bidc50YBkVUg2DryyQyQUI+SytWtTaQZ+lTKjDneJeihrosuMlJNnaQFb9xvRzFYdlfaM0wlFrfopkp7tntX+Qkbm02uGWzi39tEtSkfY/AlnaIy62hrktEdTH/AqkK+9khL7SM3484bfxjSsnxyFKNPdlNXbe6yH/6Am/Z3R96l43F0Cnu7EHPYwZVGfAU2KF1uJH8ttbj2bRdbv1UKBpgrHItKUJuxWJVCCA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1626382093; bh=YJkildxwmQmShk14pCXwhNM7YQG1OpGLxbG4d0zOtRu=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Tn40p6gpfPrAE575H/gtboaQeqdXwp5DsuPlRWx3fYa/rynkJ5LUx0PdfNGn2dRRteY4BPuNyG6jv6cwCcI5C2ZKoRzLgknDkhR1YU/tc+DfJDsARiBtPxaJK3UHjiloNAScq/4Prx2YAuGdRibWdBnDHpthSjZ+xGjKWZpPQY5jZAh7yortU2IrWoJpVtgxWWLEaR/7IM2N3CRYqlYf8zVfkyGKi8jd4BUQGwKKTnvIUxfR3lkHxBd97rZwCWAP/8Uw6Yfhm9lfJUMFN0x3ObSQYn7Z7+sYND+D//iAfWxoOPR4xi1/ObCcI0DeeqOvIQO1Y2ze7Khx0YzOKYn6RQ== X-YMail-OSG: zvGozTQVM1nGR99.ZcfYXLc8pgGWgoE1I62xCGrj3pM8A7hC026rNIAgp51SboX Yw9UzsZoamCxlOpDQIAFsAlKhL_43XFiz4rT2htqiFeK41.39LaZtNJpsOCNAUVn_oVg8mGq8zGN NaINpfJz3AlW9cMzxV7aX.q.s3XLeC.aeVEkODV0e053SxG0fax0NBEqpOtCQqvqofz.uzW8aspp ouNcU5wfRk34PIaaLuk_QcBgPWKRQO7GHWCH9iZcGQAA8UZDUsMHH1uNQMk7KDu6_STIALSx72yt c1PgLoa3HWWv1H.5d940hlKRwmYNmsDUvGa6gS6pCQ2UGlKSS_MX5vW6Fl3y8N9FTl_.yUHXGOZI 8ZxuovIFQntWb_YmDTTOaZyDxzRo1QJV3PNISwhjRmo8AqGkkk7bsMZsIS9Hk4Ndxy620oG_jxuZ V4lqSPHOyEQz9JAG8jBH33HkCYwbEVyH908FDckphxxYqow9YSlWRbUUTZN0ZaKlGXZDkXXvL.5D r2B3hLmM_8UfiDrHI2M0MaNFacOx0MABcYv.2rGKX.IfNy2CTlvUHKYQAyEsH7PprgAmoFvriru0 d_3VjU7t4n2eCfYzIENeuImUXOnHCHg.owQHBGfMS.PhAIjQHi6PaNv_qL5a2ItpMAP1tZocUpYj S6bqfWO68KWXjwXGQfH.xFB4tiQBpYqSdJ7chVJNgVnyRTcWnof6rq9jMUxKN1Xl56PMjXA5ibgC jZ9.brimxsd0kH72HXnjSCQKIcyyRtMmv84E1TeufKVlC9Uxzat77GtzSBeRJEBnM0URRrU.kPwd SLNbYB8wwdw2IqQVZ8XJPItXq_KwaPY2QjHa2uzHP5SkxAjjcUEaI7DvjqYbuLxvDfOm7VhtZYkJ 59E9qWB.xBsXeiP8ytCSm.hOQFjgn7AFFGF_3s5mO8GJADOGvZXT.UaRpQIWUYyVTk3i2rMfieCn mkUPUa9bq.CDvpnUqq5B4IZpZLjvAaGk3qcfWuxx5WdAUBVE8glguzVOvewig0uoiEJJN6Uyc_pQ R4Rvlp0XOT5MHukblRaEOrUM0s_DEn4CSbZClWCgWD1YHa0QmM2nesIy_GB8Ob0hat28zpf5rjqX lTZu2SoRN4B2MTeuTjAQQcMcoNzdbOTKxpzngs3w1xmduQzKALjdWPotLL8SxiEqczxUJHCh9gpP InBwFU0r8UiKEp._VcnHf9Bw0dwSuA109.RBlbfYWQdypFhKJGjcr4xY42FT1kavg.vQmVQo7D75 gyKSdGHn9Zs7ni7GMOuAv0umWmu1VknB857S.d2h2XWZvPbpr_l8tVGg5sWtFxsMKHA2emDTRMvI kDEmtTnXc6VeJoF6JUpCGacO7mW8fArImruQUQtrSaR4djcLTqx_rWRzo2IJVMr9bDTIjJwMM1yy UEfdScG62v7tSkcepRz0h.BmLE7AndqSwNUNGH5d60DlC45YwfFe_hILYdjox_e532OdnURDuZLf Lo8b22.lXjCldr7bBX96kmMMvaWiUbFERo.3vta.TwI.9DJJl.kyRzQMbOQQkSIMbXUxtcA7pFym wxOqjY7.iN5aS7qp1gTogVKa_A2v55QEZvRwAIJN8JF6oozbfr_Eg0G4_burpQNmaN1r4GBjYn9C 8o.LxZgk.dh4lWsI25.IkeselYYQD2QUQa3YXq8tbpu7xXzUMaxKNrPXyJ_y7JuIDGwhykVTqS5k phUIgDrCCs21QXPRkoYytEIoAu0fAUJEx8ie_YnLkiODykP1VI8Gdynf.tp.cgE7drVCvv41Bk2U .xbq_yAjUVWzkaTRqQOqHKrCLuQa55hvwV0x24P6YLlTY8kXRrGLh3J1PtUIWRTChtPsrhtqkriy p3BZNRXWibr_XiFbWc2JUxKrkyIKS1NmBv0vXeZgXw2BrvCFP4yTJ21g3Vj38rLCUlIm7CUFRCdW Vk2DGs73UDqquzWTGJjR0SL5b0h.O_OX.C2hmV8w.qPK9FiC0tyabP_ns8CVbTbl3UZyhkknxW58 TF0gx7pBdyyZHrU1zX0TAdrjl31aAUs8QGuinYw.6kd8JmEFfAAXLcrdi8ycY7UlUvSsCqmeVBPg zQR6VhV4x3OOLDVbMU.2pBrS_MNbKbQ.TowlXwckVOdK9qzbPUkwt_Af4dJv7w3a3jLI6vAfX7YC cNphZqTiHxfl4JYb3orlELzXQTeqwzTemMgTb9T6nskFC6i_dSbIsXQlMP6IJc_PkNENl.BNoKf_ WmE.pQwGh3pQvt1SSctHgY8ROM9hwf.GtHCmmXoCgHvMBYH5eZsgI8_g_asT2fdHWviJwjlO4EtC n3ldnDJGVrGCcf9sRk4GtNuCrrGdIvGB12vMH_SlofPwesEsrdGuonjihuQwGVGI9K8.Bb0P4VhO D6RqEA7HecPbTx0_.pBJg0.lYV5BALeAez35DCWqgxi_j6oidVL_zy8BWhbK_rXL4FoQO.v7ucyf mDJFDtb1jBNwEMlKLZDCi_FnB4ENN9d1DtuMJ700cuW_n6riSSKY0fluAGhsTLJ1JcaREIgY- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Thu, 15 Jul 2021 20:48:13 +0000 Received: by kubenode536.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 942a22b367400cd569665d0b60b5c35c; Thu, 15 Jul 2021 20:48:11 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: HoneyComb first-boot notes [a L3/L2/L1/RAM performance oddity: fix identified] Date: Thu, 15 Jul 2021 13:48:11 -0700 References: <8A6C415F-A57B-4F2F-861F-052B487166D6.ref@yahoo.com> <8A6C415F-A57B-4F2F-861F-052B487166D6@yahoo.com> <40AE6447-77AF-4D0E-864F-AD52D9F3346F@yahoo.com> <12A4EDD1-A2AB-4CE3-AB0E-A4B5D6FB4674@yahoo.com> <5B1B5E1A-8AE4-4889-ABE6-50C206F896FB@yahoo.com> <7DBDC8AB-C80B-4E26-B58F-251A3D29CE41@yahoo.com> <5BBF1B55-F02C-4817-B805-677EDDC5B809@yahoo.com> <0B577668-97AB-44B6-B1A7-C68F6CC299E5@yahoo.com> To: freebsd-arm In-Reply-To: Message-Id: <10364E05-8BCD-4A15-9D3A-CB5DD9952AED@yahoo.com> X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GQmfl37cpz3wPj X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=rIQSUsue; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.49 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.64.148:from]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[98.137.64.148:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.148:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.148:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-arm X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jul-11, at 18:29, Mark Millard wrote: >>>> . . . >>>=20 >>> I've run into an issue where what FreeBSD calls cpu 0 has >>> significantly different L3/L2/L1/RAM subsystem performance >>> than all the other cores (cpu 0 being worse). Similarly for >>> compared/contrasted to all 4 MACCHIATObin Double Shot cores. >>>=20 >>> A plot with curves showing the issue is at: >>>=20 >>> = https://github.com/markmi/acpphint/blob/master/acpphint_example_data/Honey= CombFreeBSDcpu0RAMAccessPerformanceIsOdd.png >>>=20 >>> The dark red curves in the plot show the expected general >>> shape for such and are for cpu 0. The lighter colored >>> curves are the MACCHIATObin curves. The darker ones are >>> the HoneyComb curves, where the L3/L2/L1 is relatively >>> effective (other than cpu 0). >>>=20 >>> My notes on Discord (so far) are . . . >>>=20 >>> The curves are from my C++ variant of the old Hierarchical >>> INTegration benchmark (historically abbreviated HINT). You >>> can read the approximate size of a level of cache from=20 >>> the x-axis for where the curve drops faster. So, right >>> (most obvious) to left (least obvious): L3 8 MiByte, L2 1 >>> MiByte (per core pair, as it turns out), L1 32 KiByte. >>>=20 >>> The curves here are for single thread benchmark >>> configurations with cpuset used to control which CPU is >>> used. I first noticed this via odd performance variations >>> in multithreading with more cores allowed than in use (so >>> migrations to a variety of cpus over time). >>>=20 >>> I explored all the CPUs (cores), not just what I plotted. >>> Only the one gets the odd performing memory access >>> structure in its curve. >>>=20 >>> FYI: The FreeBSD boot is UEFI/ACPI based for both systems, >>> not U-Boot based. >>>=20 >>=20 >> Jon Nettleton has replicated the memory access performance >> issue on the one cpu via a different HoneyComb, running >> some Linux kernel, using tinymembench as the benchmark. >>=20 >=20 > Jon reports that for HoneyCombs older and newer, EDK2's older > and newer: All show the behavior on cpu 0. "[I]t may have > always existed." >=20 > Jon also reports that U-Boot based booting does not get the > behavior. >=20 > (I've never used U-Boot to boot the HoneyComb for any OS > media that I've got around. In my U-Boot ignorance, my > quick attempts failed for FreeBSD main and Fedora 34 > Server media that I've been using with EDK2's UEFI/ACPI.) The problem in the: lx2160a_uefi/build/arm-trusted-firmware/plat/nxp/soc-lx2160a/soc.c code has been identified and my testing of the proposed fix indicates things are working. Some very early code setting up the L1 Data prefetch configuration was depending on not-well-initialized memory and an initialization routine needed to be used a little earlier in the sequencing to avoid that. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)