From nobody Sun Jan 08 08:26:48 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NqVYT1t7Mz2qp4Q for ; Sun, 8 Jan 2023 08:27:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-23.consmr.mail.gq1.yahoo.com (sonic304-23.consmr.mail.gq1.yahoo.com [98.137.68.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NqVYS3Cm0z4k49 for ; Sun, 8 Jan 2023 08:27:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1673166425; bh=ZF94J6Wuw6SUufzjvxVWjUmmHZnJKAiqKSXjblkACqs=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=rDJD5vtZ6Nyx5nRqoddbw4Z1J4CfGw6bMxONP5lVoGU+l+My8/eicgzrvNjWVkO2J/eB9qDZ79hYn+9RSO6gdAtdqcsprrG8NOM4j2bbIlM10LXSwfmlvPSK6I/jDq5+3mqBjNx6m1xaMXRrCjP/k/G5fy+//ZYUBg4ewBjwk+5Dl1lTrtpEKlOTyR4NTkmF/sRPUX0qBQJDoJV7357CiKssC6LDRmP5GHfFDbq3UqfdTC3eJi4sVpEhTo8JTvUHPhbWt9BlyK2sie3BMWThloTjHdjGb68DPcE5Lz0pGVVVgX9ck82mI35KEH7ROkkS90C7M0+NcPxeSUmZZfPVYg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1673166425; bh=jvwO0UYI1LuF8OVLESQspzVdC5j6z17xHbMGCjUSG3L=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=EABYAtdCmGUigLIxxZNVwAX5dKOiH9X1LHpUZTY8gi0hKUF9Wsnn+csN2g+ANtVzByMe9i3Wm6BKsqR1Ik2dsT1ohTOXr2Wo4WktF+YLmOWlzkEWfQgeWkQuyYWIk2/b/En6NHgVst8XG1rB+VmlSk30IlI4i+GtxL00gWKBP4dZuu2u5WvXNuY7DcqvDkgq4JTJznlvROSd4fpxXk5Ap5Ycv7E23OALj5AFtK52ygZepRyvcy8oLmQ13kkA8P6AROfAlG2o4wvMnktvwUy9Tc1rXclApPFcLuEcs5rcYMp39p3tUvZh3uKZ6ljU84cTf+a81XLaRBXtVZMovn7gbQ== X-YMail-OSG: pjhBJZAVM1kC_MOBfqSXdkX2MKzQz9qe0Y8qtRF4TbseN6USh3Utrb.1nCs0fHt WLIbA03_bKJp8ZZgu_WldVAUOjZsPFUWpcEpDClgQKX9piIV1iQeTuAwDK5pfHoZWTKDZJN6iOQH nE_CWDc6YpMoQu.jVqxAO9YQ.KzYGsA_QBE4QXlK71T7YhmhdHeFY1vdMiMxxvoRxEemNlLXPJrY T_V8j4fTaJGo6Kom5gyLgZlr71ex9Sy_fgHlRetUL5kbRKR08Qe3k0ZBSjthcVXmMUmgxALiZ7_M j8Tg.z1pSITrlFv7VGgzunr8bLGfai2uqHCoyhzmkfO5iCejKLHi8lMGsWGWs8MX6VuEV2t3thwP .7NADM8SyJnInMhlYiyfnAVpaedmE73fE2BrEd9JHPx.8j18rbXukDtTu0JUGjJH_muWVHk7Me1D KZZ9D_jIitP6FfrRkROq7KVTy6EM2WtisJkuLuszk7nw9j8X2jnbU35gKRY7QeyMw2uPxw23q9of 6wOhkbmzfv6Jd7N9vlzEKw64EP4cVM7ojGgmJs8JgMMJd3ZznFB6_BwNuttTT947K0FUPdVNbCww 3OrY0fZjki_hg9PD5kRn4RfUqyi.QiEtX7X_K82P5vzAWGUTPrYX6Mfm3.FTKdA3AtZpLkpFJtzs jvw_.yD5vjB0p1E81IEMa81j6rMGjY_3dl_X3GecqpxJrw7xHzuJyPHyW1uUum4m4_GGSzpKfD6z 7GnUSZLE2OxqmPUqU1ajex6LBe0qexdXdbkvkWoQeK9mwhkpRlwJU.7SI0FXHIVy67Pp5u39t4ii mXghkSGTihSD57rnHZ08oUoa9kXo3knwp6CoGohVIifzkAOypV52iZGNjUYBQhM_pzDAk4dVKNeI .caDERo9a5jcoJZ6znLVqzTYzRVRWKiVkF0pxMNR1PNTpNK9m.MfRaQdGmey7tWWAAUCbHM_0ZTT StfX6VCKRW6nIfuRJDOT8FAAwUHZivM7zbvZYtfFhp6b4CleNjEgFhzb.yWrl1BZHaG1f2AbOgqA bCDqNwirhR6ObZPZzwxjvqrfk8YApeAfnod5vId7b8iXM71nHTASvUvhXGAC2R60Pxna.kUwHAS7 oI5YJTkURl2NL0cYcBYivyX9DJ1qaB0AAAcP93XUP5dsesGW3BEpwedFub2pid8T2eixoJ6qd1Id hZx81rrcVoD3GyAWiBfLwVsXLWBgPDXaV3VWcyJx_Lxpt0IZHyuY90voKba_iEPD3b1AVevD9do8 MdlcVD3N0MjHz.ZQb0sBQhpfVtwIZ_jGPmFmgYkPZgPuLbS3IsBmqs0C4CbgA8OwuHpTp9TecbpL f0OzZRk3_6LMRFstEOY6.57otB2.Hl6IFwEqGl1D8.CqFyws5c_rnSVRiindVQQKk2LI0nNHLWCE f2QmkCHr4FaXJIJcDymMr.DIezMzd1CQ4Cy_RT_EnnIrEA7QHcG3cDlXVyF9.Vr33gHbD9xEySY8 cqhGxwhNiUxh.bwFkYZznNhNbjfSzXas5l3OEiCaXoGVPqLAxrpzEV9c3hQkV_EytSXKYVm4kMJo pczsw3P7bqL8KoOifEMm_UP3j8Q0ArjJhgGBEAyg8S0XGHWAuSnve4aWTLpPb3VBWOIwXHvrPias xVhqVvhJxWqRFMUCCXGLUiQgk3MNDTkZSaJYakBj6edKFScSnE4w43.eI1fqp8xTcflTOk5N8KWU inLLThvD6uvD55MAh315WQuk2hSfxwFhB1xT5L.pEypxfBMrt8Tlz9A5yNwR_KA2tTF.DCLdVlm9 hSxQF3Okk57QdQd3JxnTYtmdHIshm3lHk1qdBaw0ITNaqMNq9nujBCvXt2yRxHQN7c4g8M1VI1tQ YDdlMSDdYw4uKfuUsC7jiGKdc.rppugOTHsdRKBQGIMKmZG_vR4a3D4CYBBt2B7b5xLRBO2j2.9E F13409k_1rb1Z2YailDmg1HdLSLd89qf8eFjBk77J5lLRe9V0RFYSoMY.TTW3KvVwRCzpcjW1zqB f9ktZcHb4R6nnfUPzjwEBRop5SJm3wlBb2yQ0OuBsrM4mBP.KqeYCv9TE08Zm2kvZWcp1_alhRa1 1fATXlD7FE01OkEsnNykZIb960UJy.Iiu.e30TODHTo6RgMu_nGRRTuU8PtXqQxtu92Zzejf71of TizgrKG3aMaoANEikIUdZv2aLiFqZjldt7aqFlyOizCbhS3.d8P7YcCq9LCsyNbB3Qv6oOqTaPbn sBQzWx7sWSyJwPc.ZLpVAGRKhgTpTfA81ByqEZ0S1D7uVpUDxwG3BcqIcaGsVops9hG2msU7WJSo - X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Sun, 8 Jan 2023 08:27:05 +0000 Received: by hermes--production-bf1-5458f64d4-p2fqs (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 6d8bbeb5fe0ea5c254b378f7811cd476; Sun, 08 Jan 2023 08:27:00 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: How to make FreeBSD's kernel boot a RPi4B with modern RPi* firmware From: Mark Millard In-Reply-To: <3BE72ED7-8787-45E8-8341-FE9CF4CFB84F@googlemail.com> Date: Sun, 8 Jan 2023 00:26:48 -0800 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <9C037D3F-A440-4708-993D-117F313691BB@yahoo.com> <374EC3E5-4CB4-4336-A8B9-7A9CF6151691@yahoo.com> <9E9C739E-8308-472A-B797-05A37559DD00@googlemail.com> <217ACD33-A466-4A01-AD36-5D4A0C1B3CF0@yahoo.com> <3BE72ED7-8787-45E8-8341-FE9CF4CFB84F@googlemail.com> To: =?utf-8?Q?Klaus_K=C3=BCchemann?= X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Rspamd-Queue-Id: 4NqVYS3Cm0z4k49 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Jan 7, 2023, at 20:01, Klaus K=C3=BCchemann = wrote: >=20 >> Am 25.12.2022 um 04:15 schrieb Mark Millard : >>=20 >> The example context used below has: serial console with >> USB3 SSD boot media (not requiring a usb_pgood_delay >> setting), booting a stable/13. The RPI4B is a C0T one (no >> 3 GiByte limitation, for example: the PCIe wrapper logic >> has been corrected). >=20 > O.K., Mark, I=C2=B4ve tried to read your 4b-related reports of the = last period more closely..=20 > And now I=E2=80=99m a bit confused : > I tried now : > <> > on USB3 SSD media 4b B0T - device directly from a current img.xz and = couldn=E2=80=99t determine any problems with bcm_dma or pcie, I think you are confusing a failed experimental change with normal-software's operation --and the experimental change only applied to "C0T" parts. C0T RPi4B parts should not normally need bounce buffer activity. But I failed to make that work in my earlier experiment. (The "C0T" parts can do bounce buffer activity just fine but teh activity has the overall performance consequences when there is sustained bouncing.) You do not have my failed experimental code, so you could not test what I originally tested --even if you had a "C0T" RPi4B context available. > I don=E2=80=99t own a 4b C0T, just the C0T CM4, where my = pcie-bug-report stays unpatched since 1 year, > It needs an extended JTAG- investigation, it cannot be fixed by = firmware, it has to happen in the driver logic. > The CM4 is my personal problem, it=E2=80=99s not a supported device = (has also devmatch issues on booting), > but I can live with that on working emmc& working genet0.. I'll note that for snapshot generation there is in: https://cgit.freebsd.org/src/tree/release/arm64/RPI.conf the text: DTB_DIR=3D"/usr/local/share/rpi-firmware" DTB=3D"bcm2710-rpi-2-b.dtb bcm2710-rpi-3-b.dtb bcm2710-rpi-3-b-plus.dtb = bcm2710-rpi-cm3.dtb bcm2711-rpi-4-b.dtb" As near as I can tell, that is the list of what, at one time, was considered supported (tested?) for aarch64 RPi*'s. No CM4. > The 4b(including C0T) is a supported device(is it?) and=20 > if I understand you right : > Your 4b C0T crashes (at least sometimes) by failing in = dma-computation. You are confusing standard-code use with experimental code changes being used, changes related to avoiding unnecessary bounce buffering for "C0T" parts. > And you have a valid fix for the 4b C0T ? =E2=80=A6 I've no clue why my experiment failed. I did want to check if it would fail with modern RPi* firmware (and the matching .dtb files). But that, in turn, requires avoiding the crash for more modern RPi* firmware. Such crash avoidance could be useful to other folks as well (for other purposes than mine). > While I unfortunately cannot test 4b C0T, it=20 > sounds that if you can fix it, it=E2=80=99s a must have fix for that = device? The standard code for RPi4B's that does the bounce buffering does not have the problem that my experiment had --on any RPi4B that I've tested. I was trying to see what some performance was like without bounce buffering being significantly involved but never got such to work reliably. > Side note: have you tried fixing pcie@7d500000 : in the dts without = changing start4.elf and the likes? > Is that 4b C0T bug based on start4.elf or the dts or whatever? I've no clue what lead to the failure of my experiment. Most likely my own mistakes of some kind, given my ignorance of various things. But, that leaves me exploring based on not much understanding of what is likely to prove useful to explore, at least for now. > For my devices the EARLY_DRIVER_ would do nothing but spam dmesg with = new firmware, it can=E2=80=99t fix dma on the CM4 and=20 > Is not needed for the 4b B0T... An RPi4B with EARLY_DRIVER_ + sysutils/rpi-firmware does not produce the "spam" messages and does not crash. An RPi4B with EARLY_DRIVER_ + sufficiently-newer firmware does produce the "spam" messages --because it did not crash first. An RPi4B without EARLY_DRIVER_ but with sufficiently-newer-firmware crashes instead --before it would have generated the many "spam" messages. The difference relative to "spam" messages is the differing live Device Tree content. > but is your 4b C0T fix based on your EARLY_DRIVER_MODULE -patch, even = without touching any firmware? > If so, you have to give it to Phabricator ;-) I have no fix for my failed experiment. I'm still exploring to see what sorts of things might have contributed to the failure. Other folks may have other reasons to want to experiment with more modern firmware. aarch64 or armv7 devices not covered in the snaphots (or even in systutils/rpi-firmware) could be examples that might be involved. Some devices might post-date the sysutils/rpi-firmware content. I listed 4 kinds that sysutils/rpi-firmware does not have .dtb files for. None of the 4 are considered supported at this stage, so far as I know. But someone might work on making one of them somewhat supported. Such a person might like to avoid dealing with the bcm_dma-lack-of-initialization related crashes. > Am 08.01.2023 um 01:59 schrieb Mark Millard : >>=20 >>=20 >>=20 > Am 25.12.2022 um 04:15 schrieb Mark Millard : >=20 > The example context used below has: serial console with > USB3 SSD boot media (not requiring a usb_pgood_delay > setting), booting a stable/13. The RPI4B is a C0T one (no > 3 GiByte limitation, for example: the PCIe wrapper logic > has been corrected). >=20 > On Jan 7, 2023, at 10:58, Klaus K=C3=BCchemann = wrote: >=20 >>=20 >>> Am 07.01.2023 um 11:18 schrieb Mark Millard : >>>=20 >>>=20 >>> =E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6= ... >>>>>=20 >>>>>=20 >>>>> stable/13's source code changes are ( similarly for >>>>> releng/13.1 ): >>>>>=20 >>>>> # git -C /usr/13S-src/ diff sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c = b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>>> index cab8639bb607..6d521d6dcace 100644 >>>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>>> @@ -766,5 +766,6 @@ static driver_t bcm_dma_driver =3D { >>>>>=20 >>>>> static devclass_t bcm_dma_devclass; >>>>>=20 >>>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, = bcm_dma_devclass, 0, 0); >>>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, = bcm_dma_devclass, >>>>> + 0, 0, BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE); >>>>> MODULE_VERSION(bcm_dma, 1); >>>>>=20 >>>>>=20 >>>>> main's [so: 14's] source code changes are: >>>>>=20 >>>>> # git -C /usr/main-src/ diff = sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c = b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>>> index 5f9ecb0b7981..d901447df1e9 100644 >>>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>>> @@ -764,5 +764,6 @@ static driver_t bcm_dma_driver =3D { >>>>> sizeof(struct bcm_dma_softc), >>>>> }; >>>>>=20 >>>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0); >>>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0, >>>>> + BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE); >>>>> MODULE_VERSION(bcm_dma, 1); >>>>>=20 >>>=20 >>>=20 >>> =3D=3D=3D >>> Mark Millard >>> marklmi at yahoo.com >>>=20 >>=20 >>=20 >> =E2=80=A6=E2=80=A6.on the other hand : if your = EARLY_DRIVER_MODULE(bcm_dma=E2=80=A6 doesn=E2=80=99t do anything wrong, >> you could give it in phabricator review, why not?!.. >=20 > Yep, once I've better evidence from the RPi*'s that I have > access to. >=20 > I'll note that no vintages of the following .dtb files are > in the current sysutils/rpi-firmware port: >=20 > bcm2709-rpi-cm2.dtb > bcm2710-rpi-zero-2-w.dtb > bcm2710-rpi-zero-2.dtb > bcm2711-rpi-cm4s.dtb >=20 > I've no direct evidence of if any vintage of any of these > would end up hitting the bcm_dma issue or not. But I expect > that the EARLY_DRIVER_MODULE related patching would avoid > (just!) that specific crash problem if it would otherwise > would occur. >=20 > There is: >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261147 >=20 > reporting the absence of bcm2710-rpi-zero-2-w.dtb . So > someone might want to experiment with more recent RPi* > firmware, possibly even to develop some level of support > for bcm2710-rpi-zero-2-w.dtb (live Device Tree possibly > adjusted by the RPi* firmware) --if changes are needed. >=20 > The .dtb vintage and the RPi* start*.efi and the like > vintages are not necessarily fully independent. Mixing > and matching could be a problem, independent of any > additional FreeBSD kernel-related issues. (It is another > example of the poor level of documentation for the RPi* > context.) =3D=3D=3D Mark Millard marklmi at yahoo.com