Re: How to make FreeBSD's kernel boot a RPi4B with modern RPi* firmware

Reply: Klaus_Küchemann : "Re: How to make FreeBSD's kernel boot a RPi4B with modern RPi* firmware"
In reply to: Klaus_Küchemann : "Re: How to make FreeBSD's kernel boot a RPi4B with modern RPi* firmware"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Mark Millard <marklmi_at_yahoo.com>
Date: Sun, 08 Jan 2023 08:26:48 UTC
On Jan 7, 2023, at 20:01, Klaus Küchemann <maciphone2@googlemail.com> wrote:
> 
>> Am 25.12.2022 um 04:15 schrieb Mark Millard <marklmi@yahoo.com>:
>> 
>> The example context used below has: serial console with
>> USB3 SSD boot media (not requiring a usb_pgood_delay
>> setting), booting a stable/13. The RPI4B is a C0T one (no
>> 3 GiByte limitation, for example: the PCIe wrapper logic
>> has been corrected).
> 
> O.K., Mark, I´ve tried  to read your 4b-related reports of the last period more closely.. 
> And now I’m a bit confused :
> I tried now :
> <<FreeBSD generic 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n259905-231d75568f16: Sun Jan  1 11:28:27 UTC 2023 >>
> on USB3 SSD media 4b B0T - device directly from a current img.xz and couldn’t determine any problems with bcm_dma or pcie,

I think you are confusing a failed experimental change
with normal-software's operation --and the experimental
change only applied to "C0T" parts. C0T RPi4B parts
should not normally need bounce buffer activity. But I
failed to make that work in my earlier experiment.

(The "C0T" parts can do bounce buffer activity just fine
but teh activity has the overall performance consequences
when there is sustained bouncing.)

You do not have my failed experimental code, so you
could not test what I originally tested --even if you had
a "C0T" RPi4B context available.

> I don’t own a 4b C0T, just the C0T CM4, where my pcie-bug-report stays unpatched since 1 year,
> It needs an extended JTAG- investigation, it cannot be fixed by firmware, it has to happen in the driver logic.
> The CM4 is my personal problem, it’s not a supported device (has also devmatch issues on booting),
> but I can live with that on working emmc& working genet0..

I'll note that for snapshot generation there is in:

https://cgit.freebsd.org/src/tree/release/arm64/RPI.conf

the text:

DTB_DIR="/usr/local/share/rpi-firmware"
DTB="bcm2710-rpi-2-b.dtb bcm2710-rpi-3-b.dtb bcm2710-rpi-3-b-plus.dtb bcm2710-rpi-cm3.dtb bcm2711-rpi-4-b.dtb"

As near as I can tell, that is the list of what, at one
time, was considered supported (tested?) for aarch64
RPi*'s. No CM4.

> The 4b(including C0T) is a supported device(is it?) and 
> if I understand you right :
> Your 4b C0T crashes (at least sometimes) by failing in dma-computation.

You are confusing standard-code use with experimental
code changes being used, changes related to avoiding
unnecessary bounce buffering for "C0T" parts.

> And you have a valid fix for the 4b C0T ? …

I've no clue why my experiment failed. I did want to
check if it would fail with modern RPi* firmware (and
the matching .dtb files). But that, in turn, requires
avoiding the crash for more modern RPi* firmware. Such
crash avoidance could be useful to other folks as well
(for other purposes than mine).

> While I unfortunately cannot test 4b C0T, it 
> sounds that if you can fix it, it’s a must have fix for that device?

The standard code for RPi4B's that does the bounce
buffering does not have the problem that my experiment
had --on any RPi4B that I've tested. I was trying to
see what some performance was like without bounce
buffering being significantly involved but never got
such to work reliably.

> Side note:  have you tried fixing pcie@7d500000 : in the dts without changing start4.elf and the likes?
> Is that 4b C0T bug based on start4.elf  or the dts or whatever?

I've no clue what lead to the failure of my experiment.
Most likely my own mistakes of some kind, given my
ignorance of various things. But, that leaves me
exploring based on not much understanding of what is
likely to prove useful to explore, at least for now.

> For my devices the EARLY_DRIVER_ would do nothing but spam dmesg with new firmware, it can’t fix dma on the CM4 and 
> Is not needed for the 4b B0T...

An RPi4B with EARLY_DRIVER_ + sysutils/rpi-firmware
does not produce the "spam" messages and does not
crash.

An RPi4B with EARLY_DRIVER_ + sufficiently-newer
firmware does produce the "spam" messages --because
it did not crash first.

An RPi4B without EARLY_DRIVER_ but with
sufficiently-newer-firmware crashes instead --before
it would have generated the many "spam" messages.

The difference relative to "spam" messages is the
differing live Device Tree content.

> but is your 4b C0T fix based on your EARLY_DRIVER_MODULE -patch, even  without touching any firmware?
> If so, you have to give it to Phabricator ;-)

I have no fix for my failed experiment. I'm still
exploring to see what sorts of things might have
contributed to the failure.

Other folks may have other reasons to want to experiment
with more modern firmware. aarch64 or armv7 devices not
covered in the snaphots (or even in systutils/rpi-firmware)
could be examples that might be involved. Some devices
might post-date the sysutils/rpi-firmware content. I
listed 4 kinds that sysutils/rpi-firmware does not have
.dtb files for. None of the 4 are considered supported
at this stage, so far as I know. But someone might work
on making one of them somewhat supported. Such a person
might like to avoid dealing with the
bcm_dma-lack-of-initialization related crashes.

> Am 08.01.2023 um 01:59 schrieb Mark Millard <marklmi@yahoo.com>:
>> 
>> 
>> 
> Am 25.12.2022 um 04:15 schrieb Mark Millard <marklmi@yahoo.com>:
> 
> The example context used below has: serial console with
> USB3 SSD boot media (not requiring a usb_pgood_delay
> setting), booting a stable/13. The RPI4B is a C0T one (no
> 3 GiByte limitation, for example: the PCIe wrapper logic
> has been corrected).
> 
> On Jan 7, 2023, at 10:58, Klaus Küchemann <maciphone2@googlemail.com> wrote:
> 
>> 
>>> Am 07.01.2023 um 11:18 schrieb Mark Millard <marklmi@yahoo.com>:
>>> 
>>> 
>>> ……………………...
>>>>> 
>>>>> 
>>>>> stable/13's source code changes are ( similarly for
>>>>> releng/13.1 ):
>>>>> 
>>>>> # git -C /usr/13S-src/ diff sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>>> index cab8639bb607..6d521d6dcace 100644
>>>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>>> @@ -766,5 +766,6 @@ static driver_t bcm_dma_driver = {
>>>>> 
>>>>> static devclass_t bcm_dma_devclass;
>>>>> 
>>>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass, 0, 0);
>>>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass,
>>>>> +    0, 0, BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE);
>>>>> MODULE_VERSION(bcm_dma, 1);
>>>>> 
>>>>> 
>>>>> main's [so: 14's] source code changes are:
>>>>> 
>>>>> # git -C /usr/main-src/ diff sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>>> index 5f9ecb0b7981..d901447df1e9 100644
>>>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>>> @@ -764,5 +764,6 @@ static driver_t bcm_dma_driver = {
>>>>>    sizeof(struct bcm_dma_softc),
>>>>> };
>>>>> 
>>>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0);
>>>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0,
>>>>> +    BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE);
>>>>> MODULE_VERSION(bcm_dma, 1);
>>>>> 
>>> 
>>> 
>>> ===
>>> Mark Millard
>>> marklmi at yahoo.com
>>> 
>> 
>> 
>> …….on the other hand : if your EARLY_DRIVER_MODULE(bcm_dma… doesn’t do anything wrong,
>> you could give it in phabricator review, why not?!..
> 
> Yep, once I've better evidence from the RPi*'s that I have
> access to.
> 
> I'll note that no vintages of the following .dtb files are
> in the current sysutils/rpi-firmware port:
> 
> bcm2709-rpi-cm2.dtb
> bcm2710-rpi-zero-2-w.dtb
> bcm2710-rpi-zero-2.dtb
> bcm2711-rpi-cm4s.dtb
> 
> I've no direct evidence of if any vintage of any of these
> would end up hitting the bcm_dma issue or not. But I expect
> that the EARLY_DRIVER_MODULE related patching would avoid
> (just!) that specific crash problem if it would otherwise
> would occur.
> 
> There is:
> 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261147
> 
> reporting the absence of bcm2710-rpi-zero-2-w.dtb . So
> someone might want to experiment with more recent RPi*
> firmware, possibly even to develop some level of support
> for bcm2710-rpi-zero-2-w.dtb (live Device Tree possibly
> adjusted by the RPi* firmware) --if changes are needed.
> 
> The .dtb vintage and the RPi* start*.efi and the like
> vintages are not necessarily fully independent. Mixing
> and matching could be a problem, independent of any
> additional FreeBSD kernel-related issues. (It is another
> example of the poor level of documentation for the RPi*
> context.)



===
Mark Millard
marklmi at yahoo.com