Re: How to make FreeBSD's kernel boot a RPi4B with modern RPi* firmware

Reply: Mark Millard : "Re: How to make FreeBSD's kernel boot a RPi4B with modern RPi* firmware"
In reply to: Mark Millard : "Re: How to make FreeBSD's kernel boot a RPi4B with modern RPi* firmware"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Klaus_Küchemann <maciphone2_at_googlemail.com>
Date: Sat, 07 Jan 2023 08:37:55 UTC
Hi Mark,

`ve tested your "early_dma“ patch now on the CM4(not tested on the 4b)…
true, it can boot the latest firmware but after investigating in what’s new 
in bcm2711-rpi-cm4.dtb I guess new features are mainly  related to CPU (L2-) caches features ,
That’s why we get things like : "clk_fixed4: <Fixed clock> disabled on ofwbus0
clk_fixed4: Cannot FDT parameters.“ ...
so I think we currently won’t benefit much of the new firmware.

I had a little hope that an earlier dma could shine a bit more light on bugs like this : 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260131

But a tiny little hope can’t of course fix bugs :-) , so I think we have to focus more on fixing existing bugs and 
Adding device drivers(of course the ToDo-list is not news but still the truth;-)

Thanks for taking attention the current firmware things, always worth to take a look into.


Regards

K.

> Am 25.12.2022 um 17:37 schrieb Mark Millard <marklmi@yahoo.com>:
> 
> On Dec 24, 2022, at 20:14, Mark Millard <marklmi@yahoo.com> wrote:
> 
>> On Dec 24, 2022, at 19:15, Mark Millard <marklmi@yahoo.com> wrote:
>> 
>>> I finally ran into EARLY_DRIVER_MODULE, BUS_PASS_RESOURCE,
>>> BUS_PASS_ORDER_MIDDLE and the like and they allow being
>>> sure that the brcm,bcm2835-dma related setup has been done
>>> before any use of it is made, despite the order in the
>>> Device Tree: use an earlier pass for brcm,bcm2835-dma
>>> related attach. This avoids the kernel crashing during
>>> boot.
>>> 
>>> The example context used below has: serial console with
>>> USB3 SSD boot media (not requiring a usb_pgood_delay
>>> setting), booting a stable/13. The RPI4B is a C0T one (no
>>> 3 GiByte limitation, for example: the PCIe wrapper logic
>>> has been corrected).
>>> 
>>> stable/13's source code changes ( similarly for
>>> releng/13.1 ):
>>> 
>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>> index cab8639bb607..d8b49acfe332 100644
>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>> @@ -766,5 +766,6 @@ static driver_t bcm_dma_driver = {
>>> 
>>> static devclass_t bcm_dma_devclass;
>>> 
>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass, 0, 0);
>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass,
>>> +    0, 0, BUS_PASS_RESOURCE + BUS_PASS_ORDER_MIDDLE);
>>> MODULE_VERSION(bcm_dma, 1);
>>> 
>>> 
>>> For reference, a 13S snapshot with my kernel build replacing
>>> the snapshot's kernel, was booted with:
>>> 
>>> # strings /boot/msdos/start4.elf | grep VC_BUILD_ID_
>>> VC_BUILD_ID_USER: dom
>>> VC_BUILD_ID_TIME: 11:09:05
>>> VC_BUILD_ID_VARIANT: start
>>> VC_BUILD_ID_TIME: Oct 26 2022
>>> VC_BUILD_ID_BRANCH: bcm2711_2
>>> VC_BUILD_ID_HOSTNAME: buildbot
>>> VC_BUILD_ID_PLATFORM: raspberrypi_linux
>>> VC_BUILD_ID_VERSION: c72ad6b26ff40c91ef776b847436094ee63fabee (clean)
>>> 
>>> There are new things present that FreeBSD reports
>>> but ignores, producing messages like:
>>> 
>>> clk_fixed4: <Fixed clock> disabled on ofwbus0
>>> clk_fixed4: Cannot FDT parameters.
>>> device_attach: clk_fixed4 attach returned 6
>>> 
>>> over and over during part of the boot. It seems to
>>> retry as it goes and thus produce so many messages.
>>> 
>>> There was also:
>>> 
>>> fb0: <BCM2835 VT framebuffer driver> on simplebus0
>>> fb0: changing fb bpp from 0 to 24
>>> mbox0: mbox response error
>>> fb0: bcm2835_mbox_fb_init failed, err=5
>>> device_attach: fb0 attach returned 6
>>> 
>>> genet0 is working.
>>> 
>>> I've not checked if the microsd card slot can be used.
>>> 
>>> I used the normal FreeBSD U-Boot since I was not booting
>>> the NVM3 media that requires extra time (usb_pgood_delay
>>> would be assigned in my own U-Boot builds).
>>> 
>>> For reference, I used my typical sort of config.txt :
>>> 
>>> # more /boot/msdos/config.txt
>>> [all]
>>> arm_64bit=1
>>> dtparam=audio=on,i2c_arm=on,spi=on
>>> dtoverlay=mmc
>>> dtoverlay=disable-bt
>>> device_tree_address=0x4000
>>> kernel=u-boot.bin
>>> 
>>> [pi4]
>>> hdmi_safe=1
>>> armstub=armstub8-gic.bin
>>> 
>>> #
>>> [all]
>>> #
>>> # Local addition that avoids USB3 SSD boot failures that look like:
>>> #   uhub_reattach_port: port ? reset failed, error=USB_ERR_TIMEOUT
>>> #   uhub_reattach_port: device problem (USB_ERR_TIMEOUT), disabling port ?
>>> initial_turbo=60
>>> # U-Boot that has, for example, a built-in usb_pgood_delay assignment
>>> # for a media specific issue added:
>>> #kernel=u-boot.bin.2022.10.arm64
>>> #
>>> # Local additions:
>>> enable_uart=1
>>> uart_2ndstage=1
>>> dtdebug=1
>>> disable_commandline_tags=1
>>> disable_overscan=1
>>> #gpu_mem_1024=32
>>> #
>>> #program_usb_boot_mode=1
>>> #program_usb_boot_timeout=1
>>> 
>>> # Old RPi3's/RPi2Bv1.2's may ignore [pi4] and the like.
>>> # That would make the below inappropriate for such contexts.
>>> [pi4]
>>> # Locally avoid hdmi_safe's dislay scaling:
>>> hdmi_safe=0
>>> #
>>> armstub=armstub8-gic.bin
>>> #
>>> # Local additions:
>>> over_voltage=6
>>> arm_freq=2000
>>> sdram_freq_min=3200
>>> force_turbo=1
>>> #
>>> #total_mem=1024
>>> #total_mem=991
>>> [all]
>>> 
>>> [pi3] 
>>> armstub=armstub8.bin
>>> dtoverlay=pwm
>>> audio_pwm_mode=0
>>> [all]
>>> 
>>> 
>>> As for main [so: 14], the devclass_t use is gone, thus
>>> the source code changes are:
>>> 
>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>> index 5f9ecb0b7981..83c4c493a66b 100644
>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>> @@ -764,5 +764,6 @@ static driver_t bcm_dma_driver = {
>>>      sizeof(struct bcm_dma_softc),
>>> };
>>> 
>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0);
>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0,
>>> +    BUS_PASS_RESOURCE + BUS_PASS_ORDER_MIDDLE);
>>> MODULE_VERSION(bcm_dma, 1);
>>> 
>>> 
>> 
>> I should note that the above is not likely to be
>> the most appropriate in detail. The boot reports:
>> 
>> # dmesg -a | grep bcm_dma0
>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>> bcm_dma0: cannot allocate interrupt
>> device_attach: bcm_dma0 attach returned 6
>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>> bcm_dma0: cannot allocate interrupt
>> device_attach: bcm_dma0 attach returned 6
>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>> bcm_dma0: cannot allocate interrupt
>> device_attach: bcm_dma0 attach returned 6
>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>> bcm_dma0: cannot allocate interrupt
>> device_attach: bcm_dma0 attach returned 6
>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>> 
>> where that last (working) one has the message
>> context:
>> 
>> gic0: <ARM Generic Interrupt Controller> mem 0x40041000-0x40041fff,0x40042000-0x40043fff,0x40044000-0x40045fff,0x40046000-0x40047fff irq 30 on simplebus0
>> gic0: pn 0x2, arch 0x2, rev 0x1, implementer 0x43b irqs 256
>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>> 
>> So something involving BUS_PASS_INTERRUPT or later
>> (but before, say, BUS_PASS_SUPPORTDEV) may be more
>> appropriate. Possibly:
>> 
>> BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE
>> 
>> (so after gic0).
>> 
>> 
> 
> So, I'm now using . . .
> (leading whitespace possibly not accurately preserved)
> 
> 
> stable/13's source code changes are ( similarly for
> releng/13.1 ):
> 
> # git -C /usr/13S-src/ diff sys/arm/broadcom/bcm2835/bcm2835_dma.c
> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
> index cab8639bb607..6d521d6dcace 100644
> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
> @@ -766,5 +766,6 @@ static driver_t bcm_dma_driver = {
> 
>  static devclass_t bcm_dma_devclass;
> 
> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass, 0, 0);
> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass,
> +    0, 0, BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE);
>  MODULE_VERSION(bcm_dma, 1);
> 
> 
> main's [so: 14's] source code changes are:
> 
> # git -C /usr/main-src/ diff sys/arm/broadcom/bcm2835/bcm2835_dma.c
> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
> index 5f9ecb0b7981..d901447df1e9 100644
> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
> @@ -764,5 +764,6 @@ static driver_t bcm_dma_driver = {
>         sizeof(struct bcm_dma_softc),
>  };
> 
> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0);
> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0,
> +    BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE);
>  MODULE_VERSION(bcm_dma, 1);
> 
> ===
> Mark Millard
> marklmi at yahoo.com