Re: Help / Information request regarding FreeBSD boot on Raspberry Pi CM4

Hi Hugo,

since you wrote me in private I will forward this to the list while trying to hide your private email(hope it works).
But I would recommend you to signup for the list(if not yet done) here :
https://lists.freebsd.org/subscription/freebsd-arm and please follow-up answers  always forward to there.
, since this issue is from common interest.

O.K., like you I made a u-boot compilation with nvme stuff built and have now tested everything extensively.
I can confirm your (excellent reported) issue.

 While we now can say that the excellent patch from H.P van Braam  fixed the pcie kernel panic on cm4  & also cm4Lite 
we have a new issue with non-bootable nvme.

To answer your question exactly with a 1st estimation :

No, this is not a u-boot issue.
The compiled-in nvme-stuff doesn`t affect the detection of pcie & nvme, pcie & nvme is recognized on every u-boot-compilation ,
also in the untouched default from the current snapshot :

U-Boot> pci
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x14e4     0x2711     Bridge device           0x04
01.00.00   0x15b7     0x5009     Mass storage controller 0x08

In this case you see VendorID 0x15b7 is my WesternDigital nvme-drive.

By using some special rpi-tools I can confirm that nvme is definitely detected by  following (non fbsd-)OS bootups.

By performing a FreeBSD pxe(network)-boot and going to loader prompt I can see that pcie/nvme is NOT detected 
in the required early bootstage to bootup FreeBSD from the hard drive .

Which now leads me to a 1st idea for a possible  patch :

We need the pcie-driver to perform as something like a 

EARLY_DRIVER_MODULE_ORDERED

( https://man.freebsd.org/cgi/man.cgi?query=DRIVER_MODULE&sektion=9&n=1 )

Since something like that is on one side not a big thing it`s on the other side perhaps a sensitive thing to whole aarch64-port I would consult some involved experts 
before 
I would recommend you(Hugo) to perhaps file a new bug report here :

https://bugs.freebsd.org/bugzilla/

Recommendation :
This message also is forwarded directly to some FreeBSD guys who know a bit (or more) of the history of this pcie-driver.
Let`s wait 2 or 3 days for an answer , if nothing happens then you can file a (new) bug report.

Regards

Klaus
( written most without Google Translator ;-) Ha ha 



> Am 19.05.2025 um 17:49 schrieb Hugo Kirnbichler <xxx@xxx>:
> 
> Hello Klaus,
>     thanks for your speedy reply.
>   I've repeated my test with 
>       FreeBSD-15.0-CURRENT-arm64-aarch64-RPI-20250515-cb205f5ed808-277278.img.xz
>   This I transferred onto a NVMe SSD, but my system still doesn't boot. u-boot still fails to find a NVMe SSD.
>   I'm aware that u-boot isn't really part of FreeBSD, but as it comes with the snapshot, I assume that it's supposed to work.
>   The CM4 itself is configured to boot from NVMe (else it couldn't load u-boot).
>     What's working now (and I presume that's what your patch is about) is FreeBSD booting from SD card whilst having a NVMe SSD attached.
>   The system comes up and the SSD can be accessed via /dev/nda0.
>   This a a great improvement.
>     But I gather from the comment in your bug report that you have mastered booting from NVMe.
>   I've taken the liberty to attach the various boot messages that appear on the serial debug port.
>   Maybe you could point me in the correct direction, u-boot-wise?
>     Again, thanks a lot!
>   Regards,
> Hugo Kirnbichler
>             Sent: Monday, May 19, 2025 at 4:35 PM
> From: "Klaus Küchemann" <maciphone2@googlemail.com>
> To: "Hugo Kirnbichler“ <xxx@xxx>, freebsd-arm@freebsd.org
> Subject: Re: Help / Information request regarding FreeBSD boot on Raspberry Pi CM4
> Hi,
> 
> a quick check revealed that the patch was never MFC 'd (Merge From Current).
> So if you want to test it directly, I recommend using a snapshot of the main branch (FreeBSD 15).
> 
> Regards
> K.
> 
> 
> > Am 19.05.2025 um 13:38 schrieb Hugo Kirnbichler <xxx@xxx.xx.xx>:
> >
> > Hello Mr. Küchemann,
> > I've got your email address from the FreeBSD mailing list.
> > Trying to get FreeBSD to boot from a NVMe SSD attached to a Raspberry Pi Compute Module 4 Lite,
> > I've come across your bug report:
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260131#c0
> > in which you state that you seem to have managed to get FreeBSD to boot in aforementioned scenario.
> > This is a feat which I've failed to achieve. I managed to get it to boot from an SD card, then it works fine,
> > but only as long as there's no NVMe SSD connected.
> 
> > I'm running FreeBSD 13.3 (FreeBSD-13.3-RELEASE-arm64-aarch64-RPI.img.xz), using
> > ………
> ...devmatch_enable="NO" 
>  in /etc/rc.conf
> Using the same image with the same configuration on a NVMe SSDs stops at u-boot failing to load anything from the NVMe SSD.
 >The CM4 Lite is configured to boot from NVMe, as a test with the current release of the "raspberry pi os" shows.
 >Upon being unable to find the NVMe SSD, u-boot tries to boot from network.
> When booting from the SD card with NVMe SSD attached, FreeBSD begins to boot but runs into a kernel panic upon trying to
>access a PCI-PCI bridge on pcib1 ...
> This is how I came across your bug report in the first place.
> Detailed hardware setup:
> Raspberry Pi Compute Module 4 Lite (4 GB, no WIFI, no EMMC)
>Raspberry Pi Compute Module 4 I/O Board
>NVMe SSD on PCIe adapter board
> Boot order on CM4 "NVMe first, then SD card, repeat"
> I've got a serial port hooked up to the Pi's debug uart and have it configured to get GPU firmware boot messages,
>u-boot messages and FreeBSD messages.
> I've tried newer releases of u-boot (via sysutils/u-boot-rpi4), to no avail.
>I even tried building u-boot myself (on a system running raspberry Pi os), but the documentation leaves me somwhat baffled.
>Activating the two PCI NVMe options via "make menuconfig" does not seem to be the deal.
> Is that the right rabbit hole?
  >Thus I'd be grateful if you could point me in the right direction and enlighten me about what magic you've performed.
> I hope that contacting you this way is not bothering you too much.
> 
> .
> ….
> > Thanks for reading all this,
> > sincerely,
> > Hugo Kirnbichler
> >
>