Re: Using loader to switch kernel and root device on Pi2

[This is just a resend with the subject line fixed.
How/when I had turned into just "UORE", I do not
know.]

On Mar 30, 2025, at 21:17, bob prohaska <fbsd@www.zefox.net> wrote:

Note: My initial wording is based on not doing
any hacking to form a work around. Later I have
notes based on hacking a work around based on
your providing more description of your overall
issue.

> On Sun, Mar 30, 2025 at 06:49:48PM -0700, Mark Millard wrote:
>> On Mar 30, 2025, at 16:40, bob prohaska <fbsd@www.zefox.net> wrote:
>> 
>>> An old Raspberry Pi2 v1.1 (so, armv7) has quit booting from USB when
>>> using the "bootcode.bin" method (that file only on the MSDOS partition,
>>> the usual layout on the USB disk). The machine gets stuck in a netboot
>>> loop. 
>>> 
>>> However, it appears possible to boot using the microSD card, then load 
>>> the kernel and mount the root filesystem from USB at the loader prompt:
>>> 
>>> Hit [Enter] to boot immediately, or any other key for command prompt.
>>> Booting [/boot/kernel/kernel] in 10 seconds... 
>>> 
>>> Type '?' for a list of commands, 'help' for more detailed help.
>>> OK lsdev
>>> disk devices:
>> 
>> The below disks look like they might be 2
>> separate media, each separately being
>> bootable.
>> 
>>>  disk0:    1953525168 X 512 blocks
>>>    disk0s1: DOS/Windows
>> 
>> Does this have a bootcode.bin and other
>> RPi* firmware related files? U-Boot?
>> 
>> The above being a EFI msdsofs, likely with
>> its own EFI/BOOT/*.efi file.
>> 
>>>    disk0s2: FreeBSD
>>>      disk0s2a: FreeBSD UFS
>> 
>> The above being a UFS-based / file system,
>> likely with its own /boot/ .
>> 
>>>      disk0s2b: FreeBSD swap
>>>      disk0s2d: FreeBSD UFS
>> 
>> It is unclear if the above also has a /boot/
>> as well.
>> 
>>>  disk1:    125042688 X 512 blocks (removable)
>>>    disk1s1: DOS/Windows
>> 
>> Does this also have a bootcode.bin ? If yes,
>> the same vintage as above?  What about other
>> RPi* firmware related files? U-Boot?
>> 
>> The above also being another EFI msdsofs,
>> likely with its own EFI/BOOT/*.efi file?
>> 
>>>     disk1s2: FreeBSD
>>>      disk1s2a: FreeBSD UFS
>> 
>> The above being another UFS-based / file
>> system, likely with its own /boot/ ?
>> 
>> 
>> It that the case? The below is only for for the
>> context: "yes" to there being multiple variants
>> of various directories and files used in booting,
>> which might not be the actual type of context.
>> 
> 
> Both disks are complete FreeBSD installations.
> Disk0 is a USB mechanical hard disk. Disk1 is
> a microSD card with a FreeBSD image downloaded 
> from snapshots.

Ahh, that USB vs. microsd card context is
important to interpreting things for your
problem. (Later notes by you make it even
more so.)

> Files from that image populated
> the usb disk initially, before updating via source.

Ahh. The RPi2 v1.1 has a means of controlling
microsd vs. USB being first. But USB here is a
bus, not a specific device (unless there is only
one USB storage media present on the USB bus).

The way I read the following, bootcode.bin (and
possibly timeout) are the only RPi* firmware files
that should be on the microsd card (in standard
places) in order to then do a RPI* firmware
based USB host boot:

QUOTE
Format an SD card as FAT32 and copy over the latest bootcode.bin.
The SD card must be present in the Raspberry Pi for it to boot.
Once bootcode.bin is loaded from the SD card, the Raspberry Pi
continues booting using USB host mode.

This is useful for the Raspberry Pi 1, 2, and Zero models, which
are based on the BCM2835 and BCM2836 chips, and in situations
where a Raspberry Pi 3 fails to boot (the latest bootcode.bin
includes additional bugfixes for the Raspberry Pi 3B, compared
to the boot code burned into the BCM2837A0).
If you have a problem with a mass storage device still not
working, even with this bootcode.bin, then add a new file called
"timeout" to the SD card. This will extend to six seconds the
time for which it waits for the mass storage device to initialise.
END QUOTE

If you have more of the RPi* firmware files on the
microsd card, it may try to boot from the microsd
card instead.

NOTE added later:
Later notes are tied to having the FreeBSD
world loaded from USB by a FreeBSD kernel not
boot-time loaded from USB but from the microsd
card. Note that such is not an example of the
RPi* firmware booting from USB at all. Until
later my notes are generally based on presuming
a RPI* firmware based USB boot.

> The numbering seems backwards to me, I'd expect
> disk0 to be microSD and usb to be disk1, but no.

I expect that the disk0 vs. disk1 part of the
naming is from U-Boot's naming assignments.

>> If yes, then how do you expect the RPi2 to
>> pick which EFI msdosfs it uses each time? (That
>> is just one type of example "pick" operation.)
>> Does it need to be more stable than random for
>> which it picks?
>> 
> 
> In practice, the Raspberry Pi boots from the microSD
> card, even if a second bootable device is present.

For an RPi firmware based USB boot, things are
different than hacking in a non-RPI* firmware
based USB boot that does a partial USB based
boot in a later stage.


>> Once it picks, how do you expect the loader to
>> pick which UFS-based /boot/ it should use? Does it
>> need to be more stable than random?
>> 
> 
> I pick which ufs is used, via the
> OK set currdev="disk0s2a"
> OK set vfs.root.mountfrom="ufs:/dev/da0s2a"

That is only via a manual intervention, not the
automatic boot handling.

>> Note that which /boot/loader.conf file would be
>> used depends on which /boot/ is used if there are
>> multiple /boot/ directories present.
> 
> Disk 1 (microSD) is active at the time of booting, 
> so its /boot/loader.conf will contain the modifications.

I'll have more realted notes later.

>> As I understand things, whichever  EFT/BOOT/*.efi
>> is used, a /boot/ from the same media (not the
>> same partition or file system) is used. If there
>> is more than one such /boot/ on that media
>> (via distinct file systems on the media), then
>> the question of which /boot/ is picked is
>> repeated.
>> 
>> Can you make the disk1s1 such that is not a valid
>> RPi2* firmware source and not a valid EFI msdosfs?
>> An example could be renaming the EFI/BOOT/*.efi
>> file to be a name that is not one that would be
>> found as a standard .efi file name to be used --so
>> that it would be ignored.
> 
> No, because that is the firmware and loader that the
> Pi is able to start. It won't even try to boot usb.

The only 2 files that are needed on the microsd card
for an RPi* firmware based USB boot are: bootcode.bin
and, possibly, timeout .

It is the bootcode.bin file that enables the RPi*
fimrware based USB boot. No other files contribute
(other than timeout leading to a longer delay).

However, you later provide notes indicating that
you need the USB activity to be in a later stage
than the RPi* firmware doing the USB activity.

>>> http: (unknown)
>>> net devices:
>>>  net0:
>>> OK unload
>>> OK show currdev
>>> disk1s2a:
>>> OK set currdev="disk0s2a"
>>> OK show currdev
>>> disk0s2a
>>> OK load kernel
>>> /boot/kernel/kernel text=0x1b4 text=0x6df178 text=0x1f9cb8 data=0xc37a8 data=0x0+0x254000 0x4+0xb1af0+0x4+0x134365|
>>> OK set vfs.root.mountfrom="ufs:/dev/da0s2a"
>> 
>> You did not first show the value of
>> vfs.root.mountfrom from before you
>> changed it. Which /boot/ would it
>> have found if you had not changed
>> the value?
> 
> By default the Pi boots from the microSD, disk1 in
> this context. Presumably that means the value of
> vfs.root.mountfrom would have been  /dev/mmcsd0s2a
> or some equivalent label pointing to the same device.

A probably hypothesis, given disk1 being the
microsd card that has a lot of files that it
should not have on it for USB booting: just
bootcode.bin and, possibly, timeout relative
to RPi* firmware (in places were it such
would be found).

>>> OK boot -s
>>> Using DTB provided by EFI at 0x39f23000.
>>> Kernel entry at 0x33e00200...
>>> Kernel args: -s
>>> .....
>>> 
>>> The machine comes up with kernel and root from the USB disk, as desired.
>>> 
>>> It appears that the 
>>> currdev="disk0s2a"
>>> and 
>>> vfs.root.mountfrom="ufs:/dev/da0s2a"
>>> 
>>> could be placed in /boot/loader.conf,
>>> but where would the unload and load
>>> commands go?
>> 
>> Is there more than one /boot/ and, so, more
>> than one /boot/loader.conf that might be
>> used from one boot attempt to the next? If
>> yes, how do you know which /boot/loader.conf
>> to edit?
>> 
>> Getting which EFI/BOOT/*.efi and /boot/
>> picking to be stable and as-desired is
>> something to have in place before worrying
>> about the content of any specific
>> /boot/loader.conf or other such file.
>> (Presumes you do not make all the various
>> examples [if multiple] be duplicates of
>> each other for all the files would end up
>> involved.)
>> 
>>> Without them the machine
>>> is stuck with the microSD's kernel.
>> 
>> You should only have one RPI* firmware
>> soruce and only one EFI/BOOT/*.efi (that
>> has the right filenaming convention) and
>> only one /boot/ in normal usage contexts.
>> Having only one /boot/ means having only
>> one /boot/kernel/kernel as well. There
>> should not be alternatives that might
>> instead be found and used.
>> 
> That harks back to having bootcode.bin alone on
> the microSD dos partition, which then finds and
> boots the usb disk. That's how the system started
> out and worked through several cycles of self-hosting.
> After an upgrade bootcode.bin couldn't find the usb
> disk and got stuck in a netboot loop from which it
> couldn't be extricated. A fresh install behaved the
> same way; worked fine through a few builworld/kernel
> cycles, then started trying only to netboot.

I think the above context was missing in the earlier
information (above and the prior Email).

This a different problem: hacking a workaround
so that a later, non-RPi* firmware stage, does
a partial USB based boot after some stages of
microsd card based boot activity.

> The usual suspects (power supplies, bad cables, bad
> usb-sata bridges and faulty microSD cards) were ruled
> out by experiment.

Did your testing include testing with the timeout
file being present?

> A correspondent on comp.sys.raspberry-pi suggested
> that maybe the FreeBSD loader installed on the 
> microSD could be coaxed into booting the usb disk,
> since booting the microSD works reliably. That seems
> to be true, at least for now.

I've used USB hardware that the FreeBSD kernel
could handle but the prior U-Boot stage could
not. (It was not an RPi* context.) Such things
lead to hacks to make things limp along.

In that case I duplicated /boot/ (and a few more
files) onto the microsd card in a UFS file
system and had the RPi* firmware, U-Boot, and
EFI/BOOT/*.efi in the microsd card's msdosfs.
(I did not want later modules to be loaded from
the microsd card but from the USB media.) One
of the duplicated files had the vfs.root.mountfrom
to use to find the USB media.

I did not duplicate the rest of the world: that
was only on the USB media.

Every update needed to make sure that things
were duplicated accurately. Some files had to
be handled specially.

> currdev="disk0s2a"
> and
> vfs.root.mountfrom="ufs:/dev/da0s2a"
> both look like easy additions to /boot/loader.conf
> on the microSD card.  
> 
> The question that's gone unanswered is where to put
> the "unload" command to clear out the kernel picked
> up during loader startup so that it will be replaced
> by the new kernel from the usb disk.

What I did had no unload or reload required: I used
the duplication of content in to 2 places to avoid
such activity being needed. The initial load of
the kernel was exactly the right content and compatible
with any later kernel module loads done from the USB
media.

> It's evident my question is somewhat out of left field.
> All I really meant to ask was how to configure an
> unload command via loader.conf. The man page does not
> mention "unload", must it go elsewhere?

I suggest avoiding the need to unload and reload
by setting up to have the initial load have the
correct content in the first place. The tradeoff
is needing to keep the duplicated information
on the microsd card up to date at every update.

Note: Eventually U-Boot updated to handled the
USB context that it originally did not and I was
able to stop using the hackish technique.

So it has been a while since I've last dealt with
this sort of thing.


===
Mark Millard
marklmi at yahoo.com