Re: Using loader to switch kernel and root device on Pi2

On Mar 30, 2025, at 22:55, Mark Millard <marklmi@yahoo.com> wrote:

> [This is just a resend with the subject line fixed.
> How/when I had turned into just "UORE", I do not
> know.]
> 
> On Mar 30, 2025, at 21:17, bob prohaska <fbsd@www.zefox.net> wrote:
> 
> Note: My initial wording is based on not doing
> any hacking to form a work around. Later I have
> notes based on hacking a work around based on
> your providing more description of your overall
> issue.
> 
>> On Sun, Mar 30, 2025 at 06:49:48PM -0700, Mark Millard wrote:
>>> On Mar 30, 2025, at 16:40, bob prohaska <fbsd@www.zefox.net> wrote:
>>> 
>>>> An old Raspberry Pi2 v1.1 (so, armv7) has quit booting from USB when
>>>> using the "bootcode.bin" method (that file only on the MSDOS partition,
>>>> the usual layout on the USB disk). The machine gets stuck in a netboot
>>>> loop. 
>>>> 
>>>> However, it appears possible to boot using the microSD card, then load 
>>>> the kernel and mount the root filesystem from USB at the loader prompt:
>>>> 
>>>> Hit [Enter] to boot immediately, or any other key for command prompt.
>>>> Booting [/boot/kernel/kernel] in 10 seconds... 
>>>> 
>>>> Type '?' for a list of commands, 'help' for more detailed help.
>>>> OK lsdev
>>>> disk devices:
>>> 
>>> The below disks look like they might be 2
>>> separate media, each separately being
>>> bootable.
>>> 
>>>> disk0:    1953525168 X 512 blocks
>>>>   disk0s1: DOS/Windows
>>> 
>>> Does this have a bootcode.bin and other
>>> RPi* firmware related files? U-Boot?
>>> 
>>> The above being a EFI msdsofs, likely with
>>> its own EFI/BOOT/*.efi file.
>>> 
>>>>   disk0s2: FreeBSD
>>>>     disk0s2a: FreeBSD UFS
>>> 
>>> The above being a UFS-based / file system,
>>> likely with its own /boot/ .
>>> 
>>>>     disk0s2b: FreeBSD swap
>>>>     disk0s2d: FreeBSD UFS
>>> 
>>> It is unclear if the above also has a /boot/
>>> as well.
>>> 
>>>> disk1:    125042688 X 512 blocks (removable)
>>>>   disk1s1: DOS/Windows
>>> 
>>> Does this also have a bootcode.bin ? If yes,
>>> the same vintage as above?  What about other
>>> RPi* firmware related files? U-Boot?
>>> 
>>> The above also being another EFI msdsofs,
>>> likely with its own EFI/BOOT/*.efi file?
>>> 
>>>>    disk1s2: FreeBSD
>>>>     disk1s2a: FreeBSD UFS
>>> 
>>> The above being another UFS-based / file
>>> system, likely with its own /boot/ ?
>>> 
>>> 
>>> It that the case? The below is only for for the
>>> context: "yes" to there being multiple variants
>>> of various directories and files used in booting,
>>> which might not be the actual type of context.
>>> 
>> 
>> Both disks are complete FreeBSD installations.
>> Disk0 is a USB mechanical hard disk. Disk1 is
>> a microSD card with a FreeBSD image downloaded 
>> from snapshots.
> 
> Ahh, that USB vs. microsd card context is
> important to interpreting things for your
> problem. (Later notes by you make it even
> more so.)
> 
>> Files from that image populated
>> the usb disk initially, before updating via source.
> 
> Ahh. The RPi2 v1.1 has a means of controlling
> microsd vs. USB being first. But USB here is a
> bus, not a specific device (unless there is only
> one USB storage media present on the USB bus).
> 
> The way I read the following, bootcode.bin (and
> possibly timeout) are the only RPi* firmware files
> that should be on the microsd card (in standard
> places) in order to then do a RPI* firmware
> based USB host boot:
> 
> QUOTE
> Format an SD card as FAT32 and copy over the latest bootcode.bin.
> The SD card must be present in the Raspberry Pi for it to boot.
> Once bootcode.bin is loaded from the SD card, the Raspberry Pi
> continues booting using USB host mode.
> 
> This is useful for the Raspberry Pi 1, 2, and Zero models, which
> are based on the BCM2835 and BCM2836 chips, and in situations
> where a Raspberry Pi 3 fails to boot (the latest bootcode.bin
> includes additional bugfixes for the Raspberry Pi 3B, compared
> to the boot code burned into the BCM2837A0).
> If you have a problem with a mass storage device still not
> working, even with this bootcode.bin, then add a new file called
> "timeout" to the SD card. This will extend to six seconds the
> time for which it waits for the mass storage device to initialise.
> END QUOTE
> 
> If you have more of the RPi* firmware files on the
> microsd card, it may try to boot from the microsd
> card instead.
> 
> NOTE added later:
> Later notes are tied to having the FreeBSD
> world loaded from USB by a FreeBSD kernel not
> boot-time loaded from USB but from the microsd
> card. Note that such is not an example of the
> RPi* firmware booting from USB at all. Until
> later my notes are generally based on presuming
> a RPI* firmware based USB boot.
> 
>> The numbering seems backwards to me, I'd expect
>> disk0 to be microSD and usb to be disk1, but no.
> 
> I expect that the disk0 vs. disk1 part of the
> naming is from U-Boot's naming assignments.
> 
>>> If yes, then how do you expect the RPi2 to
>>> pick which EFI msdosfs it uses each time? (That
>>> is just one type of example "pick" operation.)
>>> Does it need to be more stable than random for
>>> which it picks?
>>> 
>> 
>> In practice, the Raspberry Pi boots from the microSD
>> card, even if a second bootable device is present.
> 
> For an RPi firmware based USB boot, things are
> different than hacking in a non-RPI* firmware
> based USB boot that does a partial USB based
> boot in a later stage.
> 
> 
>>> Once it picks, how do you expect the loader to
>>> pick which UFS-based /boot/ it should use? Does it
>>> need to be more stable than random?
>>> 
>> 
>> I pick which ufs is used, via the
>> OK set currdev="disk0s2a"
>> OK set vfs.root.mountfrom="ufs:/dev/da0s2a"
> 
> That is only via a manual intervention, not the
> automatic boot handling.
> 
>>> Note that which /boot/loader.conf file would be
>>> used depends on which /boot/ is used if there are
>>> multiple /boot/ directories present.
>> 
>> Disk 1 (microSD) is active at the time of booting, 
>> so its /boot/loader.conf will contain the modifications.
> 
> I'll have more realted notes later.
> 
>>> As I understand things, whichever  EFT/BOOT/*.efi
>>> is used, a /boot/ from the same media (not the
>>> same partition or file system) is used. If there
>>> is more than one such /boot/ on that media
>>> (via distinct file systems on the media), then
>>> the question of which /boot/ is picked is
>>> repeated.
>>> 
>>> Can you make the disk1s1 such that is not a valid
>>> RPi2* firmware source and not a valid EFI msdosfs?
>>> An example could be renaming the EFI/BOOT/*.efi
>>> file to be a name that is not one that would be
>>> found as a standard .efi file name to be used --so
>>> that it would be ignored.
>> 
>> No, because that is the firmware and loader that the
>> Pi is able to start. It won't even try to boot usb.
> 
> The only 2 files that are needed on the microsd card
> for an RPi* firmware based USB boot are: bootcode.bin
> and, possibly, timeout .
> 
> It is the bootcode.bin file that enables the RPi*
> fimrware based USB boot. No other files contribute
> (other than timeout leading to a longer delay).
> 
> However, you later provide notes indicating that
> you need the USB activity to be in a later stage
> than the RPi* firmware doing the USB activity.
> 
>>>> http: (unknown)
>>>> net devices:
>>>> net0:
>>>> OK unload
>>>> OK show currdev
>>>> disk1s2a:
>>>> OK set currdev="disk0s2a"
>>>> OK show currdev
>>>> disk0s2a
>>>> OK load kernel
>>>> /boot/kernel/kernel text=0x1b4 text=0x6df178 text=0x1f9cb8 data=0xc37a8 data=0x0+0x254000 0x4+0xb1af0+0x4+0x134365|
>>>> OK set vfs.root.mountfrom="ufs:/dev/da0s2a"
>>> 
>>> You did not first show the value of
>>> vfs.root.mountfrom from before you
>>> changed it. Which /boot/ would it
>>> have found if you had not changed
>>> the value?
>> 
>> By default the Pi boots from the microSD, disk1 in
>> this context. Presumably that means the value of
>> vfs.root.mountfrom would have been  /dev/mmcsd0s2a
>> or some equivalent label pointing to the same device.
> 
> A probably hypothesis, given disk1 being the
> microsd card that has a lot of files that it
> should not have on it for USB booting: just
> bootcode.bin and, possibly, timeout relative
> to RPi* firmware (in places were it such
> would be found).
> 
>>>> OK boot -s
>>>> Using DTB provided by EFI at 0x39f23000.
>>>> Kernel entry at 0x33e00200...
>>>> Kernel args: -s
>>>> .....
>>>> 
>>>> The machine comes up with kernel and root from the USB disk, as desired.
>>>> 
>>>> It appears that the 
>>>> currdev="disk0s2a"
>>>> and 
>>>> vfs.root.mountfrom="ufs:/dev/da0s2a"
>>>> 
>>>> could be placed in /boot/loader.conf,
>>>> but where would the unload and load
>>>> commands go?
>>> 
>>> Is there more than one /boot/ and, so, more
>>> than one /boot/loader.conf that might be
>>> used from one boot attempt to the next? If
>>> yes, how do you know which /boot/loader.conf
>>> to edit?
>>> 
>>> Getting which EFI/BOOT/*.efi and /boot/
>>> picking to be stable and as-desired is
>>> something to have in place before worrying
>>> about the content of any specific
>>> /boot/loader.conf or other such file.
>>> (Presumes you do not make all the various
>>> examples [if multiple] be duplicates of
>>> each other for all the files would end up
>>> involved.)
>>> 
>>>> Without them the machine
>>>> is stuck with the microSD's kernel.
>>> 
>>> You should only have one RPI* firmware
>>> soruce and only one EFI/BOOT/*.efi (that
>>> has the right filenaming convention) and
>>> only one /boot/ in normal usage contexts.
>>> Having only one /boot/ means having only
>>> one /boot/kernel/kernel as well. There
>>> should not be alternatives that might
>>> instead be found and used.
>>> 
>> That harks back to having bootcode.bin alone on
>> the microSD dos partition, which then finds and
>> boots the usb disk. That's how the system started
>> out and worked through several cycles of self-hosting.
>> After an upgrade bootcode.bin couldn't find the usb
>> disk and got stuck in a netboot loop from which it
>> couldn't be extricated. A fresh install behaved the
>> same way; worked fine through a few builworld/kernel
>> cycles, then started trying only to netboot.
> 
> I think the above context was missing in the earlier
> information (above and the prior Email).
> 
> This a different problem: hacking a workaround
> so that a later, non-RPi* firmware stage, does
> a partial USB based boot after some stages of
> microsd card based boot activity.
> 
>> The usual suspects (power supplies, bad cables, bad
>> usb-sata bridges and faulty microSD cards) were ruled
>> out by experiment.
> 
> Did your testing include testing with the timeout
> file being present?
> 
>> A correspondent on comp.sys.raspberry-pi suggested
>> that maybe the FreeBSD loader installed on the 
>> microSD could be coaxed into booting the usb disk,
>> since booting the microSD works reliably. That seems
>> to be true, at least for now.
> 
> I've used USB hardware that the FreeBSD kernel
> could handle but the prior U-Boot stage could
> not. (It was not an RPi* context.) Such things
> lead to hacks to make things limp along.
> 
> In that case I duplicated /boot/ (and a few more
> files) onto the microsd card in a UFS file
> system and had the RPi* firmware, U-Boot, and
> EFI/BOOT/*.efi in the microsd card's msdosfs.
> (I did not want later modules to be loaded from
> the microsd card but from the USB media.) One
> of the duplicated files had the vfs.root.mountfrom
> to use to find the USB media.
> 
> I did not duplicate the rest of the world: that
> was only on the USB media.
> 
> Every update needed to make sure that things
> were duplicated accurately. Some files had to
> be handled specially.
> 
>> currdev="disk0s2a"
>> and
>> vfs.root.mountfrom="ufs:/dev/da0s2a"
>> both look like easy additions to /boot/loader.conf
>> on the microSD card.  
>> 
>> The question that's gone unanswered is where to put
>> the "unload" command to clear out the kernel picked
>> up during loader startup so that it will be replaced
>> by the new kernel from the usb disk.
> 
> What I did had no unload or reload required: I used
> the duplication of content in to 2 places to avoid
> such activity being needed. The initial load of
> the kernel was exactly the right content and compatible
> with any later kernel module loads done from the USB
> media.
> 
>> It's evident my question is somewhat out of left field.
>> All I really meant to ask was how to configure an
>> unload command via loader.conf. The man page does not
>> mention "unload", must it go elsewhere?
> 
> I suggest avoiding the need to unload and reload
> by setting up to have the initial load have the
> correct content in the first place. The tradeoff
> is needing to keep the duplicated information
> on the microsd card up to date at every update.
> 
> Note: Eventually U-Boot updated to handled the
> USB context that it originally did not and I was
> able to stop using the hackish technique.
> 
> So it has been a while since I've last dealt with
> this sort of thing.


I no longer have media organized for the
kernel-stage USB world part of the boot
for the Rock64. Nor do I seem to have a
copy of the old instructions/reminders
file that I used while updating.

Some of the below words things as-if the
Rock64 context was more like the RPi*
context than it actually was.

Some old notes indicate:

Having a UFS file system in a partition/slice
on the microsd card in order for it to 
contain:

) A recursive copy of /boot/

) But I avoided keeping a copy of /boot/entropy on
  the microsd card as I remember. This could be a
   security issue.

) A copy of /etc/hostid (as that path is accessed
  very early, much like /boot/loader.conf is
  accessed very early, before the normal root
  file system mount).

I used rsync with a delete option and an exclude
option to deal with the recursive copies as I
remember.

The /boot/loader.conf had a:

vfs.root.mountfrom="ufs:/dev/gpt/Rock64root"

One of the considerations was if the tree at

/usr/lib/debug/boot/

should also be copied to the microsd card's UFS
partition in order to have better problem analysis
context for kernel boot failures.

I can not claim to know if the above notes
are complete vs. not.

===
Mark Millard
marklmi at yahoo.com