Booting from USB on RPI3

[Mark Millard]

On 2020-Apr-23, at 19:18, bob prohaska <fbsd at www.zefox.net> wrote:

> On Thu, Apr 23, 2020 at 06:16:14PM -0700, Mark Millard wrote:
>> On 2020-Apr-23, at 16:32, bob prohaska <fbsd at www.zefox.net> wrote:
>> 
>>> On Thu, Apr 23, 2020 at 03:24:57PM -0700, Mark Millard wrote:
>>>> 
>>>> 
>>>> 
>>>> For USB-only or for microsd card also in use,
>>>> have you tried the equivalent of:
>>>> 
>>>> OK lsdev
>>>> 
>>> Another surprise. First time I tried, after some time idle:
>>> OK lsdev
>>> unknown command
>>> but a simple 
>>> ls
>>> listed the contents of the root directory. Apparently the
>>> drive was asleep after an idle timeout.
>>> 
>>> A second attempt at lsdev produced
>>> OK lsdev
>>> disk devices:
>>>   disk0:    250085377 X 512 blocks (removable)
>>>     disk0s1: DOS/Windows
>>>     disk0s2: FreeBSD
>>>       disk0s2a: FreeBSD UFS
>>>       disk0s2b: FreeBSD swap
>>>   disk1:    78140161 X 512 blocks
>>>     disk1s1: DOS/Windows
>>>     disk1s2: FreeBSD
>>>       disk1s2a: FreeBSD UFS
>>>       disk1s2b: FreeBSD swap
>>> http: (unknown)
>>> net devices:
>>>   net0:
>> 
>> Where both the microsd card (disk0) and
>> the USB drive (disk1) set up via installing
>> FreeBSD distributed .iso's, at least for
>> how the file systems were created (even
>> if somewhat adjusted later)?
>> 
> (thinking you meant "were both...")

True. Dumb typo.

> The answer is yes, both were written with dd from a downloaded .img file.
> 
>> If yes, it may be that you have two instances
>> of each of:
>> 
>> /dev/msdosfs/MSDOSBOOT
>> /dev/ufs/rootfs
>> 
>> one per disk* for each.
>> 
> Yes, but if I'm quick enough stopping u-boot wouldn't
> only one be read per boot attempt? 

If you stop the boot at:

Hit any key to stop autoboot:  2  1  0 

then you are in u-boot/EFI before the FreeBSD
loader is loaded/in-use. This is a very
different context and does not involve or
support the various types of labels or
notations from FreeBSD. I assume you are
not referring to this rather early stage
of the boot sequence.

FreeBSD looks at disks before mounting them
and it looks at partitions/slices before they
are mounted. It creates the bindings for
the labels of various types (msdosfs, ufs,
gpt, label) before mounting is even possible.
It does so across all the media present at
the time. This is so all the labels are
available to use to pick what to mount (in
FreeBSD terms). But it means that the loader
has to pick what to do about ambiguous labeling
before you are involved.

(I've no clue if it keeps just the first-find
or last-find or what. But in the end a label
identifies only one thing, not multiple things.)

There is meta-data on the media and in the
partitions/slices/file-systems that indicates
what labels to create.

(The kernel may well repeat doing some things
that the loader had to do, for all I know.
Refinding labels could be an example.)

>> These types of naming are expecting to have
> 
>> been set up with what is after the last / being
>> a unique name relative to everything else
>> potentially present at the same time. This
>> means that two .iso installs would need
>> to have been adjusted to make the names
>> distinct before trying to use the media
>> together.
>> 
>> If these names are duplicated, then use of
>> the notations in /etc/fstab would be ambiguous
>> for identifying the drives and I've no clue
>> which instance wins --or if which wins is even
>> stable.
>> 
>> For ufs, tunefs has an option for setting the
>> file system label:
>> 
>>     -L volname
>>             Add/modify an optional file system volume label.  Legal
>>             characters are alphanumerics, dashes, and underscores.
>> 
>> Once the adjustment is seen (remount/reboot?),
>> that should make:
>> 
>> /dev/ufs/...
>> 
>> change to the new name.
>> 
>> I'm only aware of file systsem creation-time setting
>> for msdos via newfs_msdos :
>> 
>>     -L label
>>             Volume label (up to 11 characters).  The label should consist of
>>             only those characters permitted in regular DOS (8+3) filenames.
>> 
>> That should make
>> 
>> /dev/msdosfs/...
>> 
>> be the assigned name. But newfs_msdos would not
>> preserve the contents of the file system that
>> was present before (if any).
>> 
>> glabel can be used to assign labels that are
>> independent of the file system and of the
>> partition metadata, labels that would appear
>> under /dev/label/ instead of under /dev/ufs/
>> or /dev/msdosfs/ or /dev/gpt/ . (gpt is not
>> an option for MBR contexts.)
>> 
>> 
>>>> and then used the naming convention shown in
>>>> the output for any loader commands/settings
>>>> instead of using FreeBSD-stage da0 based naming?
>>>> (You might want to report what lsdev shows.)
>>>> 
>>>> Note: the disk0 and disk1 like prefixes are
>>>> copied from the u-boot notations that the
>>>> loader gets from u-boot, if I understand
>>>> right.
>>>> 
>>>> Does your loader.conf do anything to control
>>>> what is booted? 
>>> 
>>> Not in a way I recognize, but you might:
>>> OK more /boot/loader.conf
>>> *** FILE /boot/loader.conf BEGIN ***
>>> # Configure USB OTG; see usb_template(4).
>>> hw.usb.template=3
>>> umodem_load="YES"
>>> # Multiple console (serial+efi gop) enabled.
>>> boot_multicons="YES"
>>> boot_serial="YES"
>>> # Disable the beastie menu and color
>>> beastie_disable="YES"
>>> loader_color="NO"
>>> *** FILE /boot/loader.conf END ***
>> 
>> Nothing there to control what is booted.
>> 
>>> As Joanathan asked: what does
>>>> your /etc/fstab have listed? (/etc/fstab should
>>>> have FreeBSD notations, not loader device names.)
>>> 
>>> OK more /etc/fstab
>>> *** FILE /etc/fstab BEGIN ***
>>> # Custom /etc/fstab for FreeBSD embedded images
>>> /dev/ufs/rootfs   /       ufs     rw      1       1
>>> /dev/msdosfs/MSDOSBOOT /boot/msdos msdosfs rw,noatime 0 0
>>> tmpfs /tmp tmpfs rw,mode=1777,size=50m 0 0
>>> *** FILE /etc/fstab END ***
>>> OK 
>> 
>> See earlier notes: each media may be defining
>> what causes /dev/ufs/rootfs , leading to ambiguity.
>> Similarly for /dev/msdosfs/MSDOSBOOT .
>> 
>> /etc/fstab should use notation that is a unique
>> identification for each device. The media should
>> have names assigned that are unique compared to
>> anything else potentially connected at the same
>> time if labeling style names are to be used.
>> 
>> Names like /dev/da* are not stable from boot to
>> boot if multiple USB media are present at the
>> same time: which media gets which da* name could
>> vary from boot to boot. Using some form of unique
>> label assignment is a way of avoiding this.
>> 
>> However, if you stick to only one USB drive
>> being present, likely /dev/da0s2a would be
>> a stable USB drive ufs file system reference
>> for that context. Similarly for /dev/da0s1
>> being a stable reference to the USB msdosfs .
>> 
>> /dev/mmcsd0s1 would likely also be a stable
>> microsd card msdosfs reference for your context.
>> Similarly for /dev/mmcsd0s2a for being a stable
>> reference to your microsd card ufs file system.
>> 
>> For now such notations may be better for your
>> /etc/fstab file(s).
>> 
> That seems to be key. Using /dev/da... notation in /etc/fstab
> on the USB device allows a successful boot. One very important
> detail that I forgot was adding kern.cam.boot_delay="20000"
> to /boot/loader.conf. Otherwise boots are successful only if
> something else (like me) delays the boot process. The erratic
> results observed likely owe much to that oversight.

As I remember there is a RPi* way of increasing its
2 second wait for a device on a port to indicate it
is present. If you end up with problems despite the
FreeBSD stage materials indicating delays, you might
have to have the RPi3 increase its delay. The RPi3
activity for this is before FreeBSD is involved,
possibly even before u-boot is involved.

>>>> Are there any other boot-contributing files that
>>>> would be appropriate to show together in one
>>>> message for review, including any on the
>>>> msdos file system?
>>>> 
>>> 
>>> I don't know how to answer that question. Since the machine
>>> has gotten to loader and it's looking at the USB drive it's
>>> very tempting to think the msdos part is done. I'll admit
>>> that  names like /dev/ufs/rootfs seem rather obscure.
>>> Are they required for RAID?
>>> 
>> 
>> An example: does either media have in its msdos file
>> system: /EFI/FreeBSD/loader.env ? If yes, are the
>> settings present in loader.env that contribute to what
>> boots?
>> 
> Apparently not by default; nothing in microSD, but I added 
> a /boot/loader.env to the USB device, holding rootdev=disk1p2:
> It didn't seem to affect booting behavior, at least not on
> the first try. 

There is no reason to have a /boot/loader.env as far
as I know. It would be ignored: that is the wrong
place for the file.

EFI/FreeBSD/loader.env in the msdos file system is
a way to supply input to EFI/BOOT/bootaa64.efi (the
FreeBSD loader) before the loader tries to find the
FreeBSD /boot it is going to use --and before you
can type to the loader. That is why Jonathan was
indicating to create the rootdev assignment there.
(That rootdev is then used to find the /boot to use,
as I understand things.)

>> (That is something from Jonathan's exchanges with
>> you.)
>> 
>> 
>> A separate question is: Other than bootcode.bin ,
>> are the contents in the two msdos files systems
>> identical?
>> 
>> 
> Clearly not. Those on the the USB device are about six months
> newer and different in size.

You may want to make your msdosfs content matching and
keep them matching as they progress, at least for
anything that you do not have a specific reason to make
a distinction for. (Avoids odd variations in behavior.)

> For now it looks as if the immediate problem is solved. Both
> microSD and USB storage boot and run. 

Does that wording mean that you got USB-only to
work too?

Now that you can boot, you should be able to show
the ufs labels (and more) with:

tunefs -p /dev/mmcsd0s2a
tunefs -p /dev/da0s2a

The outputs will each have a line starting with:

tunefs: volume label: (-L)

and ending with the label (or empty for no label).

If the two volume labels match, you should
change at least one such that they are
distinct. After that future boots should
end up with two separate /dev/ufs/* names
that you can use to pick the ufs file system
with. The -L is the command line option
involved in specifying a volume/file-system
label.

Unfortunately, I'm not aware of anything
directly in FreeBSD analogous to "tunefs -p"
or to "tunefs -L NAME . . .". for the msdos
file systems. There seems to be only
newfs_msdos --which would wipe out the existing
msdosfs content by creating a new file system.
But it does have a -L option for indicating the
volume/file-system label to put in place.

May be after doing such the content of the
other msdos file system couple be copied into
the new/empty file system. (Not via dd. dd'ing
the slice would replace the label too, not just
the content placed in the file system.)


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)