Booting from USB on RPI3

[bob prohaska]

On Thu, Apr 23, 2020 at 09:26:53PM -0700, Mark Millard wrote:
> On 2020-Apr-23, at 19:18, bob prohaska <fbsd at www.zefox.net> wrote:
> 
> > On Thu, Apr 23, 2020 at 06:16:14PM -0700, Mark Millard wrote:
> >> On 2020-Apr-23, at 16:32, bob prohaska <fbsd at www.zefox.net> wrote:
> >> 
> >>> On Thu, Apr 23, 2020 at 03:24:57PM -0700, Mark Millard wrote:
> >>>> 
> >>>> 
> >>>> 
> >>>> For USB-only or for microsd card also in use,
> >>>> have you tried the equivalent of:
> >>>> 
> >>>> OK lsdev
> >>>> 
> >>> Another surprise. First time I tried, after some time idle:
> >>> OK lsdev
> >>> unknown command
> >>> but a simple 
> >>> ls
> >>> listed the contents of the root directory. Apparently the
> >>> drive was asleep after an idle timeout.
> >>> 
> >>> A second attempt at lsdev produced
> >>> OK lsdev
> >>> disk devices:
> >>>   disk0:    250085377 X 512 blocks (removable)
> >>>     disk0s1: DOS/Windows
> >>>     disk0s2: FreeBSD
> >>>       disk0s2a: FreeBSD UFS
> >>>       disk0s2b: FreeBSD swap
> >>>   disk1:    78140161 X 512 blocks
> >>>     disk1s1: DOS/Windows
> >>>     disk1s2: FreeBSD
> >>>       disk1s2a: FreeBSD UFS
> >>>       disk1s2b: FreeBSD swap
> >>> http: (unknown)
> >>> net devices:
> >>>   net0:
> >> 
> >> Where both the microsd card (disk0) and
> >> the USB drive (disk1) set up via installing
> >> FreeBSD distributed .iso's, at least for
> >> how the file systems were created (even
> >> if somewhat adjusted later)?
> >> 
> > (thinking you meant "were both...")
> 
> True. Dumb typo.
> 
> > The answer is yes, both were written with dd from a downloaded .img file.
> > 
> >> If yes, it may be that you have two instances
> >> of each of:
> >> 
> >> /dev/msdosfs/MSDOSBOOT
> >> /dev/ufs/rootfs
> >> 
> >> one per disk* for each.
> >> 
> > Yes, but if I'm quick enough stopping u-boot wouldn't
> > only one be read per boot attempt? 
> 
> If you stop the boot at:
> 
> Hit any key to stop autoboot:  2  1  0 
> 
> then you are in u-boot/EFI before the FreeBSD
> loader is loaded/in-use. This is a very
> different context and does not involve or
> support the various types of labels or
> notations from FreeBSD. I assume you are
> not referring to this rather early stage
> of the boot sequence.
> 
There's a very early prompt from u-boot, which
must be invoked to stop booting the microSD.
That's the point where one gets the u-boot> 
prompt, and the place to run the usb reset and
run bootcmd_usb0 commands. The next chance to
intervene is at the pause before booting the
loaded kernel, which is where I was poking around.


> FreeBSD looks at disks before mounting them
> and it looks at partitions/slices before they
> are mounted. It creates the bindings for
> the labels of various types (msdosfs, ufs,
> gpt, label) before mounting is even possible.
> It does so across all the media present at
> the time. This is so all the labels are
> available to use to pick what to mount (in
> FreeBSD terms). But it means that the loader
> has to pick what to do about ambiguous labeling
> before you are involved.
> 

I suspect that had I not used a microSD with a
runnable FreeBSD installation the boot of USB
would have been trouble-free.  

> (I've no clue if it keeps just the first-find
> or last-find or what. But in the end a label
> identifies only one thing, not multiple things.)
> 
> There is meta-data on the media and in the
> partitions/slices/file-systems that indicates
> what labels to create.
> 
Are the labels created at each boot? I thought
they were defined at partitioning. 


> (The kernel may well repeat doing some things
> that the loader had to do, for all I know.
> Refinding labels could be an example.)
> 
> >> These types of naming are expecting to have
> > 
> >> been set up with what is after the last / being
> >> a unique name relative to everything else
> >> potentially present at the same time. This
> >> means that two .iso installs would need
> >> to have been adjusted to make the names
> >> distinct before trying to use the media
> >> together.
> >> 
> >> If these names are duplicated, then use of
> >> the notations in /etc/fstab would be ambiguous
> >> for identifying the drives and I've no clue
> >> which instance wins --or if which wins is even
> >> stable.
> >> 
> >> For ufs, tunefs has an option for setting the
> >> file system label:
> >> 
> >>     -L volname
> >>             Add/modify an optional file system volume label.  Legal
> >>             characters are alphanumerics, dashes, and underscores.
> >> 
> >> Once the adjustment is seen (remount/reboot?),
> >> that should make:
> >> 
> >> /dev/ufs/...
> >> 
> >> change to the new name.
> >> 
> >> I'm only aware of file systsem creation-time setting
> >> for msdos via newfs_msdos :
> >> 
> >>     -L label
> >>             Volume label (up to 11 characters).  The label should consist of
> >>             only those characters permitted in regular DOS (8+3) filenames.
> >> 
> >> That should make
> >> 
> >> /dev/msdosfs/...
> >> 
> >> be the assigned name. But newfs_msdos would not
> >> preserve the contents of the file system that
> >> was present before (if any).
> >> 
> >> glabel can be used to assign labels that are
> >> independent of the file system and of the
> >> partition metadata, labels that would appear
> >> under /dev/label/ instead of under /dev/ufs/
> >> or /dev/msdosfs/ or /dev/gpt/ . (gpt is not
> >> an option for MBR contexts.)
> >> 
> >> 
> >>>> and then used the naming convention shown in
> >>>> the output for any loader commands/settings
> >>>> instead of using FreeBSD-stage da0 based naming?
> >>>> (You might want to report what lsdev shows.)
> >>>> 
> >>>> Note: the disk0 and disk1 like prefixes are
> >>>> copied from the u-boot notations that the
> >>>> loader gets from u-boot, if I understand
> >>>> right.
> >>>> 
> >>>> Does your loader.conf do anything to control
> >>>> what is booted? 
> >>> 
> >>> Not in a way I recognize, but you might:
> >>> OK more /boot/loader.conf
> >>> *** FILE /boot/loader.conf BEGIN ***
> >>> # Configure USB OTG; see usb_template(4).
> >>> hw.usb.template=3
> >>> umodem_load="YES"
> >>> # Multiple console (serial+efi gop) enabled.
> >>> boot_multicons="YES"
> >>> boot_serial="YES"
> >>> # Disable the beastie menu and color
> >>> beastie_disable="YES"
> >>> loader_color="NO"
> >>> *** FILE /boot/loader.conf END ***
> >> 
> >> Nothing there to control what is booted.
> >> 
> >>> As Joanathan asked: what does
> >>>> your /etc/fstab have listed? (/etc/fstab should
> >>>> have FreeBSD notations, not loader device names.)
> >>> 
> >>> OK more /etc/fstab
> >>> *** FILE /etc/fstab BEGIN ***
> >>> # Custom /etc/fstab for FreeBSD embedded images
> >>> /dev/ufs/rootfs   /       ufs     rw      1       1
> >>> /dev/msdosfs/MSDOSBOOT /boot/msdos msdosfs rw,noatime 0 0
> >>> tmpfs /tmp tmpfs rw,mode=1777,size=50m 0 0
> >>> *** FILE /etc/fstab END ***
> >>> OK 
> >> 
> >> See earlier notes: each media may be defining
> >> what causes /dev/ufs/rootfs , leading to ambiguity.
> >> Similarly for /dev/msdosfs/MSDOSBOOT .
> >> 
> >> /etc/fstab should use notation that is a unique
> >> identification for each device. The media should
> >> have names assigned that are unique compared to
> >> anything else potentially connected at the same
> >> time if labeling style names are to be used.
> >> 
> >> Names like /dev/da* are not stable from boot to
> >> boot if multiple USB media are present at the
> >> same time: which media gets which da* name could
> >> vary from boot to boot. Using some form of unique
> >> label assignment is a way of avoiding this.
> >> 
> >> However, if you stick to only one USB drive
> >> being present, likely /dev/da0s2a would be
> >> a stable USB drive ufs file system reference
> >> for that context. Similarly for /dev/da0s1
> >> being a stable reference to the USB msdosfs .
> >> 
> >> /dev/mmcsd0s1 would likely also be a stable
> >> microsd card msdosfs reference for your context.
> >> Similarly for /dev/mmcsd0s2a for being a stable
> >> reference to your microsd card ufs file system.
> >> 
> >> For now such notations may be better for your
> >> /etc/fstab file(s).
> >> 
> > That seems to be key. Using /dev/da... notation in /etc/fstab
> > on the USB device allows a successful boot. One very important
> > detail that I forgot was adding kern.cam.boot_delay="20000"
> > to /boot/loader.conf. Otherwise boots are successful only if
> > something else (like me) delays the boot process. The erratic
> > results observed likely owe much to that oversight.
> 
> As I remember there is a RPi* way of increasing its
> 2 second wait for a device on a port to indicate it
> is present. If you end up with problems despite the
> FreeBSD stage materials indicating delays, you might
> have to have the RPi3 increase its delay. The RPi3
> activity for this is before FreeBSD is involved,
> possibly even before u-boot is involved.
>
There is, by adding an empty file named timeout to the
msdos partition holding bootcode.bin, but it only gives
six seconds. I think my hard drive needs about ten. 
That's why the first run of usb scan fails to see the
hard disk. Running usb reset at the u-boot prompt finds
the disk.
 
> >>>> Are there any other boot-contributing files that
> >>>> would be appropriate to show together in one
> >>>> message for review, including any on the
> >>>> msdos file system?
> >>>> 
> >>> 
> >>> I don't know how to answer that question. Since the machine
> >>> has gotten to loader and it's looking at the USB drive it's
> >>> very tempting to think the msdos part is done. I'll admit
> >>> that  names like /dev/ufs/rootfs seem rather obscure.
> >>> Are they required for RAID?
> >>> 
> >> 
> >> An example: does either media have in its msdos file
> >> system: /EFI/FreeBSD/loader.env ? If yes, are the
> >> settings present in loader.env that contribute to what
> >> boots?
> >> 
> > Apparently not by default; nothing in microSD, but I added 
> > a /boot/loader.env to the USB device, holding rootdev=disk1p2:
> > It didn't seem to affect booting behavior, at least not on
> > the first try. 
> 
> There is no reason to have a /boot/loader.env as far
> as I know. It would be ignored: that is the wrong
> place for the file.
> 
Indeed, my mistake. And, in hindsight, probably unhelpful.

> EFI/FreeBSD/loader.env in the msdos file system is
> a way to supply input to EFI/BOOT/bootaa64.efi (the
> FreeBSD loader) before the loader tries to find the
> FreeBSD /boot it is going to use --and before you
> can type to the loader. That is why Jonathan was
> indicating to create the rootdev assignment there.
> (That rootdev is then used to find the /boot to use,
> as I understand things.)
> 
> >> (That is something from Jonathan's exchanges with
> >> you.)
> >> 
> >> 
> >> A separate question is: Other than bootcode.bin ,
> >> are the contents in the two msdos files systems
> >> identical?
> >> 
> >> 
> > Clearly not. Those on the the USB device are about six months
> > newer and different in size.
> 
> You may want to make your msdosfs content matching and
> keep them matching as they progress, at least for
> anything that you do not have a specific reason to make
> a distinction for. (Avoids odd variations in behavior.)
> 
> > For now it looks as if the immediate problem is solved. Both
> > microSD and USB storage boot and run. 
> 
> Does that wording mean that you got USB-only to
> work too?
>
Yes, bootstrapping _through_ microSD. 

> Now that you can boot, you should be able to show
> the ufs labels (and more) with:
> 
> tunefs -p /dev/mmcsd0s2a
> tunefs -p /dev/da0s2a
> 
> The outputs will each have a line starting with:
> 
> tunefs: volume label: (-L)
> 
> and ending with the label (or empty for no label).
> 
> If the two volume labels match, you should
> change at least one such that they are
> distinct. After that future boots should
> end up with two separate /dev/ufs/* names
> that you can use to pick the ufs file system
> with. The -L is the command line option
> involved in specifying a volume/file-system
> label.
>
Both microSD and USB root filesystems have the same
label. That seems to be the crux of the problem.
For now hard-coding device file names in /etc/fstab
is probably the most idiot-proof method. 

> Unfortunately, I'm not aware of anything
> directly in FreeBSD analogous to "tunefs -p"
> or to "tunefs -L NAME . . .". for the msdos
> file systems. There seems to be only
> newfs_msdos --which would wipe out the existing
> msdosfs content by creating a new file system.
> But it does have a -L option for indicating the
> volume/file-system label to put in place.
> 
> May be after doing such the content of the
> other msdos file system couple be copied into
> the new/empty file system. (Not via dd. dd'ing
> the slice would replace the label too, not just
> the content placed in the file system.)

Once, a long time ago, FreeBSD had a boot manager
that lived, I think, in the msdos partition of each
bootable hard disk. On startup, it presented a menu,
something like
1 msdos
2 freebsd
3 next disk in the boot order
defaulting to whatever was last selected. I think
that recollection gave me false hope that dual-booting
a Pi would be much simpler than it turned out.

Thanks for your help and patience!

bob prohaska