Re: Saving environment variables in u-boot

On 2021-Dec-17, at 19:12, Mark Millard <marklmi@yahoo.com> wrote:

> On 2021-Dec-17, at 19:02, Mark Millard via freebsd-arm <freebsd-arm@freebsd.org> wrote:
> 
>> On 2021-Dec-17, at 16:59, bob prohaska <fbsd@www.zefox.net> wrote:
>> 
>>> On Thu, Dec 16, 2021 at 11:00:46PM -0800, Mark Millard wrote:
>>>> On 2021-Dec-16, at 17:36, bob prohaska <fbsd@www.zefox.net> wrote:
>>>> 
>>>>> 
>>>>> Yes, the microSD contains a dd of 
>>>>> FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img
>>>>> The DOS partition is writeable, AFAIK.
>>>> 
>>>> Hmm. That means there is also a UFS root with
>>>> a kernel and world on the microsd card. Both
>>>> the microsd card and the USB drive contain
>>>> contain such, as well as each having an
>>>> /etc/fstab (and so on).
>>>> 
>>>> Having multiple such makes for a mess for
>>>> controlling which is used (and knowing which
>>>> is used). What is the first stage that you
>>>> made sure the USB drive was in use and how
>>>> did you make sure?
>>>> 
>>> 
>>> I installed a microSD containing only the FAT slice from 
>>> FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img, the FreeBSD
>>> slice being deleted using gpart. There is a timeout file.
>> 
>> Ultimately, you may need to publish a full capture of the
>> debug output that goes to the serial console when such
>> debug output has been fully enabled. This could end up
>> being for each of several variations tried.
>> 
>> Did you still have the MSDOSFS material on the USB in a
>> viable form as well?
>> 
>> https://www.raspberrypi.com/documentation/computers/config_txt.html
>> reports that there are boot options that can go in config.txt :
>> 
>> bootcode_delay
>> boot_delay
>> boot_delay_ms
>> 
>> Quoting . . .
>> 
>> QUOTE
>> bootcode_delay
>> 
>> The bootcode_delay command delays for a given number of seconds in bootcode.bin before loading start.elf: the default value is 0.
>> END QUOTE
>> 
>> QUOTE
>> boot_delay
>> 
>> The boot_delay command instructs to wait for a given number of seconds in start.elf before loading the kernel: the default value is 1. The total delay in milliseconds is calculated as (1000 x boot_delay) + boot_delay_ms. This can be useful if your SD card needs a while to get ready before Linux is able to boot from it.
>> 
>> boot_delay_ms
>> 
>> The boot_delay_ms command means wait for a given number of milliseconds in start.elf, together with boot_delay, before loading the kernel. The default value is 0.
>> END QUOTE
>> 
>> (So boot_delay_ms is only for smaller scale adjustments and can
>> be ignored here.)
>> 
>> Here "loading the kernel" means what config.txt was told was
>> the kernel: some *u-boot*.bin for the FreeBSD context.
>> 
>> 
>> So one experiment is to have only bootcode.bin and a config.txt
>> on the microsd card's MSDOSFS, specifying a significantly
>> large value for bootcode_delay . The hope would be to be able
>> to get the start*.elf file and the rest from the USB drive.
>> 
>> If that does not work, then the next is to have the RPi* firmware
>> only on the microsd card's MSDOSFS and have config.txt also
>> specify a significantly large value for boot_delay . This
>> first variation would not have a *u-boot*.bin or the FreeBSD
>> loader on the microsd card's MSDOSFS but only on the USB
>> drive's MSDOSFS.
>> 
>> If that does not work, then the next variation moves just the
>> *u-boot*.bin file to the microsd card.
>> 
>> (If that fails then controlling U-Boot's delays can be involved.
>> For now I only list the path of leaving U-Boot's delays as they
>> are: one short branch of the exploration.)
>> 
>> If that does not work, then the next variation moves just the
>> FreeBSD loader file to the microsd card.
> 
> For the above one I missed a word, making things unclear. So,
> instead:
> 
> If that does not work, then the next variation ALSO moves just
> the FreeBSD loader file to the microsd card. ( *u-boot*.bin
> stays on the microsd card from the prior step. )
> 
>> With that I'll stop listing things to test, pending results on
>> what I've listed.
>> 
>>> A hands-off warm boot reports
>>> resetting system ... 
>>> 
>>> U-Boot 2020.10 (Apr 09 2021 - 03:55:54 +0000)
>>> 
>>> DRAM:  948 MiB
>>> RPI 3 Model B (0xa02082)
>>> MMC:   mmc@7e300000: 0
>>> Loading Environment from FAT... In:    serial
>>> Out:   vidconsole
>>> Err:   vidconsole
>>> Net:   No ethernet found.
>>> starting USB...
>>> Bus usb@7e980000: USB DWC2
>>> scanning bus usb@7e980000 for devices... 5 USB Device(s) found
>>>     scanning usb for storage devices... 0 Storage Device(s) found
>> 
>> I would need to also see the RPi* firmware's prior
>> debug output to be able to interpret this. I can not
>> tell for sure which device's U-Boot copy is operating
>> for the above. I'd guess that is the the micrsd card's
>> MSDOSFS one.
>> 
>>> Hit any key to stop autoboot:  0 
>>> switch to partitions #0, OK
>>> mmc0 is current device
>>> Scanning mmc 0:1...
>> 
>> This looks to be getting a copy of the FreeBSD loader
>> from the microsd card's MSDOSFS.
>> 
>> The FreeBSD loader will only see the same Storage Devices
>> that U-Boot did: it gets the information from U-Boot.
>> 
>>> Found EFI removable media binary efi/boot/bootaa64.efi
>> 
>> There was a FreeBSD loader in the microsd card's
>> MSDOSFS.
>> 
>>> libfdt fdt_check_header(): FDT_ERR_BADMAGIC
>>> Scanning disk mmc@7e300000.blk...
>>> Found 2 disks
>>> No EFI system partition
>>> BootOrder not defined
>>> EFI boot manager: Cannot load any image
>>> 1259212 bytes read in 125 ms (9.6 MiB/s)
>>> libfdt fdt_check_header(): FDT_ERR_BADMAGIC
>>> Booting /efi\boot\bootaa64.efi
>> 
>> That FreeBSD loader is what it is using, not one
>> from the USB drive.
>> 
>>> [whitespace deleted]
>>> Consoles: EFI console  
>>>  Reading loader env vars from /efi/freebsd/loader.env
>>> Setting currdev to disk0p1:
>> 
>> "disk0p1" is U-Boot terminology for a drive, despite
>> this being the FreeBSD loader's output. The FreeBSD
>> loader uses the U-Boot information, not FreeBSD's
>> information.
>> 
>>> FreeBSD/arm64 EFI loader, Revision 1.1
>>> 
>>> Command line arguments: loader.efi
>>> Image base: 0x39e03000
>>> EFI version: 2.80
>>> EFI Firmware: Das U-Boot (rev 8224.4096)
>>> Console: comconsole (0)
>>> Load Path: /efi\boot\bootaa64.efi
>>> Load Device: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/SD(2)/SD(0)/HD(1,0x01,0,0x81f,0x18fa8)
>>> Trying ESP: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/SD(2)/SD(0)/HD(1,0x01,0,0x81f,0x18fa8)
>> 
>> This is reporting where the FreeBSD loader was found.
>> It indicates that disk0p1 was the microsd card's
>> MSDOSFS.
>> 
>>> Setting currdev to disk0p1:
>> 
>> The microsd card's MSDOSFS again.
>> 
>>> ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
>> 
>> There is no FreeBSD directory-tree there to find things
>> like /boot/lua/loader.lua in.
>> 
>> Currently we are trying to have the /boot/lua/loader.lua come
>> from the USB drive's UFS file system, not the microsd card.
>> 
>>> Type '?' for a list of commands, 'help' for more detailed help.
>>> OK 
>>> 
>>> AIUI, disk0p1 is the microSD, which has no /boot/lua/loader.lua
>>> That makes sense. If u-boot finds the USB mass storage device
>>> then running 
>>> run bootcmd_usb0 starts the system successfully.
>>> 
>>> It does appear that the FAT partition on the USB disk is involved.
>> 
>> I see no evidence of that above. (Below is a different context.)
>> 
>>> If I hide the contents of da0s1 by placing them in a subdirectory
>>> the boot fails, even if u-boot has found the USB disk:
>>> 
>>> Hit any key to stop autoboot:  0 
>>> U-Boot>   usb reset
>>> resetting USB...
>>> Bus usb@7e980000: USB DWC2
>>> scanning bus usb@7e980000 for devices... 6 USB Device(s) found
>>>     scanning usb for storage devices... 0 Storage Device(s) found
>>> 
>>> [Note that the six devices include the disk, it can be seen in usb tree.
>>> For some reason it isn't recognized as a mass storage device.]
>>> 
>>> U-Boot> editenv usb_pgood_delay
>>> edit: 2
>>> U-Boot> usb reset
>>> resetting USB...
>>> Bus usb@7e980000: USB DWC2
>>> scanning bus usb@7e980000 for devices... 6 USB Device(s) found
>>>     scanning usb for storage devices... 1 Storage Device(s) found
>>> U-Boot> run bootcmd_usb0
>>> 
>>> Device 0: Vendor: SABRENT  Rev: 1214 Prod: 
>>>          Type: Hard Disk
>>>          Capacity: 953869.7 MB = 931.5 GB (1953525168 x 512)
>>> ... is now current device
>>> Scanning usb 0:1...
>>> U-Boot> 
>>> and there it stops. Looks like both FAT partitions have a role to play.
>> 
>> That is not clear to me. But I'd rather be lookning into
>> earlier stages at this point, so that "1 Storage Device(s) found"
>> hopefully ends up being automatic.
>> 
>> If we get that to be automatic, then what is happening here
>> becomes the next area to investigate. But the investigation
>> is not far enough along to justify investigating this now.
>> 
>>> An earlier experiment tried booting the microSD card in a USB adapter,
>>> That worked correctly with nothing in the microSD slot, so the internal
>>> "boot from USB" feature might work if it could be slowed down sufficiently. 
>> 
>> Agreed, that the first thing is to get it to start up
>> the disk and wait for it. There may be more at issue
>> after that.
>> 
>>>> The order of activity is:
>>>> 
>>>> MSDOSFS: (all on one of: microsd card vs. usb drive)
>>>> RPi* firmware (I do not report the file-level sequencing)
>>>> U-Boot
>>>> FreeBSD loader (which uses information from U-Boot)
>>>> 
>>>> UFS: (both: together on one of: microsd card vs. usb drive)
>>>> FreeBSD kernel
>>>> FreeBSD world
>>>> 
>>>> (I do not specify which MSDOSFS is used when
>>>> there are multiple viable ones in separate places.
>>>> Similarly for UFS.)
>>>> 
>>> From the experiment today it seems both are required.
>> 
>> I do not get such an interpretation from your experiment.
>> Until "1 Storage Device(s) found" happens automatically
>> the context is not nicely comparable for the later stages.
>> 
>>> The first discovers the USB disk, the second uses that
>>> information to proceed further. 
>>> 
>>>> 
>>>> (It is technically possible to have kernel vs.
>>>> world in separate UFS partitions, possibly on
>>>> separate media. I've an example context that I
>>>> deal with that way to avoid a U-boot limitation
>>>> for the device: the kernel can find the world
>>>> where the U-Boot/FreeBSD-Loader can not. (The
>>>> FreeBSD loader depends on what USB can find: it
>>>> does not look elsewhere.) I only mention this
>>>> in case I need to reference it in the future,
>>>> avoiding a surprise in such a case. Otherwise
>>>> ignore the complication.)
>>>> 
>>> 
>>> Are you referring to a case where the root is on
>>> microSD but /usr and friends is on a USB device? 
>> 
>> Again: I suggest ignoring the details of this for
>> now. It would just confuse things. But if you
>> insist:
>> 
>> On the Rock64 I have a e.MMC in use instead of a
>> microsd card, and in the end the world is on a
>> USB3 SSD. But the U-Boot involved can not access
>> the USB3 --os neither can the FreeBSD loader.
>> The FreeBSD kernel is the first stage that can
>> access USB3.
>> 
>> The e.MMC has the MSDOSFS. It has an partial
>> copy of what would normally be some of the USB3
>> drive's content. (I named the mount point
>> microsd_ufs before I switched to the e.MMC .)
>> It has only (much hidden in ". . ."s).
>> 
>> # find /microsd_ufs -print | sort | less
>> /microsd_ufs
>> /microsd_ufs/.snap
>> /microsd_ufs/COPYRIGHT
>> /microsd_ufs/boot
>> /microsd_ufs/boot/beastie.4th
>> /microsd_ufs/boot/boot1.efi
>> /microsd_ufs/boot/brand-fbsd.4th
>> /microsd_ufs/boot/brand.4th
>> /microsd_ufs/boot/check-password.4th
>> /microsd_ufs/boot/color.4th
>> /microsd_ufs/boot/copy_boot_to_microsd.sh
>> /microsd_ufs/boot/defaults
>> /microsd_ufs/boot/defaults/loader.conf
>> /microsd_ufs/boot/delay.4th
>> /microsd_ufs/boot/dtb
>> . . .
>> /microsd_ufs/boot/kernel
>> /microsd_ufs/boot/kernel/accf_data.ko
>> /microsd_ufs/boot/kernel/accf_dns.ko
>> . . .
>> /microsd_ufs/boot/kernel/kernel
>> . . .
>> /microsd_ufs/boot/loader.4th
>> /microsd_ufs/boot/loader.conf
>> /microsd_ufs/boot/loader.conf.d
>> /microsd_ufs/boot/loader.efi
>> /microsd_ufs/boot/loader.help
>> /microsd_ufs/boot/loader.rc
>> /microsd_ufs/boot/loader_4th.efi
>> /microsd_ufs/boot/loader_lua.efi
>> /microsd_ufs/boot/loader_simp.efi
>> /microsd_ufs/boot/logo-beastie.4th
>> /microsd_ufs/boot/logo-beastiebw.4th
>> /microsd_ufs/boot/logo-fbsdbw.4th
>> /microsd_ufs/boot/logo-orb.4th
>> /microsd_ufs/boot/logo-orbbw.4th
>> /microsd_ufs/boot/lua
>> /microsd_ufs/boot/lua/cli.lua
>> /microsd_ufs/boot/lua/color.lua
>> /microsd_ufs/boot/lua/config.lua
>> /microsd_ufs/boot/lua/core.lua
>> /microsd_ufs/boot/lua/drawer.lua
>> /microsd_ufs/boot/lua/gfx-beastie.lua
>> /microsd_ufs/boot/lua/gfx-beastiebw.lua
>> /microsd_ufs/boot/lua/gfx-fbsdbw.lua
>> /microsd_ufs/boot/lua/gfx-orb.lua
>> /microsd_ufs/boot/lua/gfx-orbbw.lua
>> /microsd_ufs/boot/lua/hook.lua
>> /microsd_ufs/boot/lua/loader.lua
>> /microsd_ufs/boot/lua/menu.lua
>> /microsd_ufs/boot/lua/password.lua
>> /microsd_ufs/boot/lua/screen.lua
>> /microsd_ufs/boot/menu-commands.4th
>> /microsd_ufs/boot/menu.4th
>> /microsd_ufs/boot/menu.rc
>> /microsd_ufs/boot/menusets.4th
>> /microsd_ufs/boot/modules
>> /microsd_ufs/boot/screen.4th
>> /microsd_ufs/boot/shortcuts.4th
>> /microsd_ufs/boot/support.4th
>> /microsd_ufs/boot/uboot
>> /microsd_ufs/boot/version.4th
>> /microsd_ufs/boot/zfs
>> /microsd_ufs/etc
>> /microsd_ufs/etc/hostid
>> /microsd_ufs/lost+found
>> 
>> Almost no world content. /microsd_ufs/boot/loader.conf
>> has (in part):
>> 
>> vfs.root.mountfrom="ufs:/dev/gpt/Rock64root"
>> kern.cam.boot_delay=10000
>> vfs.mountroot.timeout=10
>> vfs.root_mount_always_wait=1
>> 
>> That vfs.root.mountfrom notation redirects to the
>> USB3 drive for world but the /boot/loader.conf and
>> /boot/kernel/kernel and /etc/hostid were first
>> loaded from the e.MMC media, not the USB drive.
>> 
>> As I have it, the USB3 drive also has a copy of
>> the same /boot/... and such materials and load
>> module activity is from the USB3 copy. I have to
>> keep the two places tracking so nothing ends up
>> mismatching.
>> 
>>>> You might move (not copy) the MSDOSFS material
>>>> between the microsd card and the usb drive during
>>>> experiments, avoiding having duplications. There
>>>> is the possibility of instead renaming things so
>>>> files are not found on a partition, for example.
>>>> A similar point goes for UFS materials: avoid
>>>> having multiple viable-for-booting UFS file
>>>> systems present.
>>>> 
>>>> As stands, things seem too hard to track for me
>>>> to be of much help. Please make things obvious for
>>>> what was in use by making the material uniquely
>>>> placed (for the form that can be used).
>>>> 
>>> I believe the experiment above (deleting UFS on microSD, 
>>> hiding the DOS files on USB) has been an equivalent 
>>> configuration. It looks like control passes in some way 
>>> between the DOS slices, though how is unclear.
>> 
>> I do not agree to that specific interpretation of the
>> sequence you did (on the evidence currently available).
>> Until "1 Storage Device(s) found" is automatic this
>> later-stage material is too late to yet be relevant
>> or to have an appropriate context.
>> 
>> I'm staying focused on getting the "1 Storage Device(s)
>> found" to be automatic. Absent that you are likely stuck
>> with doing something similar to be Rock64 e.MMC way of
>> working --where /boot/loader.conf and /boot/kernel/kernel
>> and /etc/hostid for early activity is from a UFS
>> partition on the microsd card.
>> 
>>>> Separate issues/questions:
>>>> 
>>>> Do you have the file "timeout" in the MSDOSFS, in
>>>> addition to config.txt and the like? If not, I
>>>> recommend including it.
>>>> 
>>> Yes, it's been tried on both microSD and USB devices. 
>>> Seems like the only one that can matter is on microSD.
>>> 
>>> 
>>>> What other RPi* firmware controls for having
>>>> various deliberate RPi* firmware delays do you
>>>> have set up?
>>>> 
>>> 
>>> The Raspberry Pi config.txt description lists several delay commands
>>> that can be placed in config.txt. None seem related to USB discovery,
>>> might any come into play early enough to be useful? They're named
>>> bootcode_delay
>>> boot_delay
>>> boot_delay_ms
>>> 
>>> I tried them casually and didn't notice much change. Are they worth
>>> revisiting more carefully? 
>> 
>> See my earlier notes.
>> 
>>>> It is not so much that these would be sufficient,
>>>> but they do establish some context before U-Boot
>>>> is even active. It could be important to
>>>> understand that context. (Unsure at this point.)
>>>> 
>>>> 
>>>>>> But that was for the u-boot-rpi4 or u-boot-rpi-arm64 ports.
>>>>>> (They also later mentioned using "usb_pgood_delay=2000\0"
>>>>>> instead, a figure they found in a bunch of configrations.)
>>>>>> 
>>>>> The Pi3 in question is capable of booting from solid-state USB
>>>>> storage without any microSD card, but fails to detect a mechanical
>>>>> disk. Which is the appropriate u-boot-rpi3 port to tamper with? I 
>>>>> tried sysutils/u-boot-rpi3 as an upgrade but that simply froze. 
>>>>> The u-boot from FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img finds
>>>>> the USB mechanical disk,  but erratically. 
>>>> 
>>>> FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img uses u-boot-rpi-arm64
>>>> as I remember: it is set up for both the RPI3 and RPI4. Given that
>>>> it works some, that would be the U-Boot port I would experiment
>>>> with if I was doing the experiments.
>>>> 
>>>>>> So something somewhat analogous might help if you are willing
>>>>>> to build and use your own u-boot port variant.
>>>>> 
>>>>> Obviously, that's a fraught enterprise at my skill level....
>>>>> I'm still somewhat hazy on the actual boot sequence when
>>>>> chaining from microSD to USB. 
>>>> 
>>>> Having the extra text in the U-Boot build does not really
>>>> depend on the chaining order question.
>>>> 
>>>> But, to repeat (sticking to the simpler context),
>>>> the order is:
>>>> 
>>>> MSDOSFS: (all on one of: microsd card vs. usb drive)
>>>> RPi* firmware (I do not report the file-level sequencing)
>>>> U-Boot
>>>> FreeBSD loader (which uses information from U-Boot)
>>>> 
>>>> UFS: (both: together on one of: microsd card vs. usb drive)
>>>> FreeBSD kernel
>>>> FreeBSD world
>>>> 
>>>> 
>>>>> Indeed, it's unclear how or if u-boot plays a role in starting 
>>>>> RasPiOS. The term u-boot isn't found on the Raspberry Pi doc 
>>>>> website, and the Pi isn't mentioned in the Denx manuals. Those
>>>>> discoveries surprised me.
>>>> 
>>>> RaspiOS and RaspiOS64 do not use U-boot. The RPi*'s can boot some
>>>> forms of a linux kernel directly and that is what RaspiOS and
>>>> RaspiOS64 are doing. 
>>> 
>>> That clears up a major misunderstanding, thank you!
>>> 
>>>> That is why the config.txt type of content
>>>> makes no mention of u-boot or of any kernel= assignment in RaspiOS
>>>> and RaspiOS64. By contrast, config.txt for FreeBSD lists kernel=
>>>> and an explicit *u-boot*.bin as the value.
>>> 
>>> Without thinking about it very carefully I tried to use the
>>> "bootcode.bin-only" method for booting an older Pi2 from a
>>> usb hard disk, as described in
>>> https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#special-bootcode-bin-only-boot-mode
>>> It worked on the Pi2 but failed on the Pi3, causing an immediate hang.
>> 
>> This gets back into the use of a config.txt with a
>> bootcode_delay assignment also being in the MSDOSFS
>> on the microsd card and the file timeout also being
>> in the MSDOSFS on the microsd card. Only if those
>> delays together can lead to the USB drive being
>> accessible will it get any farther.
>> 
>> I'd suggest such experiments. The vintage of bootcode.bin
>> matters as I understand.
>> 
>>> Might that avenue be worth pursuing? In hindsight, that it worked on a
>>> Pi2 is quite surprising.  
>>> 
>> 

One point for testing that could be a simplification
initially: booting just from a microsd card with the
USB drive powered and attached already, even though
the USB drive is not boot-media for this kind of test.

The goal would be to get the boot to report something
other than than the 0 in:

scanning usb for storage devices... 0 Storage Device(s) found

That line is not a count of bootable USB storage devices, but
a count of all the accessible USB storage devices found. it
reports useful information even for booting from a microsd
card, including reporting if the USB device was not accessible.

This could be used to learn what controls the issue without
moving things between the USB media and the microsd card.
(Once learned, there could be more to do.)


I should have noted: so far the only option that
has evidence for being sufficient is having a build
of *u-boot*.bin that has usb_pgood_delay already
assigned. I'm not aware of UEFI style U-Boot's
reading in any configuration files or executing any
scripts to control usb_pgood_delay from the outside.


===
Mark Millard
marklmi at yahoo.com