Re: Saving environment variables in u-boot

From: Mark Millard via freebsd-arm <freebsd-arm_at_freebsd.org>
Date: Sat, 18 Dec 2021 03:12:32 UTC
On 2021-Dec-17, at 19:02, Mark Millard via freebsd-arm <freebsd-arm@freebsd.org> wrote:

> On 2021-Dec-17, at 16:59, bob prohaska <fbsd@www.zefox.net> wrote:
> 
>> On Thu, Dec 16, 2021 at 11:00:46PM -0800, Mark Millard wrote:
>>> On 2021-Dec-16, at 17:36, bob prohaska <fbsd@www.zefox.net> wrote:
>>> 
>>>> 
>>>> Yes, the microSD contains a dd of 
>>>> FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img
>>>> The DOS partition is writeable, AFAIK.
>>> 
>>> Hmm. That means there is also a UFS root with
>>> a kernel and world on the microsd card. Both
>>> the microsd card and the USB drive contain
>>> contain such, as well as each having an
>>> /etc/fstab (and so on).
>>> 
>>> Having multiple such makes for a mess for
>>> controlling which is used (and knowing which
>>> is used). What is the first stage that you
>>> made sure the USB drive was in use and how
>>> did you make sure?
>>> 
>> 
>> I installed a microSD containing only the FAT slice from 
>> FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img, the FreeBSD
>> slice being deleted using gpart. There is a timeout file.
> 
> Ultimately, you may need to publish a full capture of the
> debug output that goes to the serial console when such
> debug output has been fully enabled. This could end up
> being for each of several variations tried.
> 
> Did you still have the MSDOSFS material on the USB in a
> viable form as well?
> 
> https://www.raspberrypi.com/documentation/computers/config_txt.html
> reports that there are boot options that can go in config.txt :
> 
> bootcode_delay
> boot_delay
> boot_delay_ms
> 
> Quoting . . .
> 
> QUOTE
> bootcode_delay
> 
> The bootcode_delay command delays for a given number of seconds in bootcode.bin before loading start.elf: the default value is 0.
> END QUOTE
> 
> QUOTE
> boot_delay
> 
> The boot_delay command instructs to wait for a given number of seconds in start.elf before loading the kernel: the default value is 1. The total delay in milliseconds is calculated as (1000 x boot_delay) + boot_delay_ms. This can be useful if your SD card needs a while to get ready before Linux is able to boot from it.
> 
> boot_delay_ms
> 
> The boot_delay_ms command means wait for a given number of milliseconds in start.elf, together with boot_delay, before loading the kernel. The default value is 0.
> END QUOTE
> 
> (So boot_delay_ms is only for smaller scale adjustments and can
> be ignored here.)
> 
> Here "loading the kernel" means what config.txt was told was
> the kernel: some *u-boot*.bin for the FreeBSD context.
> 
> 
> So one experiment is to have only bootcode.bin and a config.txt
> on the microsd card's MSDOSFS, specifying a significantly
> large value for bootcode_delay . The hope would be to be able
> to get the start*.elf file and the rest from the USB drive.
> 
> If that does not work, then the next is to have the RPi* firmware
> only on the microsd card's MSDOSFS and have config.txt also
> specify a significantly large value for boot_delay . This
> first variation would not have a *u-boot*.bin or the FreeBSD
> loader on the microsd card's MSDOSFS but only on the USB
> drive's MSDOSFS.
> 
> If that does not work, then the next variation moves just the
> *u-boot*.bin file to the microsd card.
> 
> (If that fails then controlling U-Boot's delays can be involved.
> For now I only list the path of leaving U-Boot's delays as they
> are: one short branch of the exploration.)
> 
> If that does not work, then the next variation moves just the
> FreeBSD loader file to the microsd card.

For the above one I missed a word, making things unclear. So,
instead:

If that does not work, then the next variation ALSO moves just
the FreeBSD loader file to the microsd card. ( *u-boot*.bin
stays on the microsd card from the prior step. )

> With that I'll stop listing things to test, pending results on
> what I've listed.
> 
>> A hands-off warm boot reports
>> resetting system ... 
>> 
>> U-Boot 2020.10 (Apr 09 2021 - 03:55:54 +0000)
>> 
>> DRAM:  948 MiB
>> RPI 3 Model B (0xa02082)
>> MMC:   mmc@7e300000: 0
>> Loading Environment from FAT... In:    serial
>> Out:   vidconsole
>> Err:   vidconsole
>> Net:   No ethernet found.
>> starting USB...
>> Bus usb@7e980000: USB DWC2
>> scanning bus usb@7e980000 for devices... 5 USB Device(s) found
>>      scanning usb for storage devices... 0 Storage Device(s) found
> 
> I would need to also see the RPi* firmware's prior
> debug output to be able to interpret this. I can not
> tell for sure which device's U-Boot copy is operating
> for the above. I'd guess that is the the micrsd card's
> MSDOSFS one.
> 
>> Hit any key to stop autoboot:  0 
>> switch to partitions #0, OK
>> mmc0 is current device
>> Scanning mmc 0:1...
> 
> This looks to be getting a copy of the FreeBSD loader
> from the microsd card's MSDOSFS.
> 
> The FreeBSD loader will only see the same Storage Devices
> that U-Boot did: it gets the information from U-Boot.
> 
>> Found EFI removable media binary efi/boot/bootaa64.efi
> 
> There was a FreeBSD loader in the microsd card's
> MSDOSFS.
> 
>> libfdt fdt_check_header(): FDT_ERR_BADMAGIC
>> Scanning disk mmc@7e300000.blk...
>> Found 2 disks
>> No EFI system partition
>> BootOrder not defined
>> EFI boot manager: Cannot load any image
>> 1259212 bytes read in 125 ms (9.6 MiB/s)
>> libfdt fdt_check_header(): FDT_ERR_BADMAGIC
>> Booting /efi\boot\bootaa64.efi
> 
> That FreeBSD loader is what it is using, not one
> from the USB drive.
> 
>> [whitespace deleted]
>> Consoles: EFI console  
>>   Reading loader env vars from /efi/freebsd/loader.env
>> Setting currdev to disk0p1:
> 
> "disk0p1" is U-Boot terminology for a drive, despite
> this being the FreeBSD loader's output. The FreeBSD
> loader uses the U-Boot information, not FreeBSD's
> information.
> 
>> FreeBSD/arm64 EFI loader, Revision 1.1
>> 
>>  Command line arguments: loader.efi
>>  Image base: 0x39e03000
>>  EFI version: 2.80
>>  EFI Firmware: Das U-Boot (rev 8224.4096)
>>  Console: comconsole (0)
>>  Load Path: /efi\boot\bootaa64.efi
>>  Load Device: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/SD(2)/SD(0)/HD(1,0x01,0,0x81f,0x18fa8)
>> Trying ESP: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/SD(2)/SD(0)/HD(1,0x01,0,0x81f,0x18fa8)
> 
> This is reporting where the FreeBSD loader was found.
> It indicates that disk0p1 was the microsd card's
> MSDOSFS.
> 
>> Setting currdev to disk0p1:
> 
> The microsd card's MSDOSFS again.
> 
>> ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
> 
> There is no FreeBSD directory-tree there to find things
> like /boot/lua/loader.lua in.
> 
> Currently we are trying to have the /boot/lua/loader.lua come
> from the USB drive's UFS file system, not the microsd card.
> 
>> Type '?' for a list of commands, 'help' for more detailed help.
>> OK 
>> 
>> AIUI, disk0p1 is the microSD, which has no /boot/lua/loader.lua
>> That makes sense. If u-boot finds the USB mass storage device
>> then running 
>> run bootcmd_usb0 starts the system successfully.
>> 
>> It does appear that the FAT partition on the USB disk is involved.
> 
> I see no evidence of that above. (Below is a different context.)
> 
>> If I hide the contents of da0s1 by placing them in a subdirectory
>> the boot fails, even if u-boot has found the USB disk:
>> 
>> Hit any key to stop autoboot:  0 
>> U-Boot>   usb reset
>> resetting USB...
>> Bus usb@7e980000: USB DWC2
>> scanning bus usb@7e980000 for devices... 6 USB Device(s) found
>>      scanning usb for storage devices... 0 Storage Device(s) found
>> 
>> [Note that the six devices include the disk, it can be seen in usb tree.
>> For some reason it isn't recognized as a mass storage device.]
>> 
>> U-Boot> editenv usb_pgood_delay
>> edit: 2
>> U-Boot> usb reset
>> resetting USB...
>> Bus usb@7e980000: USB DWC2
>> scanning bus usb@7e980000 for devices... 6 USB Device(s) found
>>      scanning usb for storage devices... 1 Storage Device(s) found
>> U-Boot> run bootcmd_usb0
>> 
>> Device 0: Vendor: SABRENT  Rev: 1214 Prod: 
>>           Type: Hard Disk
>>           Capacity: 953869.7 MB = 931.5 GB (1953525168 x 512)
>> ... is now current device
>> Scanning usb 0:1...
>> U-Boot> 
>> and there it stops. Looks like both FAT partitions have a role to play.
> 
> That is not clear to me. But I'd rather be lookning into
> earlier stages at this point, so that "1 Storage Device(s) found"
> hopefully ends up being automatic.
> 
> If we get that to be automatic, then what is happening here
> becomes the next area to investigate. But the investigation
> is not far enough along to justify investigating this now.
> 
>> An earlier experiment tried booting the microSD card in a USB adapter,
>> That worked correctly with nothing in the microSD slot, so the internal
>> "boot from USB" feature might work if it could be slowed down sufficiently. 
> 
> Agreed, that the first thing is to get it to start up
> the disk and wait for it. There may be more at issue
> after that.
> 
>>> The order of activity is:
>>> 
>>> MSDOSFS: (all on one of: microsd card vs. usb drive)
>>> RPi* firmware (I do not report the file-level sequencing)
>>> U-Boot
>>> FreeBSD loader (which uses information from U-Boot)
>>> 
>>> UFS: (both: together on one of: microsd card vs. usb drive)
>>> FreeBSD kernel
>>> FreeBSD world
>>> 
>>> (I do not specify which MSDOSFS is used when
>>> there are multiple viable ones in separate places.
>>> Similarly for UFS.)
>>> 
>> From the experiment today it seems both are required.
> 
> I do not get such an interpretation from your experiment.
> Until "1 Storage Device(s) found" happens automatically
> the context is not nicely comparable for the later stages.
> 
>> The first discovers the USB disk, the second uses that
>> information to proceed further. 
>> 
>>> 
>>> (It is technically possible to have kernel vs.
>>> world in separate UFS partitions, possibly on
>>> separate media. I've an example context that I
>>> deal with that way to avoid a U-boot limitation
>>> for the device: the kernel can find the world
>>> where the U-Boot/FreeBSD-Loader can not. (The
>>> FreeBSD loader depends on what USB can find: it
>>> does not look elsewhere.) I only mention this
>>> in case I need to reference it in the future,
>>> avoiding a surprise in such a case. Otherwise
>>> ignore the complication.)
>>> 
>> 
>> Are you referring to a case where the root is on
>> microSD but /usr and friends is on a USB device? 
> 
> Again: I suggest ignoring the details of this for
> now. It would just confuse things. But if you
> insist:
> 
> On the Rock64 I have a e.MMC in use instead of a
> microsd card, and in the end the world is on a
> USB3 SSD. But the U-Boot involved can not access
> the USB3 --os neither can the FreeBSD loader.
> The FreeBSD kernel is the first stage that can
> access USB3.
> 
> The e.MMC has the MSDOSFS. It has an partial
> copy of what would normally be some of the USB3
> drive's content. (I named the mount point
> microsd_ufs before I switched to the e.MMC .)
> It has only (much hidden in ". . ."s).
> 
> # find /microsd_ufs -print | sort | less
> /microsd_ufs
> /microsd_ufs/.snap
> /microsd_ufs/COPYRIGHT
> /microsd_ufs/boot
> /microsd_ufs/boot/beastie.4th
> /microsd_ufs/boot/boot1.efi
> /microsd_ufs/boot/brand-fbsd.4th
> /microsd_ufs/boot/brand.4th
> /microsd_ufs/boot/check-password.4th
> /microsd_ufs/boot/color.4th
> /microsd_ufs/boot/copy_boot_to_microsd.sh
> /microsd_ufs/boot/defaults
> /microsd_ufs/boot/defaults/loader.conf
> /microsd_ufs/boot/delay.4th
> /microsd_ufs/boot/dtb
> . . .
> /microsd_ufs/boot/kernel
> /microsd_ufs/boot/kernel/accf_data.ko
> /microsd_ufs/boot/kernel/accf_dns.ko
> . . .
> /microsd_ufs/boot/kernel/kernel
> . . .
> /microsd_ufs/boot/loader.4th
> /microsd_ufs/boot/loader.conf
> /microsd_ufs/boot/loader.conf.d
> /microsd_ufs/boot/loader.efi
> /microsd_ufs/boot/loader.help
> /microsd_ufs/boot/loader.rc
> /microsd_ufs/boot/loader_4th.efi
> /microsd_ufs/boot/loader_lua.efi
> /microsd_ufs/boot/loader_simp.efi
> /microsd_ufs/boot/logo-beastie.4th
> /microsd_ufs/boot/logo-beastiebw.4th
> /microsd_ufs/boot/logo-fbsdbw.4th
> /microsd_ufs/boot/logo-orb.4th
> /microsd_ufs/boot/logo-orbbw.4th
> /microsd_ufs/boot/lua
> /microsd_ufs/boot/lua/cli.lua
> /microsd_ufs/boot/lua/color.lua
> /microsd_ufs/boot/lua/config.lua
> /microsd_ufs/boot/lua/core.lua
> /microsd_ufs/boot/lua/drawer.lua
> /microsd_ufs/boot/lua/gfx-beastie.lua
> /microsd_ufs/boot/lua/gfx-beastiebw.lua
> /microsd_ufs/boot/lua/gfx-fbsdbw.lua
> /microsd_ufs/boot/lua/gfx-orb.lua
> /microsd_ufs/boot/lua/gfx-orbbw.lua
> /microsd_ufs/boot/lua/hook.lua
> /microsd_ufs/boot/lua/loader.lua
> /microsd_ufs/boot/lua/menu.lua
> /microsd_ufs/boot/lua/password.lua
> /microsd_ufs/boot/lua/screen.lua
> /microsd_ufs/boot/menu-commands.4th
> /microsd_ufs/boot/menu.4th
> /microsd_ufs/boot/menu.rc
> /microsd_ufs/boot/menusets.4th
> /microsd_ufs/boot/modules
> /microsd_ufs/boot/screen.4th
> /microsd_ufs/boot/shortcuts.4th
> /microsd_ufs/boot/support.4th
> /microsd_ufs/boot/uboot
> /microsd_ufs/boot/version.4th
> /microsd_ufs/boot/zfs
> /microsd_ufs/etc
> /microsd_ufs/etc/hostid
> /microsd_ufs/lost+found
> 
> Almost no world content. /microsd_ufs/boot/loader.conf
> has (in part):
> 
> vfs.root.mountfrom="ufs:/dev/gpt/Rock64root"
> kern.cam.boot_delay=10000
> vfs.mountroot.timeout=10
> vfs.root_mount_always_wait=1
> 
> That vfs.root.mountfrom notation redirects to the
> USB3 drive for world but the /boot/loader.conf and
> /boot/kernel/kernel and /etc/hostid were first
> loaded from the e.MMC media, not the USB drive.
> 
> As I have it, the USB3 drive also has a copy of
> the same /boot/... and such materials and load
> module activity is from the USB3 copy. I have to
> keep the two places tracking so nothing ends up
> mismatching.
> 
>>> You might move (not copy) the MSDOSFS material
>>> between the microsd card and the usb drive during
>>> experiments, avoiding having duplications. There
>>> is the possibility of instead renaming things so
>>> files are not found on a partition, for example.
>>> A similar point goes for UFS materials: avoid
>>> having multiple viable-for-booting UFS file
>>> systems present.
>>> 
>>> As stands, things seem too hard to track for me
>>> to be of much help. Please make things obvious for
>>> what was in use by making the material uniquely
>>> placed (for the form that can be used).
>>> 
>> I believe the experiment above (deleting UFS on microSD, 
>> hiding the DOS files on USB) has been an equivalent 
>> configuration. It looks like control passes in some way 
>> between the DOS slices, though how is unclear.
> 
> I do not agree to that specific interpretation of the
> sequence you did (on the evidence currently available).
> Until "1 Storage Device(s) found" is automatic this
> later-stage material is too late to yet be relevant
> or to have an appropriate context.
> 
> I'm staying focused on getting the "1 Storage Device(s)
> found" to be automatic. Absent that you are likely stuck
> with doing something similar to be Rock64 e.MMC way of
> working --where /boot/loader.conf and /boot/kernel/kernel
> and /etc/hostid for early activity is from a UFS
> partition on the microsd card.
> 
>>> Separate issues/questions:
>>> 
>>> Do you have the file "timeout" in the MSDOSFS, in
>>> addition to config.txt and the like? If not, I
>>> recommend including it.
>>> 
>> Yes, it's been tried on both microSD and USB devices. 
>> Seems like the only one that can matter is on microSD.
>> 
>> 
>>> What other RPi* firmware controls for having
>>> various deliberate RPi* firmware delays do you
>>> have set up?
>>> 
>> 
>> The Raspberry Pi config.txt description lists several delay commands
>> that can be placed in config.txt. None seem related to USB discovery,
>> might any come into play early enough to be useful? They're named
>> bootcode_delay
>> boot_delay
>> boot_delay_ms
>> 
>> I tried them casually and didn't notice much change. Are they worth
>> revisiting more carefully? 
> 
> See my earlier notes.
> 
>>> It is not so much that these would be sufficient,
>>> but they do establish some context before U-Boot
>>> is even active. It could be important to
>>> understand that context. (Unsure at this point.)
>>> 
>>> 
>>>>> But that was for the u-boot-rpi4 or u-boot-rpi-arm64 ports.
>>>>> (They also later mentioned using "usb_pgood_delay=2000\0"
>>>>> instead, a figure they found in a bunch of configrations.)
>>>>> 
>>>> The Pi3 in question is capable of booting from solid-state USB
>>>> storage without any microSD card, but fails to detect a mechanical
>>>> disk. Which is the appropriate u-boot-rpi3 port to tamper with? I 
>>>> tried sysutils/u-boot-rpi3 as an upgrade but that simply froze. 
>>>> The u-boot from FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img finds
>>>> the USB mechanical disk,  but erratically. 
>>> 
>>> FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img uses u-boot-rpi-arm64
>>> as I remember: it is set up for both the RPI3 and RPI4. Given that
>>> it works some, that would be the U-Boot port I would experiment
>>> with if I was doing the experiments.
>>> 
>>>>> So something somewhat analogous might help if you are willing
>>>>> to build and use your own u-boot port variant.
>>>> 
>>>> Obviously, that's a fraught enterprise at my skill level....
>>>> I'm still somewhat hazy on the actual boot sequence when
>>>> chaining from microSD to USB. 
>>> 
>>> Having the extra text in the U-Boot build does not really
>>> depend on the chaining order question.
>>> 
>>> But, to repeat (sticking to the simpler context),
>>> the order is:
>>> 
>>> MSDOSFS: (all on one of: microsd card vs. usb drive)
>>> RPi* firmware (I do not report the file-level sequencing)
>>> U-Boot
>>> FreeBSD loader (which uses information from U-Boot)
>>> 
>>> UFS: (both: together on one of: microsd card vs. usb drive)
>>> FreeBSD kernel
>>> FreeBSD world
>>> 
>>> 
>>>> Indeed, it's unclear how or if u-boot plays a role in starting 
>>>> RasPiOS. The term u-boot isn't found on the Raspberry Pi doc 
>>>> website, and the Pi isn't mentioned in the Denx manuals. Those
>>>> discoveries surprised me.
>>> 
>>> RaspiOS and RaspiOS64 do not use U-boot. The RPi*'s can boot some
>>> forms of a linux kernel directly and that is what RaspiOS and
>>> RaspiOS64 are doing. 
>> 
>> That clears up a major misunderstanding, thank you!
>> 
>>> That is why the config.txt type of content
>>> makes no mention of u-boot or of any kernel= assignment in RaspiOS
>>> and RaspiOS64. By contrast, config.txt for FreeBSD lists kernel=
>>> and an explicit *u-boot*.bin as the value.
>> 
>> Without thinking about it very carefully I tried to use the
>> "bootcode.bin-only" method for booting an older Pi2 from a
>> usb hard disk, as described in
>> https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#special-bootcode-bin-only-boot-mode
>> It worked on the Pi2 but failed on the Pi3, causing an immediate hang.
> 
> This gets back into the use of a config.txt with a
> bootcode_delay assignment also being in the MSDOSFS
> on the microsd card and the file timeout also being
> in the MSDOSFS on the microsd card. Only if those
> delays together can lead to the USB drive being
> accessible will it get any farther.
> 
> I'd suggest such experiments. The vintage of bootcode.bin
> matters as I understand.
> 
>> Might that avenue be worth pursuing? In hindsight, that it worked on a
>> Pi2 is quite surprising.  
>> 
> 




===
Mark Millard
marklmi at yahoo.com