Re: Boot from USB on RPi4 8GB?

Thank you for the quick and thorough reply!

> On May 30, 2021, at 3:55 AM, Mark Millard via freebsd-arm <freebsd-arm@freebsd.org> wrote:
> 
> On 2021-May-29, at 22:17, William Carson <freebsd@dsllsn.net> wrote:
> 
>> Hello Mark, sorry to unicast you but I keep getting rejected from freebsd-arm@. I was hoping you could give me some guidance, as it seems you've made quite a bit of progress in this area.
>> 
>> Is there any documentation or a howto in order to get FreeBSD to boot from a USB3 disk on the RPi4 8GB? I dd'd the latest 13.0-RELEASE image (FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img) to the USB3 SSD, but it does not boot. (The same image works just fine when written to the sdcard.) It looks like it's unable to locate the USB disk and then gets stuck in a loop trying for network boot. When I get to U-Boot prompt, it says there are no USB storage devices. If I issue "usb start", there are still no devices.
> 
> You report:
> 
>> If I try "usb reset", it seems to find it, but then it displays an error ("scanning bus xhci_pci for devices... Setup ERROR address device command for slot 1. USB device not accepting new address (error=80000000)" and then locks up. I checked that I have the latest bootloader:
> 
> (I assume no microsd card was in the microsd card slot.)

Correct.

> I've not seen or heard of that U-Boot report before.
> But you made it to U-Boot so the RPi4B firmware
> worked for getting U-Boot from the USB3 drive.
> 
> This suggests the U-Boot stage is having the problem.
> 
> So I've no specific solutions, not having seen the
> kind of report before. I've just more generic notes
> and questions.
> 
> [I'll note that I've had no problem with USB3 SSD
> booting the RPi4B's that I have access to (no
> microsd card involved).]

I'm trying to use a SAMSUNG (MZ-V7E500BW) 970 EVO SSD 500GB - M.2 NVMe, attached via the Geekworm X872 M.2 NVMe 2280/2260/2242/2230 SSD Expansion Board.

> One test is to use the microsd card that boots
> via the microsd card slot and instead put it in
> a USB3 microsd card reader, plug the reader into
> a USB3 port, and try to boot from just that.

Hmm, this worked just fine.

> If that boots, then there would seem to be some
> device incompatibility with your specific USB3
> "boot" drive. If, instead, booting via the reader
> fails, then things are rather odd. (See later
> notes about SOC vintages.)
> 
>> # rpi-eeprom-update
>> BOOTLOADER: up to date
>> CURRENT: Thu 29 Apr 16:11:25 UTC 2021 (1619712685)
>>  LATEST: Thu 29 Apr 16:11:25 UTC 2021 (1619712685)
>> RELEASE: stable (/lib/firmware/raspberrypi/bootloader/stable)
>>          Use raspi-config to change the release.
>> 
>> VL805_FW: Using bootloader EEPROM
>>   VL805: up to date
>> CURRENT: 000138a1
>>  LATEST: 000138a1
> 
> Since you got to U-Boot the RPI4B's bootloader worked.
> The above is what I currently use in the RPi4B's,
> 8 GiBYte and 4 GiByte ones.
> 
>> If I dd Raspberry Pi OS (2021-05-07-raspios-buster-armhf-lite.img) to the same disk, it boots up no problem. I'm thinking this is a U-Boot/rpi-firmware problem, but I don't really know where to begin.
> 
> FYI: if you ever want to use both a 64-bit kernel and
> a 64-bit user space, there are BETA's of such (Debian
> Buster based):
> 
> https://downloads.raspberrypi.org/raspios_lite_arm64/images/
> https://downloads.raspberrypi.org/raspios_arm64/images/
> 
> (I'm not claiming any gain from such for the problem at hand.)
> 
>> I tried building my own image using crochet,
> 
> https://wiki.freebsd.org/arm/ reports about crochet:
> 
> "Alternatively, images for many boards can be built by crochet (deprecated) or using FreeBSD's release build infrastructure."
> 
> There were no commits at https://github.com/freebsd/crochet
> between 2019-Oct-06 and 2021-Apr-23. But there are some
> on 2021-Apr-24.

Yes, this is very disappointing, but I was able to copy the board/RaspberryPi3 and modify it to use the appropriate ports/firmware files. I used a similar process for my ROCKPRO64 and Khadas EDGE-V boards and they both worked. I'm happy to confine my testing to an official FreeBSD image if it makes things easier to troubleshoot :)

> I do my own builds based on using make commands but I do
> not use the release scripts for building. I build for a
> variety of systems.
> 
>> but that didn't work either.
> 
>> I'm also not sure whether I should be using sysutils/u-boot-rpi4 (I used this) or sysutils/u-boot-rpi-arm64?
> 
> Either should generally work. (But see later notes
> about RPi4B SOC vintages.)
> 
> FreeBSD-13.0-RELEASE-arm64-aarch64-RPI.img was based on
> sysutils/u-boot-rpi-arm64 . I've historically built and
> used a variant of sysutils/u-boot-rpi4 but my history
> goes back before sysutils/u-boot-rpi-arm64 existed. I
> do not do anything were the configuration variations
> involved matter so I'm not familiar with the distinctions.
> 
> I've used both a previous 2020.10 based U-Boot and a
> more recent 2021.04 based U-Boot.
> 
>> After installing sysutils/rpi-firmware (1.20210303.g20210303), should I just copy /usr/local/share/rpi-firmware/* to the boot partition and rename config_rpi4.txt to config.txt?
> 
> I assume you mean the msdos file system partition when
> you reference "boot partition". The msdos file system
> needs to contain:
> 
> A) copies of /usr/local/share/rpi-firmware/* materials,
>   using config_rpi4.txt content as the config.txt
>   content. The copy should be recursive, to pick up
>   the likes of the overlay/ subdirectory tree.
> 
> B) a copy of an appropriate u-boot.bin  ( from
>   /usr/local/share/u-boot/u-boot-rpi-arm64/ or
>   /usr/local/share/u-boot/u-boot-rpi4/ ).
>   (See later notes as well.)
> 
> C) A copy of /boot/loader.efi (the FreeBSD loader)
>   placed at/as efi/boot/bootaa64.efi in teh msdos
>   file system.
> 
> I use a USB3 SSD that has small enough power requirements
> to not require a powered hub. (I also use a 5.1V 3.5A
> power supply as part of that context.) I've never tried
> spinning rust or higher powered USB3 media.

I can confirm those are the steps I followed. I'm not sure what's considered "high powered" but the Samsung tech specs say this particular model uses 5.7 W on average and 10.0 W maximum. But it does seem curious that the Raspberry PI OS will boot this disk without issue, so I don't think it's the drive. I also tried a Samsung 950 PRO using a different enclosure (QNINE NVME Enclosure, M.2 PCIe SSD (M Key) to USB 3.0 External Case), but it behaved the same.

> It is unclear what the dd command details were like for
> the transfer to the USB3 media. So I just assume that
> it was okay.
> 
> You may want to have an empty file named timeout in
> the msdos file system. It allows for extra time.
> (I doubt it helps, since U-Boot did load and start.)

Correct - the empty timeout file did not help. However I was able to capture the beginning of the boot process:

"scanning bus xhci_pci for devices... Device NOT ready
    Request Sense returned 02 04 01
3 USB Device(s) found
        scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot: 0
Card did not respond to voltage select! : -110

Device 0: unknown device
ethernet@7d580000 Waiting for PHY auto negotiation to complete.."

(This was with the USB3 disk attached and no sdcard in the slot; no ethernet attached either.)

> What does:
> 
> # rpi-eeprom-config 
> [all]
> BOOT_UART=0
> WAKE_ON_GPIO=1
> POWER_OFF_ON_HALT=0
> DHCP_TIMEOUT=45000
> DHCP_REQ_TIMEOUT=4000
> TFTP_FILE_TIMEOUT=30000
> ENABLE_SELF_UPDATE=1
> DISABLE_HDMI=0
> BOOT_ORDER=0xf41
> 
> report in your context? (An equivalent command is
> "vcgencmd bootloader_config". I show example output
> above.)

root@raspberrypi:~# rpi-eeprom-config
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0

> Can you see the top of the SOC? Or is there a
> heatsink or some such on it? (There is something
> there to read if it can be seen.)
> 
> One possibility is that you have newer hardware that
> the normal U-Boot vintages that FreeBSD has used do
> not handle.
> 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255080
> is about someone that instead of having only:
> 
> QUOTE
> Old Pi has the following identifying marks on the SOC package:
> BROADCOM
> 2711ZPKFSB06BOT
> TE1919
> 045-23 B3 W
> END QUOTE
> 
> the person also had an example of a 2 GiByte:
> 
> QUOTE
> New Pi has the following identifying marks on the SOC package:
> BROADCOM
> 2711ZPKFSB06C0T
> TA2105
> 054-05 B3 W
> END QUOTE
> 
> The:
> 
> 2711ZPKFSB06BOT
> vs.
> 2711ZPKFSB06C0T

Mine reads 2711ZPKFSB06B0T.

> is significant. If you have a 8GiByte "C0T" RPi4B it
> would be the first known example. In such a case, you
> might want to try the u-boot referenced in Comment 15 of
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255080 .
> It got the 2 GiByte "C0T" to boot. (But that context
> could not even boot via the microsd card slot with the
> normal media content from 13.0-RELEASE .)
> 
> I do not know if the 2021.04 U-Boot that the since
> updated port now provides would work for this context
> or not. I've no access to any "C0T" RPi4B's (or any
> Pi400's or CM4's).
> 
> There is a category of USB device that U-Boot still does
> not support as far as I know: those where the one device
> has multiple storage LUNs (instead of just one Logical
> Unit Number identifying storage).
> 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253983
> is about such a context, where we eventially figured out
> the "multiple storage LUN" issue.
> 
> But the error messages from the U-Boot of the time was
> not what you report.
> 
> 
> ===
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)
> 
>