Re: How to boot FreeBSD for arm 32 bit as DomU with u-boot on my ARM Chromebook
Date: Tue, 19 Dec 2023 22:24:16 UTC
I've asked some help on the channel #arm on Reddit and someone replied : https://www.reddit.com/r/arm/comments/18mcir8/i_cant_boot_freebsd_for_arm32_bit_as_domu_with/ Maybe his answer can be useful to understand why it does not work. On Tue, Dec 19, 2023 at 8:33 PM Stefano Stabellini <sstabellini@kernel.org> wrote: > +Michal > > Hi Mario, > > I am not sure about booting FreeBSD, but I am certain that u-boot works > fine as DomU kernel on ARMv8 (not sure about ARMv7). With this config > file: > > name="test" > kernel="u-boot.bin" > extra = "console=hvc0" > memory=256 > vcpus=1 > disk = [ 'FreeBSD-13.2-RELEASE-armv7.img,raw,xvda' ] > > I don't know for sure if you can boot FreeBSD but you should definitely > be able to see the u-boot command line prompt. The fact that you are > getting this message: > > xc: error: panic: xg_dom_core.c:689: xc_dom_find_loader: no loader found: > Invalid kernel > > Means that something is not right in the u-boot configuration or u-boot > build. Michal and Artem (CCed) might know more. From what I recall, > there was nothing special required to get u-boot.bin to boot as domU > kernel, so now I wonder if it is an ARMv7 vs. ARMv8 issue. > > Cheers, > > Stefano > > > On Tue, 19 Dec 2023, Mario Marietto wrote: > > ....I see that some other interesting files have been produced by u-boot > when I have compiled it : > > > > u-boot > > u-boot.lds > > u-boot.bin > > u-boot.map > > u-boot-nodtb.bin > > u-boot.dtb > > u-boot.srec > > u-boot-dtb.bin > > u-boot.sym > > > > So,maybe I should use a different u-boot* file for booting FreeBSD ? > > > > > > On Tue, Dec 19, 2023 at 4:28 PM Mario Marietto <marietto2008@gmail.com> > wrote: > > Hello to everyone. > > > > I have compiled the needed u-boot.bin from scratch using this procedure : > > > > # git clone https://github.com/u-boot/u-boot.git > > # cd u-boot > > # ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make snow_defconfig : this > line generates the file .config > > # nano .config and I've added these parameters : > > > > CONFIG_ARMV7_NONSEC=n > > CONFIG_EFI_GRUB_ARM32_WORKAROUND=y > > > > the uboot-bin file is generated with this command : > > > > # ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make > > > > At this point,I took a look inside the .config file and I saw that the > parameter "CONFIG_ARMV7_NONSEC=n" has been removed. So,for > > some reason,it is not accepted and this could be a problem.... > > > > These are the xen config files that I've used : > > > > nano freebsd.cfg > > > > name="test" > > kernel="u-boot.bin" > > extra = "console=hvc0" > > memory=256 > > vcpus=1 > > disk = [ 'FreeBSD-13.2-RELEASE-armv7.img,raw,xvda' ] > > > > nano start-freebsd > > > > xl create freebsd.cfg > > xl console freebsd > > > > This is what happens when I launch the vm : > > > > # ./start-freebsd > > > > Parsing config from freebsd.cfg > > xc: error: panic: xg_dom_core.c:689: xc_dom_find_loader: no loader > found: Invalid kernel > > libxl: error: libxl_dom.c:571:libxl__build_dom: xc_dom_parse_image failed > > libxl: error: libxl_create.c:1640:domcreate_rebuild_done: Domain > 1:cannot (re-)build domain: -3 > > libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain > 1:Non-existent domain > > libxl: error: libxl_domain.c:1137:domain_destroy_callback: Domain > 1:Unable to destroy guest > > libxl: error: libxl_domain.c:1064:domain_destroy_cb: Domain > 1:Destruction of domain failed > > freebsd is an invalid domain identifier (rc=-6) > > > > > > On Mon, Dec 18, 2023 at 12:39 PM Mario Marietto <marietto2008@gmail.com> > wrote: > > So,ok,I should have said "the second u-boot" ; since the first > u-boot binary is the "u-boot binary located in the RO > > memory" of the Chromebook". Sorry for the confusion. > > > > On Mon, Dec 18, 2023 at 12:35 PM Mario Marietto <marietto2008@gmail.com> > wrote: > > ---> There are no specific options in u-boot devoted to FreeBSD > > > > This is an important factor. So,what about if,instead of compiling a new > version of u-boot on the partition 2,I will > > recompile the u-boot customized version created by the virtual open > system in 2014,that should be installed on the first > > partition ? It could work if there are no differences between the u-boot > that should boot Linux and the u-boot that > > should boot FreeBSD. > > > > Can you give a look at the u-boot source code created by virtual open > systems ? You can find it on my google drive : > > > > > https://drive.google.com/file/d/1eAaZMfd6CU0xiqQfH7sq5wGVzzO09BRm/view?usp=sharing > > > > I need to understand if I can recompile it without problem so that it > can satisfy my needs (the ability of the file > > u-boot.bin to boot FreeBSD as domU under Xen,as explained by Stefano > Stabellini,the xen developer that suggested to me > > what I could do to have FreeBSD virtualized under Xen on my Arm > Chromebook) ; otherwise the risk is to find later > > problems that will make me troubles and that I will not able to fix. > > > > I gave a look at the virtual open system u-boot and I didn't see any > arndale_defconfig inside. So,If I have understood > > correctly,I should put that file inside the root of the u-boot source > code,let's say here : > > > > marietto:/home/marietto/Desktop/Files/u-boot_FreeBSD/u-boot-vos # ls > > > > .checkpatch.conf README doc > net > > .git api drivers > onenand_ipl > > .gitignore arch dts > post > > COPYING board examples > rules.mk > > CREDITS boards.cfg fs > scripts > > MAINTAINERS common include > snapshot.commit > > MAKEALL config.mk lib > spl > > Makefile cros mkconfig > test > > PRESUBMIT.cfg disk nand_spl > tools > > > > and I should do : make and make install ? and the file I need,u-boot.bin > will be generated ? > > > > I didn't find any pre made configuration file inside : > > > > u-boot-vos # find . -type f -name "exynos*" > > > > ./include/exynos-fb.h > > ./include/configs/exynos5-common.h > > ./doc/device-tree-bindings/spi/exynos-spi.txt > > ./doc/device-tree-bindings/usb/exynos-usb.txt > > ./drivers/power/exynos-tmu.c > > ./drivers/power/exynos-cpufreq.c > > ./drivers/video/exynos-fb.c > > ./drivers/spi/exynos_spi.c > > ./board/samsung/dts/exynos5250-spring.dts > > ./board/samsung/dts/exynos5250-smdk5250.dts > > ./board/samsung/dts/exynos5250-snow.dts > > ./board/samsung/dts/exynos5250-daisy.dts > > ./arch/arm/include/asm/arch-exynos5/exynos-cpufreq.h > > ./arch/arm/include/asm/arch-exynos5/exynos-tmu.h > > ./arch/arm/dts/exynos5250.dtsi > > ./arch/arm/dts/exynos-periph-id.dtsi > > ./arch/arm/cpu/armv7/exynos5/exynos_cache.c > > > > u-boot-vos # find . -type f -name "arndale*" > > > > For sure I can't use a newer version of u-boot because otherwise the > patches needed to bypass the bootloader protections > > of the Arm Chromebook (such as a lot of different patches needed to boot > correctly Linux) will be broken ; anyway,since > > it works,I don't need to use an updated version of u-boot. > > > > ----> As per my experience, you have to respect these two options, > compiling u-boot for > > FreeBSD: > https://github.com/freebsd/freebsd-ports/blob/main/sysutils/u-boot-master/files/FreeBSD_Fragment > > > > It says that I should use these parameters : > > > > CONFIG_ARMV7_NONSEC=n > > CONFIG_EFI_GRUB_ARM32_WORKAROUND=y > > > > These are the parameters used to configure a Linux kernel. I don't > understand what's the relation between the compilation > > of a linux kernel and u-boot. In the past I tried to recompile > u-boot,but I didn't have the need to set up those > > parameters,so I don't know how to do it (but I know how to recompile a > Linux kernel). > > > > ---> I'm not sure that I'm getting you right, as I don't understand what > you mean under "the first u-boot". > > > > > > I'm talking about first u-boot because the whole procedure to boot Linux > on the ARM Chromebook,that's explained here : > > > > http://www.virtualopensystems.com/en/solutions/guides/kvm-on-chromebook/ > > > > > > at some point they say : > > > > > > To be able to run KVM on ARM platforms, the kernel has to be booted in > hypervisor mode. Because of this relatively recent > > requirement (due to the introduction of the virtualization extensions), > up until now all booting methods would boot the > > kernel in the standard Supervisor mode. > > > > For the ARM Chromebook the default boot procedure doesn't allow us to > boot in hypervisor mode. Although the laptop's boot > > mechanism is based on the frequently used u-boot, the binary is located > in RO memory. Fortunately, a chained u-boot > > mechanism can be used (i.e. starting another u-boot after the original). > We can then enter hypervisor mode from our > > custom iteration of u-boot and subsequently load our kernel and > userspace. > > > > So,the first u-boot is the u-boot provided by virtual open > systems,that's able to chainload the "u-boot binary located in > > RO memory" , that does not boot Chrome OS in hypervisor mode. We don't > need it if we want to boot Linux with kvm or xen > > enabled. > > > > > > On Sun, Dec 17, 2023 at 1:28 AM Stanislav Silnicki < > stanislav.silnicki@mailgate.us> wrote: > > I'm not an expert in the topic, I only know, that ARM has divided > hardware into two worlds - Secure and > > Not-So, strictly limiting any software, running in non-secure > world with access to functions and > > resources. > https://developer.arm.com/documentation/den0013/d/Security/TrustZone-hardware-architecture?lang=en > > > > I'm not sure, that I'm getting you right, as I don't understand what you > mean under "the first u-boot". > > > > As I understand, virtualization (HYP) is running in non-secure world( > https://developer.arm.com/documentation/ddi0406/c/System-Level-Architecture/The-System-Level-Programmers--Model/The-Virtualization-Extens > > ions), so my guess (only guess!!!), virtualization software has to > prepare (configure) HW platform in the way, > > that FreeBSD kernel will not lack any resources, required to configure > MPU, VA, etc. > > So, if you lucky to boot virtualizer, which is aware of target OS, that > maybe you can boot the kernel. Although, I > > doubt, that you need to boot 'second' u-boot to boot the kernel - there > is simply ubldr, which you can hook somehow > > from virtualizer.... > > > > Stan > > > > > > > > Mario Marietto wrote: > > > > > > ---> As I understand, it makes sure that u-boot keeps in secure > mode during boot and passes control to > > ubldr, which boots FreeBSD kernel, in that mode. > > > > Can you elaborate your sentence more ? I know that the bootloader secure > mode is bypassed by the virtual open > > systems u-boot. Are you saying that when the control passes to the > second u-boot,it will happen in secure > > mode,so that the bypass that happened loading the first u-boot,is > annulled ? If this is true,maybe can I boot > > FreeBSD using the virtual-open-system custom u-boot ? Is this compatible > with FreeBSD ? Where can I find the > > u-boot.bin that the xen developer talked about ? thanks bro'. > > > > > > > > On Sun, Dec 17, 2023 at 12:35 AM Stanislav Silnicki < > stanislav.silnicki@mailgate.us> wrote: > > Hi Mario, > > > > U-Boot beast is hiding in this den: > https://source.denx.de/u-boot/u-boot.git > > I took a brief look at your post and it seems to me, that > option CONFIG_CMO_BY_VA_ONLY is irrelevant to > > your target armv7 32 bit > > platform: > https://source.denx.de/u-boot/u-boot/-/blob/master/arch/arm/cpu/armv8/Kconfig?ref_type=heads#L3 > > > > As for compiling the u-boot, it is a doable task, given that you > understand what you are doing. There > > are no specific options in u-boot devoted to FreeBSD. It is a boot > loader, whose mission to make basic > > hardware initialization, read you kernel file from some media into RAM > and then pass it control. > > > > Basically, you can grab some defconfig, prepared for any other > Exynos5250 based board (say, this one: > > > https://source.denx.de/u-boot/u-boot/-/blob/master/configs/arndale_defconfig?ref_type=heads) > and adopt > > it somehow. > > > > As per my experience, you have to respect these two options, compiling > u-boot for > > FreeBSD: > https://github.com/freebsd/freebsd-ports/blob/main/sysutils/u-boot-master/files/FreeBSD_Fragment > > > > As I understand, it makes sure, that u-boot keeps in secure mode during > boot and passes control to > > ubldr, which boots FreBSD kernel, in that mode. Otherwise, there a lot > of surprises you may realize. > > > > Hope, this will help to progress you tasks > > Stan > > > > Mario Marietto wrote: > > > > > > Hello. > > > > I'm trying to boot FreeBSD for arm32 bit as DomU on my ARM > Chromebook. Basically there are > > two ways to accomplish this task : > > > > 1) to write a patch that allows the FreeBSD kernel to boot as a > zImage file. This could be > > accomplished applying this patch to a specific file that's on the > source code of FreeBSD : > > > > > > > https://xenbits.xen.org/gitweb/?p=p...8;hb=0782e25d98cc1391472717035f986c979edef0c9 > > > > > > This patch was written by Julien Grall a lot of time ago and now > it does not work anymore. > > This is the reason : > > > > > > It appears FreeBSD-CURRENT removed the last step converting > the kernel file to > > kernel.bin. The patch can be readily rebased, but without > kernel.bin that > > doesn't do too much. > > > > > > > > So,without a rebase of that patch the first option is not applicable. > And I'm not able to fix it. > > > > 2) booting FreeBSD using U-Boot,as explained to me by a xen developer : > > > > > > I was trying to explain why and how Julien's patch works so that > you could be the one > > to re-do something similar or fix the patch on the FreeBSD kernel > that you are > > working with. I am happy to help review and write patches but I > don't work with the > > FreeBSD kernel so I wouldn't be able to help you quickly. However, > I might have a > > suggestion. Do you know if FreeBSD can be booted by U-Boot ? > Because U-Boot > > definitely boots as Xen on ARM guest firmware/bootloader. You > should be able to build > > U-Boot and use the U-Boot binary as Xen guest kernel, then U-Boot > could load FreeBSD > > from disk or network and start it. For instance as domU config > file: > > > > kernel="/home/petalinux/u-boot.bin" > > disk = [ '/home/petalinux/test.img,raw,xvda' ] > > > > I know it is important to build u-boot with the following config > to make it work on > > Xen. > > > > CONFIG_CMO_BY_VA_ONLY=y > > > > > > > > This option seems more doable to me according to my knowledge. But I > need to understand how to do > > it. > > > > Well,let's say that on the ARM Chromebook I'm forced to use and install > a customized version of > > u-boot,created by virtual open systems,because it is the only one that > allows bypassing its > > bootloader protection. You can find more information here : > > > > > http://www.virtualopensystems.com/en/solutions/guides/kvm-on-chromebook/?vos=tech > > > > This is the relevant section to read : > > > > > > Bootloader : > > > > If you wish to skip this chapter you can download a pre-compiled > binary of the > > bootloader: > > > > > > $ wget > > > http://www.virtualopensystems.com/downloads/guides/kvm_on_chromebook/nv_u-boot-snow.kpart > > > > > > To be able to run KVM on ARM platforms, the kernel has to be > booted in hypervisor > > mode. Because of this relatively recent requirement (due to the > introduction of the > > virtualization extensions), up until now all booting methods would > boot the kernel in > > the standard Supervisor mode. For the ARM Chromebook the default > boot procedure > > doesn't allow us to boot in hypervisor mode. Although the laptop's > boot mechanism is > > based on the frequently used u-boot, the binary is located in RO > memory. Fortunately, > > a chained u-boot mechanism can be used (i.e. starting another > u-boot after the > > original). We can then enter hypervisor mode from our custom > iteration of u-boot and > > subsequently load our kernel and userspace. > > > > Checkout the needed u-boot code : > > > > > > $ git clone git://github.com/virtualopensystems/u-boot.git$ cd > u-boot$ > > ./scripts/build.sh > > > > > > If successful, a message about how to copy the bootloader on the > USB flash disk or SD > > card will appear. We will use it later when preparing the boot > medium to start our > > system. If you have followed the Setting up the boot medium > chapter and you have a > > prepared boot device, then you can update u-boot by running : > > > > > > $ sudo dd if=nv_uboot-snow.kpart of=/dev/sdX1 > > > > > > > > so,the needed u-boot that we must use should be installed on the first > partition of the sd card. > > > > There is another relevant section to read : > > > > > > Setting up the boot medium > > > > Now it is time to copy all the relevant files that we created in > the previous > > chapters,and use them to boot Chromebook with a different kernel > and OS. In all these > > examples the device /dev/sdX is used. Take extra care to change > the examples to the > > device that you have attached. Insert the boot medium on your > workstation and > > carefully execute the following step. First we need to properly > format the boot > > medium. > > > > In the uboot source directory : > > > > > > $ sudo ./scripts/sdcard.sh /dev/sdX > > > > > > This will erase all data and create 4 partitions in the medium, > along with copying > > the u-boot binary to the first partition: > > > > > > Partition 1 = ChromeOS signed binary (V.O.S chained u-boot) > > Partition 2 = not used > > Partition 3 = EXT2 partition for u-boot files (uImage and > exynos5250-snow.dtb) > > Partition 4 = EXT4 partition for userspace files > > > > > > With u-boot being copied, next is the kernel image and DTB file. > From the kernel > > source execute : > > > > > > $ mkdir ../mnt/ > > $ sudo mount /dev/sdX3 ../mnt/ > > $ sudo cp arch/arm/boot/uImage ../mnt/ > > $ sudo cp arch/arm/boot/dts/exynos5250-snow.dtb ../mnt/ > > $ sudo umount /dev/sdX3 > > > > > > Finally, we have to copy the Ubuntu userspace filesystem that we > created earlier: > > > > > > $ sudo mount /dev/sdX4 mnt/$ sudo cp -a ./precise/* mnt/$ sudo > umount /dev/sdX4 > > > > > > > > Now,my idea is to chainload the already chain loaded u-boot created by > V.O.S to the new u-boot > > that we need for booting FreeBSD and that can be installed in the > partition n.2,as shown in this > > scheme,because it is not used : > > > > > > Partition 1 = ChromeOS signed binary (V.O.S chained u-boot) > > Partition 2 = not used (maybe we can install the u-boot for arm 32 > bit,compatible with FreeBSD on > > this partition) > > Partition 3 = EXT2 partition for u-boot files (uImage and > exynos5250-snow.dtb) > > Partition 4 = EXT4 partition for userspace files > > > > > > Take in consideration that default boot string is hardcoded here,in the > snow.h file of the custom > > u-boot created by VOS : > > > > > > > https://github.com/virtualopensyste...18a39b6c177dff58a/include/configs/snow.h#L101 > > > > > > and it needs to be recompiled because it should point to the partition > n.2,where I will install > > the u-boot files as explained here : > > > > > > https://wiki.freebsd.org/arm/Chromebook > > > > > > I have some questions to ask before I start working on this. > > > > 1) The xen developer said : > > > > > > You should be able to build U-Boot and use the U-Boot binary as > Xen guest kernel... > > > > > > > > where is the u-boot binary,according to this document ? > > > > https://wiki.freebsd.org/arm/Chromebook > > > > I don't see it. > > > > > > 2) where is the source code of the file that I can get here : > > > > > http://commondatastorage.googleapis.com/chromeos-localmirror/distfiles/nv_uboot-snow-simplefb.kpart.bz2 > > > > I need the source code if I want to recompile u-boot so that it can > point to the partition 4. > > > > Maybe it can be found on this link : > > > > http://linux-exynos.org/dist/chromebook/nv_uboot/ > > > > but it can't be opened.... > > > > > > 3) in this specific scenario the source code of u-boot should run on arm > 32 bit,not on arm > > 64,because I have the Samsung Chromebook "SNOW" model XE303C12,that's > powered by a Samsung Exynos > > 5250 (ARMv7 32 bit Cortex A15) Soc. > > > > > > 4) I'm not sure if I can chainload the customized u-boot created by > V.O.S that should be > > installed on the first partition with the u-boot tailored for booting > FreeBSD that should be > > installed on the partition 2.... > > > > > > 5) the xen developer said that u-boot should be compiled enabling this > option : > > > > > > Code: > > > > CONFIG_CMO_BY_VA_ONLY=y > > > > > > Well,can you provide some good source that can help me to understand how > I can recompile u-boot > > for FreeBSD ? thanks. > > > > -- > > Mario. > > > > > > > > -- > > Mario. > > > > > > > > -- > > Mario. > > > > > > > > -- > > Mario. > > > > > > > > -- > > Mario. > > > > > > > > -- > > Mario. > > > > -- Mario.