Booting kernel.bin directly on Raspberry Pi / external DTB support

Fri Mar 25 21:35:53 UTC 2016

> Le 25 mars 2016 à 19:18, Warner Losh <imp at bsdimp.com> a écrit :
> 
> Have you looked at NetBSD to see what they do? It sounds like the SD driver in the boot roms is much faster than the one in u-boot.bin. Perhaps there's some init code we could import? Also, what are the kernel sizes? Maybe their kernel is smaller than ours for some reason and we should look at why...

NetBSD’s kernel size for RPI2 is around 5.5MB which is similar to our kernel.

It is booted directly by the firmware and no specific config.txt is required. So it looks exactly like a linux kernel, it even has a cmdline.txt file which allows the passing of boot parameters (root=ld0a console=fb)

You mentioned the ARM side of the RPI needs to be set up before being able to run the FreeBSD kernel: for my understanding, any idea about what the FreeBSD kernel requires exactly (apart from the metadata and all the module loading stuff)? I mean what does it need in terms of board set up - i.e. in other words what does u-boot does that is required by the FreeBSD kernel? I was under the understanding that the firmware sets up everything and even patches the DTB with board specific values (memory size, etc). Given that the firmware passes the patched DTB to the binary it loads, I expected the kernel.bin to boot directly, but I am obviously wrong and lack some embedded knowledge. 

I am sorry to ask here, I guess my questions aren’t of interest to many people, but I want to learn more and can’t find documentation on this matter.  

Cheers,
Sylvain. 

> On Fri, Mar 25, 2016 at 12:14 PM, Sylvain Garrigues <sylgar at gmail.com <mailto:sylgar at gmail.com>> wrote:
> u-boot> fatload mmc 0 0x100000 kernel.bin
> 
> takes around the same time than ubldr (actually slightly less, let’s say 4s - but that’s still so slow...)
> 
> Happy to help or test anything to get FreeBSD’s kernel first line appear on the screen near « instantaneously » (~ 1s) like our NetBSD or Linux friends.
> 
> Sylvain
> 
> 
>> Le 25 mars 2016 à 19:01, Warner Losh <imp at bsdimp.com <mailto:imp at bsdimp.com>> a écrit :
>> 
>> 
>> 
>> On Fri, Mar 25, 2016 at 11:55 AM, Sylvain Garrigues <sylgar at gmail.com <mailto:sylgar at gmail.com>> wrote:
>> Thank you. 
>> 
>> I wanted to boot the kernel directly so as to:
>> 1/ remove the dependency on a 3rd party loader
>> 2/ reduce boot time and make it comparable with NetBSD and Linux (which don’t use u-boot)
>> 
>> NetBSD and Linux on the Pi boot so much faster than our FreeBSD image: less than 1 second after power-up, the first kernel copyright line appears on my LCD. For the FreeBSD images, the longest part is ubldr loading the kernel. It takes around 4/5 seconds (with the same SD card).
>> 
>> I used recent u-boot versions (the current u-boot-rpi2 port is getting old and is not in sync with the u-boot-rpi port recently updated), tried with dcache enabled, but the boot process still takes several seconds, most of the time being spent in ubldr loading the kernel.
>> 
>> I have no idea how we could improve the booting time? I wanted to experiment with booting the kernel directly as the first ARM program on my Pi, but as you explained I need some init code. I managed to compile Andrew’s one (https://github.com/freebsd/freebsd/commit/074d37d46c3f9b282cd2d849d997b1b39acd710c <https://github.com/freebsd/freebsd/commit/074d37d46c3f9b282cd2d849d997b1b39acd710c>) but I don’t know how to use it. Andrew, if you read this ;-)
>> 
>> I wonder if the loader could be more efficient at loading off the SD card. The SD cards can do 10-20MB/s easily, and the ARM kernel is around 5MB last time I checked. 5 seconds is 1MB/s, which is much slower than we know the hardware can do. I don't know if that's because the callback mechanism in u-boot.bin is so slow (that's what ubldr uses to load the kernel), or there's something inherently slow about our loader. Some analysis here might be quite useful. I'm guessing that we're not getting all the benefits from streaming read mode because we're doing I/Os that are tiny for some reason. But I haven't looked to make sure.
>> 
>> What's the speed when booting directly from u-boot.bin?
>> 
>> Warner
>> 
>> 
>> 
>>  
>>> Le 25 mars 2016 à 06:48, Warner Losh <imp at bsdimp.com <mailto:imp at bsdimp.com>> a écrit :
>>> 
>>> 
>>> 
>>> On Thu, Mar 24, 2016 at 11:25 PM, Bernd Walter <ticso at cicely7.cicely.de <mailto:ticso at cicely7.cicely.de>> wrote:
>>> On Thu, Mar 24, 2016 at 04:29:26PM +0100, Sylvain Garrigues wrote:
>>> > Hello,
>>> >
>>> > I have written a small (ugly) patch to be able to boot a kernel directly without the ubldr loader while still using an external DTB (Linux-style booting may pass the DTB location pointer in the r2 register).
>>> > The patch is here: https://reviews.freebsd.org/differential/diff/14577/ <https://reviews.freebsd.org/differential/diff/14577/>
>>> >
>>> > I tested my patch successfully with the QEMU emulator with the -dtb option available in recent versions, using a VERSATILEPB kernel without FDT_STATIC.
>>> > # qemu-system-arm -M versatilepb -m 128M -kernel versatile.flash -cpu arm1176 -dtb versatilepb.dtb
>>> > FYI, once the kernel is built, here is the script to build the versatile.flash (adapted from gonzo, no longer need to clear the r0-r3 registers): https://reviews.freebsd.org/P92 <https://reviews.freebsd.org/P92> <https://reviews.freebsd.org/P92 <https://reviews.freebsd.org/P92>>
>>> >
>>> > I also tested successfully my patch on my Raspberry Pi 2 using U-BOOT and the RPI2 kernel with my patch applied (and the LINUX_BOOT_ABI option):
>>> > u-boot> fatload mmc 0 0x200000 kernel.bin
>>> > u-boot> go 0x200000
>>> >
>>> > That works.
>>> >
>>> > So now I thought I could even bypass u-boot and launch kernel.bin directly from the Pi firmware??? But it doesn???t work, I don???t understand why, and that is why I am writing here.
>>> > Here is the config.txt which I thought would work:
>>> >
>>> > kernel=kernel.bin (instead of u-boot.bin)
>>> > kernel_address=0x200000 (line added because a FreeBSD kernel needs to be loaded on a 1MB or 2MB boundary)
>>> > device_tree=rpi2.dtb
>>> > device_tree_address=0x100
>>> > disable_commandline_tags=1
>>> >
>>> > What am I missing? Is it even possible to boot kernel.bin directly on the Pi (with my patch)? I found this static minimalist loader from Andrew here: https://github.com/freebsd/freebsd/commit/074d37d46c3f9b282cd2d849d997b1b39acd710c <https://github.com/freebsd/freebsd/commit/074d37d46c3f9b282cd2d849d997b1b39acd710c><https://github.com/freebsd/freebsd/commit/074d37d46c3f9b282cd2d849d997b1b39acd710c <https://github.com/freebsd/freebsd/commit/074d37d46c3f9b282cd2d849d997b1b39acd710c>> - does it mean such a loader is necessary before the kernel?
>>> >
>>> > Thanks,
>>> > Sylvain
>>> >
>>> > PS: I know the « official and supported » way of booting FreeBSD on the Pi is the u-boot + ubldr combination. I just would like to finish this experiment and understand why kernel.bin cannot be booted directly.
>>> 
>>> I don't know if it is part of ubldr, or if there is any additional code,
>>> which runs before ubldr, but the Pis won't boot with the ARM CPU.
>>> Somewhere in the boot path is GPU bootcode, which then enables the ARM
>>> CPU.
>>> You also need low level HW init, such as setting up the clocks and RAM,
>>> which is not done in the Kernel.
>>> 
>>> What you are missing it the hardware init code that's in u-boot.bin. This code sets up
>>> the board after the on-board GPU loads it into the ARM's address space. That code
>>> sets up the ARM side of the world and hands off the code to ubldr which finds the dtb
>>> and does the normal /boot/loader things as well (loading modules, setting tunables
>>> and the like) then hands off to the kernel. You can eliminate ubldr without a huge
>>> amount of effort, as you've found. However, eliminating u-boot.bin is going to be
>>> a lot more work/
>>> 
>>> tl;dr: The interface between u-boot.bin and ubldr/kernel.bin is quite a bit different
>>> than between the initial bootstrap and u-boot.bin. So you can't just drop in kernel.bin
>>> and have it work without replicating that interface.
>>> 
>>> Warner
> 
> 