Re: Should we boot the FreeBSD kernel in ELF format or in zImage format ? How?

I really don't know where to start. It sounds too technical for me. I don't
even know what to ask to understand where to start. mmm...can someone
provide some documentation ? but not too much dense with technicalities.
Thanks.

On Sat, Nov 25, 2023 at 6:02 PM Warner Losh <imp@bsdimp.com> wrote:

>
>
> On Sat, Nov 25, 2023 at 4:41 AM Mario Marietto <marietto2008@gmail.com>
> wrote:
>
>> Hello to everyone.
>>
>> we have just virtualized Debian 12 on our arm (32 bit) Chromebook. As
>> host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It
>> works great. But our goal is different. We want to virtualize FreeBSD as
>> domU. Can we have a working Xen PV network driver for a FreeBSD arm guest
>> ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm.
>> I would like to know if Julien's work was accepted upstream by FreeBSD, in
>> which case FreeBSD as a Xen guest on arm should work if we enable the Xen
>> PV drivers in the FreeBSD on arm kernel. If Julien's work was not accepted
>> upstream by FreeBSD, we will have to find his patches and apply them
>> ourselves to the FreeBSD on arm kernel.
>>
>> We found these slides :
>>
>>
>> https://events.static.linuxfound.org/sites/events/files/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf
>>
>> Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what
>> we want to find.
>>
>> It looks like when that slide presentation was written, there were some
>> limitations on FreeBSD Xen guests. For example, for our debian bookworm
>> guest, I am using vcpus = '2' to match the number of real cpus on our
>> Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD
>> guest, so I will need to change that vcpus = '1' in the FreeBSD guest
>> config unless support for 2 or more vcpus was added later, which is
>> possible because that slide presentation is 9 years old.
>>
>> Here is where I would expect to find the XENHVM FreeBSD on arm kernel
>> config file:
>>
>> https://cgit.freebsd.org/src/tree/sys/arm/conf
>>
>> But it is not there unless I am not understanding something correctly.
>> For now, unfortunately conclude that the support for Xen on arm that Julien
>> Grall mentioned in that slide presentation 9 years ago was never added to
>> the official FreeBSD source code. I am searching the web now to see if the
>> patches that Julien Grall wrote are still posted somewhere online. If we
>> cannot find them, we can ask here and on the xen-users mailing list. Julien
>> regularly reads that list and responds to question about Xen on arm, so I
>> think he will tell us how to find the patches if we cannot find them online.
>>
>> According to this page from the FreeBSD wiki:
>>
>> https://wiki.freebsd.org/Xen
>>
>> I think FreeBSD only supports Xen on x86, not arm. So this is going to be
>> a bit of a challenge to get a Xen FreeBSD guest on arm working. We know
>> Julien Grall has some patches that made it work in the past !
>>
>> I found a slightly newer slide presentation by Julien here:
>>
>> https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd
>>
>> It is about the same, but it mentions the GENERIC FreeBSD kernel supports
>> Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen on
>> arm 32 bit, which I haven't found online yet.
>>
>> Please,take a look at this output of the linux kernel that can boot on
>> Xen, and the FreeBSD kernel that cannot :
>>
>>
>> % file zImage-6.1.59-stb-xen-cbe+
>> zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little-endian)
>>
>> % file FREEBSD-XENVIRT
>> FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not stripped
>>
>>
>> The FreeBSD kernel that won't boot is in ELF format but the Linux kernel
>> that does boot is in zImage format.
>>
>> I spent time reading the docs on xenbits.xenproject.org, and according
>> to those docs Xen on arm only knows how to boot a kernel in the zImage
>> format, so the FreeBSD kernel is in a format that modern Xen incorrectly
>> detects as an x86 kernel.
>>
>> I also watched Julien Grall's 30 minute video presentation of his work to
>> boot FreeBSD/arm on Xen at FOSDEM 2014 here :
>>
>> https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/
>>
>> In that video, and in other places, Julien mentions that the boot ABI for
>> FreeBSD/arm on Xen was not yet developed and he was getting occasional
>> crashes and needed to investigate the problem. He mentioned the zImage ABI
>> that Linux uses, but pointed out FreeBSD does not use that format, and back
>> then it was an open question which format to use to boot FreeBSD/arm on
>> Xen. Unfortunately, nine years later, the only supported format is still
>> the zImage format that Linux uses.
>>
>> It looks like Julien's work back then was using an ELF binary to boot
>> FreeBSD/arm on Xen instead of the supported zImage format that Linux uses
>> and the modern Xen toolstack exits with an error when trying to boot the
>> FreeBSD ELF formatted binary that Julien's patch creates. So the best
>> solution would be to try to port the rules to build a FreeBSD kernel in the
>> zImage format instead of the ELF format. I have been studying the Makefiles
>> in Linux to see how Linux builds the Linux arm kernel in the zImage format,
>> but it is not trivial to understand
>>
>
> Look at kernel.bin in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. It
> should be easy to adapt the target to build that. I've done similar things
> with u-boot formats in the past, but that was 4 employers and 20 years ago
> now.
>
> This path is not well trod. I do know that arm64 virtualization with bhyve
> is hitting the tree. I'm not sure how easy/hard this will be to modernize.
> I'm interested to see how your explorations go.
>
> Warner
>


-- 
Mario.