From nobody Sat Nov 25 22:20:33 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sd5sh32Cpz528pS; Sat, 25 Nov 2023 22:21:12 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sd5sh0FJrz3fc5; Sat, 25 Nov 2023 22:21:12 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-a00191363c1so445272766b.0; Sat, 25 Nov 2023 14:21:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700950870; x=1701555670; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=t/lSNxASWRvsK2cphJI1ZwX564P/D/fkfRr4BwwJkyA=; b=jkrnr40BYs43XvUxtYhA+UdOh6tOUfNhMjLZTZBkmoTgDQG+XGp5JniHEN6Sqefl37 VIZ+LdGQUvf0omiX/o+lHFVeEOwFrqUEQZr1KVdCETfeOcR8Tj2Cce/+ESvuJiG2gpH3 daKP0mlClK4KLAFBF62bT66HAnO7qmGgjSKG7s16c3syIHWLmJsJQyJ3+OEbF+Cce21K HTaZOJr6FNSZZO28IpEmto4LWkUAabtt3Tz+mlJby3SPzJRqPyMrkODjXx+9CvxfG7NI 3Vuvy3vjDPUeM31FuTwh8DaSkXGMVYfQo/hQ9KV8qxJSgPSwbnROB0Y0/bVOW9oiZ5wH T4PA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700950870; x=1701555670; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=t/lSNxASWRvsK2cphJI1ZwX564P/D/fkfRr4BwwJkyA=; b=INwETliex+aPCGivvr38h9lXRAahV/MFa+nLqEVuqCiyI/WPomVY5b8U/6mboKAx06 jrKdeMDVNQpHTx5Su4O2Y4ylFdHcCLqjanfMjHzUMWEX0VQUwRtaO4Haj6J5x2T1q8kg BYrlok0kyP6OjAfcdMnSH0tIpG+hlYS5TPfiVMFYf+muvXa5o1+boWNnzTlLBtEz4CqM EbUIL6Iytnr94zO3FScwgBM8krwPj6PW9SSwkcMsUdUoG0+UVbPHW5ueTJIb6y5G1ggO rqGCHXooLdJA6D7fBCzHHzl/O8Ho5mG0Ob7ei+rctXfLS0lwmzXE71Qjr2yUROHIRY3K c2YA== X-Gm-Message-State: AOJu0YwQBu0HE7pjK1sEKJY81qTvK6+G9QvCiBHkru5/HMpjZHYYju4Q mI2uJC+pc9mVHMJ50IVrgtuUUj9MooAm4kOREFHY7zOyeKY= X-Google-Smtp-Source: AGHT+IEhkzwilMD6ERaL2bwBn3rOcuTDkx1lGUDZ5xkUD+poku47ky/zmkiWI0tfZHohL3AQulxcriuBDhJofy1Va5I= X-Received: by 2002:a17:906:dfcf:b0:a04:937a:f8b0 with SMTP id jt15-20020a170906dfcf00b00a04937af8b0mr5281464ejc.28.1700950869800; Sat, 25 Nov 2023 14:21:09 -0800 (PST) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Sat, 25 Nov 2023 23:20:33 +0100 Message-ID: Subject: Re: Should we boot the FreeBSD kernel in ELF format or in zImage format ? How? To: Warner Losh Cc: freebsd-hackers , freebsd-arm@freebsd.org, FreeBSD Current , FreeBSD Mailing List , freebsd-xen@freebsd.org, royger@freebsd.org Content-Type: multipart/alternative; boundary="0000000000005ff95f060b017cde" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4Sd5sh0FJrz3fc5 --0000000000005ff95f060b017cde Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I really don't know where to start. It sounds too technical for me. I don't even know what to ask to understand where to start. mmm...can someone provide some documentation ? but not too much dense with technicalities. Thanks. On Sat, Nov 25, 2023 at 6:02=E2=80=AFPM Warner Losh wrote: > > > On Sat, Nov 25, 2023 at 4:41=E2=80=AFAM Mario Marietto > wrote: > >> Hello to everyone. >> >> we have just virtualized Debian 12 on our arm (32 bit) Chromebook. As >> host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It >> works great. But our goal is different. We want to virtualize FreeBSD as >> domU. Can we have a working Xen PV network driver for a FreeBSD arm gues= t >> ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on ar= m. >> I would like to know if Julien's work was accepted upstream by FreeBSD, = in >> which case FreeBSD as a Xen guest on arm should work if we enable the Xe= n >> PV drivers in the FreeBSD on arm kernel. If Julien's work was not accept= ed >> upstream by FreeBSD, we will have to find his patches and apply them >> ourselves to the FreeBSD on arm kernel. >> >> We found these slides : >> >> >> https://events.static.linuxfound.org/sites/events/files/slides/Porting%2= 0FreeBSD%20on%20Xen%20on%20ARM%20.pdf >> >> Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what >> we want to find. >> >> It looks like when that slide presentation was written, there were some >> limitations on FreeBSD Xen guests. For example, for our debian bookworm >> guest, I am using vcpus =3D '2' to match the number of real cpus on our >> Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD >> guest, so I will need to change that vcpus =3D '1' in the FreeBSD guest >> config unless support for 2 or more vcpus was added later, which is >> possible because that slide presentation is 9 years old. >> >> Here is where I would expect to find the XENHVM FreeBSD on arm kernel >> config file: >> >> https://cgit.freebsd.org/src/tree/sys/arm/conf >> >> But it is not there unless I am not understanding something correctly. >> For now, unfortunately conclude that the support for Xen on arm that Jul= ien >> Grall mentioned in that slide presentation 9 years ago was never added t= o >> the official FreeBSD source code. I am searching the web now to see if t= he >> patches that Julien Grall wrote are still posted somewhere online. If we >> cannot find them, we can ask here and on the xen-users mailing list. Jul= ien >> regularly reads that list and responds to question about Xen on arm, so = I >> think he will tell us how to find the patches if we cannot find them onl= ine. >> >> According to this page from the FreeBSD wiki: >> >> https://wiki.freebsd.org/Xen >> >> I think FreeBSD only supports Xen on x86, not arm. So this is going to b= e >> a bit of a challenge to get a Xen FreeBSD guest on arm working. We know >> Julien Grall has some patches that made it work in the past ! >> >> I found a slightly newer slide presentation by Julien here: >> >> https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd >> >> It is about the same, but it mentions the GENERIC FreeBSD kernel support= s >> Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen o= n >> arm 32 bit, which I haven't found online yet. >> >> Please,take a look at this output of the linux kernel that can boot on >> Xen, and the FreeBSD kernel that cannot : >> >> >> % file zImage-6.1.59-stb-xen-cbe+ >> zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (lit= tle-endian) >> >> % file FREEBSD-XENVIRT >> FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV),= dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), = not stripped >> >> >> The FreeBSD kernel that won't boot is in ELF format but the Linux kernel >> that does boot is in zImage format. >> >> I spent time reading the docs on xenbits.xenproject.org, and according >> to those docs Xen on arm only knows how to boot a kernel in the zImage >> format, so the FreeBSD kernel is in a format that modern Xen incorrectly >> detects as an x86 kernel. >> >> I also watched Julien Grall's 30 minute video presentation of his work t= o >> boot FreeBSD/arm on Xen at FOSDEM 2014 here : >> >> https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/ >> >> In that video, and in other places, Julien mentions that the boot ABI fo= r >> FreeBSD/arm on Xen was not yet developed and he was getting occasional >> crashes and needed to investigate the problem. He mentioned the zImage A= BI >> that Linux uses, but pointed out FreeBSD does not use that format, and b= ack >> then it was an open question which format to use to boot FreeBSD/arm on >> Xen. Unfortunately, nine years later, the only supported format is still >> the zImage format that Linux uses. >> >> It looks like Julien's work back then was using an ELF binary to boot >> FreeBSD/arm on Xen instead of the supported zImage format that Linux use= s >> and the modern Xen toolstack exits with an error when trying to boot the >> FreeBSD ELF formatted binary that Julien's patch creates. So the best >> solution would be to try to port the rules to build a FreeBSD kernel in = the >> zImage format instead of the ELF format. I have been studying the Makefi= les >> in Linux to see how Linux builds the Linux arm kernel in the zImage form= at, >> but it is not trivial to understand >> > > Look at kernel.bin in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. I= t > should be easy to adapt the target to build that. I've done similar thing= s > with u-boot formats in the past, but that was 4 employers and 20 years ag= o > now. > > This path is not well trod. I do know that arm64 virtualization with bhyv= e > is hitting the tree. I'm not sure how easy/hard this will be to modernize= . > I'm interested to see how your explorations go. > > Warner > --=20 Mario. --0000000000005ff95f060b017cde Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I really don't know where to start. It sounds too tech= nical for me. I don't even know what to ask to understand where to star= t. mmm...can someone provide some documentation ? but not too much dense wi= th technicalities. Thanks.

On Sat, Nov 25, 2023 at 6:02=E2=80=AFPM Warn= er Losh <imp@bsdimp.com> wrote:=


On Sat, Nov 25, 2023 at 4:41=E2=80=AFAM Mario Mariet= to <marietto= 2008@gmail.com> wrote:
= Hello to everyone.

we have just virtualized Debian 12 on our ar= m (32 bit) Chromebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It=20 works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm=20 guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by= =20 FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we=20 enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's wor= k was not accepted upstream by FreeBSD, we will have to find his patches=20 and apply them ourselves to the FreeBSD on arm kernel.

We found these slides :

https://events.static.linuxfound.org/sites/events/file= s/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf

Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what = we want to find.

It looks like when that slide presentation was written, there were=20 some limitations on FreeBSD Xen guests. For example, for our debian=20 bookworm guest, I am using vcpus =3D '2' to match the number of rea= l cpus=20 on our Chromebook, but slide 13 mentions support for only 1 VCPU with a=20 FreeBSD guest, so I will need to change that vcpus =3D '1' in the F= reeBSD=20 guest config unless support for 2 or more vcpus was added later, which=20 is possible because that slide presentation is 9 years old.

Here is where I would expect to find the XENHVM FreeBSD on arm kernel co= nfig file:

https://cgit.freebsd.org/src/tree/sys/arm/= conf

But it is not there unless I am not understanding something=20 correctly. For now, unfortunately conclude that the support for Xen on=20 arm that Julien Grall mentioned in that slide presentation 9 years ago=20 was never added to the official FreeBSD source code. I am searching the=20 web now to see if the patches that Julien Grall wrote are still posted=20 somewhere online. If we cannot find them, we can ask here and on the=20 xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the=20 patches if we cannot find them online.

According to this page from the FreeBSD wiki:

https://wiki.freebsd.org/Xen

I think FreeBSD only supports Xen on x86, not arm. So this is going=20 to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past !

I found a slightly newer slide presentation by Julien here:

https://www.slide= share.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd

It is about the same, but it mentions the GENERIC FreeBSD kernel=20 supports Xen on arm64, but still says we need the XENHVM FreeBSD config=20 for Xen on arm 32 bit, which I haven't found online yet.

Please,take a look at this output of the linux kernel that can boot on X= en, and the FreeBSD kernel that cannot :


% file zImage-6.1.59-stb-xen-cbe+
zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little=
-endian)

% file FREEBSD-XENVIRT         =20
FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dy=
namically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not=
 stripped


The FreeBSD kernel that won't boot is in ELF format but t= he Linux kernel that does boot is in zImage format.

I spent time reading the docs on xenbits.xenproject.org, and=20 according to those docs Xen on arm only knows how to boot a kernel in=20 the zImage format, so the FreeBSD kernel is in a format that modern Xen=20 incorrectly detects as an x86 kernel.

I also watched Julien Grall's 30 minute video presentation of his wo= rk to boot FreeBSD/arm on Xen at FOSDEM 2014 here :

https://archive.fosdem.or= g/2014/schedule/event/freebsd_xen_arm/

In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting=20 occasional crashes and needed to investigate the problem. He mentioned=20 the zImage ABI that Linux uses, but pointed out FreeBSD does not use=20 that format, and back then it was an open question which format to use=20 to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only=20 supported format is still the zImage format that Linux uses.

It looks like Julien's work back then was using an ELF binary to boo= t FreeBSD/arm on Xen instead of the supported zImage format that Linux=20 uses and the modern Xen toolstack exits with an error when trying to=20 boot the FreeBSD ELF formatted binary that Julien's patch creates. So= =20 the best solution would be to try to port the rules to build a FreeBSD=20 kernel in the zImage format instead of the ELF format. I have been=20 studying the Makefiles in Linux to see how Linux builds the Linux arm=20 kernel in the zImage format, but it is not trivial to understand

<= /div>

Look at kernel.bin = in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. It should be e= asy to adapt the target to build that. I've done similar things with u-= boot formats in the past, but that was 4 employers and 20 years ago now.

This path is not well trod. I do know that arm64= virtualization with bhyve is hitting the tree. I'm not sure how easy/h= ard this will be to modernize. I'm interested to see how your explorati= ons go.

Warner


--
Mario.
--0000000000005ff95f060b017cde--