From nobody Sat Nov 25 11:41:03 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Scqgp4tp8z52FHM; Sat, 25 Nov 2023 11:41:42 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Scqgn5Cnzz4Vnh; Sat, 25 Nov 2023 11:41:41 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=hoiORWyH; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::534 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-548f0b7ab11so3744841a12.1; Sat, 25 Nov 2023 03:41:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700912500; x=1701517300; darn=freebsd.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=azjIXGhUpx8mRf9YywHFybitCenT4RIf9yxVGY4Q2P0=; b=hoiORWyHZ5ZxPXQkm9pn65hjiDLuv7C/MJ+Vxz5xypdrXDMwnzlV8JWaI53IszeI+x sJDEygaYXlOkCGV/PCWqekyH627QP09CegAcO9Fc1Mxqp781bWofyG43JTwTUyoEAqDr iUnRBD0uOuopLKrfNdJonaBS8MtKD4p3NaBqvS7BjiFKDYAXZJTmMy4HOFCG+6jtbJIT qSOQGpn9Mo/Xk/yIV6i/d1853ykmlW0fHsQPbO+M8+fRvpKFXV8Bsm3NWnbMcb50kTcj 5L/IgqnD6wN1Ksx4HkARTgp+HCsPA7dHA1Q2xJVZUxbvY9H1NtHAOxNi6fRZ/iExKBQc goTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700912500; x=1701517300; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=azjIXGhUpx8mRf9YywHFybitCenT4RIf9yxVGY4Q2P0=; b=vW1Is14FHhJjV0zzIrmZvrhJ11v5iETF1ImEnukYrtwYoDKjJJ7UToOeRfHBLqhFT2 ANl4vj8X+RHgBcvoiwfbJtJR53NTOE1wSc12MpMZaIhZJnQujozrGqpi82Tt4glIe/ov qpk+K5VdlXjQXDqY86kIkfjg0K8oK/hO87Dk+AYv3ZPZBSJBKSY7JZWstRInkhgeLsWL T5q4uLuPsNULsFKjC4V4wZf+xPf1QCEcvogWl4VPnV3uqflR+qxhbRCOLqVkmJygHIbN Q9Q/ulFue3MBCXgJG263suEISC2K8X98l4h9IKSz9WPytRrh9CNI60Y+W8/Uxru6LaVA iuqw== X-Gm-Message-State: AOJu0YzO05HLZ5j3VJQfFJZVZoGmeHBUQse1MK6iXGWzzXWvR+4THN30 v0VuaTWrFjJldNB0EGVQDbe5cUMif6pn7zNx+SeVVh1eTvo= X-Google-Smtp-Source: AGHT+IGpCIUAyjFh3G2HUz4/okVYKzia3wH/NAaFC75uH5L3c2mUsSVPHkOyuVI0/FhJpw74VS3WFG/LcW4W1pbsWhA= X-Received: by 2002:a17:906:15d:b0:a0b:6d32:2e09 with SMTP id 29-20020a170906015d00b00a0b6d322e09mr1386684ejh.36.1700912499758; Sat, 25 Nov 2023 03:41:39 -0800 (PST) List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Sat, 25 Nov 2023 12:41:03 +0100 Message-ID: Subject: Fwd: Should we boot the FreeBSD kernel in ELF format or in zImage format ? How? To: freebsd-hackers , freebsd-arm@freebsd.org, FreeBSD Current , FreeBSD Mailing List , freebsd-xen@freebsd.org, royger@freebsd.org Content-Type: multipart/alternative; boundary="00000000000057995f060af88d0b" X-Spamd-Result: default: False [-2.48 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.995]; NEURAL_HAM_LONG(-0.85)[-0.854]; NEURAL_HAM_MEDIUM(-0.63)[-0.626]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::534:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-arm@freebsd.org,freebsd-current@freebsd.org,freebsd-questions@freebsd.org,freebsd-xen@freebsd.org]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4Scqgn5Cnzz4Vnh X-Spamd-Bar: -- --00000000000057995f060af88d0b Content-Type: text/plain; charset="UTF-8" Hello to everyone. we have just virtualized Debian 12 on our arm (32 bit) Chromebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's work was not accepted upstream by FreeBSD, we will have to find his patches and apply them ourselves to the FreeBSD on arm kernel. We found these slides : https://events.static.linuxfound.org/sites/events/files/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what we want to find. It looks like when that slide presentation was written, there were some limitations on FreeBSD Xen guests. For example, for our debian bookworm guest, I am using vcpus = '2' to match the number of real cpus on our Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD guest, so I will need to change that vcpus = '1' in the FreeBSD guest config unless support for 2 or more vcpus was added later, which is possible because that slide presentation is 9 years old. Here is where I would expect to find the XENHVM FreeBSD on arm kernel config file: https://cgit.freebsd.org/src/tree/sys/arm/conf But it is not there unless I am not understanding something correctly. For now, unfortunately conclude that the support for Xen on arm that Julien Grall mentioned in that slide presentation 9 years ago was never added to the official FreeBSD source code. I am searching the web now to see if the patches that Julien Grall wrote are still posted somewhere online. If we cannot find them, we can ask here and on the xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the patches if we cannot find them online. According to this page from the FreeBSD wiki: https://wiki.freebsd.org/Xen I think FreeBSD only supports Xen on x86, not arm. So this is going to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past ! I found a slightly newer slide presentation by Julien here: https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd It is about the same, but it mentions the GENERIC FreeBSD kernel supports Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen on arm 32 bit, which I haven't found online yet. Please,take a look at this output of the linux kernel that can boot on Xen, and the FreeBSD kernel that cannot : % file zImage-6.1.59-stb-xen-cbe+ zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little-endian) % file FREEBSD-XENVIRT FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not stripped The FreeBSD kernel that won't boot is in ELF format but the Linux kernel that does boot is in zImage format. I spent time reading the docs on xenbits.xenproject.org, and according to those docs Xen on arm only knows how to boot a kernel in the zImage format, so the FreeBSD kernel is in a format that modern Xen incorrectly detects as an x86 kernel. I also watched Julien Grall's 30 minute video presentation of his work to boot FreeBSD/arm on Xen at FOSDEM 2014 here : https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/ In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting occasional crashes and needed to investigate the problem. He mentioned the zImage ABI that Linux uses, but pointed out FreeBSD does not use that format, and back then it was an open question which format to use to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only supported format is still the zImage format that Linux uses. It looks like Julien's work back then was using an ELF binary to boot FreeBSD/arm on Xen instead of the supported zImage format that Linux uses and the modern Xen toolstack exits with an error when trying to boot the FreeBSD ELF formatted binary that Julien's patch creates. So the best solution would be to try to port the rules to build a FreeBSD kernel in the zImage format instead of the ELF format. I have been studying the Makefiles in Linux to see how Linux builds the Linux arm kernel in the zImage format, but it is not trivial to understand. -- Mario. --00000000000057995f060af88d0b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello to every= one.

we have just virtualized Debian 12 on our arm (32 bit) Chr= omebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It=20 works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm=20 guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by= =20 FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we=20 enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's wor= k was not accepted upstream by FreeBSD, we will have to find his patches=20 and apply them ourselves to the FreeBSD on arm kernel.

We found these slides :

https://events.static.linuxfound.org/sites/events/file= s/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf

Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what = we want to find.

It looks like when that slide presentation was written, there were=20 some limitations on FreeBSD Xen guests. For example, for our debian=20 bookworm guest, I am using vcpus =3D '2' to match the number of rea= l cpus=20 on our Chromebook, but slide 13 mentions support for only 1 VCPU with a=20 FreeBSD guest, so I will need to change that vcpus =3D '1' in the F= reeBSD=20 guest config unless support for 2 or more vcpus was added later, which=20 is possible because that slide presentation is 9 years old.

Here is where I would expect to find the XENHVM FreeBSD on arm kernel co= nfig file:

https://cgit.freebsd.org/src/tree/sys/arm/= conf

But it is not there unless I am not understanding something=20 correctly. For now, unfortunately conclude that the support for Xen on=20 arm that Julien Grall mentioned in that slide presentation 9 years ago=20 was never added to the official FreeBSD source code. I am searching the=20 web now to see if the patches that Julien Grall wrote are still posted=20 somewhere online. If we cannot find them, we can ask here and on the=20 xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the=20 patches if we cannot find them online.

According to this page from the FreeBSD wiki:

https://wiki.freebsd.org/Xen

I think FreeBSD only supports Xen on x86, not arm. So this is going=20 to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past !

I found a slightly newer slide presentation by Julien here:

https://www.slide= share.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd

It is about the same, but it mentions the GENERIC FreeBSD kernel=20 supports Xen on arm64, but still says we need the XENHVM FreeBSD config=20 for Xen on arm 32 bit, which I haven't found online yet.

Please,take a look at this output of the linux kernel that can boot on X= en, and the FreeBSD kernel that cannot :


% file zImage-6.1.59-stb-xen-cbe+
zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little=
-endian)

% file FREEBSD-XENVIRT         =20
FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dy=
namically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not=
 stripped


The FreeBSD kernel that won't boot is in ELF format but t= he Linux kernel that does boot is in zImage format.

I spent time reading the docs on xenbits.xenproject.org, and=20 according to those docs Xen on arm only knows how to boot a kernel in=20 the zImage format, so the FreeBSD kernel is in a format that modern Xen=20 incorrectly detects as an x86 kernel.

I also watched Julien Grall's 30 minute video presentation of his wo= rk to boot FreeBSD/arm on Xen at FOSDEM 2014 here :

https://archive.fosdem.or= g/2014/schedule/event/freebsd_xen_arm/

In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting=20 occasional crashes and needed to investigate the problem. He mentioned=20 the zImage ABI that Linux uses, but pointed out FreeBSD does not use=20 that format, and back then it was an open question which format to use=20 to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only=20 supported format is still the zImage format that Linux uses.

It looks like Julien's work back then was using an ELF binary to boo= t FreeBSD/arm on Xen instead of the supported zImage format that Linux=20 uses and the modern Xen toolstack exits with an error when trying to=20 boot the FreeBSD ELF formatted binary that Julien's patch creates. So= =20 the best solution would be to try to port the rules to build a FreeBSD=20 kernel in the zImage format instead of the ELF format. I have been=20 studying the Makefiles in Linux to see how Linux builds the Linux arm=20 kernel in the zImage format, but it is not trivial to understand.

=
--
M= ario.
--00000000000057995f060af88d0b--