From nobody Thu Jan 11 14:34:44 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T9nJX2kQQz568Lq; Thu, 11 Jan 2024 14:35:24 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T9nJW1xwGz4XRq; Thu, 11 Jan 2024 14:35:23 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=I1Z2cFie; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::52d as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-ed1-x52d.google.com with SMTP id 4fb4d7f45d1cf-558b5f4cf29so166642a12.3; Thu, 11 Jan 2024 06:35:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704983721; x=1705588521; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=wQfoGh9ejTq8YSQGbzdFQeIoOb24oCgvFpuh71KirxI=; b=I1Z2cFieXopE3LxqNktP+m69opTFsQ7sOjxRyfIv9HvfA5C1lDYSMrNdaE1TTLankZ 97toZamkg7/a/sDhBc5BuqS/XKa5F6fiJpGotVQBNJyDyhIvlUsSJE/uXhvGv64PeRcs SHgi2691pE0u1I6IqcjPf9Xf6I1j8rPdObyjkpKBMNarM2L4EoV34noZ8dDw3kLU8Hkv rMBahlLigP2eGZkHAz2+KRD/2SnRCMB9nNQcOi0hdcQIJ1vWzQExWXR7ZwgDxq9FoReO t15z8JvTi73r2D9KPT/1wvhxcvG3O/B8GM2SXYRJOqOzv4XDnIp4p9p2b7s8IRWDn3J0 ryvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704983721; x=1705588521; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=wQfoGh9ejTq8YSQGbzdFQeIoOb24oCgvFpuh71KirxI=; b=ExxAnFAvTObLyL/caHbAnLN4+aQFQMqqMNvAKzXmjvAx1KfzhNC1RDAtraMaMt/ka8 razZ7fGaP679p67EkbaVenrPIw3iklzoT7gn7W3QB9lZ9yQt99tUMBbpXJ/JEE7DMFXY /rTistsBSVgdHdgDuqUt75yejLrZMrLTxAHcBKmHQsR6s93BVCQvuJiZiAo1yVdYi3H2 VKKz/W1z7lOdamXQk6zUidSOz8iX/jUPxphWTtwyTzHCCH0MzvfsC+2jemRNZrkiIKDS NKBQ3nrCyH9xNIc6XB//0p+FCoF4k6sbyAHYA58iqb/skjhw8zC/B79vDTBtjZtd5F9Q LSJg== X-Gm-Message-State: AOJu0YwIdUwlIAlefmqBJwWoHmozYfu9S0NFjBx2Nd/fmQqwf6H8FXVi rkXn8/WBCxisJWgmmCAczczp5yoTdDOc5BnvAJYuMMht0HQ= X-Google-Smtp-Source: AGHT+IG4qu5DlaMbyL8Oxz7EUjSYvfK2PBKhCDPKTIaTF1fjfkmOsXXzFWjsC02mhnCy41j/0VtlhLKGZbST7FEwquM= X-Received: by 2002:a17:907:7da1:b0:a29:445f:a1 with SMTP id oz33-20020a1709077da100b00a29445f00a1mr917859ejc.140.1704983721038; Thu, 11 Jan 2024 06:35:21 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 From: Mario Marietto Date: Thu, 11 Jan 2024 15:34:44 +0100 Message-ID: Subject: Unknown option "XENHVM" for Arm 32 and for arm 64 ; missing code detected. To: freebsd-hackers , FreeBSD virtualization , FreeBSD Mailing List Content-Type: multipart/alternative; boundary="0000000000000a58cd060eac75c0" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.54 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; NEURAL_HAM_MEDIUM(-0.54)[-0.542]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52d:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-virtualization@freebsd.org,freebsd-questions@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+] X-Rspamd-Queue-Id: 4T9nJW1xwGz4XRq --0000000000000a58cd060eac75c0 Content-Type: text/plain; charset="UTF-8" Hello. I'm trying to boot FreeBSD for arm32 bit as DomU on my ARM Chromebook. Basically,with the help of the xen developers I have patched the FreeBSD kernel and I've been able to boot it as if it was a zImage format file. I've enabled all the xen options inside the Linux kernel file used for the domU and this is what happened when I tried to boot the FreeBSD kernel file : mario@devuan-bunsen:/mnt/zroot2/zroot2/OS/Chromebook/freebsd-xen/domU-freebsd# ./start-freebsd Parsing config from freebsd.cfg libxl: debug: libxl_create.c:2081:do_domain_create: ao 0x442780: create: how=(nil) callback=(nil) poller=0x43cc50 libxl: detail: libxl_create.c:662:libxl__domain_make: passthrough: disabled libxl: debug: libxl_arm.c:148:libxl__arch_domain_prepare_config: Configure the domain libxl: debug: libxl_arm.c:151:libxl__arch_domain_prepare_config: - Allocate 0 SPIs libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=unknown specification=xen libxl: debug: libxl_device.c:452:libxl__device_disk_set_backend: Disk vdev=xvda, using backend phy libxl: debug: libxl_create.c:1342:initiate_domain_create: Domain 1:running bootloader libxl: debug: libxl_bootloader.c:417:libxl__bootloader_run: Domain 1:no bootloader configured, using user supplied kernel libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x43d8f0: deregister unregistered domainbuilder: detail: xc_dom_allocate: cmdline="console=hvc0", features="" domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x40008+0x966 at 0xb5972000 domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x40008000 -> 0x4096e000 (pfn 0x40008 + 0x966 pages) domainbuilder: detail: xc_dom_load_zimage_kernel: called domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg 0x40008000-0x4096e000 domainbuilder: detail: xc_dom_load_zimage_kernel: copy 9851212 bytes from blob 0xb62d8000 to dst 0xb5972000 domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x48000+0x1 at 0xb6f0a000 domainbuilder: detail: xc_dom_alloc_segment: devicetree : 0x48000000 -> 0x48001000 (pfn 0x48000 + 0x1 pages) domainbuilder: detail: alloc_magic_pages: called domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x48001000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: bootearly: doing nothing domainbuilder: detail: start_info_arm: called domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 58 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 9620 kB domainbuilder: detail: domU mmap : 9628 kB domainbuilder: detail: vcpu_arm32: called domainbuilder: detail: Initial state CPSR 0x400001d3 PC 0x40008000 domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[0] -> d0 0x39000 domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[1] -> d0 0x39001 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=phy specification=xen libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: register slotnum=3 libxl: debug: libxl_create.c:2120:do_domain_create: ao 0x442780: inprogress: poller=0x43cc50, flags=i libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: event epath=/local/domain/0/backend/vbd/1/51712/state libxl: debug: libxl_event.c:1054:devstate_callback: backend /local/domain/0/backend/vbd/1/51712/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: event epath=/local/domain/0/backend/vbd/1/51712/state libxl: debug: libxl_event.c:1051:devstate_callback: backend /local/domain/0/backend/vbd/1/51712/state wanted state 2 ok libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: deregister slotnum=3 libxl: debug: libxl_device.c:1150:device_backend_callback: Domain 1:calling device_backend_cleanup libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x4410e4: deregister unregistered libxl: debug: libxl_linux.c:194:libxl__hotplug_disk: Domain 1:Args and environment ready libxl: debug: libxl_device.c:1251:device_hotplug: Domain 1:calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_device.c:1252:device_hotplug: Domain 1:extra args: libxl: debug: libxl_device.c:1260:device_hotplug: Domain 1:env: libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: script: /etc/xen/scripts/block libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_TYPE: vbd libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_PATH: backend/vbd/1/51712 libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_BASE_PATH: backend libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block add libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x441168: deregister unregistered libxl: debug: libxl_linux.c:211:libxl__get_hotplug_script_info: Domain 1:num_exec 1, not running hotplug scripts libxl: debug: libxl_device.c:1236:device_hotplug: Domain 1:No hotplug script to execute libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x441168: deregister unregistered libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: register slotnum=3 libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: event epath=/local/domain/1/console/tty libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: event epath=/local/domain/1/console/tty libxl: debug: libxl_event.c:2403:libxl__ao_progress_report: ao 0x442780: progress report: ignored libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: deregister slotnum=3 libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x43e7cc: deregister unregistered libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete, rc=0 libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy libxl: debug: libxl_domain.c:704:libxl_domain_unpause: Domain 1:ao 0x442780: create: how=(nil) callback=(nil) poller=0x43cc50 libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete, rc=0 libxl: debug: libxl_domain.c:712:libxl_domain_unpause: Domain 1:ao 0x442780: inprogress: poller=0x43cc50, flags=ic libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy xencall:buffer: debug: total allocations:98 total releases:98 xencall:buffer: debug: current allocations:0 maximum allocations:3 xencall:buffer: debug: cache current size:3 xencall:buffer: debug: cache hits:86 misses:3 toobig:9 xencall:buffer: debug: total allocations:0 total releases:0 xencall:buffer: debug: current allocations:0 maximum allocations:0 xencall:buffer: debug: cache current size:0 xencall:buffer: debug: cache hits:0 misses:0 toobig:0 FROZEN. I see these two processes enabled...but I don't know how to use FreeBSD... # ps ax 2606 ? Ssl 0:00 xl -vvvv create freebsd.cfg 2607 pts/0 Sl+ 0:00 /usr/lib/xen-4.17/bin/xenconsole 1 --num 0 --type pv now the fun begins...because,based on what say the xen developers : ... the ps output, it seems that ``xl create`` completed and you have the console open. So the freeze you mention is just because your FreeBSD guest is not outputting anything. As mentioned earlier, I don't think a lot of testing has been done for 32-bit Arm FreeBSD. So it is quite possible that there are some pieces of code missing. The first step is to check if the FreeBSD kernel was built with Xen options. If they are, then you will need to find out where FreeBSD is stuck (or why the console is not enabled). Unfortunately, we don't have any support to use GDB on the guest kernel. So you will have to modify FreeBSD a bit to check where it freezes. Assuming you have a debug build of the hypervisor, then you can sprinkle the FreeBSD boot code with the assembly instruction 'hvc 0xfffc'. When this is reached, this will issue an hypercall that will print on the Xen console that the given instruction has reached (the PC will be printed). There are other useful 'hvc' calls implemented by Xen for low level debugging. You can look at do_debug_trap() in Xen code. I've added this to GENERIC kernel of FreeBSD guest : # Xen HVM Guest Optimizations # NOTE: XENHVM depends on xenpci. They must be added or removed together. options XENHVM # Xen HVM kernel infrastructure device xenpci # Xen HVM Hypervisor services driver Unfortunately,it is not accepted : unknown option "XENHVM" I've found these parameters ONLY on the file /usr/src/sys/conf/files.amd64 : x86/xen/pv.c optional xenhvm x86/xen/pvcpu_enum.c optional xenhvm x86/xen/xen_pci_bus.c optional xenhvm Look at the prefix x86. It is not for arm. It seems there is a piece of code that hasn't been written. Is there a chance to add it ? Does anyone want to help ? No arch/arm64 does not contain Xen in KERNCONF either. Surprisingly files.x86 invokes a different set of files. I don't know what build it is. x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xenhvm x86/xen/xen_apic.c optional xenhvm x86/xen/xenpv.c optional xenhvm x86/xen/xen_msi.c optional xenhvm x86/xen/xen_nexus.c optional xenhvm -- Mario. --0000000000000a58cd060eac75c0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello.

I'm trying to boot FreeBSD for arm32 bit as DomU on my ARM Chromebook.= =20 Basically,with the help of the xen developers I have patched the FreeBSD kernel and I've been able to boot it as if it was a zImage format file= .

I've enabled all the xen options inside= =20 the Linux kernel file used for the domU and this is what happened when I tried to boot the FreeBSD kernel file :


mario@devuan-bunsen:/mnt/zroot2/zroot2/OS/Chromebo=
ok/freebsd-xen/domU-freebsd# ./start-freebsd

Parsing config from freebsd.cfg

libxl: debug: libxl_create.c:2081:do_domain_create: ao 0x442780: create: ho=
w=3D(nil) callback=3D(nil) poller=3D0x43cc50

libxl: detail: libxl_create.c:662:libxl__domain_make: passthrough: disabled
libxl: debug: libxl_arm.c:148:libxl__arch_domain_prepare_config: Configure =
the domain
libxl: debug: libxl_arm.c:151:libxl__arch_domain_prepare_config: - Allocate=
 0 SPIs

libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk vdev=
=3Dxvda spec.backend=3Dunknown specification=3Dxen

libxl: debug: libxl_device.c:452:libxl__device_disk_set_backend: Disk vdev=
=3Dxvda, using backend phy
libxl: debug: libxl_create.c:1342:initiate_domain_create: Domain 1:running =
bootloader
libxl: debug: libxl_bootloader.c:417:libxl__bootloader_run: Domain 1:no boo=
tloader configured, using user supplied kernel

libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=3D0x4=
3d8f0: deregister unregistered domainbuilder: detail: xc_dom_allocate: cmdl=
ine=3D"console=3Dhvc0", features=3D""

domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x4000=
8+0x966 at 0xb5972000
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x40008000 -&=
gt; 0x4096e000  (pfn 0x40008 + 0x966 pages)
domainbuilder: detail: xc_dom_load_zimage_kernel: called
domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg 0x40008000-0x4=
096e000
domainbuilder: detail: xc_dom_load_zimage_kernel: copy 9851212 bytes from b=
lob 0xb62d8000 to dst 0xb5972000
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x4800=
0+0x1 at 0xb6f0a000
domainbuilder: detail: xc_dom_alloc_segment:   devicetree   : 0x48000000 -&=
gt; 0x48001000  (pfn 0x48000 + 0x1 pages)
domainbuilder: detail: alloc_magic_pages: called
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x48001000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: bootearly: doing nothing
domainbuilder: detail: start_info_arm: called
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 58 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 9620 kB
domainbuilder: detail:       domU mmap          : 9628 kB
domainbuilder: detail: vcpu_arm32: called
domainbuilder: detail: Initial state CPSR 0x400001d3 PC 0x40008000
domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[0] -> d0 0x39000
domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[1] -> d0 0x39001
domainbuilder: detail: xc_dom_release: called

libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk vdev=
=3Dxvda spec.backend=3Dphy specification=3Dxen

libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch w=3D0x441=
0e4 wpath=3D/local/domain/0/backend/vbd/1/51712/state token=3D3/0: register=
 slotnum=3D3

libxl: debug: libxl_create.c:2120:do_domain_create: ao 0x442780: inprogress=
: poller=3D0x43cc50, flags=3Di

libxl: debug: libxl_event.c:750:watchfd_callback: watch w=3D0x4410e4 wpath=
=3D/local/domain/0/backend/vbd/1/51712/state token=3D3/0: event epath=3D/lo=
cal/domain/0/backend/vbd/1/51712/state

libxl: debug: libxl_event.c:1054:devstate_callback: backend /local/domain/0=
/backend/vbd/1/51712/state wanted state 2 still waiting state 1

libxl: debug: libxl_event.c:750:watchfd_callback: watch w=3D0x4410e4 wpath=
=3D/local/domain/0/backend/vbd/1/51712/state token=3D3/0: event epath=3D/lo=
cal/domain/0/backend/vbd/1/51712/state

libxl: debug: libxl_event.c:1051:devstate_callback: backend /local/domain/0=
/backend/vbd/1/51712/state wanted state 2 ok

libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch w=3D0x4=
410e4 wpath=3D/local/domain/0/backend/vbd/1/51712/state token=3D3/0: deregi=
ster slotnum=3D3

libxl: debug: libxl_device.c:1150:device_backend_callback: Domain 1:calling=
 device_backend_cleanup
libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=3D0x4=
410e4: deregister unregistered
libxl: debug: libxl_linux.c:194:libxl__hotplug_disk: Domain 1:Args and envi=
ronment ready
libxl: debug: libxl_device.c:1251:device_hotplug: Domain 1:calling hotplug =
script: /etc/xen/scripts/block add
libxl: debug: libxl_device.c:1252:device_hotplug: Domain 1:extra args:
libxl: debug: libxl_device.c:1260:device_hotplug: Domain 1:env:
libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: script: /etc/xe=
n/scripts/block
libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_TYPE: vb=
d
libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_PATH: ba=
ckend/vbd/1/51712
libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_BASE_PAT=
H: backend
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execu=
te: /etc/xen/scripts/block add
libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=3D0x4=
41168: deregister unregistered
libxl: debug: libxl_linux.c:211:libxl__get_hotplug_script_info: Domain 1:nu=
m_exec 1, not running hotplug scripts
libxl: debug: libxl_device.c:1236:device_hotplug: Domain 1:No hotplug scrip=
t to execute
libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=3D0x4=
41168: deregister unregistered

libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch w=3D0x43e=
7cc wpath=3D/local/domain/1/console/tty token=3D3/1: register slotnum=3D3

libxl: debug: libxl_event.c:750:watchfd_callback: watch w=3D0x43e7cc wpath=
=3D/local/domain/1/console/tty token=3D3/1: event epath=3D/local/domain/1/c=
onsole/tty

libxl: debug: libxl_event.c:750:watchfd_callback: watch w=3D0x43e7cc wpath=
=3D/local/domain/1/console/tty token=3D3/1: event epath=3D/local/domain/1/c=
onsole/tty

libxl: debug: libxl_event.c:2403:libxl__ao_progress_report: ao 0x442780: pr=
ogress report: ignored

libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch w=3D0x4=
3e7cc wpath=3D/local/domain/1/console/tty token=3D3/1: deregister slotnum=
=3D3

libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=3D0x4=
3e7cc: deregister unregistered
libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete,=
 rc=3D0
libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy

libxl: debug: libxl_domain.c:704:libxl_domain_unpause: Domain 1:ao 0x442780=
: create: how=3D(nil) callback=3D(nil) poller=3D0x43cc50

libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete,=
 rc=3D0

libxl: debug: libxl_domain.c:712:libxl_domain_unpause: Domain 1:ao 0x442780=
: inprogress: poller=3D0x43cc50, flags=3Dic

libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy

xencall:buffer: debug: total allocations:98 total releases:98
xencall:buffer: debug: current allocations:0 maximum allocations:3
xencall:buffer: debug: cache current size:3
xencall:buffer: debug: cache hits:86 misses:3 toobig:9
xencall:buffer: debug: total allocations:0 total releases:0
xencall:buffer: debug: current allocations:0 maximum allocations:0
xencall:buffer: debug: cache current size:0
xencall:buffer: debug: cache hits:0 misses:0 toobig:0

FROZEN.

I see these two processes enabled...but I don't know how to use FreeBSD= ...

=09 =09 
# ps ax
2606 ?        Ssl    0:00 xl -vvvv create freebsd.cfg
2607 pts/0    Sl+    0:00 /usr/lib/xen-4.17/bin/xenconsole 1 --num 0 --type=
 pv

now the fun begins...because,based on what say the xen developers :

=09
=09
... the ps output, it seems that ``xl create`` completed and you have the console open. So the freeze you mention is just because your=20 FreeBSD guest is not outputting anything. As mentioned earlier, I don't= =20 think a lot of testing has been done for 32-bit Arm FreeBSD. So it is=20 quite possible that there are some pieces of code missing. The first=20 step is to check if the FreeBSD kernel was built with Xen=20 options. If they are, then you will need to find out where FreeBSD is=20 stuck (or why the console is not enabled). Unfortunately, we don't have= =20 any support to use GDB on the guest kernel. So you will have to modify=20 FreeBSD a bit to check where it freezes. Assuming you have a debug=20 build of the hypervisor, then you can sprinkle the FreeBSD boot code=20 with the assembly instruction 'hvc 0xfffc'. When this is reached, t= his=20 will issue an hypercall that will print on the Xen console that the given= =20 instruction has reached (the PC will be printed). There are other useful=20 'hvc' calls implemented by Xen for low level debugging. You can loo= k at=20 do_debug_trap() in Xen code.
=09

I've added this to GENERIC kernel of FreeBS= D guest :

=09 =09 
# Xen HVM Guest Optimizations

# NOTE: XENHVM depends on xenpci.  They must be added or removed togeth=
er.

options     XENHVM            # Xen HVM kernel infrastructure
device      xenpci            # Xen HVM Hypervisor services driver

<=
/code>
Unfortunately,it is not accepted :

unknown option "XENHVM"


I've found these parameters ONLY on the file /usr/src/sys/conf/files.amd64 :

<=
/code>

		
x86/xen/pv.c            optional    xenhvm
x86/xen/pvcpu_enum.c    optional    xenhvm
x86/xen/xen_pci_bus.c   optional    xenhvm

Look at the prefix x86. It is not for arm. It seems there is a piece of code that hasn't been written. Is there a chance t= o add it ? Does anyone want to help ?

No arch/arm64= does not contain Xen in KERNCONF either.

Surprisingly files.x86 invokes a diff= erent set of files. I don't know what build it is.

=09 =09 
x86/xen/hvm.c            optional    xenhvm
x86/xen/xen_intr.c       optional    xenhvm
x86/xen/xen_apic.c       optional    xenhvm
x86/xen/xenpv.c          optional    xenhvm
x86/xen/xen_msi.c        optional    xenhvm
x86/xen/xen_nexus.c      optional    xenhvm
<= /div>
--
Mario.
=
--0000000000000a58cd060eac75c0--