From nobody Wed Oct 09 15:22:43 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XNxTv52zjz5Yfp9 for ; Wed, 09 Oct 2024 15:22:59 +0000 (UTC) (envelope-from shamaz.mazum@gmail.com) Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XNxTs10jXz4p7n; Wed, 9 Oct 2024 15:22:57 +0000 (UTC) (envelope-from shamaz.mazum@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=kzERx3Wu; spf=pass (mx1.freebsd.org: domain of shamaz.mazum@gmail.com designates 2a00:1450:4864:20::131 as permitted sender) smtp.mailfrom=shamaz.mazum@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-53988c54ec8so8229238e87.0; Wed, 09 Oct 2024 08:22:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728487375; x=1729092175; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=mhdPuyLC0+7VYwR8lKpMmcoDbmwduvCzZV8i3kDuNPE=; b=kzERx3Wuy90mPtqRnyTup7akdXZpT2K6ieQlDoS05INg8BxUEJZiJEFTvqm2nCW/jN OgIAJVO6tpqhutUbT6/Mp5soDLYXTkm5nzK8SwaLlQ+rZqVU4YQmXRifF/8mouc+h3eY GQIia1x9gLPbQH0npqJZWeCvjayxOMdNgv0gqqKRdm7j4bU6Uiq0cW5Diymqd+PxNVTV WFMEJPZpyHKyFSZ6Uoqimk0VWT9Xo0CObAawF8F+RPLRvfSNi/rDyr7czZtiDrUVMjDE SXtpnrSO/k+3uTxjcewASlar8B+ulBOUvpWazkLp1S44yu2OF+7HSHw56LZkUooffqK0 lQag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728487375; x=1729092175; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mhdPuyLC0+7VYwR8lKpMmcoDbmwduvCzZV8i3kDuNPE=; b=gHa5EKekkJedHqjrIAy+HkGFieBoht5HEdLzKol2xcG0EfWVt2GSMozj1/FufY6uCH mk6IN3oW0ZJnTrzzVQ4J/NbmQr2h2ycOPGsEUYUE4/Vwgn5f5N3yhIM7jKKIs9zSNWho 0dfxurHAv8ZHdVjhEUyQCvia78b2qJIscA5p/VQPaFZUd9akGE4926G1mv6ev9EZv0mH xLJz0fQyZ+ap1HMLLF7boCT8hKu0GQC+lH1fz7MgPY41OPKtf75p3bIT+yv1k7a8AoxV mvawbEaNAYXMzuxJpR/2BuKUmVHF4WMRisgZ419EwwazLfTSILsDaF7AoYku80ipYwl6 Q0Vg== X-Gm-Message-State: AOJu0Yx5CeDRVI5HEAzorPhEetTgT1FBL8/C4x/w/+qVFnS1FHtxy7ih PtI/xJ506QU7MHGLWrJZo4nO1mvaCUWGuAWfo5u+G3NvFFMIEpwhEpaWsOMc8+uJbDoft8BK/DQ +2xsrpJ/dN5EcDKCZnfAnC7fl05yhp4XL X-Google-Smtp-Source: AGHT+IGi007blXc5amWrxeayWhuE93/NXk5iCCBgegYt5bsMHBgQjNtydRzciMYmeFMwvZhbI/+oQycZIjWlObUncno= X-Received: by 2002:a05:6512:1096:b0:539:959e:f0e8 with SMTP id 2adb3069b0e04-539c4897565mr1614471e87.21.1728487374341; Wed, 09 Oct 2024 08:22:54 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: <17f4077d-647d-4848-9d6f-97f9886ef636@freebsd.org> In-Reply-To: From: Vasily Postnicov Date: Wed, 9 Oct 2024 15:22:43 +0000 Message-ID: Subject: Re: Running Mezzano in bhyve To: Peter Grehan Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000f2501306240cd3e0" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROMTLD(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TAGGED_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::131:from] X-Rspamd-Queue-Id: 4XNxTs10jXz4p7n X-Spamd-Bar: --- --000000000000f2501306240cd3e0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Turns out, panic always happens in the same place when Mezzano is trying to initialize an AHCI controller. Qemu uses an old PATA controller by default, which works. When forcing SATA, it fails just like bhyve. Virtio block devices are also claimed to work, but using virtio-blk option in bhyve also results in panic. When I tried Mezzano on my PC (bare metal), I used a USB stick. Can a USB stick be emulated in bhyve? I guess, only using USB controller passthrough? =D1=81=D1=80, 9 =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2 12:18, Vasi= ly Postnicov : > Thanks! It worked, but Mezanno crashes after a short period of time: > > Begin PCI scan. > PCI:0:1F:0 8086:7000 Intel Corporation - 82371SB PIIX3 ISA [Natoma/Triton > II] 6:1:0 rid: 0 hdr: 0 intr: FF > PCI:0:1D:0 FB5D:40FB NIL - NIL 3:0:0 rid: 0 hdr: 0 intr: FF > 40: MSI 84 > PCI:0:5:0 1AF4:1000 Red Hat, Inc - Virtio network device 2:0:0 rid: 0 hdr= : > 0 intr: A > 40: Unknown capability 11 > 4C: MSI 80 > PCI:0:3:0 8086:2821 Intel Corporation - 82801HR/HO/HH (ICH8R/DO/DH) 6 por= t > SATA Controller [AHCI mode] 1:6:1 rid: 0 hdr: 0 intr: B > 40: MSI 88 > Detected AHCI ABAR at C1002000 > AHCI IRQ is B > Host Capabilities FF30FF3F > Global Host Control 80000000 > Interrupt Status 0 > Ports Implemented 1 > Version 10300 > Command Completion Coalescing Control 0 > Command Completion Coalescing Ports 0 > Enclosure Management Location 0 > Enclosure Management Control 0 > Host Capabilities Extended 4 > BIOS/OS Handoff Control and Status 0 > AHCI HBA version 1.300 > Handler: 0 > Config register: 17 > ----- PANIC ----- > Early page fault on address 80000035B0 > Local CPU is #<7A4AF9> > Run queues: > Thread #<1468079 Thread Initial thread> holds the world > Run queue #<1498139>/SUPERVISOR: > Run queue #<1498159>/HIGH: > Run queue #<1498179>/NORMAL: > Run queue #<1498199>/LOW: > IRQ state: > IRQ #<1497DF9 Irq :Number 0> - 0 (86 delivered) > #<1499119> PIT [exclusive] > IRQ #<1497E29 Irq :Number 1> - 1 (1 delivered) > #<1499699> INTEL-8042-KEYBOARD [exclusive] > IRQ #<1498009 Irq :Number B> - B (0 delivered) > #<149C959> #<149C909> > IRQ #<1498039 Irq :Number C> - C (0 delivered) > #<14993D9> INTEL-8042-AUXILIARY [exclusive] > Active timers: (current time is 147260) > Thread #<1468079 Thread Initial thread> ACTIVE #<1498669 Wait-Queue > Heartbeat wait queue> > TIFH: 1 TPFH: NIL > 20000121FB90 7FFF892B07 DEBUG-DUMP > 20000121FD20 7FFF800FAE (LAMBDA IN PANIC-1) > 20000121FE30 7FFF80073F PANIC-1 > 20000121FE50 7FFF800633 PANIC > 20000121FEC0 7FFF8820B1 FATAL-PAGE-FAULT > 20000121FEF0 7FFF88114B %PAGE-FAULT-HANDLER > 20000121FF40 7FFF87F3BB %%INTERRUPT-SERVICE-ROUTINES > 20000121FFD0 80000035B0 # > 200000A1FFF0 7FFF800841 %CALL-ON-WIRED-STACK-WITHOUT-INTERRUPTS > 20000221FE60 7FFF958E90 (SETF PCI-CONFIG/16) > 20000221FE90 7FFF9B5670 AHCI-PCI-REGISTER > 20000221FEF0 7FFF95B9C9 (FLET DX-LAMBDA IN PCI-DETECT) > 20000221FF50 7FFF95B22E MAP-PCI-DEVICES > 20000221FF90 7FFF955671 PCI-DETECT > 20000221FFA0 7FFF93C8F4 INITIALIZE-PLATFORM > 20000221FFC0 7FFF8D7F03 BOOTLOADER-ENTRY-POINT > > Looks like it executes an interrupt handler with calling something "on a > wired stack without interrupts". Postmortem rflags: > root@vonbraun:~ # bhyvectl --vm=3Dubuntu --get-rflags > rflags[0] 0x0000000000000046 > > I do not know what to think, that %cal-on-wired-stack-without-interrupts > seems to be legit to me: > https://github.com/froggey/Mezzano/blob/master/supervisor/x86-64/interrup= ts.lisp > Anyway, thanks again for the help! > > =D1=81=D1=80, 9 =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2 11:20, Pe= ter Grehan : > >> > Hi! Mezzano is an operating system written in Common lisp. I would lik= e >> > to test it in bhyve. When I launch it, it spins infinitely in a loop >> > measuring how many CPU cycles it makes in a unit of time (I understood >> > it by adding various debug messages). >> > >> > The timer it uses is something called a PIT: >> > https://wiki.osdev.org/Programmable_Interval_Timer >> > >> > >> > The problem is, once the timer is configured it fires an interrupt onl= y >> > once, while it obviously must fire interrupts constantly at some >> > frequency (100 Hz, as I understand from the code). >> > >> > Is the PIT supported by bhyve? Does FreeBSD use it anywhere in its >> kernel? >> >> Yes, though mode 3 of the timer isn't supported by the device >> emulation. It appears that Mezzano uses that mode >> >> >> https://github.com/froggey/Mezzano/blob/master/supervisor/x86-64/time.li= sp#L20 >> >> Try this patch to the PIT code to see if it helps >> https://people.freebsd.org/~grehan/bhyve_vatpit.diff >> >> later, >> >> Peter. >> >> --000000000000f2501306240cd3e0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Turns out, panic always happens in the same place when Mez= zano is trying to initialize an AHCI controller. Qemu uses an old PATA cont= roller by=C2=A0default, which works. When forcing SATA, it fails just like = bhyve. Virtio block devices are also claimed to work, but using virtio-blk = option in bhyve also results in panic. When I tried Mezzano on my PC (bare = metal), I used a USB stick. Can a USB stick be emulated in bhyve? I guess, = only using USB controller passthrough?

=
=D1=81=D1=80, 9 =D0=BE=D0=BA=D1=82. 2= 024=E2=80=AF=D0=B3. =D0=B2 12:18, Vasily Postnicov <shamaz.mazum@gmail.com>:
Thanks! It worked, but Mezanno crash= es after a short=C2=A0period of time:

Begin PCI scan.PCI:0:1F:0 8086:7000 Intel Corporation - 82371SB PIIX3 ISA [Natoma/Triton = II] 6:1:0 rid: 0 hdr: 0 intr: FF
PCI:0:1D:0 FB5D:40FB NIL - NIL 3:0:0 ri= d: 0 hdr: 0 intr: FF
=C2=A0 =C2=A0 40: MSI 84
PCI:0:5:0 1AF4:1000 Red= Hat, Inc - Virtio network device 2:0:0 rid: 0 hdr: 0 intr: A
=C2=A0 =C2= =A0 40: Unknown capability 11
=C2=A0 =C2=A0 4C: MSI 80
PCI:0:3:0 8086= :2821 Intel Corporation - 82801HR/HO/HH (ICH8R/DO/DH) 6 port SATA Controlle= r [AHCI mode] 1:6:1 rid: 0 hdr: 0 intr: B
=C2=A0 =C2=A0 40: MSI 88
De= tected AHCI ABAR at C1002000
AHCI IRQ is B
Host Capabilities FF30FF3F=
Global Host Control 80000000
Interrupt Status 0
Ports Implemented= 1
Version 10300
Command Completion Coalescing Control 0
Command C= ompletion Coalescing Ports 0
Enclosure Management Location 0
Enclosur= e Management Control 0
Host Capabilities Extended 4
BIOS/OS Handoff C= ontrol and Status 0
AHCI HBA version 1.300
Handler: 0
Config regis= ter: 17
----- PANIC -----
Early page fault on address 80000035B0
L= ocal CPU is #<7A4AF9>
Run queues:
Thread #<1468079 Thread In= itial thread> holds the world
Run queue #<1498139>/SUPERVISOR:<= br>Run queue #<1498159>/HIGH:
Run queue #<1498179>/NORMAL:Run queue #<1498199>/LOW:
IRQ state:
IRQ #<1497DF9 Irq :Nu= mber 0> - 0 (86 delivered)
=C2=A0 #<1499119> PIT [exclusive]IRQ #<1497E29 Irq :Number 1> - 1 (1 delivered)
=C2=A0 #<149969= 9> INTEL-8042-KEYBOARD [exclusive]
IRQ #<1498009 Irq :Number B>= - B (0 delivered)
=C2=A0 #<149C959> #<149C909>
IRQ #<= 1498039 Irq :Number C> - C (0 delivered)
=C2=A0 #<14993D9> INTE= L-8042-AUXILIARY [exclusive]
Active timers: (current time is 147260)
= Thread #<1468079 Thread Initial thread> ACTIVE #<1498669 Wait-Queu= e Heartbeat wait queue>
TIFH: 1 TPFH: NIL
20000121FB90 7FFF892B07 = DEBUG-DUMP
20000121FD20 7FFF800FAE (LAMBDA IN PANIC-1)
20000121FE30 7= FFF80073F PANIC-1
20000121FE50 7FFF800633 PANIC
20000121FEC0 7FFF8820= B1 FATAL-PAGE-FAULT
20000121FEF0 7FFF88114B %PAGE-FAULT-HANDLER
20000= 121FF40 7FFF87F3BB %%INTERRUPT-SERVICE-ROUTINES
20000121FFD0 80000035B0 = #<unknown>
200000A1FFF0 7FFF800841 %CALL-ON-WIRED-STACK-WITHOUT-IN= TERRUPTS
20000221FE60 7FFF958E90 (SETF PCI-CONFIG/16)
20000221FE90 7F= FF9B5670 AHCI-PCI-REGISTER
20000221FEF0 7FFF95B9C9 (FLET DX-LAMBDA IN PC= I-DETECT)
20000221FF50 7FFF95B22E MAP-PCI-DEVICES
20000221FF90 7FFF95= 5671 PCI-DETECT
20000221FFA0 7FFF93C8F4 INITIALIZE-PLATFORM
20000221F= FC0 7FFF8D7F03 BOOTLOADER-ENTRY-POINT

Looks li= ke it executes an interrupt handler with calling something "on a wired= stack without interrupts". Postmortem rflags:
root@vonbraun= :~ # bhyvectl --vm=3Dubuntu --get-rflags
rflags[0] 0x0000000000000046

I do not know what to think, that %cal-on-wired-= stack-without-interrupts seems to be legit to me:=C2=A0https://github.com/froggey/Mezzano/blob/master/supervisor/x= 86-64/interrupts.lisp
Anyway, thanks again for the help!

=D1=81=D1=80, 9 =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2 11:20, Pet= er Grehan <greha= n@freebsd.org>:
> Hi! Mezzano is an ope= rating system written in Common lisp. I would like
> to test it in bhyve. When I launch it, it spins infinitely in a loop <= br> > measuring how many CPU cycles it makes in a unit of time (I understood=
> it by adding various debug messages).
>
> The timer it uses is something called a PIT:
> https://wiki.osdev.org/Programmable_Interval_= Timer
> <https://wiki.osdev.org/Programmable_Inter= val_Timer>
>
> The problem is, once the timer is configured it fires an interrupt onl= y
> once, while it obviously must fire interrupts constantly at some
> frequency (100 Hz, as I understand from the code).
>
> Is the PIT supported by bhyve? Does FreeBSD use it anywhere in its ker= nel?

=C2=A0 Yes, though mode 3 of the timer isn't supported by the device emulation. It appears that Mezzano uses that mode

https://github.com/fro= ggey/Mezzano/blob/master/supervisor/x86-64/time.lisp#L20

=C2=A0 Try this patch to the PIT code to see if it helps
=C2=A0 =C2=A0 =C2=A0https://people.freebsd.org/~= grehan/bhyve_vatpit.diff

later,

Peter.

--000000000000f2501306240cd3e0--