From nobody Thu Oct 10 19:43:28 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XPgDH037jz5YWt6 for ; Thu, 10 Oct 2024 19:43:43 +0000 (UTC) (envelope-from shamaz.mazum@gmail.com) Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XPgDF5cBRz4J5c; Thu, 10 Oct 2024 19:43:41 +0000 (UTC) (envelope-from shamaz.mazum@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=XXnF8IwB; spf=pass (mx1.freebsd.org: domain of shamaz.mazum@gmail.com designates 2a00:1450:4864:20::136 as permitted sender) smtp.mailfrom=shamaz.mazum@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-539973829e7so1378660e87.0; Thu, 10 Oct 2024 12:43:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728589420; x=1729194220; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Plwfv8DFoejiLnDijLXCD3eTe5ZuGBDiYsjackQlWdE=; b=XXnF8IwBn3BVMfVLRgkeKz0IP9v/o1MnS0lkocqD34hDg5cqHTSxmIT7XfIMiIr7hX mI8esKSbRuxZQ8qpQiqVnaJR2hEgHUT91Dm1zzGqX3W9fiyECt2p96QdZJwzijLL3ZA1 49cULsU61ly4VfA2gCiKxk65v6KhOm9X1FG9rtosQLtv5rURSe7/wJArMH+AzegfgApk RZb/TuM0XcHwTgaBeYu9vmf/hWa5MPRLLW9tt2mLpUN8Pp0t+oVmoxogVZ7rc6H0GnZX 1c5J8C80e7pPtQuMntOlJDNeABnRo7NvAMc63gS37BiDjaXNCB3FkokXnXWhWbUxLZdW 55Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728589420; x=1729194220; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Plwfv8DFoejiLnDijLXCD3eTe5ZuGBDiYsjackQlWdE=; b=YX8RH0wX9Gk8j030OvEmc2xje3sArpLtYvqF/StahF/SjwwVmJSFISTy+qox1JVoi+ sOrUv/IqBTF9Ey4qVqKgrDyDb0mjVC38ZgvyqICCLQQPdg3AG9n4vM/s1wjpGfOyVReM b5gygI4aOMFGPIrFVC0Bhai5MUSgkGx0IxJMLxHcqrXupHmw0jMnlRwKxnuLNRwRO4by cjzlrwJotg8tWoeVcfWCp9othEwD5GnD0Eclp3ujxzqQdizG3tTI1NTyuAdVVs9xwzgj pVNqm7ip+BbsqYwLhIu2t5GeA8MIZEv1kqR39i0qBFxTkRfyDq7kmf8mCYnT7uCS/Oi3 NRww== X-Gm-Message-State: AOJu0YwD/DFFxQrayKSf6IybxVFOashlw9chcptRAppiwGntcqNSjrrk 3E0Lsp2jC1RULudFNi1R4u4tXkUlqm9RdtbXQV9MqcWpjrzFA4FV5T0jikRy2NH+2wG8lIvTmCX BnkG7Ms0/kd/4VRCYB5ykswEP7UIRAqBulCweWw== X-Google-Smtp-Source: AGHT+IEDfVR+Sv26y7M4DTC0lSEfqBm2itAAWrx8KOrFg8QTiDOe+sjHk3stsHmcwajq3NHSfaVjgsc2SRal7tsK7dk= X-Received: by 2002:a05:6512:2399:b0:536:54db:ddd0 with SMTP id 2adb3069b0e04-539d472b288mr459550e87.0.1728589419455; Thu, 10 Oct 2024 12:43:39 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: <17f4077d-647d-4848-9d6f-97f9886ef636@freebsd.org> <8b249b64-d041-4f12-b6cb-fdb528837f22@freebsd.org> In-Reply-To: From: Vasily Postnicov Date: Thu, 10 Oct 2024 19:43:28 +0000 Message-ID: Subject: Re: Running Mezzano in bhyve To: Peter Grehan Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="0000000000004f2cfa06242496a6" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROMTLD(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TAGGED_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::136:from] X-Rspamd-Queue-Id: 4XPgDF5cBRz4J5c X-Spamd-Bar: --- --0000000000004f2cfa06242496a6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I suspect PCI interrupts are not functioning correctly. Look at this code: ;; Attach interrupt handler. (sup:debug-print-line "Handler: " (ahci-irq-handler ahci)) (sup:irq-attach (sup:platform-irq (pci:pci-intr-line location)) (ahci-irq-handler-function ahci) ahci) and this (defun pci-intr-line (device) (pci-config/8 device +pci-config-intr-line+)) ;; comment by me: the constant is #x3c I found that "PCI 0x3c" means PCI interrupt pin. AFAIK, interrupt pins are not supported by bhyve, is that correct? If it's true, I need either to teach bhyve how to deal with legacy interrupts or to teach Mezzano to understand MSI. What would be easier in your opinion? =D1=87=D1=82, 10 =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2 17:12, Vas= ily Postnicov : > I was able to fix panics in both virtio and AHCI. This is what I found: > > 1) Virtio had a stupid bug, namely Mezzano tried to find an accessor to > some IO port in the runtime doing something like (funcall (intern (format > nil "~a-~a" bus-name slot-name)) ...). Surely, the creator made an error = in > the name of one of the accessors, so FUNCALL tried to call an unbound > symbol, hence the page fault. > 2) AHCI had the following code: > > ;; Magic hacks for Intel devices? > ;; Set port enable bits in Port Control and Status on Intel controllers. > (when (eql (pci:pci-config/16 location pci:+pci-config-vendorid+) #x8086) > (let* ((n-ports (1+ (ldb (byte +ahci-CAP-NP-size+ +ahci-CAP-NP-position= +) > (ahci-global-register ahci > +ahci-register-CAP+)))) > (pcs (pci:pci-config/16 location #x92))) > (setf (pci:pci-config/16 location #x92) (logior pcs > (ash #xFF (- (- 8 > n-ports))))))) > > I checked the value of N-PORTS, it's 20, so (ash #xff (- (- 8 n-ports))) > is 1044480 which is bigger than 2^16-1. I recompiled bhyve with MAX_PORTS= =3D > 6 in bhyve/pci_ahci.c and the panic disappeared. Now I have this output: > > Detected AHCI ABAR at C1002000 > AHCI IRQ is B > Host Capabilities FF30FF25 > Global Host Control 80000000 > Interrupt Status 0 > Ports Implemented 1 > Version 10300 > Command Completion Coalescing Control 0 > Command Completion Coalescing Ports 0 > Enclosure Management Location 0 > Enclosure Management Control 0 > Host Capabilities Extended 4 > BIOS/OS Handoff Control and Status 0 > AHCI HBA version 1.300 > Handler: 0 > Config register: 17 > Port 0 > Waiting for CR/FR to stop. > Allocated port data at 105C33000 > Command List at 105C33000 > Received FIS at 105C33400 > Command Tabl at 105C33500 > Initializing device on port 0 > Command List Base Address 5C33000 > Command List Base Address Upper 32-bits 1 > FIS Base Address 5C33400 > FIS Base Address Upper 32-bits 1 > Interrupt Status 0 > Interrupt Enable 7D80003F > Command and Status 1C017 > Task File Data 50 > Signature 101 > SATA Status (SCR0: SStatus) 133 > SATA Control (SCR2: SControl) 300 > SATA Error (SCR1: SError) 0 > SATA Active (SCR3: SActive) 0 > Command Issue 0 > SATA Notification (SCR4: SNotification) 0 > FIS-based Switching Control 0 > *** AHCI-RUN-COMMAND TIMEOUT EXPIRED! *** > Command completed. > 105C33600: 28A20040 100000 0 3F > 105C33610: 0 59564248 4644452D 2D413239 > 105C33620: 382D4136 39433646 0 30300000 > 105C33630: 20203120 42482020 45205956 54415341 > 105C33640: 49532044 20204B20 20202020 20202020 > 105C33650: 20202020 20202020 20202020 80802020 > 105C33660: B000000 4000 60000 0 > 105C33670: 0 0 A00000 70000 > 105C33680: 780003 780078 40200078 0 > 105C33690: 0 1F0000 40010E 0 > 105C336A0: 2803F0 74004068 40684000 4000B400 > 105C336B0: 7F 0 0 0 > 105C336C0: 0 0 A00000 0 > 105C336D0: 10000 6008 0 0 > 105C336E0: 0 0 0 40080000 > 105C336F0: 4008 0 0 0 > 105C33700: 0 0 0 0 > 105C33710: 0 0 0 0 > 105C33720: 0 0 0 0 > 105C33730: 0 0 0 0 > 105C33740: 0 0 0 0 > 105C33750: 10000 0 0 0 > 105C33760: 0 0 0 0 > 105C33770: 0 0 0 0 > 105C33780: 0 0 0 0 > 105C33790: 0 0 0 0 > 105C337A0: 40000000 0 0 0 > 105C337B0: 0 0 0 1020 > 105C337C0: 0 0 0 0 > 105C337D0: 0 0 0 0 > 105C337E0: 0 0 0 0 > 105C337F0: 0 0 0 78A50000 > Features (83): 7400 > Sector size: 200 > Sector count: A00000 > Serial: BHYVE-FD29-AA68-6F9C > Model: BHYVE SATA DISK > Registered new R/W disk #<149CAC9> sectors:A00000 > Host Capabilities FF30FF25 > Global Host Control 80000002 > Interrupt Status 1 > Ports Implemented 1 > Version 10300 > Command Completion Coalescing Control 0 > Command Completion Coalescing Ports 0 > Enclosure Management Location 0 > Enclosure Management Control 0 > Host Capabilities Extended 4 > BIOS/OS Handoff Control and Status 0 > PCI:0:0:0 1022:7432 NIL - NIL 6:0:0 rid: 0 hdr: 0 intr: FF > 40: Unknown capability 10 > *** AHCI-RUN-COMMAND TIMEOUT EXPIRED! *** > *** AHCI-RUN-COMMAND TIMEOUT EXPIRED! *** > Detected MBR style parition table on disk #<149CAC9> > Detected partition 0 on disk #<149CAC9>. Start: 800 size: 800 > Registered new R/W disk #<149CCD9> sectors:800 > Detected partition 1 on disk #<149CAC9>. Start: 1000 size: 800 > Registered new R/W disk #<149CD89> sectors:800 > Detected partition 2 on disk #<149CAC9>. Start: 2000 size: 9FE000 > Registered new R/W disk #<149CE39> sectors:9FE000 > Looking for paging disk with UUID > 5C:F6:EE:79:2C:DF:45:E1:BA:2B:63:25:C4:1A:5F:10 > *** AHCI-RUN-COMMAND TIMEOUT EXPIRED! *** > Found image with UUID 5C:F6:EE:79:2C:DF:45:E1:BA:2B:63:25:C4:1A:5F:10 on > disk #<149CE39> > Found boot image on disk #<149CE39>! > BML4 at -7FFFFFEFD000 > Store freelist block is 2 > > It seems it is booting, but very very slowly with those "TIMEOUT EXPIRED" > messages. For virtio-blk, it's almost the same with an exception that it > hangs completely. I'll try to investigate further. Meanwhile, can you mak= e > any suggestions why those magic intel AHCI controller hacks are required > and why sc->ports can get bigger than DEF_PORTS in pci_ahci_init in bhyve= ? > --0000000000004f2cfa06242496a6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I suspect PCI interrupts are not function= ing correctly.

Look at this code:
=C2=A0 =C2=A0 ;; Attach interrupt handler.
=C2=A0 =C2=A0 (sup:debu= g-print-line "Handler: " (ahci-irq-handler ahci))
=C2=A0 =C2= =A0 (sup:irq-attach (sup:platform-irq (pci:pci-intr-line location))
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (ahci-ir= q-handler-function ahci)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 ahci)

and this

(defun pci-intr-line (device)
=C2=A0 (pci-config/8 device +pci-c= onfig-intr-line+)) ;; comment by me: the constant is #x3c
I found that "PCI 0x3c" means PCI interrupt pin. AFAI= K, interrupt pins are not supported by bhyve, is that=C2=A0correct? If it&#= 39;s true, I need either to teach bhyve how to deal with legacy interrupts = or to teach Mezzano to understand MSI. What would be easier in your=C2=A0op= inion?

=D1=87=D1=82, 10 =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2= 17:12, Vasily Postnicov <sham= az.mazum@gmail.com>:
I wa= s able to fix panics in both virtio and AHCI. This is what=C2=A0I found:
1) Virtio had a stupid bug, namely Mezzano tried to find a= n accessor to some IO port in the runtime doing something like (funcall (in= tern (format nil "~a-~a" bus-name slot-name)) ...). Surely, the c= reator made an error in the name of one of the accessors, so FUNCALL tried = to call an unbound symbol, hence the page fault.
2) AHCI had the = following code:

;; Magic hacks for Intel devices?<= br>;; Set port enable bits in Port Control and Status on Intel controllers.=
(when (eql (pci:pci-config/16 location pci:+pci-config-vendorid+)= #x8086)
=C2=A0 (let* ((n-ports (1+ (ldb (byte +ahci-CAP-NP-size+ +ahci-= CAP-NP-position+)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(ahci-global-register ahci +ah= ci-register-CAP+))))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(pcs (pci:pci-con= fig/16 location #x92)))
=C2=A0 =C2=A0 (setf (pci:pci-config/16 location = #x92) (logior pcs
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (ash #xFF (- (-= 8 n-ports)))))))

I checked the value of N-PORTS, = it's 20, so (ash #xff (- (- 8 n-ports))) is=C2=A01044480 which is bigge= r than 2^16-1. I recompiled bhyve with MAX_PORTS =3D 6 in bhyve/pci_ahci.c = and the panic disappeared. Now I have this output:

Detected AHCI ABAR at C1002000
AHCI IRQ is B
Host Capabilities FF30F= F25
Global Host Control 80000000
Interrupt Status 0
Ports Implemen= ted 1
Version 10300
Command Completion Coalescing Control 0
Comman= d Completion Coalescing Ports 0
Enclosure Management Location 0
Enclo= sure Management Control 0
Host Capabilities Extended 4
BIOS/OS Handof= f Control and Status 0
AHCI HBA version 1.300
Handler: 0
Config re= gister: 17
Port 0
Waiting for CR/FR to stop.
Allocated port data a= t 105C33000
Command List at 105C33000
Received FIS at 105C33400
Co= mmand Tabl at 105C33500
Initializing device on port 0
=C2=A0Command L= ist Base Address 5C33000
=C2=A0Command List Base Address Upper 32-bits 1=
=C2=A0FIS Base Address 5C33400
=C2=A0FIS Base Address Upper 32-bits = 1
=C2=A0Interrupt Status 0
=C2=A0Interrupt Enable 7D80003F
=C2=A0C= ommand and Status 1C017
=C2=A0Task File Data 50
=C2=A0Signature 101=C2=A0SATA Status (SCR0: SStatus) 133
=C2=A0SATA Control (SCR2: SContr= ol) 300
=C2=A0SATA Error (SCR1: SError) 0
=C2=A0SATA Active (SCR3: SA= ctive) 0
=C2=A0Command Issue 0
=C2=A0SATA Notification (SCR4: SNotifi= cation) 0
=C2=A0FIS-based Switching Control 0
*** AHCI-RUN-COMMAND TI= MEOUT EXPIRED! ***
Command completed.
105C33600: 28A20040 100000 0 3F=
105C33610: 0 59564248 4644452D 2D413239
105C33620: 382D4136 39433646= 0 30300000
105C33630: 20203120 42482020 45205956 54415341
105C33640:= 49532044 20204B20 20202020 20202020
105C33650: 20202020 20202020 202020= 20 80802020
105C33660: B000000 4000 60000 0
105C33670: 0 0 A00000 700= 00
105C33680: 780003 780078 40200078 0
105C33690: 0 1F0000 40010E 0105C336A0: 2803F0 74004068 40684000 4000B400
105C336B0: 7F 0 0 0
10= 5C336C0: 0 0 A00000 0
105C336D0: 10000 6008 0 0
105C336E0: 0 0 0 4008= 0000
105C336F0: 4008 0 0 0
105C33700: 0 0 0 0
105C33710: 0 0 0 0105C33720: 0 0 0 0
105C33730: 0 0 0 0
105C33740: 0 0 0 0
105C337= 50: 10000 0 0 0
105C33760: 0 0 0 0
105C33770: 0 0 0 0
105C33780: 0= 0 0 0
105C33790: 0 0 0 0
105C337A0: 40000000 0 0 0
105C337B0: 0 0= 0 1020
105C337C0: 0 0 0 0
105C337D0: 0 0 0 0
105C337E0: 0 0 0 0105C337F0: 0 0 0 78A50000
Features (83): 7400
Sector size: 200
S= ector count: A00000
Serial: BHYVE-FD29-AA68-6F9C
Model: BHYVE SATA DI= SK
Registered new R/W disk #<149CAC9> sectors:A00000
Host Capab= ilities FF30FF25
Global Host Control 80000002
Interrupt Status 1
P= orts Implemented 1
Version 10300
Command Completion Coalescing Contro= l 0
Command Completion Coalescing Ports 0
Enclosure Management Locati= on 0
Enclosure Management Control 0
Host Capabilities Extended 4
B= IOS/OS Handoff Control and Status 0
PCI:0:0:0 1022:7432 NIL - NIL 6:0:0 = rid: 0 hdr: 0 intr: FF
=C2=A0 =C2=A0 40: Unknown capability 10
*** AH= CI-RUN-COMMAND TIMEOUT EXPIRED! ***
*** AHCI-RUN-COMMAND TIME= OUT EXPIRED! ***
Detected MBR style parition table on disk #&= lt;149CAC9>
Detected partition 0 on disk #<149CAC9>. Start: 800= size: 800
Registered new R/W disk #<149CCD9> sectors:800
Detec= ted partition 1 on disk #<149CAC9>. Start: 1000 size: 800
Register= ed new R/W disk #<149CD89> sectors:800
Detected partition 2 on dis= k #<149CAC9>. Start: 2000 size: 9FE000
Registered new R/W disk #&l= t;149CE39> sectors:9FE000
Looking for paging disk with UUID 5C:F6:EE:= 79:2C:DF:45:E1:BA:2B:63:25:C4:1A:5F:10
*** AHCI-RUN-COMMAND T= IMEOUT EXPIRED! ***
Found image with UUID 5C:F6:EE:79:2C:DF:45:E1:= BA:2B:63:25:C4:1A:5F:10 on disk #<149CE39>
Found boot image on dis= k #<149CE39>!
BML4 at -7FFFFFEFD000
Store freelist block is 2
It seems it is booting, but very very slowly with those &= quot;TIMEOUT EXPIRED" messages. For virtio-blk, it's almost the sa= me with an exception=C2=A0that=C2=A0it hangs completely. I'll try to in= vestigate further. Meanwhile, can you make any suggestions why those magic = intel AHCI controller hacks are required and why sc->ports can get bigge= r than DEF_PORTS in pci_ahci_init in bhyve?
--0000000000004f2cfa06242496a6--