From nobody Fri Oct 18 06:31:07 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XVFGK36XSz5Yn92 for ; Fri, 18 Oct 2024 06:31:21 +0000 (UTC) (envelope-from SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XVFGG39ZLz4b4k for ; Fri, 18 Oct 2024 06:31:15 +0000 (UTC) (envelope-from SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nt.com.au header.s=dkim header.b=VfWUtYUQ; spf=pass (mx1.freebsd.org: domain of "SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au" designates 203.13.68.12 as permitted sender) smtp.mailfrom="SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au"; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 92BA120B4981 for ; Fri, 18 Oct 2024 16:31:08 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 865472127CB8 for ; Fri, 18 Oct 2024 16:31:08 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=nt.com.au; h= content-transfer-encoding:content-type:content-type:in-reply-to :content-language:references:to:subject:subject:from:from :user-agent:mime-version:date:date:message-id; s=dkim; t= 1729233068; x=1731825069; bh=W0rXtUgnBUUjFb8kGjEaeaI1tKbiPqSnjUz 9VVe+gno=; b=VfWUtYUQ9mER8TsJ33rVeA12at55xi0ube1/qUbRZbrJFJsQMtL TVr3AGXdsuu45doPjaIdQ+mpPdo4GO0ZQkZSV71u+NYWvXq7bpPL++C0GiDW0+YB 5yhUjs3/TsaX3mwaeu1Nxh48UB9dPv7QyPNXRvAoJ8FJm+AzpCW0kIfs= Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id XKryYq-mBM5y for ; Fri, 18 Oct 2024 16:31:08 +1000 (AEST) Received: from [192.168.1.101] (otn-120-29-24-249.broadband.onthenet.net [120.29.24.249]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 5384D2127CB7; Fri, 18 Oct 2024 16:31:08 +1000 (AEST) Message-ID: <2540b716-8f7e-4ff8-a582-613977bdb8c0@freebsd.org> Date: Fri, 18 Oct 2024 16:31:07 +1000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Peter Grehan Subject: Re: Running Mezzano in bhyve To: Vasily Postnicov Cc: freebsd-virtualization@freebsd.org References: <17f4077d-647d-4848-9d6f-97f9886ef636@freebsd.org> <8b249b64-d041-4f12-b6cb-fdb528837f22@freebsd.org> <106b8500-a0ef-4095-af20-8c0f110ea739@freebsd.org> Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.4 cv=Fu4D/Xrq c=1 sm=1 tr=0 ts=671200ac a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=0UtLtLgMz4NvdyIsuxvgLw==:17 a=IkcTkHD0fZMA:10 a=DAUX931o1VcA:10 a=bi0XHdcepdgA:10 a=a4NEJbfMAAAA:8 a=QyXUC8HyAAAA:8 a=2LeWwqssq1ttuCmCoZgA:9 a=QEXdDO2ut3YA:10 X-Spamd-Result: default: False [-3.29 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[grehan@freebsd.org,SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au]; R_SPF_ALLOW(-0.20)[+ip4:203.13.68.0/24]; R_DKIM_ALLOW(-0.20)[nt.com.au:s=dkim]; RCVD_IN_DNSWL_LOW(-0.10)[203.13.68.12:from]; RWL_MAILSPIKE_GOOD(-0.10)[203.13.68.12:from]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[grehan@freebsd.org,SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au]; DKIM_TRACE(0.00)[nt.com.au:+]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:9313, ipnet:203.13.68.0/24, country:AU]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4XVFGG39ZLz4b4k X-Spamd-Bar: --- > Regarding items 3) and 4): > > 3) Indeed, bhyve does not explicitly forbid writing to 0x3c. I meant > the following. The interrupt line is set is pci_emul.c in bhyve: > pci_set_cfgdata8(pi, PCIR_INTLINE, pirq_irq(ii->ii_pirq_pin)); Bhyve > asserts interrupts with pci_irq_assert in amd64/pci_irq.c. We need > this line: vm_isa_assert_irq(pi->pi_vmctx, pirq->reg & PIRQ_IRQ, > pi->pi_lintr.ioapic_irq); pirq->reg & PIRQ_IRQ is literally the same > as pirq_irq(ii->ii_pirq_pin). Now, if something (e.g. UEFI firmware, > bootloader) writes to PCIR_INTLINE bhyve will still send interrupts > with the number that was there before the write, while the OS will > expect an interrupt with the new number. I treat this as a bug in > bhyve (but it affects nobody, because newer OSes do not use the 8259 > interrupt controller). Ah yes, I think that's correct. I'm assuming that you're using EFI to boot, and bhyve/EFI recently changed to have PCI setup done by EFI and not in bhyve as before. Sounds like this is an extreme corner case that was broken by that. Not surprising: any o/s that supports EFI boot could be expected to support IOAPIC interrupts at a minimum. > 4) It's commenting the lock what makes an effect. I commented > pci_generate_msi just in case because it's not needed for Mezzano, > but runs protected by the mutex which is now gone. This is a > backtrace and thread list when bhyve hangs up if the mutex is not > commented out: I suspect this might be related to the above. > I think implementing IOAPIC in MEzzano is the best option indeed, > but I have a little experience. I'll see what I can do. See figures 3-2 through 3-5 in the Intel MP spec for an explanation of how it all fits together https://web.archive.org/web/20121002210153/http://download.intel.com/design/archives/processors/pro/docs/24201606.pdf later, Peter.