From nobody Sun Oct 27 22:28:06 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XcB4M410Fz5bWRj; Sun, 27 Oct 2024 22:28:19 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from vogon.madpilot.net (vogon.madpilot.net [IPv6:2a01:4f8:1c1c:11e5::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XcB4L49rFz537H; Sun, 27 Oct 2024 22:28:18 +0000 (UTC) (envelope-from mad@madpilot.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=madpilot.net header.s=cyq4qetkgngm header.b="N dO2t7i"; spf=pass (mx1.freebsd.org: domain of mad@madpilot.net designates 2a01:4f8:1c1c:11e5::1 as permitted sender) smtp.mailfrom=mad@madpilot.net; dmarc=pass (policy=quarantine) header.from=madpilot.net Received: from mail (mail [IPv6:fd5c:5351:d272::3]) by vogon.madpilot.net (Postfix) with ESMTP id 4XcB491Tz3zL7yv; Sun, 27 Oct 2024 23:28:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=madpilot.net; h= content-transfer-encoding:content-type:content-type:in-reply-to :from:from:content-language:references:subject:subject:date:date :message-id:received; s=cyq4qetkgngm; t=1730068087; x= 1731882488; bh=P2ERgjEae+SmXupx4Tp47i2JMrrQiZuUAGA5R6ZDGeg=; b=N dO2t7iZRbjxxOzD64PwPupvTvsUWBk+BNbVJCpAujnekFXtdLba3pTFLk17CpunI byIhQ4JaU1GTwqpqM2dvRkzrlTSwSJtU/gAuSuyrvbC0wutmYGIveW2MK/blcaYW dLxml/1s2Znfem65Vfd6ER43JeDdIgy8l9VLTJ13hgeRzE9iqKwoPCvnA2DnJ6WC wkRjHTDHwIi3uNjyfcozkz35ZTvIFkWogwdLK4WtNXbg2Jxu0KQr2ceBrQ+bWSE0 NZwAwp0r5DTxZ+S6n2d/+lrfGah1Tv2TnorUBYzLljDxgwGyCSBzl4a3kMMtb30K xaiH4jnHCRH9NhPrZTFmw== Received: from vogon.madpilot.net ([IPv6:fd5c:5351:d272::3]) by mail (vogon.madpilot.net [IPv6:fd5c:5351:d272::3]) (amavisd-new, port 10026) with ESMTP id q1gn94FX5dyx; Sun, 27 Oct 2024 23:28:07 +0100 (CET) Message-ID: <63102fa6-6676-4aa3-ba72-dc212a73dabf@madpilot.net> Date: Sun, 27 Oct 2024 23:28:06 +0100 Subject: Re: bhyve regression (head): windows VMs failing with error 0xc000021a To: markj@freebsd.org Cc: freebsd-current@freebsd.org, virtualization@freebsd.org References: <6a2a37b3-353e-40ca-b8a9-f4ef97733da8@madpilot.net> <48eb1621-4b22-4080-8e80-059d475244f3@madpilot.net> Content-Language: en-US From: Guido Falsi Autocrypt: addr=mad@madpilot.net; keydata= xsBNBE+G+l0BCADi/WBQ0aRJfnE7LBPsM0G3m/m3Yx7OPu4iYFvS84xawmRHtCNjWIntsxuX fptkmEo3Rsw816WUrek8dxoUAYdHd+EcpBcnnDzfDH5LW/TZ4gbrFezrHPdRp7wdxi23GN80 qPwHEwXuF0X4Wy5V0OO8B6VT/nA0ADYnBDhXS52HGIJ/GCUjgqJn+phDTdCFLvrSFdmgx4Wl c0W5Z1p5cmDF9l8L/hc959AeyNf7I9dXnjekGM9gVv7UDUYzCifR3U8T0fnfdMmS8NeI9NC+ wuREpRO4lKOkTnj9TtQJRiptlhcHQiAlG1cFqs7EQo57Tqq6cxD1FycZJLuC32bGbgalABEB AAHNHkd1aWRvIEZhbHNpIDxtYWRAbWFkcGlsb3QubmV0PsLAeQQTAQgAIwIbAwIeAQIXgAUL CQgHAwUVCgkICwQWAgMBBQJS79AgAhkBAAoJEBrmhg5Wy9KTc0kH/RO64ORBlTbTHaUaOj8F Je5O5NU2Pt9Cyt5ZWBRvxntr1zPTJGKRPS9ihlIfqT4ZvEngQGp57EUyFbCpI0UWasTerImM tt5WACnGmCzUTB39UXx8Oy4b1EgWeTJQ747e/F1mQLXTNa6ijRBE9fYlTb4gAkPN88/wVV9v 3PZozKLTg16ghBzHM/P7Lk8L7clPEZChX1FTa/6eSt3nvzfCuTMZbBPJF/ph+q1KyPqRgVfh tyhu5dvgMoPz/ni41IfeSrkJTD5RXzdyGR9q4Z1NYeBsLkRjC4LxKAP5KqUsvlOUjKvO1byj ApYdMarol+IGkaSk9e3zVYAJkWKjn/ni8XbOwU0EUxB7QQEQAKFhrDceoPdK/IHDSmoj6SQY isvM7VdhcleS7E9DoEAVt7yMbf6HbbMVTTY6ckvwTWQssywLBXNVqxgc4WLJjzfUhgef+WE7 5M3+WFYlOVQLGZY/zEVgma1raYnOHNAOzeHLDmEXjbZP6vGAeDyBbGfQPpE7qGYZ7ubeT3Xw QO+PklcCrvOPj2ZPcAxGNS2xVU/LzONqCrJqLMJSIcCdsbiSP4G5PnDFHtMokaTY6OEr8OEQ fOAerhcHUa/z7Uu8YtmaqKH+QGkE/WEgaRqSiTnv0JOTD+DxehaqvoKPPZ++2NpCZMHB2i6A /xifmQwEiIjEXtcueBRzkNUQkxhqZyS13SrhocL9ydtaVPBzZatAEjUDDEJmAMLVFs45qfyh MiNapHJo2n3MW/E5omqCvEkDdWX/en3P7CK2TemeaDghMsgkNKax/z0wNo5UZCkOPOz0xpNi UilOVbkuezZZNg65741qee2lfXhQIaZ66yT7hphc/N/z3PIAtLeze4u1VR2EXAuZ2sWAdlKC NTlJMsaU/x70BV11Wd/ypnVzM68dfdQIIAj1iMFAD/lXGlEUmKXg5Ov2VQDlTntQoanCYrAg +8CttPzjrydgLZFq3hrtQmfc0se5yv1WHS69+BsUOG09RvvawUDZxUjW19kyeN9THaNRgow3 kSuArUp6zSmJABEBAAHCwF8EGAEIAAkFAlMQe0ECGwwACgkQGuaGDlbL0pMN5wgA4bCkX/qw EVC06ToeR6C2putmSWQMgpDaqrv65Hubo+QGmg2P4ewTYQQ4g6oYWS03qHxqVVWhKz7FjfrV +dH8qbCLfSgIcvdBha7ayGZVrsiuMLKGbw36fcmkZPpSDOfHcP0XH8Z+u9CWj0xUkTxAlZ/7 i6gYSUpG2JWNtdmE/X8VVEyXusCLwy0K0BI60A/4dRTIX3C4QKrJ3ZbUXegz70ynjHf+lQMZ 9IZKASoRMuS5FozPQh6abvmwZEPdf5I9riUElzvHrqJ8Bx0t3Pujdoth+yNHpnBxrtO8LkQd rQ58P0SwcaIX33T2U9pG8bhu5YVR88FQ8OQ0cEsPBpDncg== In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-1.99 / 15.00]; MISSING_MIME_VERSION(2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; DMARC_POLICY_ALLOW(-0.50)[madpilot.net,quarantine]; R_DKIM_ALLOW(-0.20)[madpilot.net:s=cyq4qetkgngm]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org,virtualization@freebsd.org]; RCPT_COUNT_THREE(0.00)[3]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[madpilot.net:+] X-Rspamd-Queue-Id: 4XcB4L49rFz537H X-Spamd-Bar: - List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org On 26/10/24 15:28, Mark Johnston wrote: > On Fri, Oct 25, 2024 at 11:12:48PM +0200, Guido Falsi wrote: >> On 25/10/24 22:49, Mark Johnston wrote: >>> On Fri, Oct 25, 2024 at 09:24:13PM +0200, Guido Falsi wrote: >>>> Hi, >>>> >>>> I've recently updated my current machines to git commit >>>> 525a177c165740fc697df3de5b92e58b3b41477c (Sun Oct 20 22:43:41 2024 -0800) >>>> and just have Windows 10 VMs fail to start in bhyve with the error in the >>>> subject. >>>> >>>> I've been unable to recover them with usual tricks (automatic recovery, >>>> chkdsk, and other tools provided by the OS). Looks like the machine fails to >>>> read C: after boot. >>>> >>>> These machines were working fine before the update, so my suspect is that >>>> some recent change in bhyve is causing the issue and the VMs would be >>>> otherwise fine. >>>> >>>> The VMs have their filesystems in compressed zvols. >>>> >>>> Anyone has an idea or can point to some change I can test reverting? >>> >>> Just a guess, but you might try adding "-o pci.enable_bars=true" to the >>> bhyve command line arguments >> >> Tried but it looks like it made no difference. >> >>> >>>> I an also try bisecting, I'd guess the issue is quite recent. >>> >>> Which revision did you update from? >> >> I updated from git commit 450a6690f557493bd33d8f3666b22ddc5150703b (Thu Sep >> 19 11:49:40 2024 -0500) >> >> In the while I noticed some commits to TPM emulation/passthorugh, maybe >> they're related? > > It might be, but I don't see any TPM devices configured in the > invocation below. > > There was a number of changes to usr.sbin/bhyve in that window, so > reverting them one by one would probably turn up the culprit. Or, you > need to pick up commit 8c8ebbb045185396083cd3e4d333fe1851930ee7, given > that you're using AHCI emulation. HI again, thanks for the suggestion, updating to a newer head including this commit "fixed" the issue. SO if there was a regression it's already been fixed. Thanks again for your help! -- Guido Falsi