From nobody Mon Nov 21 14:18:37 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NG8dF4bHsz4j8Nl; Mon, 21 Nov 2022 14:18:41 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NG8dF4FNYz4F1W; Mon, 21 Nov 2022 14:18:41 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1669040321; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=wjfki+S9HP7uwb/AML6k5pc2e9yTcQIXq33OVleR/IA=; b=KJTdAVdk2HhmxPbpe/4iFxBtNqb8ObGlhUVnvjjc6IGQo4dQ+kWIZMNHFbw7091s4uBwGf xu7kVDR4OWgg1k3cnQJhGeMmxJJJ0/RscVX1MgEBCbZdqjV535WbLL9niR0TTWte2v418f 0KyKgGAYJvzvDqcTRt/ogk7FD0/zNpCG3caNa3c11JkiHgVajdmgOSVgpwKrtc/1p+Iwfu ClftqyIB9gdzH2eWMcHJXxb88DUDhuZCIvtlBN8s4A0DjDV86emKy1N2GRsIqBaXMMXOrv w+HvnJH8BdDRlX6DSmiBUSZu5IxZ7frcv6r/Z1b6n1eN+pFcLb2FxBnGlIsLdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1669040321; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=wjfki+S9HP7uwb/AML6k5pc2e9yTcQIXq33OVleR/IA=; b=Hz2HDMC8nwIOi7cn2gY/L/c47Wcv0+6hwiUqgdS5AUgMupwoajcfq5WHjumeaYtfzNFOFy xGiRY0ly33u8Wl4ZIDgmfT00NgKMMTcAlvY4YOSax8BjVMScfxk11KGYA8AbKEtqBUK78F KBXWhRrsdIbovkTZtfvOwOF49LOQ6Z2PTqiF7PzuYtiPbMgxM9+9WFG4hqKPGD+6VyXzM5 7z5dz9uhOY5Tfmag4/zELLcV3CTO99/QX3yWbNoH1X6dm3wqqRwORH2HddLbnyHvumrPaV MxNRwnRskMa51H+VA2lAgQS3uD6k5vK93TaRBGvoybWb6vPbBEEfWZXj5P7y0Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1669040321; a=rsa-sha256; cv=none; b=soh/uBUxmHePv2eP5agY6a/Zry23T2hNymRf5/2apU3kzR/xbusQvpZHEkVGICt+Rxhlxn 80APkSnihCBJ6kzNYPIfHxCsTk3fkKJdJGdTNTM0YruzGTICebgFJhV/qHnuCVdbuTtD4F mHPzkiWeZ9bL8BljCjbHT2Nmv6kIBvdtwmLlbG8j9uLnG7QMpUA/VT3fbh1bg+sdA8kTXO /n5ljFV3Hxjk6u/geDozhzyiuYSqYafIsi79rljj/jLDqorKpT4yPMsyrEPDcBrL1uZZcy KaE2lIw5iOdZF5zQhYsJsKJ8TXusCan+qt3Bveg22vmi9NVT6+vxmIP4vUYK6A== Received: from [192.168.9.9] (p578013d9.dip0.t-ipconnect.de [87.128.19.217]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NG8dD3wV0z1FMp; Mon, 21 Nov 2022 14:18:40 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: <39b67bfb-6533-74ef-2bdb-782fb162951f@FreeBSD.org> Date: Mon, 21 Nov 2022 15:18:37 +0100 List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.5.0 Subject: Re: git: c0f35dbf19c3 - main - vmm: Use a cpuset_t for vCPUs waiting for STARTUP IPIs. To: Kristof Provost , John Baldwin Cc: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org References: <202211181826.2AIIQssl030757@gitrepo.freebsd.org> From: =?UTF-8?Q?Corvin_K=c3=b6hne?= In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------AMx8XKtN7ijnKPl15jRb3Rqa" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------AMx8XKtN7ijnKPl15jRb3Rqa Content-Type: multipart/mixed; boundary="------------HNQIGZa4vhvfzWnws28XTU10"; protected-headers="v1" From: =?UTF-8?Q?Corvin_K=c3=b6hne?= To: Kristof Provost , John Baldwin Cc: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Message-ID: <39b67bfb-6533-74ef-2bdb-782fb162951f@FreeBSD.org> Subject: Re: git: c0f35dbf19c3 - main - vmm: Use a cpuset_t for vCPUs waiting for STARTUP IPIs. References: <202211181826.2AIIQssl030757@gitrepo.freebsd.org> In-Reply-To: --------------HNQIGZa4vhvfzWnws28XTU10 Content-Type: multipart/alternative; boundary="------------2nNzeS0r8mpJ3AS0wfJoygtT" --------------2nNzeS0r8mpJ3AS0wfJoygtT Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMTEvMjEvMjAyMiAyOjE1IFBNLCBLcmlzdG9mIFByb3Zvc3Qgd3JvdGU6DQo+DQo+IE9u IDE4IE5vdiAyMDIyLCBhdCAxOToyNiwgSm9obiBCYWxkd2luIHdyb3RlOg0KPg0KPiAgICAg VGhlIGJyYW5jaCBtYWluIGhhcyBiZWVuIHVwZGF0ZWQgYnkgamhiOg0KPg0KPiAgICAgVVJM Og0KPiAgICAgaHR0cHM6Ly9jZ2l0LkZyZWVCU0Qub3JnL3NyYy9jb21taXQvP2lkPWMwZjM1 ZGJmMTljM2M4ODI1YmQyYjMyMWQ4ZWZkNTgyODA3ZDE5NDANCj4gICAgIDxodHRwczovL2Nn aXQuRnJlZUJTRC5vcmcvc3JjL2NvbW1pdC8/aWQ9YzBmMzVkYmYxOWMzYzg4MjViZDJiMzIx ZDhlZmQ1ODI4MDdkMTk0MD4NCj4NCj4gICAgIGNvbW1pdCBjMGYzNWRiZjE5YzNjODgyNWJk MmIzMjFkOGVmZDU4MjgwN2QxOTQwDQo+ICAgICBBdXRob3I6IEpvaG4gQmFsZHdpbiA8amhi QEZyZWVCU0Qub3JnPg0KPiAgICAgQXV0aG9yRGF0ZTogMjAyMi0xMS0xOCAxODowNToxMCAr MDAwMA0KPiAgICAgQ29tbWl0OiBKb2huIEJhbGR3aW4gPGpoYkBGcmVlQlNELm9yZz4NCj4g ICAgIENvbW1pdERhdGU6IDIwMjItMTEtMTggMTg6MjU6MzggKzAwMDANCj4NCj4gICAgIHZt bTogVXNlIGEgY3B1c2V0X3QgZm9yIHZDUFVzIHdhaXRpbmcgZm9yIFNUQVJUVVAgSVBJcy4N Cj4NCj4gICAgIFJldGlyZSB0aGUgYm9vdF9zdGF0ZSBtZW1iZXIgb2Ygc3RydWN0IHZsYXBp YyBhbmQgaW5zdGVhZCB1c2UgYQ0KPiAgICAgY3B1c2V0DQo+ICAgICBpbiB0aGUgVk0gdG8g dHJhY2sgdkNQVXMgd2FpdGluZyBmb3IgU1RBUlRVUCBJUElzLiBJTklUIElQSXMgYWRkDQo+ ICAgICB2Q1BVcyB0byB0aGlzIHNldCwgYW5kIFNUQVJUVVAgSVBJcyByZW1vdmUgdkNQVXMg ZnJvbSB0aGUgc2V0Lg0KPiAgICAgU1RBUlRVUCBJUElzIGFyZSBvbmx5IHJlcG9ydGVkIHRv IHVzZXJsYW5kIGZvciB2Q1BVcyB0aGF0IHdlcmUNCj4gICAgIHJlbW92ZWQNCj4gICAgIGZy b20gdGhlIHNldC4NCj4NCj4gICAgIEluIHBhcnRpY3VsYXIsIHRoaXMgcGVybWl0cyBhIHN1 YnNlcXVlbnQgY2hhbmdlIHRvIGFsbG9jYXRlIHZDUFVzIG9uDQo+ICAgICBkZW1hbmQgd2hl biB0aGUgdkNQVSBtYXkgbm90IGJlIGFsbG9jYXRlZCB1bnRpbCBhZnRlciBhIFNUQVJUVVAN Cj4gICAgIElQSSBpcw0KPiAgICAgcmVwb3J0ZWQgdG8gdXNlcmxhbmQuDQo+DQo+ICAgICBS ZXZpZXdlZCBieTogY29ydmluaywgbWFya2oNCj4gICAgIERpZmZlcmVudGlhbCBSZXZpc2lv bjogaHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0QzNzE3Mw0KPiAgICAgLS0tDQo+ICAg ICBzeXMvYW1kNjQvaW5jbHVkZS92bW0uaCB8IDMgKysrDQo+ICAgICBzeXMvYW1kNjQvdm1t L2lvL3ZsYXBpYy5jIHwgNDYNCj4gICAgICsrKysrKysrKystLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLQ0KPiAgICAgc3lzL2FtZDY0L3ZtbS9pby92bGFwaWNfcHJpdi5oIHwg NyAtLS0tLS0tDQo+ICAgICBzeXMvYW1kNjQvdm1tL3ZtbS5jIHwgMjcgKysrKysrKysrKysr KysrKysrKysrKysrKw0KPiAgICAgNCBmaWxlcyBjaGFuZ2VkLCA0MSBpbnNlcnRpb25zKCsp LCA0MiBkZWxldGlvbnMoLSkNCj4NCj4NCj4gSeKAmW0gc2VlaW5nIGEgcGFuaWMgc3RhcnRp bmcgYSBiaHl2ZSBndWVzdCwgYW5kIEkgdGhpbmsgdGhpcyBjb21taXQgDQo+IG1pZ2h0IGJl IHRoZSByZXNwb25zaWJsZSBvbmU6DQo+DQo+IHxsb2dpbjogcGFuaWM6IGFjcXVpcmluZyBi bG9ja2FibGUgc2xlZXAgbG9jayB3aXRoIHNwaW5sb2NrIG9yIA0KPiBjcml0aWNhbCBzZWN0 aW9uIGhlbGQgKHNsZWVwIG11dGV4KSB2bSByZW5kZXp2b3VzIGxvY2sgQCANCj4gL3Vzci9z cmMvc3lzL2FtZDY0L3ZtbS92bW0uYzoyNTA4IGNwdWlkID0gMTQgdGltZSA9IDE2NjkwMzUy MTIgS0RCOiANCj4gc3RhY2sgYmFja3RyYWNlOiBkYl90cmFjZV9zZWxmX3dyYXBwZXIoKSBh dCANCj4gZGJfdHJhY2Vfc2VsZl93cmFwcGVyKzB4MmIvZnJhbWUgMHhmZmZmZmUwMTVmMmRk NTMwIHZwYW5pYygpIGF0IA0KPiB2cGFuaWMrMHgxNTEvZnJhbWUgMHhmZmZmZmUwMTVmMmRk NTgwIHBhbmljKCkgYXQgcGFuaWMrMHg0My9mcmFtZSANCj4gMHhmZmZmZmUwMTVmMmRkNWUw IHdpdG5lc3NfY2hlY2tvcmRlcigpIGF0IA0KPiB3aXRuZXNzX2NoZWNrb3JkZXIrMHhkM2Uv ZnJhbWUgMHhmZmZmZmUwMTVmMmRkN2EwIF9fbXR4X2xvY2tfZmxhZ3MoKSANCj4gYXQgX19t dHhfbG9ja19mbGFncysweDk0L2ZyYW1lIDB4ZmZmZmZlMDE1ZjJkZDdmMCB2bV9zdGFydF9j cHVzKCkgYXQgDQo+IHZtX3N0YXJ0X2NwdXMrMHgzMS9mcmFtZSAweGZmZmZmZTAxNWYyZGQ4 MjAgDQo+IHZsYXBpY19pY3Jsb193cml0ZV9oYW5kbGVyKCkgYXQgdmxhcGljX2ljcmxvX3dy aXRlX2hhbmRsZXIrMHgzMGMvZnJhbWUgDQo+IDB4ZmZmZmZlMDE1ZjJkZDhhMCB2bXhfcnVu KCkgYXQgdm14X3J1bisweDIwZWQvZnJhbWUgDQo+IDB4ZmZmZmZlMDE1ZjJkZDljMCB2bV9y dW4oKSBhdCB2bV9ydW4rMHgxZDIvZnJhbWUgMHhmZmZmZmUwMTVmMmRkYWEwIA0KPiB2bW1k ZXZfaW9jdGwoKSBhdCB2bW1kZXZfaW9jdGwrMHg2NDQvZnJhbWUgMHhmZmZmZmUwMTVmMmRk YjQwIA0KPiBkZXZmc19pb2N0bCgpIGF0IGRldmZzX2lvY3RsKzB4Y2QvZnJhbWUgMHhmZmZm ZmUwMTVmMmRkYjkwIHZuX2lvY3RsKCkgDQo+IGF0IHZuX2lvY3RsKzB4MTMxL2ZyYW1lIDB4 ZmZmZmZlMDE1ZjJkZGNhMCBkZXZmc19pb2N0bF9mKCkgYXQgDQo+IGRldmZzX2lvY3RsX2Yr MHgxZS9mcmFtZSAweGZmZmZmZTAxNWYyZGRjYzAga2Vybl9pb2N0bCgpIGF0IA0KPiBrZXJu X2lvY3RsKzB4MjAyL2ZyYW1lIDB4ZmZmZmZlMDE1ZjJkZGQzMCBzeXNfaW9jdGwoKSBhdCAN Cj4gc3lzX2lvY3RsKzB4MTJhL2ZyYW1lIDB4ZmZmZmZlMDE1ZjJkZGUwMCBhbWQ2NF9zeXNj YWxsKCkgYXQgDQo+IGFtZDY0X3N5c2NhbGwrMHgxMmUvZnJhbWUgMHhmZmZmZmUwMTVmMmRk ZjMwIGZhc3Rfc3lzY2FsbF9jb21tb24oKSBhdCANCj4gZmFzdF9zeXNjYWxsX2NvbW1vbisw eGY4L2ZyYW1lIDB4ZmZmZmZlMDE1ZjJkZGYzMCAtLS0gc3lzY2FsbCAoNTQsIA0KPiBGcmVl QlNEIEVMRjY0LCBpb2N0bCksIHJpcCA9IDB4M2RkOWNiZDdmOTRhLCByc3AgPSAweDNkZGMx ZDQ0YWU1OCwgcmJwIA0KPiA9IDB4M2RkYzFkNDRhZjEwIC0tLSB8DQo+DQo+IEFuZCBrZ2Ri 4oCZcyBiYWNrdHJhY2U6DQo+DQo+IHwoa2dkYikgYnQgIzAgX19jdXJ0aHJlYWQgKCkgYXQg DQo+IC91c3Ivc3JjL3N5cy9hbWQ2NC9pbmNsdWRlL3BjcHVfYXV4Lmg6NTkgIzEgZHVtcF9z YXZlY3R4ICgpIGF0IA0KPiAvdXNyL3NyYy9zeXMva2Vybi9rZXJuX3NodXRkb3duLmM6NDA1 ICMyIDB4ZmZmZmZmZmY4MGJlYzY3OCBpbiBkdW1wc3lzIA0KPiAoZGk9MHgwKSBhdCAvdXNy L3NyYy9zeXMveDg2L2luY2x1ZGUvZHVtcC5oOjg3ICMzIGRvYWR1bXAgDQo+ICh0ZXh0ZHVt cD10ZXh0ZHVtcEBlbnRyeT0wKSBhdCAvdXNyL3NyYy9zeXMva2Vybi9rZXJuX3NodXRkb3du LmM6NDM0IA0KPiAjNCAweGZmZmZmZmZmODA0YjQwM2EgaW4gZGJfZHVtcCAoZHVtbXk9PG9w dGltaXplZCBvdXQ+LCANCj4gZHVtbXkyPTx1bmF2YWlsYWJsZT4sIGR1bW15Mz08dW5hdmFp bGFibGU+LCBkdW1teTQ9PHVuYXZhaWxhYmxlPikgYXQgDQo+IC91c3Ivc3JjL3N5cy9kZGIv ZGJfY29tbWFuZC5jOjU5MyAjNSAweGZmZmZmZmZmODA0YjNlNDAgaW4gZGJfY29tbWFuZCAN Cj4gKGxhc3RfY21kcD08b3B0aW1pemVkIG91dD4sIGNtZF90YWJsZT08b3B0aW1pemVkIG91 dD4sIGRvcGFnZXI9dHJ1ZSkgDQo+IGF0IC91c3Ivc3JjL3N5cy9kZGIvZGJfY29tbWFuZC5j OjUwNiAjNiAweGZmZmZmZmZmODA0YjNiMGQgaW4gDQo+IGRiX2NvbW1hbmRfbG9vcCAoKSBh dCAvdXNyL3NyYy9zeXMvZGRiL2RiX2NvbW1hbmQuYzo1NTMgIzcgDQo+IDB4ZmZmZmZmZmY4 MDRiNzFhNiBpbiBkYl90cmFwICh0eXBlPTxvcHRpbWl6ZWQgb3V0PiwgY29kZT08b3B0aW1p emVkIA0KPiBvdXQ+KSBhdCAvdXNyL3NyYy9zeXMvZGRiL2RiX21haW4uYzoyNzAgIzggMHhm ZmZmZmZmZjgwYzNiODllIGluIA0KPiBrZGJfdHJhcCAodHlwZT10eXBlQGVudHJ5PTMsIGNv ZGU9PHVuYXZhaWxhYmxlPiwgY29kZUBlbnRyeT0wLCANCj4gdGY9dGZAZW50cnk9MHhmZmZm ZmUwMTVmMmRkNDcwKSBhdCAvdXNyL3NyYy9zeXMva2Vybi9zdWJyX2tkYi5jOjc0NSAjOSAN Cj4gMHhmZmZmZmZmZjgxMGNlNTc3IGluIHRyYXAgKGZyYW1lPTB4ZmZmZmZlMDE1ZjJkZDQ3 MCkgYXQgDQo+IC91c3Ivc3JjL3N5cy9hbWQ2NC9hbWQ2NC90cmFwLmM6NjExICMxMCA8c2ln bmFsIGhhbmRsZXIgY2FsbGVkPiAjMTEgDQo+IGtkYl9lbnRlciAod2h5PTxvcHRpbWl6ZWQg b3V0PiwgbXNnPTxvcHRpbWl6ZWQgb3V0PikgYXQgDQo+IC91c3Ivc3JjL3N5cy9rZXJuL3N1 YnJfa2RiLmM6NTA5ICMxMiAweGZmZmZmZmZmODBiZWM4MjIgaW4gdnBhbmljIA0KPiAoZm10 PTxvcHRpbWl6ZWQgb3V0PiwgYXA9YXBAZW50cnk9MHhmZmZmZmUwMTVmMmRkNWMwKSBhdCAN Cj4gL3Vzci9zcmMvc3lzL2tlcm4va2Vybl9zaHV0ZG93bi5jOjk2NyAjMTMgMHhmZmZmZmZm ZjgwYmVjNWMzIGluIHBhbmljIA0KPiAoZm10PTB4ZmZmZmZmZmY4MWU4Y2U3MCA8Y25wdXRz X210eD4gIlwzMTQ6KVwyMDFcMzc3XDM3N1wzNzdcMzc3IikgYXQgDQo+IC91c3Ivc3JjL3N5 cy9rZXJuL2tlcm5fc2h1dGRvd24uYzo5MDMgIzE0IDB4ZmZmZmZmZmY4MGM1ZWE0ZSBpbiAN Cj4gd2l0bmVzc19jaGVja29yZGVyIChsb2NrPTB4ZmZmZmY4MDRkNTg1ZDEzOCwgZmxhZ3M9 PG9wdGltaXplZCBvdXQ+LCANCj4gZmlsZT08b3B0aW1pemVkIG91dD4sIGxpbmU9MjUwOCwg aW50ZXJsb2NrPTxvcHRpbWl6ZWQgb3V0PikgYXQgDQo+IC91c3Ivc3JjL3N5cy9rZXJuL3N1 YnJfd2l0bmVzcy5jOjEyMDIgIzE1IDB4ZmZmZmZmZmY4MGJjNmQwNCBpbiANCj4gX19tdHhf bG9ja19mbGFncyAoYz0weGZmZmZmODA0ZDU4NWQxNTAsIG9wdHM9MCwgDQo+IGZpbGU9MHhm ZmZmZmZmZjgzMjI1NDJkICIvdXNyL3NyYy9zeXMvYW1kNjQvdm1tL3ZtbS5jIiwgbGluZT0y NTA4KSBhdCANCj4gL3Vzci9zcmMvc3lzL2tlcm4va2Vybl9tdXRleC5jOjI3OCAjMTYgMHhm ZmZmZmZmZjgzMjA0MTAxIGluIA0KPiB2bV9zdGFydF9jcHVzICh2bT0weGZmZmZmODA0ZDU4 NWQwMDAsIA0KPiB0b3N0YXJ0PXRvc3RhcnRAZW50cnk9MHhmZmZmZmUwMTVmMmRkODU4KSBh dCANCj4gL3Vzci9zcmMvc3lzL2FtZDY0L3ZtbS92bW0uYzoyNTA4ICMxNyAweGZmZmZmZmZm ODMyMTFiNWMgaW4gDQo+IHZsYXBpY19pY3Jsb193cml0ZV9oYW5kbGVyICh2bGFwaWM9dmxh cGljQGVudHJ5PTB4ZmZmZmY4MDQ4ZGMxNGE4MCwgDQo+IHJldHU9cmV0dUBlbnRyeT0weGZm ZmZmZTAxNWYyZGQ5NTApIGF0IA0KPiAvdXNyL3NyYy9zeXMvYW1kNjQvdm1tL2lvL3ZsYXBp Yy5jOjExNzIgIzE4IDB4ZmZmZmZmZmY4MzIxOTc3ZCBpbiANCj4gdm14X2hhbmRsZV9hcGlj X3dyaXRlICh2Y3B1PTB4ZmZmZmY4MDQ4ZGIzYWMwMCwgDQo+IHZsYXBpYz0weGZmZmZmODA0 OGRjMTRhODAsIHF1YWw9NzY4KSBhdCANCj4gL3Vzci9zcmMvc3lzL2FtZDY0L3ZtbS9pbnRl bC92bXguYzoyMTg0ICMxOSB2bXhfZXhpdF9wcm9jZXNzIA0KPiAodm14PTB4ZmZmZmY4MDRk NTg1YjAwMCwgdmNwdT0weGZmZmZmODA0OGRiM2FjMDAsIA0KPiB2bWV4aXQ9MHhmZmZmZjgw NGE3YzFhNjg4KSBhdCAvdXNyL3NyYy9zeXMvYW1kNjQvdm1tL2ludGVsL3ZteC5jOjI3Njcg DQo+ICMyMCB2bXhfcnVuICh2Y3B1aT0weGZmZmZmODA0OGRiM2FjMDAsIHJpcD08b3B0aW1p emVkIG91dD4sIA0KPiBwbWFwPTB4ZmZmZmZlMDAzZGNlMTUzMCwgZXZpbmZvPTB4ZmZmZmZl MDE1ZjJkZDlkOCkgYXQgDQo+IC91c3Ivc3JjL3N5cy9hbWQ2NC92bW0vaW50ZWwvdm14LmM6 MzE3NCAjMjEgMHhmZmZmZmZmZjgzMjAyNTcyIGluIA0KPiB2bV9ydW4gKHZjcHU9dmNwdUBl bnRyeT0weGZmZmZmODA0YTdjMWE2MDAsIA0KPiB2bWVfdXNlcj12bWVfdXNlckBlbnRyeT0w eGZmZmZmODAwMzI2MDFiMDgpIGF0IA0KPiAvdXNyL3NyYy9zeXMvYW1kNjQvdm1tL3ZtbS5j OjE4NzMgIzIyIDB4ZmZmZmZmZmY4MzIwNWFiNCBpbiANCj4gdm1tZGV2X2lvY3RsIChjZGV2 PTxvcHRpbWl6ZWQgb3V0PiwgY21kPTMyMzA2OTI4NjUsIGRhdGE9PG9wdGltaXplZCANCj4g b3V0PiwgZmZsYWc9PG9wdGltaXplZCBvdXQ+LCB0ZD08b3B0aW1pemVkIG91dD4pIGF0IA0K PiAvdXNyL3NyYy9zeXMvYW1kNjQvdm1tL3ZtbV9kZXYuYzo1NjUgIzIzIDB4ZmZmZmZmZmY4 MGE3YmU3ZCBpbiANCj4gZGV2ZnNfaW9jdGwgKGFwPTB4ZmZmZmZlMDE1ZjJkZGJhOCkgYXQg DQo+IC91c3Ivc3JjL3N5cy9mcy9kZXZmcy9kZXZmc192bm9wcy5jOjkzMyAjMjQgMHhmZmZm ZmZmZjgwY2Y2NDIxIGluIA0KPiB2bl9pb2N0bCAoZnA9MHhmZmZmZjgwMDBjZTJjYjQwLCBj b209PG9wdGltaXplZCBvdXQ+LCANCj4gZGF0YT0weGZmZmZmODAwMzI2MDFiMDAsIGFjdGl2 ZV9jcmVkPTB4ZmZmZmY4MDAwYzRkMDgwMCwgdGQ9MHgxMCkgYXQgDQo+IC91c3Ivc3JjL3N5 cy9rZXJuL3Zmc192bm9wcy5jOjE2OTkgIzI1IDB4ZmZmZmZmZmY4MGE3YzUyZSBpbiANCj4g ZGV2ZnNfaW9jdGxfZiAoZnA9MHhmZmZmZmZmZjgxZThjZTcwIDxjbnB1dHNfbXR4PiwgY29t PTAsIA0KPiBkYXRhPTB4ZmZmZmZmZmY4MTJhMDAwMCwgY3JlZD0weDEsIHRkPTB4MTApIGF0 IA0KPiAvdXNyL3NyYy9zeXMvZnMvZGV2ZnMvZGV2ZnNfdm5vcHMuYzo4NjQgIzI2IDB4ZmZm ZmZmZmY4MGM2NDY3MiBpbiANCj4gZm9faW9jdGwgKGZwPTB4ZmZmZmY4MDAwY2UyY2I0MCwg Y29tPTMyMzA2OTI4NjUsIGRhdGE9MHgxZDAsIA0KPiBhY3RpdmVfY3JlZD0weDEsIHRkPTxv cHRpbWl6ZWQgb3V0PikgYXQgL3Vzci9zcmMvc3lzL3N5cy9maWxlLmg6MzY1IA0KPiAjMjcg a2Vybl9pb2N0bCAodGQ9dGRAZW50cnk9MHhmZmZmZmUwMTYwOTM2MDAwLCBmZD08b3B0aW1p emVkIG91dD4sIA0KPiBjb209Y29tQGVudHJ5PTMyMzA2OTI4NjUsIGRhdGE9MHgxZDAgPGVy cm9yOiBDYW5ub3QgYWNjZXNzIG1lbW9yeSBhdCANCj4gYWRkcmVzcyAweDFkMD4sIGRhdGFA ZW50cnk9MHhmZmZmZjgwMDMyNjAxYjAwICIiKSBhdCANCj4gL3Vzci9zcmMvc3lzL2tlcm4v c3lzX2dlbmVyaWMuYzo4MDMgIzI4IDB4ZmZmZmZmZmY4MGM2NDNiYSBpbiANCj4gc3lzX2lv Y3RsICh0ZD0weGZmZmZmZTAxNjA5MzYwMDAsIHVhcD0weGZmZmZmZTAxNjA5MzYzZjgpIGF0 IA0KPiAvdXNyL3NyYy9zeXMva2Vybi9zeXNfZ2VuZXJpYy5jOjcxMSAjMjkgMHhmZmZmZmZm ZjgxMGNmM2JlIGluIA0KPiBzeXNjYWxsZW50ZXIgKHRkPTxvcHRpbWl6ZWQgb3V0PikgYXQg DQo+IC91c3Ivc3JjL3N5cy9hbWQ2NC9hbWQ2NC8uLi8uLi9rZXJuL3N1YnJfc3lzY2FsbC5j OjE4OSAjMzAgDQo+IGFtZDY0X3N5c2NhbGwgKHRkPTB4ZmZmZmZlMDE2MDkzNjAwMCwgdHJh Y2VkPTApIGF0IA0KPiAvdXNyL3NyYy9zeXMvYW1kNjQvYW1kNjQvdHJhcC5jOjEyMDAgIzMx IDxzaWduYWwgaGFuZGxlciBjYWxsZWQ+ICMzMiANCj4gMHgwMDAwM2RkOWNiZDdmOTRhIGlu ID8/ICgpIEJhY2t0cmFjZSBzdG9wcGVkOiBDYW5ub3QgYWNjZXNzIG1lbW9yeSBhdCANCj4g YWRkcmVzcyAweDNkZGMxZDQ0YWU1OCAoa2dkYikgZnIgMTYgIzE2IDB4ZmZmZmZmZmY4MzIw NDEwMSBpbiANCj4gdm1fc3RhcnRfY3B1cyAodm09MHhmZmZmZjgwNGQ1ODVkMDAwLCANCj4g dG9zdGFydD10b3N0YXJ0QGVudHJ5PTB4ZmZmZmZlMDE1ZjJkZDg1OCkgYXQgDQo+IC91c3Iv c3JjL3N5cy9hbWQ2NC92bW0vdm1tLmM6MjUwOCAyNTA4IG10eF9sb2NrKCZ2bS0+cmVuZGV6 dm91c19tdHgpOyB8DQo+DQo+IEkgYmVsaWV2ZSBXSVRORVNTIGlzIHVwc2V0IHRoYXQgd2Xi gJlyZSBnb2luZyB0byBwb3RlbnRpYWxseSBzbGVlcCBkb2luZyANCj4gbXR4X2xvY2soJnZt LT5yZW5kZXp2b3VzX210eCk7IGluIHZtX3N0YXJ0X2NwdXMoKSB3aGVuIHdl4oCZdmUgZG9u ZSANCj4gY3JpdGljYWxfZW50ZXIoKSBpbiB2bV9ydW4oKS4NCj4NCj4gQmVzdCByZWdhcmRz LA0KPiBLcmlzdG9mDQo+DQoNCkNhbiB5b3UgcGxlYXNlIHRlc3QgaHR0cHM6Ly9yZXZpZXdz LmZyZWVic2Qub3JnL0QzNzQ1Mj8NCg0KDQotLSANCkJlc3QgcmVnYXJkcywNCkNvcnZpbg0K DQo= --------------2nNzeS0r8mpJ3AS0wfJoygtT Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On 11/21/2022 2:15 PM, Kristof Provost= wrote:

On 18 Nov 2022, at 19:26, John Baldwin wrote:

The branch main has been updated by jhb:

URL: https://cgit.FreeBSD.org/src/com= mit/?id=3Dc0f35dbf19c3c8825bd2b321d8efd582807d1940

commit c0f35dbf19c3c8825bd2b321d8efd582807d1940
Author: John Baldwin <jhb@FreeBSD.org>
AuthorDate: 2022-11-18 18:05:10 +0000
Commit: John Baldwin <jhb@FreeBSD.org>
CommitDate: 2022-11-18 18:25:38 +0000

vmm: Use a cpuset_t for vCPUs waiting for STARTUP IPIs.

Retire the boot_state member of struct vlapi= c and instead use a cpuset
in the VM to track vCPUs waiting for STARTUP IPIs. INIT IPIs add
vCPUs to this set, and STARTUP IPIs remove vCPUs from the set.
STARTUP IPIs are only reported to userland for vCPUs that were removed
from the set.

In particular, this permits a subsequent change to allocate vCPUs on
demand when the vCPU may not be allocated until after a STARTUP IPI is
reported to userland.

Reviewed by: corvink, markj
Differential Revision: = https://reviews.freebsd.org/D37173
---
sys/amd64/include/vmm.h | 3 +++
sys/amd64/vmm/io/vlapic.c | 46 ++++++++++--------------------------------
sys/amd64/vmm/io/vlapic_priv.h | 7 -------
sys/amd64/vmm/vmm.c | 27 +++++++++++++++++++++++++
4 files changed, 41 insertions(+), 42 deletions(-)


I=E2=80=99m seeing a panic starting a bhyve gue= st, and I think this commit might be the responsible one:

login: panic: acquiring blockable sleep lock with spinlock or cri=
tical section held (sleep mutex) vm rendezvous lock @ /usr/src/sys/amd64/=
vmm/vmm.c:2508
cpuid =3D 14
time =3D 1669035212
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe015f2=
dd530
vpanic() at vpanic+0x151/frame 0xfffffe015f2dd580
panic() at panic+0x43/frame 0xfffffe015f2dd5e0
witness_checkorder() at witness_checkorder+0xd3e/frame 0xfffffe015f2dd7a0=

__mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xfffffe015f2dd7f0
vm_start_cpus() at vm_start_cpus+0x31/frame 0xfffffe015f2dd820
vlapic_icrlo_write_handler() at vlapic_icrlo_write_handler+0x30c/frame 0x=
fffffe015f2dd8a0
vmx_run() at vmx_run+0x20ed/frame 0xfffffe015f2dd9c0
vm_run() at vm_run+0x1d2/frame 0xfffffe015f2ddaa0
vmmdev_ioctl() at vmmdev_ioctl+0x644/frame 0xfffffe015f2ddb40
devfs_ioctl() at devfs_ioctl+0xcd/frame 0xfffffe015f2ddb90
vn_ioctl() at vn_ioctl+0x131/frame 0xfffffe015f2ddca0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe015f2ddcc0
kern_ioctl() at kern_ioctl+0x202/frame 0xfffffe015f2ddd30
sys_ioctl() at sys_ioctl+0x12a/frame 0xfffffe015f2dde00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe015f2ddf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe015f2ddf3=
0
--- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x3dd9cbd7f94a, rsp =3D 0=
x3ddc1d44ae58, rbp =3D 0x3ddc1d44af10 ---

And kgdb=E2=80=99s backtrace:

(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
#1  dump_savectx () at /usr/src/sys/kern/kern_shutdown.c:405
#2  0xffffffff80bec678 in dumpsys (di=3D0x0) at /usr/src/sys/x86/include/=
dump.h:87
#3  doadump (textdump=3Dtextdump@entry=3D0) at /usr/src/sys/kern/kern_shu=
tdown.c:434
#4  0xffffffff804b403a in db_dump (dummy=3D<optimized out>, dummy2=3D=
<unavailable>, dummy3=3D<unavailable>, dummy4=3D<unavailab=
le>) at /usr/src/sys/ddb/db_command.c:593
#5  0xffffffff804b3e40 in db_command (last_cmdp=3D<optimized out>, =
cmd_table=3D<optimized out>, dopager=3Dtrue) at /usr/src/sys/ddb/db=
_command.c:506
#6  0xffffffff804b3b0d in db_command_loop () at /usr/src/sys/ddb/db_comma=
nd.c:553
#7  0xffffffff804b71a6 in db_trap (type=3D<optimized out>, code=3D&=
lt;optimized out>) at /usr/src/sys/ddb/db_main.c:270
#8  0xffffffff80c3b89e in kdb_trap (type=3Dtype@entry=3D3, code=3D<una=
vailable>, code@entry=3D0, tf=3Dtf@entry=3D0xfffffe015f2dd470) at /usr=
/src/sys/kern/subr_kdb.c:745
#9  0xffffffff810ce577 in trap (frame=3D0xfffffe015f2dd470) at /usr/src/s=
ys/amd64/amd64/trap.c:611
#10 <signal handler called>
#11 kdb_enter (why=3D<optimized out>, msg=3D<optimized out>) =
at /usr/src/sys/kern/subr_kdb.c:509
#12 0xffffffff80bec822 in vpanic (fmt=3D<optimized out>, ap=3Dap@en=
try=3D0xfffffe015f2dd5c0) at /usr/src/sys/kern/kern_shutdown.c:967
#13 0xffffffff80bec5c3 in panic (fmt=3D0xffffffff81e8ce70 <cnputs_mtx&=
gt; "\314:)\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:90=
3
#14 0xffffffff80c5ea4e in witness_checkorder (lock=3D0xfffff804d585d138, =
flags=3D<optimized out>, file=3D<optimized out>, line=3D2508,=
 interlock=3D<optimized out>)
    at /usr/src/sys/kern/subr_witness.c:1202
#15 0xffffffff80bc6d04 in __mtx_lock_flags (c=3D0xfffff804d585d150, opts=3D=
0, file=3D0xffffffff8322542d "/usr/src/sys/amd64/vmm/vmm.c", line=3D2508)=
 at /usr/src/sys/kern/kern_mutex.c:278
#16 0xffffffff83204101 in vm_start_cpus (vm=3D0xfffff804d585d000, tostart=
=3Dtostart@entry=3D0xfffffe015f2dd858) at /usr/src/sys/amd64/vmm/vmm.c:25=
08
#17 0xffffffff83211b5c in vlapic_icrlo_write_handler (vlapic=3Dvlapic@ent=
ry=3D0xfffff8048dc14a80, retu=3Dretu@entry=3D0xfffffe015f2dd950) at /usr/=
src/sys/amd64/vmm/io/vlapic.c:1172
#18 0xffffffff8321977d in vmx_handle_apic_write (vcpu=3D0xfffff8048db3ac0=
0, vlapic=3D0xfffff8048dc14a80, qual=3D768) at /usr/src/sys/amd64/vmm/int=
el/vmx.c:2184
#19 vmx_exit_process (vmx=3D0xfffff804d585b000, vcpu=3D0xfffff8048db3ac00=
, vmexit=3D0xfffff804a7c1a688) at /usr/src/sys/amd64/vmm/intel/vmx.c:2767=

#20 vmx_run (vcpui=3D0xfffff8048db3ac00, rip=3D<optimized out>, pma=
p=3D0xfffffe003dce1530, evinfo=3D0xfffffe015f2dd9d8) at /usr/src/sys/amd6=
4/vmm/intel/vmx.c:3174
#21 0xffffffff83202572 in vm_run (vcpu=3Dvcpu@entry=3D0xfffff804a7c1a600,=
 vme_user=3Dvme_user@entry=3D0xfffff80032601b08) at /usr/src/sys/amd64/vm=
m/vmm.c:1873
#22 0xffffffff83205ab4 in vmmdev_ioctl (cdev=3D<optimized out>, cmd=
=3D3230692865, data=3D<optimized out>, fflag=3D<optimized out>=
;, td=3D<optimized out>) at /usr/src/sys/amd64/vmm/vmm_dev.c:565
#23 0xffffffff80a7be7d in devfs_ioctl (ap=3D0xfffffe015f2ddba8) at /usr/s=
rc/sys/fs/devfs/devfs_vnops.c:933
#24 0xffffffff80cf6421 in vn_ioctl (fp=3D0xfffff8000ce2cb40, com=3D<op=
timized out>, data=3D0xfffff80032601b00, active_cred=3D0xfffff8000c4d0=
800, td=3D0x10) at /usr/src/sys/kern/vfs_vnops.c:1699
#25 0xffffffff80a7c52e in devfs_ioctl_f (fp=3D0xffffffff81e8ce70 <cnpu=
ts_mtx>, com=3D0, data=3D0xffffffff812a0000, cred=3D0x1, td=3D0x10) at=
 /usr/src/sys/fs/devfs/devfs_vnops.c:864
#26 0xffffffff80c64672 in fo_ioctl (fp=3D0xfffff8000ce2cb40, com=3D323069=
2865, data=3D0x1d0, active_cred=3D0x1, td=3D<optimized out>) at /us=
r/src/sys/sys/file.h:365
#27 kern_ioctl (td=3Dtd@entry=3D0xfffffe0160936000, fd=3D<optimized ou=
t>, com=3Dcom@entry=3D3230692865, data=3D0x1d0 <error: Cannot acces=
s memory at address 0x1d0>, data@entry=3D0xfffff80032601b00 "")
    at /usr/src/sys/kern/sys_generic.c:803
#28 0xffffffff80c643ba in sys_ioctl (td=3D0xfffffe0160936000, uap=3D0xfff=
ffe01609363f8) at /usr/src/sys/kern/sys_generic.c:711
#29 0xffffffff810cf3be in syscallenter (td=3D<optimized out>) at /u=
sr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189
#30 amd64_syscall (td=3D0xfffffe0160936000, traced=3D0) at /usr/src/sys/a=
md64/amd64/trap.c:1200
#31 <signal handler called>
#32 0x00003dd9cbd7f94a in ?? ()
Backtrace stopped: Cannot access memory at address 0x3ddc1d44ae58
(kgdb) fr 16
#16 0xffffffff83204101 in vm_start_cpus (vm=3D0xfffff804d585d000, tostart=
=3Dtostart@entry=3D0xfffffe015f2dd858) at /usr/src/sys/amd64/vmm/vmm.c:25=
08
2508		mtx_lock(&vm->rendezvous_mtx);

I believe WITNESS is upset that we=E2=80=99re g= oing to potentially sleep doing mtx_lock(&vm->rendezvous_mtx); in vm_start_cpus() when we=E2=80=99ve done critical_enter() in vm_run().

Best regards,
Kristof


Can you please test https://reviews.freebsd.org/D37452?<= br> 


--=20
Best regards,
Corvin
--------------2nNzeS0r8mpJ3AS0wfJoygtT-- --------------HNQIGZa4vhvfzWnws28XTU10-- --------------AMx8XKtN7ijnKPl15jRb3Rqa Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmN7iL0FAwAAAAAACgkQ2FTaVjFeAmp9 qRAAwtA9RPCim2H0RC1EseE205i8Dnipl53OZwY73hifamTJsxVOKerubqOwm+7zU5CCwXxRFixR E3MNOG3kME6QGDuNtncPwa2paFXyGIJ3sPgh7mhjAfQ2Td4lqf4VxdwCp5GRFSRvRUfb4eIM6+Iy 2Xl//MHeFxFiACtaak1m/5BpUlPxeeQD5Ek69U1Kmq1Sa5hT0U7GLhTconl/7spNi0yZIrsBr6aO djPo9KeQHtVpmI26Y9j9K7hL3+RulufMddkRpbs0c27E+5M3CkfPYBlZD1mobPeeeoLgxJ4HjWuv J4WSsNOstVN65QIxXCPGJ9zhGmPZQONuJmvE3bSMxbfCbWuuNbWzTb3sOnlQ2v9JpQGJ6Sw0blR2 0Jj6/MOgpXz5PZ3MdYvR4L8B3wCRTlNU7zC7Mce/L6qkjB/hCIz34CVfAO5zvjDguy02pDJN/Dms XzkQe5gTdo2YAM1ivUSL+M7RsWiZV347J6Hu0Sim8kizGqoCIYwVn/trNiE2nfR5F3Q5eL3+LFsn 3jGI60heHYN9NhNtMY+FOm92tcK3jQYGj0jlO+fCyFsTg8zdcEHf79skGTOeSmSmy6WZQ+astsOW wDlWSwNko6IztT+3Hzdt8hGphXfrNX3zpXWqCR6UDA5nqd8FmOQngWnT6LGJL20Jnom1taGu+cQU zKc= =EtqY -----END PGP SIGNATURE----- --------------AMx8XKtN7ijnKPl15jRb3Rqa--