From nobody Wed Dec 29 17:08:45 2021 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0B92619256AC; Wed, 29 Dec 2021 17:08:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JPHtP5QxWz4fWw; Wed, 29 Dec 2021 17:08:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9A36285B; Wed, 29 Dec 2021 17:08:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1BTH8jq1008748; Wed, 29 Dec 2021 17:08:45 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1BTH8jFx008747; Wed, 29 Dec 2021 17:08:45 GMT (envelope-from git) Date: Wed, 29 Dec 2021 17:08:45 GMT Message-Id: <202112291708.1BTH8jFx008747@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: "Bjoern A. Zeeb" Subject: git: f1442847c940 - main - bhyve: passthru: enable BARs before possibly mmap(2)ing them List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: bz X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: f1442847c9404d4bc5f5524a0c3362dd39cb14f9 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1640797725; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=pqo5ODQvYZlnXmX2B//8ZVpZCEEVQ9USmtYN9Gb/4bw=; b=CesI5hZo69fElta1RHRrxv2bObskR4E9GcdUJxG/mjx2G8xkORDDm60covd0WwfmdJh93z /EKG9IQUNzhFj1pdvFlUie7fhnwCds0Pnl4TFImpiV9qqA9wZdrzu6A58KQNsrQFl0WWjp w+2T32hK40cHHZrWPn7Yg7otN7nBEhSQUzsJoNLOmmucP+HywYgPVavvqu4z2fg8TzvZ69 G4VKjNTEUcyNIKO7O8KhBq4nPkcXWZlaTILGaOG/wA9Ry2EQ5BKTXbbiAZLbLimEU/i64e 5AkI1mwCNSsRKHmAHg7BUREyTd+/U8YUwMLEAPPLbIWL/JiGIxaxRjhAx3etJA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1640797725; a=rsa-sha256; cv=none; b=iiomZkghi7QSFqd27U8JQBHtBJ49ICJW4jT6SiBpucJ2/mUYQJbiFlQUhb7vqlQKG5ZGBv uY8lh9Sl946bNZKP3mR3NcI7GrN83WNcS58ruxPvBYp7mkKJt+ZXVypWqiQGez8AYW1KkW Q8SyXoEINzOqqNmN6ehgI6EuQBi5j82qHz9hZo6aBxEpDO8gO4umsekc7p3B8UNiQPlqiu r187U9M4fQN6rlLvahCxx4wZxkYwaMoNcbN6DpWV9X4DzpWo/aMoRQAlutRf8ExsmCfmBE FKhQeex6zA1XXWxlvlhZjoji4guUgpvfDtDqHDSLTeMqJkhKIJ6jVJz7VDTMlw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by bz: URL: https://cgit.FreeBSD.org/src/commit/?id=f1442847c9404d4bc5f5524a0c3362dd39cb14f9 commit f1442847c9404d4bc5f5524a0c3362dd39cb14f9 Author: Bjoern A. Zeeb AuthorDate: 2021-12-23 14:59:49 +0000 Commit: Bjoern A. Zeeb CommitDate: 2021-12-29 17:01:05 +0000 bhyve: passthru: enable BARs before possibly mmap(2)ing them The first time we start bhyve with a passthru device everything is fine as on boot we do enable BARs. If a driver (unload) inside bhyve disables the BAR(s) as some Linux drivers do, we need to make sure we re-enable them on next bhyve start. If we are trying to mmap a disabled BAR for MSI-X (PCIOCBARMMAP) the kernel will give us an EBUSY. While we were re-enabling the BAR(s) in the current code loop cfginit() was writing the changes out too late to the real hardware. Move the call to init_msix_table() after the register on the real hardware was updated. That way the kernel will be happy and the mmap will succeed and bhyve will start. Also simplify the code given the last argument to init_msix_table() is unused we do not need to do checks for each bar. [1] MFC after: 3 days PR: 260148 Pointed out by: markj [1] Sponsored by: The FreeBSD Foundation Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D33628 --- usr.sbin/bhyve/pci_passthru.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/usr.sbin/bhyve/pci_passthru.c b/usr.sbin/bhyve/pci_passthru.c index 9ad19f440e37..769d7c160037 100644 --- a/usr.sbin/bhyve/pci_passthru.c +++ b/usr.sbin/bhyve/pci_passthru.c @@ -420,7 +420,7 @@ msix_table_write(struct vmctx *ctx, int vcpu, struct passthru_softc *sc, } static int -init_msix_table(struct vmctx *ctx, struct passthru_softc *sc, uint64_t base) +init_msix_table(struct vmctx *ctx, struct passthru_softc *sc) { struct pci_devinst *pi = sc->psc_pi; struct pci_bar_mmap pbm; @@ -548,13 +548,6 @@ cfginitbar(struct vmctx *ctx, struct passthru_softc *sc) sc->psc_bar[i].lobits = lobits; pi->pi_bar[i].lobits = lobits; - /* The MSI-X table needs special handling */ - if (i == pci_msix_table_bar(pi)) { - error = init_msix_table(ctx, sc, base); - if (error) - return (-1); - } - /* * 64-bit BAR takes up two slots so skip the next one. */ @@ -596,7 +589,17 @@ cfginit(struct vmctx *ctx, struct pci_devinst *pi, int bus, int slot, int func) write_config(&sc->psc_sel, PCIR_COMMAND, 2, pci_get_cfgdata16(pi, PCIR_COMMAND)); - error = 0; /* success */ + /* + * We need to do this after PCIR_COMMAND got possibly updated, e.g., + * a BAR was enabled, as otherwise the PCIOCBARMMAP might fail on us. + */ + error = init_msix_table(ctx, sc); + if (error != 0) { + warnx("failed to initialize MSI-X table for PCI %d/%d/%d: %d", + bus, slot, func, error); + goto done; + } + done: return (error); }