git: 7ea16192a01e - stable/13 - bhyve: passthru: enable BARs before possibly mmap(2)ing them

The branch stable/13 has been updated by bz:

URL: https://cgit.FreeBSD.org/src/commit/?id=7ea16192a01ea3de7b088ef6dc3cbf29d8e21a54

commit 7ea16192a01ea3de7b088ef6dc3cbf29d8e21a54
Author:     Bjoern A. Zeeb <bz@FreeBSD.org>
AuthorDate: 2021-12-23 14:59:49 +0000
Commit:     Bjoern A. Zeeb <bz@FreeBSD.org>
CommitDate: 2022-01-09 02:38:58 +0000

    bhyve: passthru: enable BARs before possibly mmap(2)ing them
    
    The first time we start bhyve with a passthru device everything is fine
    as on boot we do enable BARs.  If a driver (unload) inside bhyve disables
    the BAR(s) as some Linux drivers do, we need to make sure we re-enable
    them on next bhyve start.
    
    If we are trying to mmap a disabled BAR for MSI-X (PCIOCBARMMAP)
    the kernel will give us an EBUSY.
    While we were re-enabling the BAR(s) in the current code loop
    cfginit() was writing the changes out too late to the real hardware.
    
    Move the call to init_msix_table() after the register on the real
    hardware was updated.  That way the kernel will be happy and the
    mmap will succeed and bhyve will start.
    Also simplify the code given the last argument to init_msix_table()
    is unused we do not need to do checks for each bar. [1]
    
    Some passthru devices only support MSI instead of MSI-X. For those
    devices the initialization of MSI-X table will fail. Re-add (or in
    the MFC case keep) the check erroneously removed in the initial commit. [2]
    
    PR:             260148
    Pointed out by: markj [1]
    Sponsored by:   The FreeBSD Foundation
    Reviewed by:    markj
    Differential Revision: https://reviews.freebsd.org/D33628
    Submitted by:   (C.Koehne beckhoff.com) [2]
    Reviewed by:    manu, bz
    Differential Revision:  https://reviews.freebsd.org/D33728
    
    (cherry picked from commit f1442847c9404d4bc5f5524a0c3362dd39cb14f9)
    (cherry picked from commit 338a1be836308f6d807f8bfe9b335463d537abc4)
---
 usr.sbin/bhyve/pci_passthru.c | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/usr.sbin/bhyve/pci_passthru.c b/usr.sbin/bhyve/pci_passthru.c
index 2e274829d783..436091cbe8ee 100644
--- a/usr.sbin/bhyve/pci_passthru.c
+++ b/usr.sbin/bhyve/pci_passthru.c
@@ -420,7 +420,7 @@ msix_table_write(struct vmctx *ctx, int vcpu, struct passthru_softc *sc,
 }
 
 static int
-init_msix_table(struct vmctx *ctx, struct passthru_softc *sc, uint64_t base)
+init_msix_table(struct vmctx *ctx, struct passthru_softc *sc)
 {
 	struct pci_devinst *pi = sc->psc_pi;
 	struct pci_bar_mmap pbm;
@@ -538,13 +538,6 @@ cfginitbar(struct vmctx *ctx, struct passthru_softc *sc)
 		if (error)
 			return (-1);
 
-		/* The MSI-X table needs special handling */
-		if (i == pci_msix_table_bar(pi)) {
-			error = init_msix_table(ctx, sc, base);
-			if (error) 
-				return (-1);
-		}
-
 		/*
 		 * 64-bit BAR takes up two slots so skip the next one.
 		 */
@@ -586,6 +579,20 @@ cfginit(struct vmctx *ctx, struct pci_devinst *pi, int bus, int slot, int func)
 	pci_set_cfgdata16(pi, PCIR_COMMAND, read_config(&sc->psc_sel,
 	    PCIR_COMMAND, 2));
 
+	/*
+	 * We need to do this after PCIR_COMMAND got possibly updated, e.g.,
+	 * a BAR was enabled, as otherwise the PCIOCBARMMAP might fail on us.
+	 */
+	if (pci_msix_table_bar(pi) >= 0) {
+		error = init_msix_table(ctx, sc);
+		if (error != 0) {
+			warnx(
+			    "failed to initialize MSI-X table for PCI %d/%d/%d: %d",
+			    bus, slot, func, error);
+			goto done;
+		}
+	}
+
 	error = 0;				/* success */
 done:
 	return (error);