From nobody Tue Jun 04 23:52:15 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vv6p75v4Zz5N07r; Tue, 04 Jun 2024 23:52:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Vv6p71QwBz4K8R; Tue, 4 Jun 2024 23:52:15 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1717545135; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=+mi29rZzINty6FUmV6hUH/rzbPkjPE+XCCGD5emOfCI=; b=anAzfiAS3ihf4AWTIv5D1emrG1h6aZlUw1XWeNdeGKz2ETFYoLegp5xtheZu/pxzM4O9Lm uJiTjeKP2nc+ruqWNUGkb94u75RyiMHi6xoUkpvV4gb3Lyzi0IiVWeBVJriSY3U757Yizu L03PJkkWNuuSmapnGufwTk8qA4ThDgsE2zH9QMAIdYmExt/1eBF56+ZoLVTb0gR9UpZCY1 4LEc3+pKxWmxzVAYXVAivazBmiLCIgP0HL5goe50KLP5ZUNBi8FhOJwKONydX7QDtWib4v KJKQplc36H+A/Qci8t5A0N+mSTtG0InkYtfXT0Wy0pOGtat0eWpkF2NlpluOCA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1717545135; a=rsa-sha256; cv=none; b=r4pbT/YL8I6wM9O2jg/VlW2nZCAQYKfjX2t81XXcd3fZNe3f4Qiny0iqrgaREmzgyviGb9 jRCsbnDSOJi54C7nr8G9/MA8rQ4zjHV3CxM0hbrZo8IzHFiI+wWrCDOIpfoWTxTSY2scay PMtbKtdqTyAEkSk8lKCHYWY0X0YdRg5KlT4nKB6JZpVGsIApo/T7/e3O5qiZrNs8fWbIxX u6o/jNbuOXb+QVDwUhhpX77TiinN1/g16rOB8EbinsXfeAz+7Q5GxOO0iaiyIMgcC/ZY05 Kfy8gAfHnL+0fUqgaYA3tf1mHwXffg1Ua8enHCbT7RKfPTUR+jRTWQPkq567gA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1717545135; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=+mi29rZzINty6FUmV6hUH/rzbPkjPE+XCCGD5emOfCI=; b=AxiALfY6gO37fIC973yPo3CFx4BINP8QKhs9npins+1H9s5F/w9zzP321Nh3gPiUdxzbJG qwqR3DrL4lXlULUncNSPaQn7MnEDdtX4D45k+WJ3PaeK0+LZb59z5F2U4Nmm5eSiIbWEpq yytqZsqhH2ybB5t24Q7LlQ4tRqAfnNZR1avqZiSqTh56Qc5b6ACkp+6mLG0vSzoAWOIigl VGGyvtxAnt+RkT20ogK2l4xvqUTFzIlpAaseG4XF7N4ZIJPYmZlBURUGbAqmNpjE8BR3bj y62zLt9K3tLDGhW9W1mY57T5hs5zpr0KbWCQF3Wfxdpov4C+yzNML00fhk7VOw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Vv6p712QFzffH; Tue, 4 Jun 2024 23:52:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 454NqFQt061331; Tue, 4 Jun 2024 23:52:15 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 454NqFVj061328; Tue, 4 Jun 2024 23:52:15 GMT (envelope-from git) Date: Tue, 4 Jun 2024 23:52:15 GMT Message-Id: <202406042352.454NqFVj061328@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 871b33ad65ba - main - pci: Consistently use pci_vf_* for suballocated VF memory resources List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 871b33ad65baf07c92cce120a4fc1978c2ed7b3b Auto-Submitted: auto-generated The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=871b33ad65baf07c92cce120a4fc1978c2ed7b3b commit 871b33ad65baf07c92cce120a4fc1978c2ed7b3b Author: John Baldwin AuthorDate: 2024-06-04 23:51:37 +0000 Commit: John Baldwin CommitDate: 2024-06-04 23:51:37 +0000 pci: Consistently use pci_vf_* for suballocated VF memory resources Some of the bus resource methods were passing these up to the parent which triggered rman mismatch assertions in INVARIANTS kernels. Reported by: kp Reviewed by: imp Tested by: kp (earlier version) Differential Revision: https://reviews.freebsd.org/D45406 --- sys/dev/pci/pci.c | 118 ++++++++++++++++++++++++++++++++++-- sys/dev/pci/pci_iov.c | 151 ++++++++++++++++++++++++++++++++++++++++++++++ sys/dev/pci/pci_private.h | 19 ++++++ 3 files changed, 284 insertions(+), 4 deletions(-) diff --git a/sys/dev/pci/pci.c b/sys/dev/pci/pci.c index 2cb8b4ce9fcc..2093d6a8b5ef 100644 --- a/sys/dev/pci/pci.c +++ b/sys/dev/pci/pci.c @@ -164,10 +164,18 @@ static device_method_t pci_methods[] = { DEVMETHOD(bus_get_resource, bus_generic_rl_get_resource), DEVMETHOD(bus_delete_resource, pci_delete_resource), DEVMETHOD(bus_alloc_resource, pci_alloc_resource), +#ifdef PCI_IOV + DEVMETHOD(bus_adjust_resource, pci_adjust_resource), +#else DEVMETHOD(bus_adjust_resource, bus_generic_adjust_resource), +#endif DEVMETHOD(bus_release_resource, pci_release_resource), DEVMETHOD(bus_activate_resource, pci_activate_resource), DEVMETHOD(bus_deactivate_resource, pci_deactivate_resource), +#ifdef PCI_IOV + DEVMETHOD(bus_map_resource, pci_map_resource), + DEVMETHOD(bus_unmap_resource, pci_unmap_resource), +#endif DEVMETHOD(bus_child_deleted, pci_child_deleted), DEVMETHOD(bus_child_detached, pci_child_detached), DEVMETHOD(bus_child_pnpinfo, pci_child_pnpinfo_method), @@ -5687,14 +5695,30 @@ pci_activate_resource(device_t dev, device_t child, struct resource *r) struct pci_devinfo *dinfo; int error, rid, type; - error = bus_generic_activate_resource(dev, child, r); + dinfo = device_get_ivars(child); +#ifdef PCI_IOV + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + error = EINVAL; + break; + case SYS_RES_MEMORY: + error = pci_vf_activate_mem_resource(dev, child, r); + break; + default: + error = bus_generic_activate_resource(dev, child, r); + break; + } + } else +#endif + error = bus_generic_activate_resource(dev, child, r); if (error) return (error); /* Enable decoding in the command register when activating BARs. */ if (device_get_parent(child) == dev) { /* Device ROMs need their decoding explicitly enabled. */ - dinfo = device_get_ivars(child); rid = rman_get_rid(r); type = rman_get_type(r); if (type == SYS_RES_MEMORY && PCIR_IS_BIOS(&dinfo->cfg, rid)) @@ -5716,13 +5740,29 @@ pci_deactivate_resource(device_t dev, device_t child, struct resource *r) struct pci_devinfo *dinfo; int error, rid, type; - error = bus_generic_deactivate_resource(dev, child, r); + dinfo = device_get_ivars(child); +#ifdef PCI_IOV + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + error = EINVAL; + break; + case SYS_RES_MEMORY: + error = pci_vf_deactivate_mem_resource(dev, child, r); + break; + default: + error = bus_generic_deactivate_resource(dev, child, r); + break; + } + } else +#endif + error = bus_generic_deactivate_resource(dev, child, r); if (error) return (error); /* Disable decoding for device ROMs. */ if (device_get_parent(child) == dev) { - dinfo = device_get_ivars(child); rid = rman_get_rid(r); type = rman_get_type(r); if (type == SYS_RES_MEMORY && PCIR_IS_BIOS(&dinfo->cfg, rid)) @@ -5732,6 +5772,76 @@ pci_deactivate_resource(device_t dev, device_t child, struct resource *r) return (0); } +#ifdef PCI_IOV +int +pci_adjust_resource(device_t dev, device_t child, struct resource *r, + rman_res_t start, rman_res_t end) +{ + struct pci_devinfo *dinfo; + + dinfo = device_get_ivars(child); + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + return (EINVAL); + case SYS_RES_MEMORY: + return (pci_vf_adjust_mem_resource(dev, child, r, + start, end)); + } + + /* Fall through for other types of resource allocations. */ + } + + return (bus_generic_adjust_resource(dev, child, r, start, end)); +} + +int +pci_map_resource(device_t dev, device_t child, struct resource *r, + struct resource_map_request *argsp, struct resource_map *map) +{ + struct pci_devinfo *dinfo; + + dinfo = device_get_ivars(child); + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + return (EINVAL); + case SYS_RES_MEMORY: + return (pci_vf_map_mem_resource(dev, child, r, argsp, + map)); + } + + /* Fall through for other types of resource allocations. */ + } + + return (bus_generic_map_resource(dev, child, r, argsp, map)); +} + +int +pci_unmap_resource(device_t dev, device_t child, struct resource *r, + struct resource_map *map) +{ + struct pci_devinfo *dinfo; + + dinfo = device_get_ivars(child); + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + return (EINVAL); + case SYS_RES_MEMORY: + return (pci_vf_unmap_mem_resource(dev, child, r, map)); + } + + /* Fall through for other types of resource allocations. */ + } + + return (bus_generic_unmap_resource(dev, child, r, map)); +} +#endif + void pci_child_deleted(device_t dev, device_t child) { diff --git a/sys/dev/pci/pci_iov.c b/sys/dev/pci/pci_iov.c index c8e139f043c9..1f1138b2d336 100644 --- a/sys/dev/pci/pci_iov.c +++ b/sys/dev/pci/pci_iov.c @@ -1070,6 +1070,12 @@ pci_vf_release_mem_resource(device_t dev, device_t child, struct resource *r) dinfo = device_get_ivars(child); + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &dinfo->cfg.iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + if (rman_get_flags(r) & RF_ACTIVE) { error = bus_deactivate_resource(child, r); if (error != 0) @@ -1086,3 +1092,148 @@ pci_vf_release_mem_resource(device_t dev, device_t child, struct resource *r) return (rman_release_resource(r)); } + +int +pci_vf_activate_mem_resource(device_t dev, device_t child, struct resource *r) +{ +#ifdef INVARIANTS + struct pci_devinfo *dinfo = device_get_ivars(child); +#endif + struct resource_map map; + int error; + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &dinfo->cfg.iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + error = rman_activate_resource(r); + if (error != 0) + return (error); + + if ((rman_get_flags(r) & RF_UNMAPPED) == 0) { + error = BUS_MAP_RESOURCE(dev, child, r, NULL, &map); + if (error != 0) { + rman_deactivate_resource(r); + return (error); + } + + rman_set_mapping(r, &map); + } + return (0); +} + +int +pci_vf_deactivate_mem_resource(device_t dev, device_t child, struct resource *r) +{ +#ifdef INVARIANTS + struct pci_devinfo *dinfo = device_get_ivars(child); +#endif + struct resource_map map; + int error; + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &dinfo->cfg.iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + error = rman_deactivate_resource(r); + if (error != 0) + return (error); + + if ((rman_get_flags(r) & RF_UNMAPPED) == 0) { + rman_get_mapping(r, &map); + BUS_UNMAP_RESOURCE(dev, child, r, &map); + } + return (0); +} + +int +pci_vf_adjust_mem_resource(device_t dev, device_t child, struct resource *r, + rman_res_t start, rman_res_t end) +{ +#ifdef INVARIANTS + struct pci_devinfo *dinfo = device_get_ivars(child); +#endif + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &dinfo->cfg.iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + return (rman_adjust_resource(r, start, end)); +} + +static struct resource * +pci_vf_find_parent_resource(struct pcicfg_iov *iov, struct resource *r) +{ + struct resource *pres; + + for (u_int i = 0; i <= PCIR_MAX_BAR_0; i++) { + pres = iov->iov_bar[i].res; + if (pres != NULL) { + if (rman_get_start(pres) <= rman_get_start(r) && + rman_get_end(pres) >= rman_get_end(r)) + return (pres); + } + } + return (NULL); +} + +int +pci_vf_map_mem_resource(device_t dev, device_t child, struct resource *r, + struct resource_map_request *argsp, struct resource_map *map) +{ + struct pci_devinfo *dinfo = device_get_ivars(child); + struct pcicfg_iov *iov = dinfo->cfg.iov; + struct resource_map_request args; + struct resource *pres; + rman_res_t length, start; + int error; + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + /* Resources must be active to be mapped. */ + if (!(rman_get_flags(r) & RF_ACTIVE)) + return (ENXIO); + + resource_init_map_request(&args); + error = resource_validate_map_request(r, argsp, &args, &start, &length); + if (error) + return (error); + + pres = pci_vf_find_parent_resource(dinfo->cfg.iov, r); + if (pres == NULL) + return (ENOENT); + + args.offset = start - rman_get_start(pres); + args.length = length; + return (bus_map_resource(iov->iov_pf, pres, &args, map)); +} + +int +pci_vf_unmap_mem_resource(device_t dev, device_t child, struct resource *r, + struct resource_map *map) +{ + struct pci_devinfo *dinfo = device_get_ivars(child); + struct pcicfg_iov *iov = dinfo->cfg.iov; + struct resource *pres; + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + pres = pci_vf_find_parent_resource(iov, r); + if (pres == NULL) + return (ENOENT); + return (bus_unmap_resource(iov->iov_pf, pres, map)); +} diff --git a/sys/dev/pci/pci_private.h b/sys/dev/pci/pci_private.h index f97a4df5471b..6dc7e3c505d1 100644 --- a/sys/dev/pci/pci_private.h +++ b/sys/dev/pci/pci_private.h @@ -65,9 +65,16 @@ bus_get_dma_tag_t pci_get_dma_tag; bus_get_resource_list_t pci_get_resource_list; bus_delete_resource_t pci_delete_resource; bus_alloc_resource_t pci_alloc_resource; +#ifdef PCI_IOV +bus_adjust_resource_t pci_adjust_resource; +#endif bus_release_resource_t pci_release_resource; bus_activate_resource_t pci_activate_resource; bus_deactivate_resource_t pci_deactivate_resource; +#ifdef PCI_IOV +bus_map_resource_t pci_map_resource; +bus_unmap_resource_t pci_unmap_resource; +#endif bus_child_deleted_t pci_child_deleted; bus_child_detached_t pci_child_detached; bus_child_pnpinfo_t pci_child_pnpinfo_method; @@ -158,4 +165,16 @@ struct resource *pci_vf_alloc_mem_resource(device_t dev, device_t child, rman_res_t count, u_int flags); int pci_vf_release_mem_resource(device_t dev, device_t child, struct resource *r); +int pci_vf_activate_mem_resource(device_t dev, device_t child, + struct resource *r); +int pci_vf_deactivate_mem_resource(device_t dev, device_t child, + struct resource *r); +int pci_vf_adjust_mem_resource(device_t dev, device_t child, + struct resource *r, rman_res_t start, rman_res_t end); +int pci_vf_map_mem_resource(device_t dev, device_t child, + struct resource *r, struct resource_map_request *argsp, + struct resource_map *map); +int pci_vf_unmap_mem_resource(device_t dev, device_t child, + struct resource *r, struct resource_map *map); + #endif /* _PCI_PRIVATE_H_ */