From nobody Thu Feb 27 14:17:30 2025 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Z3YMH08Dvz5qG2Q; Thu, 27 Feb 2025 14:17:31 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Z3YMG6Pkzz3xQ2; Thu, 27 Feb 2025 14:17:30 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1740665850; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=paMZPKJ+jjnRZvy3hoV2aQDOQVhFcSF+4FXhsdFGXng=; b=tGUACg9KevAdDn1xQ/SquLwewiytdG5acglAv6BT/vugde5x+8cgMUl0vPgU83su73R7Ph a5pIVg1E6n40XeWJPtzcCQ1pYMkM6Y976iTBQttdC1lP169D5Bv1PSRJMU65pZKJ5gHDiK 76uAWee0lqr3m26E9tKLMtrR7rCArYXO1hOBYDV/0RgfQRZnGoBVtTiCAzxVx5uy1cLo28 511+kSgJlxbsIGAg0V1gC2P9xGIzwZh9ofIpiuI1a0JsGCoKTlB/wyZWpbOV1jYoUvcHy3 P8XQK5z4XkNEPItAxcDTgUZJRnv0LEzb28lUZrHDI8l1+n4/luEQvKTaxwk85g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1740665850; a=rsa-sha256; cv=none; b=Xr7S0c5x50xLBuMPX3v1tI19M5sxti6jF12wxkf9WXT2jTwdj2ROmYKHJmmkmdN/KcPD0s 9IKaJy+ys5kLIngLNuWSoSH4faDfUCazKdGjq6twvc1mnv0SJakleeXOn1whDkHMeUSXux YpmklBvdtTAuRE/8JdQ30cmPM5CkBm8OJF3JwmiyCA0O5safKbjfENu/xQgaPUZSVY7rrt LTe7qkxL88zcV4raKbNXCDU7dID95RqtdvLnGz1GeRxgQpzZFJN4ro8e1e4m2JfP78ZCpP JNkdmwmYWjN2MFjvwUewOZjpzV9tvbsBcoVv+SW3nTfr4YhwVQkg3WAvZJ7qFQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1740665850; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=paMZPKJ+jjnRZvy3hoV2aQDOQVhFcSF+4FXhsdFGXng=; b=LPco3nWq8M1YV6I2QbN8VfdiC67fisBAGayKS45kmJbGuiBKYH4oZ0go+g30HWPgOIYOBV 1LD3UZ1QeMG2elYKlojQLenA9QW1JhZHNYMS2WQr43zOJZNLt1GfO6XwSliyIOfvuvxygT LDxHSYgYgvamH0LUdhpuLzH9xDpcg+N8NUA1BJbwhzRRxQZUV+I0foV3OykOtto7iovBal 0ZukcAzQFNSiQO5+m2rHJgckZe+aEETbKyqGUfNSU+qQDzHzdFNGH+2hhTWxo7DzG+MS9C frL3ylahN+G3C/I4Vk/wiLPgXbM1gln/qh8Z/tcA6a4XaTL450vkQRF2RK5scQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Z3YMG5v6Rzmsd; Thu, 27 Feb 2025 14:17:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 51REHUBd017505; Thu, 27 Feb 2025 14:17:30 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 51REHUtI017502; Thu, 27 Feb 2025 14:17:30 GMT (envelope-from git) Date: Thu, 27 Feb 2025 14:17:30 GMT Message-Id: <202502271417.51REHUtI017502@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: John Baldwin Subject: git: 9f54ed946430 - stable/14 - pci: Consistently use pci_vf_* for suballocated VF memory resources List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 9f54ed9464308593f130f7b36bd4052e34af8c35 Auto-Submitted: auto-generated The branch stable/14 has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=9f54ed9464308593f130f7b36bd4052e34af8c35 commit 9f54ed9464308593f130f7b36bd4052e34af8c35 Author: John Baldwin AuthorDate: 2024-06-04 23:51:37 +0000 Commit: John Baldwin CommitDate: 2025-02-27 13:09:23 +0000 pci: Consistently use pci_vf_* for suballocated VF memory resources Some of the bus resource methods were passing these up to the parent which triggered rman mismatch assertions in INVARIANTS kernels. Reported by: kp Reviewed by: imp Tested by: kp (earlier version) Differential Revision: https://reviews.freebsd.org/D45406 (cherry picked from commit 871b33ad65baf07c92cce120a4fc1978c2ed7b3b) --- sys/dev/pci/pci.c | 121 ++++++++++++++++++++++++++++++++++-- sys/dev/pci/pci_iov.c | 152 ++++++++++++++++++++++++++++++++++++++++++++++ sys/dev/pci/pci_private.h | 19 ++++++ 3 files changed, 288 insertions(+), 4 deletions(-) diff --git a/sys/dev/pci/pci.c b/sys/dev/pci/pci.c index 2c00b44d941a..8ff31beca696 100644 --- a/sys/dev/pci/pci.c +++ b/sys/dev/pci/pci.c @@ -164,10 +164,18 @@ static device_method_t pci_methods[] = { DEVMETHOD(bus_get_resource, bus_generic_rl_get_resource), DEVMETHOD(bus_delete_resource, pci_delete_resource), DEVMETHOD(bus_alloc_resource, pci_alloc_resource), +#ifdef PCI_IOV + DEVMETHOD(bus_adjust_resource, pci_adjust_resource), +#else DEVMETHOD(bus_adjust_resource, bus_generic_adjust_resource), +#endif DEVMETHOD(bus_release_resource, pci_release_resource), DEVMETHOD(bus_activate_resource, pci_activate_resource), DEVMETHOD(bus_deactivate_resource, pci_deactivate_resource), +#ifdef PCI_IOV + DEVMETHOD(bus_map_resource, pci_map_resource), + DEVMETHOD(bus_unmap_resource, pci_unmap_resource), +#endif DEVMETHOD(bus_child_deleted, pci_child_deleted), DEVMETHOD(bus_child_detached, pci_child_detached), DEVMETHOD(bus_child_pnpinfo, pci_child_pnpinfo_method), @@ -5694,14 +5702,31 @@ pci_activate_resource(device_t dev, device_t child, int type, int rid, struct pci_devinfo *dinfo; int error; - error = bus_generic_activate_resource(dev, child, type, rid, r); + dinfo = device_get_ivars(child); +#ifdef PCI_IOV + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + error = EINVAL; + break; + case SYS_RES_MEMORY: + error = pci_vf_activate_mem_resource(dev, child, r); + break; + default: + error = bus_generic_activate_resource(dev, child, type, + rid, r); + break; + } + } else +#endif + error = bus_generic_activate_resource(dev, child, type, rid, r); if (error) return (error); /* Enable decoding in the command register when activating BARs. */ if (device_get_parent(child) == dev) { /* Device ROMs need their decoding explicitly enabled. */ - dinfo = device_get_ivars(child); if (type == SYS_RES_MEMORY && PCIR_IS_BIOS(&dinfo->cfg, rid)) pci_write_bar(child, pci_find_bar(child, rid), rman_get_start(r) | PCIM_BIOS_ENABLE); @@ -5722,13 +5747,31 @@ pci_deactivate_resource(device_t dev, device_t child, int type, struct pci_devinfo *dinfo; int error; - error = bus_generic_deactivate_resource(dev, child, type, rid, r); + dinfo = device_get_ivars(child); +#ifdef PCI_IOV + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + error = EINVAL; + break; + case SYS_RES_MEMORY: + error = pci_vf_deactivate_mem_resource(dev, child, r); + break; + default: + error = bus_generic_deactivate_resource(dev, child, + type, rid, r); + break; + } + } else +#endif + error = bus_generic_deactivate_resource(dev, child, type, rid, + r); if (error) return (error); /* Disable decoding for device ROMs. */ if (device_get_parent(child) == dev) { - dinfo = device_get_ivars(child); if (type == SYS_RES_MEMORY && PCIR_IS_BIOS(&dinfo->cfg, rid)) pci_write_bar(child, pci_find_bar(child, rid), rman_get_start(r)); @@ -5736,6 +5779,76 @@ pci_deactivate_resource(device_t dev, device_t child, int type, return (0); } +#ifdef PCI_IOV +int +pci_adjust_resource(device_t dev, device_t child, int type, struct resource *r, + rman_res_t start, rman_res_t end) +{ + struct pci_devinfo *dinfo; + + dinfo = device_get_ivars(child); + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + return (EINVAL); + case SYS_RES_MEMORY: + return (pci_vf_adjust_mem_resource(dev, child, r, + start, end)); + } + + /* Fall through for other types of resource allocations. */ + } + + return (bus_generic_adjust_resource(dev, child, type, r, start, end)); +} + +int +pci_map_resource(device_t dev, device_t child, int type, struct resource *r, + struct resource_map_request *argsp, struct resource_map *map) +{ + struct pci_devinfo *dinfo; + + dinfo = device_get_ivars(child); + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + return (EINVAL); + case SYS_RES_MEMORY: + return (pci_vf_map_mem_resource(dev, child, r, argsp, + map)); + } + + /* Fall through for other types of resource allocations. */ + } + + return (bus_generic_map_resource(dev, child, type, r, argsp, map)); +} + +int +pci_unmap_resource(device_t dev, device_t child, int type, struct resource *r, + struct resource_map *map) +{ + struct pci_devinfo *dinfo; + + dinfo = device_get_ivars(child); + if (dinfo->cfg.flags & PCICFG_VF) { + switch (rman_get_type(r)) { + /* VFs can't have I/O BARs. */ + case SYS_RES_IOPORT: + return (EINVAL); + case SYS_RES_MEMORY: + return (pci_vf_unmap_mem_resource(dev, child, r, map)); + } + + /* Fall through for other types of resource allocations. */ + } + + return (bus_generic_unmap_resource(dev, child, type, r, map)); +} +#endif + void pci_child_deleted(device_t dev, device_t child) { diff --git a/sys/dev/pci/pci_iov.c b/sys/dev/pci/pci_iov.c index ee1387966b00..b9e22fafe103 100644 --- a/sys/dev/pci/pci_iov.c +++ b/sys/dev/pci/pci_iov.c @@ -1076,6 +1076,12 @@ pci_vf_release_mem_resource(device_t dev, device_t child, int rid, dinfo = device_get_ivars(child); + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &dinfo->cfg.iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + if (rman_get_flags(r) & RF_ACTIVE) { error = bus_deactivate_resource(child, SYS_RES_MEMORY, rid, r); if (error != 0) @@ -1091,3 +1097,149 @@ pci_vf_release_mem_resource(device_t dev, device_t child, int rid, return (rman_release_resource(r)); } + +int +pci_vf_activate_mem_resource(device_t dev, device_t child, struct resource *r) +{ +#ifdef INVARIANTS + struct pci_devinfo *dinfo = device_get_ivars(child); +#endif + struct resource_map map; + int error; + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &dinfo->cfg.iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + error = rman_activate_resource(r); + if (error != 0) + return (error); + + if ((rman_get_flags(r) & RF_UNMAPPED) == 0) { + error = BUS_MAP_RESOURCE(dev, child, SYS_RES_MEMORY, r, NULL, + &map); + if (error != 0) { + rman_deactivate_resource(r); + return (error); + } + + rman_set_mapping(r, &map); + } + return (0); +} + +int +pci_vf_deactivate_mem_resource(device_t dev, device_t child, struct resource *r) +{ +#ifdef INVARIANTS + struct pci_devinfo *dinfo = device_get_ivars(child); +#endif + struct resource_map map; + int error; + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &dinfo->cfg.iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + error = rman_deactivate_resource(r); + if (error != 0) + return (error); + + if ((rman_get_flags(r) & RF_UNMAPPED) == 0) { + rman_get_mapping(r, &map); + BUS_UNMAP_RESOURCE(dev, child, SYS_RES_MEMORY, r, &map); + } + return (0); +} + +int +pci_vf_adjust_mem_resource(device_t dev, device_t child, struct resource *r, + rman_res_t start, rman_res_t end) +{ +#ifdef INVARIANTS + struct pci_devinfo *dinfo = device_get_ivars(child); +#endif + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &dinfo->cfg.iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + return (rman_adjust_resource(r, start, end)); +} + +static struct resource * +pci_vf_find_parent_resource(struct pcicfg_iov *iov, struct resource *r) +{ + struct resource *pres; + + for (u_int i = 0; i <= PCIR_MAX_BAR_0; i++) { + pres = iov->iov_bar[i].res; + if (pres != NULL) { + if (rman_get_start(pres) <= rman_get_start(r) && + rman_get_end(pres) >= rman_get_end(r)) + return (pres); + } + } + return (NULL); +} + +int +pci_vf_map_mem_resource(device_t dev, device_t child, struct resource *r, + struct resource_map_request *argsp, struct resource_map *map) +{ + struct pci_devinfo *dinfo = device_get_ivars(child); + struct pcicfg_iov *iov = dinfo->cfg.iov; + struct resource_map_request args; + struct resource *pres; + rman_res_t length, start; + int error; + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + /* Resources must be active to be mapped. */ + if (!(rman_get_flags(r) & RF_ACTIVE)) + return (ENXIO); + + resource_init_map_request(&args); + error = resource_validate_map_request(r, argsp, &args, &start, &length); + if (error) + return (error); + + pres = pci_vf_find_parent_resource(dinfo->cfg.iov, r); + if (pres == NULL) + return (ENOENT); + + args.offset = start - rman_get_start(pres); + args.length = length; + return (bus_map_resource(iov->iov_pf, pres, &args, map)); +} + +int +pci_vf_unmap_mem_resource(device_t dev, device_t child, struct resource *r, + struct resource_map *map) +{ + struct pci_devinfo *dinfo = device_get_ivars(child); + struct pcicfg_iov *iov = dinfo->cfg.iov; + struct resource *pres; + + KASSERT(rman_get_type(r) == SYS_RES_MEMORY, + ("%s: invalid resource %p", __func__, r)); + KASSERT(rman_is_region_manager(r, &iov->rman), + ("%s: rman %p doesn't match for resource %p", __func__, + &dinfo->cfg.iov->rman, r)); + + pres = pci_vf_find_parent_resource(iov, r); + if (pres == NULL) + return (ENOENT); + return (bus_unmap_resource(iov->iov_pf, pres, map)); +} diff --git a/sys/dev/pci/pci_private.h b/sys/dev/pci/pci_private.h index ea0efc54c829..d3dd41abdb62 100644 --- a/sys/dev/pci/pci_private.h +++ b/sys/dev/pci/pci_private.h @@ -65,9 +65,16 @@ bus_get_dma_tag_t pci_get_dma_tag; bus_get_resource_list_t pci_get_resource_list; bus_delete_resource_t pci_delete_resource; bus_alloc_resource_t pci_alloc_resource; +#ifdef PCI_IOV +bus_adjust_resource_t pci_adjust_resource; +#endif bus_release_resource_t pci_release_resource; bus_activate_resource_t pci_activate_resource; bus_deactivate_resource_t pci_deactivate_resource; +#ifdef PCI_IOV +bus_map_resource_t pci_map_resource; +bus_unmap_resource_t pci_unmap_resource; +#endif bus_child_deleted_t pci_child_deleted; bus_child_detached_t pci_child_detached; bus_child_pnpinfo_t pci_child_pnpinfo_method; @@ -158,4 +165,16 @@ struct resource *pci_vf_alloc_mem_resource(device_t dev, device_t child, rman_res_t count, u_int flags); int pci_vf_release_mem_resource(device_t dev, device_t child, int rid, struct resource *r); +int pci_vf_activate_mem_resource(device_t dev, device_t child, + struct resource *r); +int pci_vf_deactivate_mem_resource(device_t dev, device_t child, + struct resource *r); +int pci_vf_adjust_mem_resource(device_t dev, device_t child, + struct resource *r, rman_res_t start, rman_res_t end); +int pci_vf_map_mem_resource(device_t dev, device_t child, + struct resource *r, struct resource_map_request *argsp, + struct resource_map *map); +int pci_vf_unmap_mem_resource(device_t dev, device_t child, + struct resource *r, struct resource_map *map); + #endif /* _PCI_PRIVATE_H_ */