From nobody Mon Nov 03 23:17:43 2025 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d0nYh2DxNz6Fylg; Mon, 03 Nov 2025 23:17:44 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d0nYh1lp1z3hcB; Mon, 03 Nov 2025 23:17:44 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1762211864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zDq2D9GlDhFYCfGZ71pgZloKM8S+7zoHkvJyz8aVNS4=; b=jhtFfc5T1JVVpvlN1Pa8Vo3LmnpZvSSoqjMVmECV8AUuXfnpQai4uWkP/gP5yArK50vvmI y9TY7EuokB5Sp6Z8rgv6O5Dge5D9N8lD+F3y6rctq/UKNOXEeF4Ec2Vbfge0Drg4ycDPWf DLAcCikS23nbOFdceQSNiFgOryy5w37Xt19Kv+4EAPGRZtjGUjfpziM2dC/vgCM7HNCFWC soPq+v7w7rrAtvFW3PrManSjpdqSzfk3zxpc7OfNys5FdAGMp6Sn1eAWNcS9M5aNdrQWbR fN+ZHkPRMl6R5BgWmikJEKjgH4SIMajRMBd3y0d9oJxgfMUdx2PVo8lxiv1WXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1762211864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zDq2D9GlDhFYCfGZ71pgZloKM8S+7zoHkvJyz8aVNS4=; b=b22AfUVlclYB2Dnc6QYf0p9zdYmw2XYSfdUfCukU5TBPaUZASuoekAQZZ6t9YplmnEW3uB BtAssTSJvcCq5I1NxW7oLyPcQscNFv7F5qFTtLj7X4MR3Wz8Ry9amp0S+V5tLYk/m/TF11 VwyCnHREKFNHCKCwIFQ9I8okANfyeb0zTHOR9ZU+TLj8U5Xb6a0CTEkEgvuv3oW08jON+D Cke2tegHfmaszfIEOJV1SkErIefU9aGJiiqrHbvEvUXbGKAwQ54nii09mxm3avLBRBjvbn CdB5qG3WqTR/RgJDIVUXlxuP/VChebMHJacawbsuG5Z8PQ1HMiVxPe9VND4qMA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1762211864; a=rsa-sha256; cv=none; b=nshfX0SC8NXA35/zNEb6b4Tzf9Xl6B3hIq5Bj+5QBUMSw/EjrPigyUI+Ac4Gs6iXirjxZF lWublyqannPk7ugJTO54EPomnVyZONtzNTqOnNXw+Aq5boKF2xh69tLRoFu04B2eRyYkwv E6uY19xk9tdiqwwgckTYBjcwCbg3psdFIxHEAJIk8enxfij6zPAljVMnZqKElphOi0j9pC u/CwO7NsGj5FCByxZ1WikQq1fYOrWX5kh9q1qTe0Ad99RtI1hwnlofnFpdiMpX4lUhB5Vw iKaF0+j22N4ulKbMKmVv3dMOm0818E1DVvso6yDQKSvP9JogXRzStfEkfmqGrA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4d0nYh0xpyzXfC; Mon, 03 Nov 2025 23:17:44 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 5A3NHh6j070295; Mon, 3 Nov 2025 23:17:43 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 5A3NHhJR070292; Mon, 3 Nov 2025 23:17:43 GMT (envelope-from git) Date: Mon, 3 Nov 2025 23:17:43 GMT Message-Id: <202511032317.5A3NHhJR070292@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Alexander Ziaee Subject: git: 6c321dc15d52 - stable/15 - nvme: Add handling for bar5 List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ziaee X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: 6c321dc15d5234742c6452e975f41f4a24b52e1d Auto-Submitted: auto-generated The branch stable/15 has been updated by ziaee: URL: https://cgit.FreeBSD.org/src/commit/?id=6c321dc15d5234742c6452e975f41f4a24b52e1d commit 6c321dc15d5234742c6452e975f41f4a24b52e1d Author: Jasper Tran O'Leary AuthorDate: 2025-10-28 20:43:35 +0000 Commit: Alexander Ziaee CommitDate: 2025-11-03 22:04:30 +0000 nvme: Add handling for bar5 The NVMe spec allows the Table BIR (TBIR) and PBA DIR (PBIR) to be 0, 4, or 5. The existing NVMe driver basically only has support for 4, perhaps under the assumption that BAR4 is 64-bit and also occupies BAR5. This change adds support for BAR5, covering the case where BAR4 and BAR5 might both be present and 32-bit, where the Table BIR might be 4 and the PBA BIR might be 5, or vice versa. The NVMe spec (in the SR-IOV section) also permits VFs to use BIR=2, so I haven't added stricter checks on which BIR will be permitted by the driver. This enables FreeBSD on Google Compute Engine C4 Machines. MFC after: 3 days Reviewed by: imp Sponsored by: Google Co-authored-by: Matt Delco Signed-off-by: Jasper Tran O'Leary Differential Revision: https://reviews.freebsd.org/D53140 (cherry picked from commit 7b32f4f0a7fe9b1b2f5a3905ca15f656713255ad) --- sys/dev/nvme/nvme_ctrlr.c | 9 +++++++-- sys/dev/nvme/nvme_pci.c | 48 ++++++++++++++++++++++++++++++++++----------- sys/dev/nvme/nvme_private.h | 6 ++++-- 3 files changed, 48 insertions(+), 15 deletions(-) diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c index fc912c1342f4..6ed9cb33f991 100644 --- a/sys/dev/nvme/nvme_ctrlr.c +++ b/sys/dev/nvme/nvme_ctrlr.c @@ -1746,9 +1746,14 @@ noadminq: bus_release_resource(ctrlr->dev, SYS_RES_IRQ, rman_get_rid(ctrlr->res), ctrlr->res); - if (ctrlr->bar4_resource != NULL) { + if (ctrlr->msix_table_resource != NULL) { bus_release_resource(dev, SYS_RES_MEMORY, - ctrlr->bar4_resource_id, ctrlr->bar4_resource); + ctrlr->msix_table_resource_id, ctrlr->msix_table_resource); + } + + if (ctrlr->msix_pba_resource != NULL) { + bus_release_resource(dev, SYS_RES_MEMORY, + ctrlr->msix_pba_resource_id, ctrlr->msix_pba_resource); } bus_release_resource(dev, SYS_RES_MEMORY, diff --git a/sys/dev/nvme/nvme_pci.c b/sys/dev/nvme/nvme_pci.c index c07a68d2f0dc..cecb05ca0a92 100644 --- a/sys/dev/nvme/nvme_pci.c +++ b/sys/dev/nvme/nvme_pci.c @@ -152,11 +152,15 @@ static int nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) { ctrlr->resource_id = PCIR_BAR(0); + ctrlr->msix_table_resource_id = -1; + ctrlr->msix_table_resource = NULL; + ctrlr->msix_pba_resource_id = -1; + ctrlr->msix_pba_resource = NULL; ctrlr->resource = bus_alloc_resource_any(ctrlr->dev, SYS_RES_MEMORY, &ctrlr->resource_id, RF_ACTIVE); - if(ctrlr->resource == NULL) { + if (ctrlr->resource == NULL) { nvme_printf(ctrlr, "unable to allocate pci resource\n"); return (ENOMEM); } @@ -166,15 +170,32 @@ nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) ctrlr->regs = (struct nvme_registers *)ctrlr->bus_handle; /* - * The NVMe spec allows for the MSI-X table to be placed behind - * BAR 4/5, separate from the control/doorbell registers. Always - * try to map this bar, because it must be mapped prior to calling - * pci_alloc_msix(). If the table isn't behind BAR 4/5, - * bus_alloc_resource() will just return NULL which is OK. + * The NVMe spec allows for the MSI-X tables to be placed behind + * BAR 4 and/or 5, separate from the control/doorbell registers. */ - ctrlr->bar4_resource_id = PCIR_BAR(4); - ctrlr->bar4_resource = bus_alloc_resource_any(ctrlr->dev, SYS_RES_MEMORY, - &ctrlr->bar4_resource_id, RF_ACTIVE); + + ctrlr->msix_table_resource_id = pci_msix_table_bar(ctrlr->dev); + ctrlr->msix_pba_resource_id = pci_msix_pba_bar(ctrlr->dev); + + if (ctrlr->msix_table_resource_id >= 0 && + ctrlr->msix_table_resource_id != ctrlr->resource_id) { + ctrlr->msix_table_resource = bus_alloc_resource_any(ctrlr->dev, + SYS_RES_MEMORY, &ctrlr->msix_table_resource_id, RF_ACTIVE); + if (ctrlr->msix_table_resource == NULL) { + nvme_printf(ctrlr, "unable to allocate msi-x table resource\n"); + return (ENOMEM); + } + } + if (ctrlr->msix_pba_resource_id >= 0 && + ctrlr->msix_pba_resource_id != ctrlr->resource_id && + ctrlr->msix_pba_resource_id != ctrlr->msix_table_resource_id) { + ctrlr->msix_pba_resource = bus_alloc_resource_any(ctrlr->dev, + SYS_RES_MEMORY, &ctrlr->msix_pba_resource_id, RF_ACTIVE); + if (ctrlr->msix_pba_resource == NULL) { + nvme_printf(ctrlr, "unable to allocate msi-x pba resource\n"); + return (ENOMEM); + } + } return (0); } @@ -200,9 +221,14 @@ bad: ctrlr->resource_id, ctrlr->resource); } - if (ctrlr->bar4_resource != NULL) { + if (ctrlr->msix_table_resource != NULL) { + bus_release_resource(dev, SYS_RES_MEMORY, + ctrlr->msix_table_resource_id, ctrlr->msix_table_resource); + } + + if (ctrlr->msix_pba_resource != NULL) { bus_release_resource(dev, SYS_RES_MEMORY, - ctrlr->bar4_resource_id, ctrlr->bar4_resource); + ctrlr->msix_pba_resource_id, ctrlr->msix_pba_resource); } if (ctrlr->tag) diff --git a/sys/dev/nvme/nvme_private.h b/sys/dev/nvme/nvme_private.h index 52e9fcbbebcd..a425a6a5ad62 100644 --- a/sys/dev/nvme/nvme_private.h +++ b/sys/dev/nvme/nvme_private.h @@ -235,8 +235,10 @@ struct nvme_controller { * separate from the control registers which are in BAR 0/1. These * members track the mapping of BAR 4/5 for that reason. */ - int bar4_resource_id; - struct resource *bar4_resource; + int msix_table_resource_id; + struct resource *msix_table_resource; + int msix_pba_resource_id; + struct resource *msix_pba_resource; int msi_count; uint32_t enable_aborts;