From nobody Mon Mar 09 13:39:57 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fTymt5xKtz6VlYP; Mon, 09 Mar 2026 13:39:58 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fTymt5Nw7z46GT; Mon, 09 Mar 2026 13:39:58 +0000 (UTC) (envelope-from jhb@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1773063598; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s8KUlrtztL3NBWNOzi4/qWI//tRvZIDQZr+gpp62EWU=; b=Pk0zHJ233HC9nYOGZJANjSVgNPQScNXDhP68r3+8rk0uoqKqisL0JBDJKxncI0NAhyMVmo st9DBuYsPWlghk7GOqSTR4HmlzbE9umgZpJMTUcoGNRt2jejhvG25vT1NdiOmD0sAfTFxX XkGXdgljKPAfk99E69q0hVYDhcuG84KBVj7nsRMyotGFSThYWDj35wjpM4m5V1z5kFB4wJ IqwJQpf3A44LFubShkWwNYZxrE79dxsKUrv/4VT5uzp3oMYJMQZ56GCXwFjdJ8iZeG5zx1 9rEZYLip/VtjrZ3rtGwY2GD1ZzHB4YPKZnBRzMrfZXkf83tmx8eJeCu+X2IddA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1773063598; a=rsa-sha256; cv=none; b=coxqgOgXgN7fdEgGZKY+dQgQE9x5KyL49gwthlf2XDH1afl10HP2TKC6zrgfvoqSyVBPts sjfjF7i2z9EGkIoaQrZnJZTGgTjGvMRdAhkn6Wsnb+BL25NxMtbUiEG9ydQ30uguFpXwOv pNcn611LEU1F7mhL3KesFzVBbtBubn6XKevjuoj9dhM6gln/6B9tz5FMZ4EbNDxumWibqD PWgt9+LezxtIZ3GMjikFoq0dlinJIlfjp+wFLW2qFuKAhbEzJ7ymU7irr6Yvw/17fysU0t D1HR62rxiBhCjTI3h2Qx4dX9hXHsiFD4YLIhGl7MVopMuy1wdiTYOL7ab7p6bg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1773063598; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s8KUlrtztL3NBWNOzi4/qWI//tRvZIDQZr+gpp62EWU=; b=NaE+jaXc7sZBNrUspRNzopRQCZlLQrakY9mUzOU05aNjWR5IGGgASR9ihBo96mt6hpecd7 BaoJ2y1zusC0AfXUsa+WN9PzZw1opjJoer/lGQ3II7Eep1D6+1yT3Etb6VEd93Ja1znIVI EdtGPKko9dCJigRzt+E3MCKukscDC1bAVok7PwG6yAx5XSKF3d8o6thmgv4/f6jSwPWr3+ H0bLIQhGNT82afdxnSEVDY5m7Yzqw66rlIjCqgMgjLUZFWKBX/ZqOhpHzphhAN5mlUlQsa DhyJ5tdUGhC0MmzbuT8TEF3Jk8/n7vTYhsC2R2OczPFmehcS66zXlhtJZBUVAQ== Received: from [IPV6:2601:5c0:4202:5670:5853:86ec:97b9:344a] (unknown [IPv6:2601:5c0:4202:5670:5853:86ec:97b9:344a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 4fTymt39krzLVX; Mon, 09 Mar 2026 13:39:58 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Message-ID: Date: Mon, 9 Mar 2026 09:39:57 -0400 List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: git: b3d9e5013f3e - main - nvme: Don't active memory space until all BARs are configured Content-Language: en-US To: Alexander Ziaee , src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Cc: Matt Delco References: <69ab0f9b.18db4.5196b3f7@gitrepo.freebsd.org> From: John Baldwin In-Reply-To: <69ab0f9b.18db4.5196b3f7@gitrepo.freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 3/6/26 12:32, Alexander Ziaee wrote: > The branch main has been updated by ziaee: > > URL: https://cgit.FreeBSD.org/src/commit/?id=b3d9e5013f3e5016ffbd3d3d6091194658af2b92 > > commit b3d9e5013f3e5016ffbd3d3d6091194658af2b92 > Author: Matt Delco > AuthorDate: 2026-03-06 17:23:03 +0000 > Commit: Alexander Ziaee > CommitDate: 2026-03-06 17:28:41 +0000 > > nvme: Don't active memory space until all BARs are configured > > In the current current behavior the 2nd and 3rd BARs can be activated > when they're configured with address zero. This change defers the > activation of all BARs until after they've all been configured with an > address. > > This enables FreeBSD on Google Compute Engine C4-LSSD Machines. This hack is fine for now, but the real fix for this is that the PCI bus driver needs to ensure resources are reserved for all of the MEM or IO BARs before enabling MEM or IO decoding at which point this can (and should) be reverted. That's kind of a longstanding bug in the PCI bus driver that we haven't had to solve previously because the firmware in most systems assigns BARs during POST so that in practice we never have to deal with unreserved BARs. > Sponsored by: Google > Tested by: NetApp (previous version) > Reviewed by: gallatin, imp > Discussed with: jrtc27 (improved error reporting) > Differential Revision: https://reviews.freebsd.org/D55541 > --- > sys/dev/nvme/nvme_pci.c | 44 +++++++++++++++++++++++++++++++++++++------- > 1 file changed, 37 insertions(+), 7 deletions(-) > > diff --git a/sys/dev/nvme/nvme_pci.c b/sys/dev/nvme/nvme_pci.c > index 5784c6d1be96..74191df52058 100644 > --- a/sys/dev/nvme/nvme_pci.c > +++ b/sys/dev/nvme/nvme_pci.c > @@ -151,24 +151,28 @@ nvme_pci_probe (device_t device) > static int > nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) > { > + int error; > + > ctrlr->resource_id = PCIR_BAR(0); > ctrlr->msix_table_resource_id = -1; > ctrlr->msix_table_resource = NULL; > ctrlr->msix_pba_resource_id = -1; > ctrlr->msix_pba_resource = NULL; > > + /* > + * Using RF_ACTIVE will set the Memory Space bit in the PCI command register. > + * The remaining BARs will get mapped in before they've been programmed with > + * an address. To avoid this we'll not set this flag and instead call > + * bus_activate_resource() after all the BARs have been programmed. > + */ > ctrlr->resource = bus_alloc_resource_any(ctrlr->dev, SYS_RES_MEMORY, > - &ctrlr->resource_id, RF_ACTIVE); > + &ctrlr->resource_id, 0); > > if (ctrlr->resource == NULL) { > nvme_printf(ctrlr, "unable to allocate pci resource\n"); > return (ENOMEM); > } > > - ctrlr->bus_tag = rman_get_bustag(ctrlr->resource); > - ctrlr->bus_handle = rman_get_bushandle(ctrlr->resource); > - ctrlr->regs = (struct nvme_registers *)ctrlr->bus_handle; > - > /* > * The NVMe spec allows for the MSI-X tables to be placed behind > * BAR 4 and/or 5, separate from the control/doorbell registers. > @@ -180,7 +184,7 @@ nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) > if (ctrlr->msix_table_resource_id >= 0 && > ctrlr->msix_table_resource_id != ctrlr->resource_id) { > ctrlr->msix_table_resource = bus_alloc_resource_any(ctrlr->dev, > - SYS_RES_MEMORY, &ctrlr->msix_table_resource_id, RF_ACTIVE); > + SYS_RES_MEMORY, &ctrlr->msix_table_resource_id, 0); > if (ctrlr->msix_table_resource == NULL) { > nvme_printf(ctrlr, "unable to allocate msi-x table resource\n"); > return (ENOMEM); > @@ -190,13 +194,39 @@ nvme_ctrlr_allocate_bar(struct nvme_controller *ctrlr) > ctrlr->msix_pba_resource_id != ctrlr->resource_id && > ctrlr->msix_pba_resource_id != ctrlr->msix_table_resource_id) { > ctrlr->msix_pba_resource = bus_alloc_resource_any(ctrlr->dev, > - SYS_RES_MEMORY, &ctrlr->msix_pba_resource_id, RF_ACTIVE); > + SYS_RES_MEMORY, &ctrlr->msix_pba_resource_id, 0); > if (ctrlr->msix_pba_resource == NULL) { > nvme_printf(ctrlr, "unable to allocate msi-x pba resource\n"); > return (ENOMEM); > } > } > > + error = bus_activate_resource(ctrlr->dev, ctrlr->resource); > + if (error) { > + nvme_printf(ctrlr, "unable to activate pci resource: %d\n", error); > + return (error); > + } > + if (ctrlr->msix_table_resource != NULL) { > + error = bus_activate_resource(ctrlr->dev, ctrlr->msix_table_resource); > + if (error) { > + nvme_printf(ctrlr, "unable to activate msi-x table resource: %d\n", > + error); > + return (error); > + } > + } > + if (ctrlr->msix_pba_resource != NULL) { > + error = bus_activate_resource(ctrlr->dev, ctrlr->msix_pba_resource); > + if (error) { > + nvme_printf(ctrlr, "unable to activate msi-x pba resource: %d\n", > + error); > + return (error); > + } > + } > + > + ctrlr->bus_tag = rman_get_bustag(ctrlr->resource); > + ctrlr->bus_handle = rman_get_bushandle(ctrlr->resource); > + ctrlr->regs = (struct nvme_registers *)ctrlr->bus_handle; This is a gross hack that needs to die (ctlr->regs), and also, bus_tag and bus_handle should probably be removed as well. -- John Baldwin