From nobody Sun May 17 18:45:13 2026 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gJVHH59FSz6cntM for ; Sun, 17 May 2026 18:45:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R13" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gJVHH46Jxz3F1h for ; Sun, 17 May 2026 18:45:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1779043515; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z9zxtf6V9qkAwz8M869108wEK0GHGckUr+vnPEzfuoQ=; b=DWNwTYcL1XzzZPGYiQXJSXx3K6H3VOSt0fWiouVNIeTYAy8Kg1QjGgt19D32mwzX+IcL17 vZWNPDnFDr6YT0ae83ht47Apfvc1uEHQALr9gC6bAn89NkFF+BhITrMvaXxXzUl2hWt60H KjKtSrg3ji4MXbyM9MoQg9zSiQOnCp9/rKWfeNR1z/QVFisXudKZSZZeqWYx2zowLHrhvh 65/tB1fnQPeIO9vX5gZ42rR2c2GLCmTOB84cqTrAA2B0BtXOhYvdOywots+AoDHQokG6GE byd6M0wIjlxBaiqKL9FdRgdsd7CMS2qh88cdcCnZ6DgMvFTmTjqcISr5mcfm/A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1779043515; a=rsa-sha256; cv=none; b=V535XLpUrzW7gUjU2V0BIkVolNkWOQtgCaLz2FLycs0lEuhI869Z7N3bkMuU4DEEEXLE+A /CQEMmDr/DBdv4z0ALc2vZ7XJItbIzxFWKmq4qk9qxTp9IuIBJqqR9032/XRLx1tlNGwdP wRA7bK97J93YJ1KO6CpFyB8uRqsGphqndWNxcJwwTjf0DJ4d6XiwSYt+BRuXC3W6YHwcNP dp7LtpuiAeQrcxXjsEMaAUR2h6IQhoezwHicmBUKx5Vqa1SDHbdxxTNC9i6532rjJ5jiVy VBS0BtP5stJYBZ3+YSByq1YFj/bniwAr2ddwYXt59WR2QsZofFJQzIJH4SLr/A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1779043515; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z9zxtf6V9qkAwz8M869108wEK0GHGckUr+vnPEzfuoQ=; b=wawzZqqSt0iS1kO3wp5+3iekg4wP0r9+L3vil/z9JTPu/iYwmTPB1SLrNPMbQ9M/6y+O81 1Hfz4t/qTIs5cKocDPVL90IwM4zbz0YSuhqK4PnwHrCtcZsloBW4lxZ8y/4bfj3JD9hNVS njL4TqHIYVVmjE6cMunVmxf4+xsU4+PEOAacHoe/XC6hsigiaLQGu/im1aBT3DlNqVpWAT Iqgf/DkAcdvl9XF8ELINKOLa7b18rHPO9AP4bY4APfDW1LtT1Si732T6ozEl9i/uTZkwPH o02R5z0vcScA/Io4+p3iFrTJRfaylmp4btjH4pVI2mrdrwio08iCiV0vaTnQBw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4gJVHH3g1Bzs8t for ; Sun, 17 May 2026 18:45:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 64HIjFYo096519 for ; Sun, 17 May 2026 18:45:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 64HIjFic096518 for bugs@FreeBSD.org; Sun, 17 May 2026 18:45:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 243225] mpr0: Out of chain frames leads to boot hang Date: Sun, 17 May 2026 18:45:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.3-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: steve@witopia.net X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D243225 Steve Shippa changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |steve@witopia.net --- Comment #19 from Steve Shippa --- Adding detailed findings and a one-line fix for this bug. HARDWARE: Dell PowerEdge R740xd, Dell HBA330 Mini (SAS3008/mpr), 2x Intel DC P4500 1.6TB NVMe U.2, 2x Xeon Gold 6248 (80 threads), 128GB RAM, BIOS 2.27.= 0, HBA330 firmware 16.17.01.00. OS: FreeBSD 15.0-RELEASE-p8, UEFI boot, GENERIC kernel. SYMPTOM: Intermittent (~50%) boot hang with "mpr0: Out of chain frames, consider increasing hw.mpr.max_chains." Occurs with two NVMe drives install= ed. One NVMe or zero NVMe boots 100% reliably with all 8 HBA drives. ROOT CAUSE: All three controllers (nvme0 at pci0:62:0:0, nvme1 at pci0:63:0= :0, mpr0 at pci0:65:0:0) share the same PCIe root complex on NUMA domain 0. With per_cpu_io_queues enabled (default), each NVMe controller creates 40 I/O qu= eues (one per core) with 40 MSI-X vectors. Two controllers =3D 80 queues + 80 MS= I-X vectors allocated simultaneously through the same PCIe switch the mpr drive= r is initializing through. This overwhelms the PCIe resource allocation and caus= es mpr init to fail. The "out of chain frames" message is misleading -- it is = not a chain frame shortage. FIX: Add to /boot/loader.conf: hw.nvme.per_cpu_io_queues=3D"0" This reduces each NVMe controller from 40 queues to 1 queue, eliminating the PCIe resource conflict. 7+ consecutive boots (cold and warm) with zero failures. Minimal performance impact for typical workloads. THINGS THAT DID NOT WORK: - hw.mpr.max_chains=3D"200000" (tunable confirmed set in loader, kernel sti= ll failed) - hw.mpr.disable_msix=3D"1" - kern.maxphys=3D"524288" - loader_delay=3D"3" - kern.cam.boot_delay=3D"10000" - hint.nvme.1.disabled=3D"1" in loader.conf (not passed to kernel on UEFI/L= ua boot -- kenv shows nothing) - hint.nvme.1.disabled=3D"1" in device.hints (works as workaround but requi= res rc.local to devctl enable post-boot) - Lua loader delay in /boot/lua/loader.lua (delay is pre-kernel, kernel re-enumerates PCIe independently) - Forth loader.rc modifications (completely ignored on FreeBSD 15 UEFI -- u= ses Lua loader) - BIOS MMIO Base changes (56TB/12TB/512GB) - BIOS Slot Disablement / Boot Driver Disabled (NVMe bays are backplane-connected, not in slot list) ADDITIONAL NOTES: 1. FreeBSD 15 UEFI boot uses the Lua loader. /boot/loader.rc (Forth) is ign= ored entirely. This may not be widely known and caught us off guard during debugging. 2. hint.* set in loader.conf does not appear in kenv on UEFI/Lua boot. Hints must go in /boot/device.hints. 3. Terry Kennedy's earlier analysis of this being a timing issue with a misleading error message was correct. --=20 You are receiving this mail because: You are the assignee for the bug.=