From nobody Fri Aug 28 19:02:45 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hWnnx2Dmpz6qtGC for ; Fri, 28 Aug 2026 19:02:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hWnnx1M90z41rk for ; Fri, 28 Aug 2026 19:02:45 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787943765; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=o7gy2utRuE0XOUP7a0EE0UUhWmu4FHr8JKWVRaqS+Us=; b=flD9AQk9PcatRpj9bkBwNtux3/DRLmalB1el+fGj/T0c2OGpg8rMrb4S44pP0OhwKsd3ax sp0FmiDYnVtSPkTUQH+zQjCvxXcXW901jUpdxSfTSod0rM+POeofub7lOXPR9JmOGsMeg7 carctlIr+kC0Bx7SFW/oHxV4uHUFYVfzFybBPDFfiw1zUk4vnIINK3Hnz/Oh/HgxNquuLK f65q3hhE7SWYxAf6bw3INrfb33na5tjXxkogtksZckQk7cjWLBXLoQ/HmmyBlqboGLSM08 PxlpVzvquXjNTto75IgInCGqvUAy0H9I9+oxFihGAfnQrGRtFU0sWwBz2JMS4w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1787943765; a=rsa-sha256; cv=none; b=L53edeXuHRXm/um8PMgcOgsnoGQfq/oK+ZFrq69GPXkYsogjD3X13vJ56gWBpd5i8fZelw x6iTuKxw7sizUqxnUpI387+iYNcR231VOBI6vLLCop5MzZ5llnZIxpT4HEU9iPyz5irCGO DW7WqvOZoK2rOJbrOqg3kXSrg0zsgsesG4qMJgN66HjiQrXSR/Vz+bwBqw1zVHuMicl+Mb MgrvoTjuek4ihIaIsgASFX53Gi38QLRAQneAh9GLY9jSAgo6ssFGFQ7pLgP4l2+aXpzAu1 6dTkNMU8se2VbANrWaAKQWKzp4U+IQJu4Qp0WdBprlW+G6Apv7mBE+BUN+Wung== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787943765; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=o7gy2utRuE0XOUP7a0EE0UUhWmu4FHr8JKWVRaqS+Us=; b=SlQcodNyJ2NCXPER0Cg834CfgIvKQsurxA37D9fGM1u9re/w4/9W1zA10JrlrMA3cdr4df cixm8gmc7gi4tYZ7fjU7N8upH/ebtBHePuLIv15KkOg63ebRQUQYkMFmvrlc4ujG1A8+lh 5rJomBTn0o30pd5hlfEv4aPHafbTU6Yib0boZZySab05q9TjIhuGK87E0lKt1wHT+jfEWL C3hYoVGe0oT4DELKjbZfRSn1O9VjlG30vo0X2KOVH16qz+ECshtRy0DE2/G4LiQfSw0uqQ WXO52TVEMDoYe2FLYYVbqa3+pjN2crkAY+/bEyaUZFWAKenwt4EjbZe9GuucGw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hWnnx04Zdz1LnL for ; Fri, 28 Aug 2026 19:02:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 31da8 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 28 Aug 2026 19:02:45 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alexander Ziaee Subject: git: 59f363d0c21a - main - nvme.4: Canonicalize LOADER TUNABLES List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ziaee X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 59f363d0c21a5bc5897d7fd04bb06cd6eccfb1d9 Auto-Submitted: auto-generated Date: Fri, 28 Aug 2026 19:02:45 +0000 Message-Id: <6a91db55.31da8.6e4a2a4@gitrepo.freebsd.org> The branch main has been updated by ziaee: URL: https://cgit.FreeBSD.org/src/commit/?id=59f363d0c21a5bc5897d7fd04bb06cd6eccfb1d9 commit 59f363d0c21a5bc5897d7fd04bb06cd6eccfb1d9 Author: Alexander Ziaee AuthorDate: 2026-08-28 18:59:04 +0000 Commit: Alexander Ziaee CommitDate: 2026-08-28 19:01:05 +0000 nvme.4: Canonicalize LOADER TUNABLES Improve style for consistency with the rest of the manual and add the tunables to the search database. MFC after: 3 days Reviewed by: seuros Differential Revision: https://reviews.freebsd.org/D59165 --- share/man/man4/nvme.4 | 149 +++++++++++++++++++++----------------------------- 1 file changed, 63 insertions(+), 86 deletions(-) diff --git a/share/man/man4/nvme.4 b/share/man/man4/nvme.4 index 5391862f6669..10f0a067f48a 100644 --- a/share/man/man4/nvme.4 +++ b/share/man/man4/nvme.4 @@ -31,7 +31,7 @@ .\" .\" Author: Jim Harris .\" -.Dd August 23, 2026 +.Dd August 27, 2026 .Dt NVME 4 .Os .Sh NAME @@ -107,7 +107,46 @@ to expose NVM Express namespaces as disk devices which can be partitioned. Note that in NVM Express terms, a namespace is roughly equivalent to a SCSI LUN. -.Sh CONFIGURATION +.Sh KERNEL CONFIGURATION +The following settings can be configured at kernel compile time, see +.Xr build 7 : +.Bl -tag -width indent +.It Cd options NVME_2X_RESET +Reset controller twice upon boot. +This should not be necessary. +.It Cd options NVME_USE_NVD=0 +Use the +.Xr nda 4 +driver to provide a disk device instead of the default +.Xr nvd 4 . +The +.Xr nvd 4 +driver performs better with smaller transactions and few TRIM +commands. +It sends all commands directly to the drive immediately. +The +.Xr nda 4 +driver performs better with larger transactions and also collapses +TRIM commands giving better performance. +It can queue commands to the drive; combine +.Dv BIO_DELETE +commands into a single trip; and +use the CAM I/O scheduler to bias one type of operation over another. +.Pp +This value can also be set at boot time with the +.Va hw.nvme.use_nvd=0 +tunable. +.El +.Sh LOADER TUNABLES +The following settings can be configured in +.Xr loader.conf 5 +or at the boot +.Xr loader 8 +prompt during runtime: +.Pp +.Bl -tag -width indent +.It Va hw.nvme.per_cpu_io_queues=0 +Force a single I/O queue pair shared by all CPUs. By default, .Nm will create an I/O queue pair for each CPU, provided enough MSI-X vectors @@ -115,109 +154,47 @@ and NVMe queue pairs can be allocated. If not enough vectors or queue pairs are available, nvme(4) will use a smaller number of queue pairs and assign multiple CPUs per queue pair. -.Pp -To force a single I/O queue pair shared by all CPUs, set the following -tunable value in -.Xr loader.conf 5 : -.Bd -literal -offset indent -hw.nvme.per_cpu_io_queues=0 -.Ed -.Pp -To assign more than one CPU per I/O queue pair, thereby reducing the number -of MSI-X vectors consumed by the device, set the following tunable value in -.Xr loader.conf 5 : -.Bd -literal -offset indent -hw.nvme.min_cpus_per_ioq=X -.Ed -.Pp -To force legacy interrupts for all +.It Va hw.nvme.min_cpus_per_ioq=X +Assign more than one CPU per I/O queue pair, +thereby reducing the number of MSI-X vectors consumed by the device. +.It Va hw.nvme.force_intx=1 +Force legacy interrupts for all .Nm -driver instances, set the following tunable value in -.Xr loader.conf 5 : -.Bd -literal -offset indent -hw.nvme.force_intx=1 -.Ed -.Pp +driver instances. Note that use of INTx implies disabling of per-CPU I/O queue pairs. -.Pp -To control maximum amount of system RAM in bytes to use as Host Memory -Buffer for capable devices, set the following tunable: -.Bd -literal -offset indent -hw.nvme.hmb_max -.Ed -.Pp +.It Va hw.nvme.hmb_max +Control maximum amount of system RAM in bytes +to use as Host Memory Buffer for capable devices. The default value is 5% of physical memory size per device. -.Pp -To enable Autonomous Power State Transition (APST), set the following -tunable value in -.Xr loader.conf 5 : -.Bd -literal -offset indent -hw.nvme.apst_enable=1 -.Ed -.Pp -The default vendor-provided settings, if any, will be applied. -To override this, set the following tunable: -.Bd -literal -offset indent -hw.nvme.apst_data -.Ed -.Pp -The string must contain up to 32 encoded integers, e.g. "0x6418 0 -0 0x3e820". +.It Va hw.nvme.apst_enable=1 +Enable Autonomous Power State Transition +.Pq APST . +.It Va hw.nvme.apst_data +Override default APST settings. +If unset, the default vendor-provided settings, if any, will be applied. +The string must contain up to 32 encoded integers, e.g. +.Dq 0x6418 0 0 0x3e820 . Each value corresponds to a specific available power state starting from the lowest, and defines the target state (bits 3..7) to transition to, as well as the idle time in milliseconds (bits 8..31) to wait before that transition. Bits 0..2 must be zero. -.Pp +.It Va hw.nvme.use_nvd=0 The .Xr nvd 4 driver is used to provide a disk driver to the system by default. The .Xr nda 4 driver can also be used instead. -The -.Xr nvd 4 -driver performs better with smaller transactions and few TRIM -commands. -It sends all commands directly to the drive immediately. -The -.Xr nda 4 -driver performs better with larger transactions and also collapses -TRIM commands giving better performance. -It can queue commands to the drive; combine -.Dv BIO_DELETE -commands into a single trip; and -use the CAM I/O scheduler to bias one type of operation over another. -To select the -.Xr nda 4 -driver, set the following tunable value in -.Xr loader.conf 5 : -.Bd -literal -offset indent -hw.nvme.use_nvd=0 -.Ed .Pp This value may also be set in the kernel config file with -.Bd -literal -offset indent -.Cd options NVME_USE_NVD=0 -.Ed -.Pp +.Cd options NVME_USE_NVD=0 . +.It Va hw.nvme.verbose_cmd_dump=1 +Enable verbose error messages. When there is an error, .Nm prints only the most relevant information about the command by default. -To enable dumping of all information about the command, set the following tunable -value in -.Xr loader.conf 5 : -.Bd -literal -offset indent -hw.nvme.verbose_cmd_dump=1 -.Ed -.Pp -Prior versions of the driver reset the card twice on boot. -This proved to be unnecessary and inefficient, so the driver now resets drive -controller only once. -The old behavior may be restored in the kernel config file with -.Bd -literal -offset indent -.Cd options NVME_2X_RESET -.Ed +.El .Sh SYSCTL VARIABLES The following controller-level sysctls are currently implemented: .Bl -tag -width indent