Re: Call for testers with NVMe
- In reply to: S. Ross Gohlke: "Re: Call for testers with NVMe"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Mon, 21 Sep 2026 23:40:25 UTC
On 9/21/26 18:18, S. Ross Gohlke wrote: > > > On 9/15/26 07:12, S. Ross Gohlke wrote: >> >> >> On 9/14/26 11:31, Kevin Bowling wrote: >>> On Sun, Sep 13, 2026 at 9:41 AM S. Ross Gohlke<ross@bisd.ro> wrote: >>>> On 9/13/26 08:10, Abdelkader Boudih wrote: >>>>> Hi Again >>>>> >>>>> The last round endup with 80% of the Diffs merged and 50% already MFCed. >>>>> >>>>> imp@ still want to try one diff in a Netflix server... >>>>> So if there is an outage that spoil your favorite show, I'm the one to >>>>> blame. :) >>>>> >>>>> Marcin Cieslak reported a wedge in Apple S3x . (I still trying to >>>>> reproduce it, but maybe this stack fix it) >>>>> >>>>> New round of NVME testing : >>>>> >>>>> D59637 nvme: do not touch INTMS/INTMC when configured for MSI-X >>>>> D59636 nvme: enforce the Abort Command Limit >>>>> D59634 nvme: do not complete a command when its Abort is not performed >>>>> D59633 nvme: delete the I/O queues in the system shutdown path >>>>> D59631 nvme: honor CRTO for the controller ready timeout >>>>> D59628 nvme: derive CC.CSS from CAP.CSS instead of hardcoding the NVM set >>>>> D59627 nvme: honor FLBAS Format Index Upper when selecting the LBA format >>>>> D59626 nvme: bound the AER error log byte-swap by the fetched length >>>>> D59625 nvme: reject namespaces formatted with metadata >>>>> >>>>> This all extracted fromhttps://nvmexpress.org/specifications/ (Spec >>>>> 2.4 , August 2026) >>>>> Link to the top of the stackhttps://reviews.freebsd.org/D59636 >>>>> >>>>> Mostly Spec review, previous versions of the spec was either vague, or >>>>> non-existant and nobody touched the code since then. >>>>> >>>>> I request more people to test it so we add Quirks to drives that are >>>>> not spec compliant, but first we need to get the driver to spec >>>>> complianciancy first. >>>>> >>>>> I tested this on my fleet for 10 days, and the machines are daily >>>>> building kernels and running as CI. >>>>> >>>>> Best, >>>>> Abdelkader >>>>> >>>> I upgraded CURRENT via pkgbase last night, from 1600021 to 1600025. >>>> * FreeBSD-src-16.snap20260912223212 >>>> * FreeBSD-utilities-16.snap20260912182422 >>>> I run a custom kernel based on MINIMAL_MMCCAM, but nvme is in the kernel. >>>> ``` >>>> % sysctl kern.conftxt | grep nvm >>>> device nvme >>>> ``` >>>> Also, I tested with GENERIC kernel and got the same result. >>>> >>>> System is ThinkPad L14 Gen 2 Tiger Lake. >>>> >>>> I have everything working as expected, but intr shows unusually high >>>> utilization, around >>>> 50% of a core, constantly, in top(1). >>> Did you identify _which_ intr? It sounds like it could be a22eb754c4bb. >> >> I guess I don't know how to identify _which_ intr. >> >> I am seeing no output in the usual places -- dmesg, /var/log/messages. >> >> I tried enabling hw.nvme.verbose_cmd_dump: 1, but that makes no >> difference. >> >> Maybe I am wrong about nvme being the cause, I just ran out of things >> to check. >> >> All I know is, this is definitely new behavior and I have exhausted >> my troubleshooting playbook. >> >> From "top -PCS": >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 11 root 8 199 ki31 0B 128K RUN 0 60.2H >> 788.76% idle >> 12 root 29 -54 - 0B 464K WAIT 0 290:43 >> 60.54% intr >> >> If there is something else to check related to intr, please let me >> know which manual page to start reading. >> >> Also, I will look for a22eb754c4bb. Thanks for the pointer. >> >>>> I tried everything I could think of, with no change. >>>> * unload kernel modules >>>> * stop all services >>>> Then remembered the nvme upgrades. >>>> I think this is the cause of the problem. >>>> >>>> I don't know anything substantial about storage hardware in general or >>>> nvme in specific. >>>> I tried different values in loader.conf(5) for some tunables mentioned >>>> in nvme(4) that >>>> seemed plausibly related. >>>> * hw.nvme.apst_enable >>>> * hw.nvme.force_intx >>>> They do not even show up in sysctl(8) after reboot. >>>> >>>> ``` >>>> % doas nvmecontrol devlist >>>> nvme0: SAMSUNG MZALQ256HBJD-00BL2 >>>> nvme0ns1 (244198MB) >>>> >>>> % doas pciconf -lv >>>> ... >>>> nvme0@pci0:2:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d >>>> device=0xa809 subvendor=0x144d subdevice=0xa801 >>>> vendor = 'Samsung Electronics Co Ltd' >>>> device = 'NVMe SSD Controller 980 (DRAM-less)' >>>> class = mass storage >>>> subclass = NVM >>>> ... >>>> >>>> % doas nvmecontrol identify nvme0 >>>> Controller Capabilities/Features >>>> ================================ >>>> Vendor ID: 144d >>>> Subsystem Vendor ID: 144d >>>> Serial Number: S65FNE1R713832 >>>> Model Number: SAMSUNG MZALQ256HBJD-00BL2 >>>> Firmware Version: 7L2QFXM7 >>>> Recommended Arb Burst: 2 >>>> IEEE OUI Identifier: 00 25 38 >>>> Multi-Path I/O Capabilities: Not Supported >>>> Max Data Transfer Size: 2097152 bytes >>>> Sanitize Crypto Erase: Not Supported >>>> Sanitize Block Erase: Supported >>>> Sanitize Overwrite: Not Supported >>>> Sanitize NDI: Supported >>>> Sanitize NODMMAS: No >>>> Controller ID: 0x0005 >>>> Version: 1.4.0 >>>> Traffic Based Keep Alive: Not Supported >>>> Controller Type: I/O Controller >>>> Keep Alive Timer Not Supported >>>> Maximum Outstanding Commands Not Specified >>>> >>>> Admin Command Set Attributes >>>> ============================ >>>> Security Send/Receive: Supported >>>> Format NVM: Supported >>>> Firmware Activate/Download: Supported >>>> Namespace Management: Not Supported >>>> Device Self-test: Supported >>>> Directives: Not Supported >>>> NVMe-MI Send/Receive: Not Supported >>>> Virtualization Management: Not Supported >>>> Doorbell Buffer Config: Not Supported >>>> Get LBA Status: Not Supported >>>> Sanitize: block, >>>> Abort Command Limit: 8 >>>> Async Event Request Limit: 4 >>>> Number of Firmware Slots: 3 >>>> Firmware Slot 1 Read-Only: No >>>> Per-Namespace SMART Log: Yes >>>> Error Log Page Entries: 64 >>>> Number of Power States: 5 >>>> Total NVM Capacity: 256060514304 bytes >>>> Unallocated NVM Capacity: 0 bytes >>>> Firmware Update Granularity: 04 (16384 bytes) >>>> Host Buffer Preferred Size: 67108864 bytes >>>> Host Buffer Minimum Size: 16777216 bytes >>>> >>>> NVM Command Set Attributes >>>> ========================== >>>> Submission Queue Entry Size >>>> Max: 64 >>>> Min: 64 >>>> Completion Queue Entry Size >>>> Max: 16 >>>> Min: 16 >>>> Number of Namespaces: 1 >>>> Compare Command: Supported >>>> Write Uncorrectable Command: Supported >>>> Dataset Management Command: Supported >>>> Write Zeroes Command: Not Supported >>>> Save Features: Supported >>>> Reservations: Not Supported >>>> Timestamp feature: Supported >>>> Verify feature: Not Supported >>>> Fused Operation Support: Not Supported >>>> Format NVM Attributes: Per-NS Erase, Per-NS Format >>>> Volatile Write Cache: Present, flush all >>>> >>>> NVM Subsystem Name: nqn.1994-11.com.samsung:nvme:PM991a:M.2:S65FNE1R713832 >>>> >>>> % doas nvmecontrol identify nvme0ns1 >>>> Size: 500118192 blocks >>>> Capacity: 500118192 blocks >>>> Utilization: 484394000 blocks >>>> Thin Provisioning: Not Supported >>>> Number of LBA Formats: 1 >>>> Current LBA Format: LBA Format #00 >>>> Metadata Capabilities >>>> Extended: Not Supported >>>> Separate: Not Supported >>>> Data Protection Caps: Not Supported >>>> Data Protection Settings: Not Enabled >>>> Multi-Path I/O Capabilities: Not Supported >>>> Reservation Capabilities: Not Supported >>>> Format Progress Indicator: 0% remains >>>> Deallocate Logical Block: Read Not Reported >>>> Optimal I/O Boundary: 0 blocks >>>> NVM Capacity: 256060514304 bytes >>>> Preferred Write Granularity: 32 blocks >>>> Preferred Write Alignment: 8 blocks >>>> Preferred Deallocate Granul: 10368 blocks >>>> Preferred Deallocate Align: 10368 blocks >>>> Optimal Write Size: 256 blocks >>>> Globally Unique Identifier: 00000000000000000000000000000000 >>>> IEEE EUI64: 002538d711133242 >>>> LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: Best >>>> >>>> % sysctl hw.nvme >>>> hw.nvme.verbose_cmd_dump: 1 >>>> hw.nvme.use_nvd: 0 >>>> >>>> % sysctl dev.nvme >>>> dev.nvme.0.alignment_splits: 0 >>>> dev.nvme.0.ioq.3.dump_debug: 0 >>>> dev.nvme.0.ioq.3.recovery: 0 >>>> dev.nvme.0.ioq.3.num_recovery_nolock: 0 >>>> dev.nvme.0.ioq.3.num_ignored: 0 >>>> dev.nvme.0.ioq.3.num_failures: 0 >>>> dev.nvme.0.ioq.3.num_retries: 0 >>>> dev.nvme.0.ioq.3.num_intr_handler_calls: 10915 >>>> dev.nvme.0.ioq.3.num_cmds: 11049 >>>> dev.nvme.0.ioq.3.cq_head: 41 >>>> dev.nvme.0.ioq.3.sq_tail: 41 >>>> dev.nvme.0.ioq.3.sq_head: 41 >>>> dev.nvme.0.ioq.3.num_trackers: 128 >>>> dev.nvme.0.ioq.3.num_entries: 256 >>>> dev.nvme.0.ioq.2.dump_debug: 0 >>>> dev.nvme.0.ioq.2.recovery: 0 >>>> dev.nvme.0.ioq.2.num_recovery_nolock: 0 >>>> dev.nvme.0.ioq.2.num_ignored: 0 >>>> dev.nvme.0.ioq.2.num_failures: 0 >>>> dev.nvme.0.ioq.2.num_retries: 0 >>>> dev.nvme.0.ioq.2.num_intr_handler_calls: 5321 >>>> dev.nvme.0.ioq.2.num_cmds: 5410 >>>> dev.nvme.0.ioq.2.cq_head: 34 >>>> dev.nvme.0.ioq.2.sq_tail: 34 >>>> dev.nvme.0.ioq.2.sq_head: 34 >>>> dev.nvme.0.ioq.2.num_trackers: 128 >>>> dev.nvme.0.ioq.2.num_entries: 256 >>>> dev.nvme.0.ioq.1.dump_debug: 0 >>>> dev.nvme.0.ioq.1.recovery: 0 >>>> dev.nvme.0.ioq.1.num_recovery_nolock: 0 >>>> dev.nvme.0.ioq.1.num_ignored: 0 >>>> dev.nvme.0.ioq.1.num_failures: 0 >>>> dev.nvme.0.ioq.1.num_retries: 0 >>>> dev.nvme.0.ioq.1.num_intr_handler_calls: 10896 >>>> dev.nvme.0.ioq.1.num_cmds: 11014 >>>> dev.nvme.0.ioq.1.cq_head: 6 >>>> dev.nvme.0.ioq.1.sq_tail: 6 >>>> dev.nvme.0.ioq.1.sq_head: 6 >>>> dev.nvme.0.ioq.1.num_trackers: 128 >>>> dev.nvme.0.ioq.1.num_entries: 256 >>>> dev.nvme.0.ioq.0.dump_debug: 0 >>>> dev.nvme.0.ioq.0.recovery: 0 >>>> dev.nvme.0.ioq.0.num_recovery_nolock: 0 >>>> dev.nvme.0.ioq.0.num_ignored: 0 >>>> dev.nvme.0.ioq.0.num_failures: 0 >>>> dev.nvme.0.ioq.0.num_retries: 0 >>>> dev.nvme.0.ioq.0.num_intr_handler_calls: 10407 >>>> dev.nvme.0.ioq.0.num_cmds: 10503 >>>> dev.nvme.0.ioq.0.cq_head: 7 >>>> dev.nvme.0.ioq.0.sq_tail: 7 >>>> dev.nvme.0.ioq.0.sq_head: 7 >>>> dev.nvme.0.ioq.0.num_trackers: 128 >>>> dev.nvme.0.ioq.0.num_entries: 256 >>>> dev.nvme.0.adminq.dump_debug: 0 >>>> dev.nvme.0.adminq.recovery: 0 >>>> dev.nvme.0.adminq.num_recovery_nolock: 0 >>>> dev.nvme.0.adminq.num_ignored: 0 >>>> dev.nvme.0.adminq.num_failures: 0 >>>> dev.nvme.0.adminq.num_retries: 0 >>>> dev.nvme.0.adminq.num_intr_handler_calls: 225 >>>> dev.nvme.0.adminq.num_cmds: 229 >>>> dev.nvme.0.adminq.cq_head: 97 >>>> dev.nvme.0.adminq.sq_tail: 101 >>>> dev.nvme.0.adminq.sq_head: 101 >>>> dev.nvme.0.adminq.num_trackers: 16 >>>> dev.nvme.0.adminq.num_entries: 128 >>>> dev.nvme.0.fail_on_reset: 0 >>>> dev.nvme.0.cap_hi: 48 >>>> dev.nvme.0.cap_lo: 1006845951 >>>> dev.nvme.0.reset_stats: 0 >>>> dev.nvme.0.num_recovery_nolock: 0 >>>> dev.nvme.0.num_ignored: 0 >>>> dev.nvme.0.num_failures: 0 >>>> dev.nvme.0.num_retries: 0 >>>> dev.nvme.0.num_intr_handler_calls: 37764 >>>> dev.nvme.0.num_cmds: 38205 >>>> dev.nvme.0.timeout_period: 30 >>>> dev.nvme.0.admin_timeout_period: 60 >>>> dev.nvme.0.int_coal_threshold: 0 >>>> dev.nvme.0.int_coal_time: 0 >>>> dev.nvme.0.num_io_queues: 4 >>>> dev.nvme.0.wake: 0 >>>> dev.nvme.0.%iommu: rid=0x200 >>>> dev.nvme.0.%parent: pci1 >>>> dev.nvme.0.%pnpinfo: vendor=0x144d device=0xa809 subvendor=0x144d >>>> subdevice=0xa801 class=0x010802 >>>> dev.nvme.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 >>>> handle=\_SB_.PC00.PEG0.PEGP >>>> dev.nvme.0.%driver: nvme >>>> dev.nvme.0.%desc: Generic NVMe Device >>>> dev.nvme.%parent: >>>> ``` >>>> >>>> If there is more information I need to provide, let me know. >>>> >>>> I usually just wait for such things to work themselves out, and they >>>> usually do over time, but given the call for testers I thought this >>>> feedback might be useful... and it would be nice to have my power >>>> sipping setup back. >>>> >>>> >>>> Regards, >>>> >>>> Ross >>>> >>>> >>>> > I upgraded pkgbase yesterday, from 1600025 to 1600026. > > * FreeBSD-src-16.snap20260920013406 > > * FreeBSD-utilities-16.snap20260919182608 > > intr utilization is down to 3-5% from 50-60%. > > Thanks. > > Regards, > > Ross > I forgot my tests. I am happy to run more. doas nvmecontrol perftest -n 16 -o read -s 4096 -t 30 nvme0ns1 Threads: 16 Size: 4096 READ Time: 30 IO/s: 34572 MB/s: 135 doas nvmecontrol perftest -n 16 -o read -s 131072 -t 30 nvme0ns1 Threads: 16 Size: 131072 READ Time: 30 IO/s: 5019 MB/s: 627 doas nvmecontrol perftest -n 32 -o read -s 4096 -t 30 nvme0ns1 Threads: 32 Size: 4096 READ Time: 30 IO/s: 33295 MB/s: 130 doas nvmecontrol perftest -n 32 -o read -s 131072 -t 30 nvme0ns1 Threads: 32 Size: 131072 READ Time: 30 IO/s: 5538 MB/s: 692 doas nvmecontrol perftest -n 64 -o read -s 4096 -t 30 nvme0ns1 Threads: 64 Size: 4096 READ Time: 30 IO/s: 33066 MB/s: 129 doas nvmecontrol perftest -n 64 -o read -s 131072 -t 30 nvme0ns1 Threads: 64 Size: 131072 READ Time: 30 IO/s: 5833 MB/s: 729