Re: Call for testers with NVMe
- Reply: S. Ross Gohlke: "Re: Call for testers with NVMe"
- In reply to: Kevin Bowling : "Re: Call for testers with NVMe"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Tue, 15 Sep 2026 12:12:25 UTC
On 9/14/26 11:31, Kevin Bowling wrote: > On Sun, Sep 13, 2026 at 9:41 AM S. Ross Gohlke<ross@bisd.ro> wrote: >> >> On 9/13/26 08:10, Abdelkader Boudih wrote: >>> Hi Again >>> >>> The last round endup with 80% of the Diffs merged and 50% already MFCed. >>> >>> imp@ still want to try one diff in a Netflix server... >>> So if there is an outage that spoil your favorite show, I'm the one to >>> blame. :) >>> >>> Marcin Cieslak reported a wedge in Apple S3x . (I still trying to >>> reproduce it, but maybe this stack fix it) >>> >>> New round of NVME testing : >>> >>> D59637 nvme: do not touch INTMS/INTMC when configured for MSI-X >>> D59636 nvme: enforce the Abort Command Limit >>> D59634 nvme: do not complete a command when its Abort is not performed >>> D59633 nvme: delete the I/O queues in the system shutdown path >>> D59631 nvme: honor CRTO for the controller ready timeout >>> D59628 nvme: derive CC.CSS from CAP.CSS instead of hardcoding the NVM set >>> D59627 nvme: honor FLBAS Format Index Upper when selecting the LBA format >>> D59626 nvme: bound the AER error log byte-swap by the fetched length >>> D59625 nvme: reject namespaces formatted with metadata >>> >>> This all extracted fromhttps://nvmexpress.org/specifications/ (Spec >>> 2.4 , August 2026) >>> Link to the top of the stackhttps://reviews.freebsd.org/D59636 >>> >>> Mostly Spec review, previous versions of the spec was either vague, or >>> non-existant and nobody touched the code since then. >>> >>> I request more people to test it so we add Quirks to drives that are >>> not spec compliant, but first we need to get the driver to spec >>> complianciancy first. >>> >>> I tested this on my fleet for 10 days, and the machines are daily >>> building kernels and running as CI. >>> >>> Best, >>> Abdelkader >>> >> I upgraded CURRENT via pkgbase last night, from 1600021 to 1600025. >> * FreeBSD-src-16.snap20260912223212 >> * FreeBSD-utilities-16.snap20260912182422 >> I run a custom kernel based on MINIMAL_MMCCAM, but nvme is in the kernel. >> ``` >> % sysctl kern.conftxt | grep nvm >> device nvme >> ``` >> Also, I tested with GENERIC kernel and got the same result. >> >> System is ThinkPad L14 Gen 2 Tiger Lake. >> >> I have everything working as expected, but intr shows unusually high >> utilization, around >> 50% of a core, constantly, in top(1). > Did you identify _which_ intr? It sounds like it could be a22eb754c4bb. I guess I don't know how to identify _which_ intr. I am seeing no output in the usual places -- dmesg, /var/log/messages. I tried enabling hw.nvme.verbose_cmd_dump: 1, but that makes no difference. Maybe I am wrong about nvme being the cause, I just ran out of things to check. All I know is, this is definitely new behavior and I have exhausted my troubleshooting playbook. From "top -PCS": PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 8 199 ki31 0B 128K RUN 0 60.2H 788.76% idle 12 root 29 -54 - 0B 464K WAIT 0 290:43 60.54% intr If there is something else to check related to intr, please let me know which manual page to start reading. Also, I will look for a22eb754c4bb. Thanks for the pointer. > >> I tried everything I could think of, with no change. >> * unload kernel modules >> * stop all services >> Then remembered the nvme upgrades. >> I think this is the cause of the problem. >> >> I don't know anything substantial about storage hardware in general or >> nvme in specific. >> I tried different values in loader.conf(5) for some tunables mentioned >> in nvme(4) that >> seemed plausibly related. >> * hw.nvme.apst_enable >> * hw.nvme.force_intx >> They do not even show up in sysctl(8) after reboot. >> >> ``` >> % doas nvmecontrol devlist >> nvme0: SAMSUNG MZALQ256HBJD-00BL2 >> nvme0ns1 (244198MB) >> >> % doas pciconf -lv >> ... >> nvme0@pci0:2:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d >> device=0xa809 subvendor=0x144d subdevice=0xa801 >> vendor = 'Samsung Electronics Co Ltd' >> device = 'NVMe SSD Controller 980 (DRAM-less)' >> class = mass storage >> subclass = NVM >> ... >> >> % doas nvmecontrol identify nvme0 >> Controller Capabilities/Features >> ================================ >> Vendor ID: 144d >> Subsystem Vendor ID: 144d >> Serial Number: S65FNE1R713832 >> Model Number: SAMSUNG MZALQ256HBJD-00BL2 >> Firmware Version: 7L2QFXM7 >> Recommended Arb Burst: 2 >> IEEE OUI Identifier: 00 25 38 >> Multi-Path I/O Capabilities: Not Supported >> Max Data Transfer Size: 2097152 bytes >> Sanitize Crypto Erase: Not Supported >> Sanitize Block Erase: Supported >> Sanitize Overwrite: Not Supported >> Sanitize NDI: Supported >> Sanitize NODMMAS: No >> Controller ID: 0x0005 >> Version: 1.4.0 >> Traffic Based Keep Alive: Not Supported >> Controller Type: I/O Controller >> Keep Alive Timer Not Supported >> Maximum Outstanding Commands Not Specified >> >> Admin Command Set Attributes >> ============================ >> Security Send/Receive: Supported >> Format NVM: Supported >> Firmware Activate/Download: Supported >> Namespace Management: Not Supported >> Device Self-test: Supported >> Directives: Not Supported >> NVMe-MI Send/Receive: Not Supported >> Virtualization Management: Not Supported >> Doorbell Buffer Config: Not Supported >> Get LBA Status: Not Supported >> Sanitize: block, >> Abort Command Limit: 8 >> Async Event Request Limit: 4 >> Number of Firmware Slots: 3 >> Firmware Slot 1 Read-Only: No >> Per-Namespace SMART Log: Yes >> Error Log Page Entries: 64 >> Number of Power States: 5 >> Total NVM Capacity: 256060514304 bytes >> Unallocated NVM Capacity: 0 bytes >> Firmware Update Granularity: 04 (16384 bytes) >> Host Buffer Preferred Size: 67108864 bytes >> Host Buffer Minimum Size: 16777216 bytes >> >> NVM Command Set Attributes >> ========================== >> Submission Queue Entry Size >> Max: 64 >> Min: 64 >> Completion Queue Entry Size >> Max: 16 >> Min: 16 >> Number of Namespaces: 1 >> Compare Command: Supported >> Write Uncorrectable Command: Supported >> Dataset Management Command: Supported >> Write Zeroes Command: Not Supported >> Save Features: Supported >> Reservations: Not Supported >> Timestamp feature: Supported >> Verify feature: Not Supported >> Fused Operation Support: Not Supported >> Format NVM Attributes: Per-NS Erase, Per-NS Format >> Volatile Write Cache: Present, flush all >> >> NVM Subsystem Name: nqn.1994-11.com.samsung:nvme:PM991a:M.2:S65FNE1R713832 >> >> % doas nvmecontrol identify nvme0ns1 >> Size: 500118192 blocks >> Capacity: 500118192 blocks >> Utilization: 484394000 blocks >> Thin Provisioning: Not Supported >> Number of LBA Formats: 1 >> Current LBA Format: LBA Format #00 >> Metadata Capabilities >> Extended: Not Supported >> Separate: Not Supported >> Data Protection Caps: Not Supported >> Data Protection Settings: Not Enabled >> Multi-Path I/O Capabilities: Not Supported >> Reservation Capabilities: Not Supported >> Format Progress Indicator: 0% remains >> Deallocate Logical Block: Read Not Reported >> Optimal I/O Boundary: 0 blocks >> NVM Capacity: 256060514304 bytes >> Preferred Write Granularity: 32 blocks >> Preferred Write Alignment: 8 blocks >> Preferred Deallocate Granul: 10368 blocks >> Preferred Deallocate Align: 10368 blocks >> Optimal Write Size: 256 blocks >> Globally Unique Identifier: 00000000000000000000000000000000 >> IEEE EUI64: 002538d711133242 >> LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: Best >> >> % sysctl hw.nvme >> hw.nvme.verbose_cmd_dump: 1 >> hw.nvme.use_nvd: 0 >> >> % sysctl dev.nvme >> dev.nvme.0.alignment_splits: 0 >> dev.nvme.0.ioq.3.dump_debug: 0 >> dev.nvme.0.ioq.3.recovery: 0 >> dev.nvme.0.ioq.3.num_recovery_nolock: 0 >> dev.nvme.0.ioq.3.num_ignored: 0 >> dev.nvme.0.ioq.3.num_failures: 0 >> dev.nvme.0.ioq.3.num_retries: 0 >> dev.nvme.0.ioq.3.num_intr_handler_calls: 10915 >> dev.nvme.0.ioq.3.num_cmds: 11049 >> dev.nvme.0.ioq.3.cq_head: 41 >> dev.nvme.0.ioq.3.sq_tail: 41 >> dev.nvme.0.ioq.3.sq_head: 41 >> dev.nvme.0.ioq.3.num_trackers: 128 >> dev.nvme.0.ioq.3.num_entries: 256 >> dev.nvme.0.ioq.2.dump_debug: 0 >> dev.nvme.0.ioq.2.recovery: 0 >> dev.nvme.0.ioq.2.num_recovery_nolock: 0 >> dev.nvme.0.ioq.2.num_ignored: 0 >> dev.nvme.0.ioq.2.num_failures: 0 >> dev.nvme.0.ioq.2.num_retries: 0 >> dev.nvme.0.ioq.2.num_intr_handler_calls: 5321 >> dev.nvme.0.ioq.2.num_cmds: 5410 >> dev.nvme.0.ioq.2.cq_head: 34 >> dev.nvme.0.ioq.2.sq_tail: 34 >> dev.nvme.0.ioq.2.sq_head: 34 >> dev.nvme.0.ioq.2.num_trackers: 128 >> dev.nvme.0.ioq.2.num_entries: 256 >> dev.nvme.0.ioq.1.dump_debug: 0 >> dev.nvme.0.ioq.1.recovery: 0 >> dev.nvme.0.ioq.1.num_recovery_nolock: 0 >> dev.nvme.0.ioq.1.num_ignored: 0 >> dev.nvme.0.ioq.1.num_failures: 0 >> dev.nvme.0.ioq.1.num_retries: 0 >> dev.nvme.0.ioq.1.num_intr_handler_calls: 10896 >> dev.nvme.0.ioq.1.num_cmds: 11014 >> dev.nvme.0.ioq.1.cq_head: 6 >> dev.nvme.0.ioq.1.sq_tail: 6 >> dev.nvme.0.ioq.1.sq_head: 6 >> dev.nvme.0.ioq.1.num_trackers: 128 >> dev.nvme.0.ioq.1.num_entries: 256 >> dev.nvme.0.ioq.0.dump_debug: 0 >> dev.nvme.0.ioq.0.recovery: 0 >> dev.nvme.0.ioq.0.num_recovery_nolock: 0 >> dev.nvme.0.ioq.0.num_ignored: 0 >> dev.nvme.0.ioq.0.num_failures: 0 >> dev.nvme.0.ioq.0.num_retries: 0 >> dev.nvme.0.ioq.0.num_intr_handler_calls: 10407 >> dev.nvme.0.ioq.0.num_cmds: 10503 >> dev.nvme.0.ioq.0.cq_head: 7 >> dev.nvme.0.ioq.0.sq_tail: 7 >> dev.nvme.0.ioq.0.sq_head: 7 >> dev.nvme.0.ioq.0.num_trackers: 128 >> dev.nvme.0.ioq.0.num_entries: 256 >> dev.nvme.0.adminq.dump_debug: 0 >> dev.nvme.0.adminq.recovery: 0 >> dev.nvme.0.adminq.num_recovery_nolock: 0 >> dev.nvme.0.adminq.num_ignored: 0 >> dev.nvme.0.adminq.num_failures: 0 >> dev.nvme.0.adminq.num_retries: 0 >> dev.nvme.0.adminq.num_intr_handler_calls: 225 >> dev.nvme.0.adminq.num_cmds: 229 >> dev.nvme.0.adminq.cq_head: 97 >> dev.nvme.0.adminq.sq_tail: 101 >> dev.nvme.0.adminq.sq_head: 101 >> dev.nvme.0.adminq.num_trackers: 16 >> dev.nvme.0.adminq.num_entries: 128 >> dev.nvme.0.fail_on_reset: 0 >> dev.nvme.0.cap_hi: 48 >> dev.nvme.0.cap_lo: 1006845951 >> dev.nvme.0.reset_stats: 0 >> dev.nvme.0.num_recovery_nolock: 0 >> dev.nvme.0.num_ignored: 0 >> dev.nvme.0.num_failures: 0 >> dev.nvme.0.num_retries: 0 >> dev.nvme.0.num_intr_handler_calls: 37764 >> dev.nvme.0.num_cmds: 38205 >> dev.nvme.0.timeout_period: 30 >> dev.nvme.0.admin_timeout_period: 60 >> dev.nvme.0.int_coal_threshold: 0 >> dev.nvme.0.int_coal_time: 0 >> dev.nvme.0.num_io_queues: 4 >> dev.nvme.0.wake: 0 >> dev.nvme.0.%iommu: rid=0x200 >> dev.nvme.0.%parent: pci1 >> dev.nvme.0.%pnpinfo: vendor=0x144d device=0xa809 subvendor=0x144d >> subdevice=0xa801 class=0x010802 >> dev.nvme.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 >> handle=\_SB_.PC00.PEG0.PEGP >> dev.nvme.0.%driver: nvme >> dev.nvme.0.%desc: Generic NVMe Device >> dev.nvme.%parent: >> ``` >> >> If there is more information I need to provide, let me know. >> >> I usually just wait for such things to work themselves out, and they >> usually do over time, but given the call for testers I thought this >> feedback might be useful... and it would be nice to have my power >> sipping setup back. >> >> >> Regards, >> >> Ross >> >> >>