Re: Call for testers with NVMe
- Reply: Kevin Bowling : "Re: Call for testers with NVMe"
- In reply to: Abdelkader Boudih: "Re: Call for testers with NVMe"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sun, 13 Sep 2026 16:41:09 UTC
On 9/13/26 08:10, Abdelkader Boudih wrote: > Hi Again > > The last round endup with 80% of the Diffs merged and 50% already MFCed. > > imp@ still want to try one diff in a Netflix server... > So if there is an outage that spoil your favorite show, I'm the one to > blame. :) > > Marcin Cieslak reported a wedge in Apple S3x . (I still trying to > reproduce it, but maybe this stack fix it) > > New round of NVME testing : > > D59637 nvme: do not touch INTMS/INTMC when configured for MSI-X > D59636 nvme: enforce the Abort Command Limit > D59634 nvme: do not complete a command when its Abort is not performed > D59633 nvme: delete the I/O queues in the system shutdown path > D59631 nvme: honor CRTO for the controller ready timeout > D59628 nvme: derive CC.CSS from CAP.CSS instead of hardcoding the NVM set > D59627 nvme: honor FLBAS Format Index Upper when selecting the LBA format > D59626 nvme: bound the AER error log byte-swap by the fetched length > D59625 nvme: reject namespaces formatted with metadata > > This all extracted from https://nvmexpress.org/specifications/ (Spec > 2.4 , August 2026) > Link to the top of the stack https://reviews.freebsd.org/D59636 > > Mostly Spec review, previous versions of the spec was either vague, or > non-existant and nobody touched the code since then. > > I request more people to test it so we add Quirks to drives that are > not spec compliant, but first we need to get the driver to spec > complianciancy first. > > I tested this on my fleet for 10 days, and the machines are daily > building kernels and running as CI. > > Best, > Abdelkader > I upgraded CURRENT via pkgbase last night, from 1600021 to 1600025. * FreeBSD-src-16.snap20260912223212 * FreeBSD-utilities-16.snap20260912182422 I run a custom kernel based on MINIMAL_MMCCAM, but nvme is in the kernel. ``` % sysctl kern.conftxt | grep nvm device nvme ``` Also, I tested with GENERIC kernel and got the same result. System is ThinkPad L14 Gen 2 Tiger Lake. I have everything working as expected, but intr shows unusually high utilization, around 50% of a core, constantly, in top(1). I tried everything I could think of, with no change. * unload kernel modules * stop all services Then remembered the nvme upgrades. I think this is the cause of the problem. I don't know anything substantial about storage hardware in general or nvme in specific. I tried different values in loader.conf(5) for some tunables mentioned in nvme(4) that seemed plausibly related. * hw.nvme.apst_enable * hw.nvme.force_intx They do not even show up in sysctl(8) after reboot. ``` % doas nvmecontrol devlist nvme0: SAMSUNG MZALQ256HBJD-00BL2 nvme0ns1 (244198MB) % doas pciconf -lv ... nvme0@pci0:2:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa809 subvendor=0x144d subdevice=0xa801 vendor = 'Samsung Electronics Co Ltd' device = 'NVMe SSD Controller 980 (DRAM-less)' class = mass storage subclass = NVM ... % doas nvmecontrol identify nvme0 Controller Capabilities/Features ================================ Vendor ID: 144d Subsystem Vendor ID: 144d Serial Number: S65FNE1R713832 Model Number: SAMSUNG MZALQ256HBJD-00BL2 Firmware Version: 7L2QFXM7 Recommended Arb Burst: 2 IEEE OUI Identifier: 00 25 38 Multi-Path I/O Capabilities: Not Supported Max Data Transfer Size: 2097152 bytes Sanitize Crypto Erase: Not Supported Sanitize Block Erase: Supported Sanitize Overwrite: Not Supported Sanitize NDI: Supported Sanitize NODMMAS: No Controller ID: 0x0005 Version: 1.4.0 Traffic Based Keep Alive: Not Supported Controller Type: I/O Controller Keep Alive Timer Not Supported Maximum Outstanding Commands Not Specified Admin Command Set Attributes ============================ Security Send/Receive: Supported Format NVM: Supported Firmware Activate/Download: Supported Namespace Management: Not Supported Device Self-test: Supported Directives: Not Supported NVMe-MI Send/Receive: Not Supported Virtualization Management: Not Supported Doorbell Buffer Config: Not Supported Get LBA Status: Not Supported Sanitize: block, Abort Command Limit: 8 Async Event Request Limit: 4 Number of Firmware Slots: 3 Firmware Slot 1 Read-Only: No Per-Namespace SMART Log: Yes Error Log Page Entries: 64 Number of Power States: 5 Total NVM Capacity: 256060514304 bytes Unallocated NVM Capacity: 0 bytes Firmware Update Granularity: 04 (16384 bytes) Host Buffer Preferred Size: 67108864 bytes Host Buffer Minimum Size: 16777216 bytes NVM Command Set Attributes ========================== Submission Queue Entry Size Max: 64 Min: 64 Completion Queue Entry Size Max: 16 Min: 16 Number of Namespaces: 1 Compare Command: Supported Write Uncorrectable Command: Supported Dataset Management Command: Supported Write Zeroes Command: Not Supported Save Features: Supported Reservations: Not Supported Timestamp feature: Supported Verify feature: Not Supported Fused Operation Support: Not Supported Format NVM Attributes: Per-NS Erase, Per-NS Format Volatile Write Cache: Present, flush all NVM Subsystem Name: nqn.1994-11.com.samsung:nvme:PM991a:M.2:S65FNE1R713832 % doas nvmecontrol identify nvme0ns1 Size: 500118192 blocks Capacity: 500118192 blocks Utilization: 484394000 blocks Thin Provisioning: Not Supported Number of LBA Formats: 1 Current LBA Format: LBA Format #00 Metadata Capabilities Extended: Not Supported Separate: Not Supported Data Protection Caps: Not Supported Data Protection Settings: Not Enabled Multi-Path I/O Capabilities: Not Supported Reservation Capabilities: Not Supported Format Progress Indicator: 0% remains Deallocate Logical Block: Read Not Reported Optimal I/O Boundary: 0 blocks NVM Capacity: 256060514304 bytes Preferred Write Granularity: 32 blocks Preferred Write Alignment: 8 blocks Preferred Deallocate Granul: 10368 blocks Preferred Deallocate Align: 10368 blocks Optimal Write Size: 256 blocks Globally Unique Identifier: 00000000000000000000000000000000 IEEE EUI64: 002538d711133242 LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: Best % sysctl hw.nvme hw.nvme.verbose_cmd_dump: 1 hw.nvme.use_nvd: 0 % sysctl dev.nvme dev.nvme.0.alignment_splits: 0 dev.nvme.0.ioq.3.dump_debug: 0 dev.nvme.0.ioq.3.recovery: 0 dev.nvme.0.ioq.3.num_recovery_nolock: 0 dev.nvme.0.ioq.3.num_ignored: 0 dev.nvme.0.ioq.3.num_failures: 0 dev.nvme.0.ioq.3.num_retries: 0 dev.nvme.0.ioq.3.num_intr_handler_calls: 10915 dev.nvme.0.ioq.3.num_cmds: 11049 dev.nvme.0.ioq.3.cq_head: 41 dev.nvme.0.ioq.3.sq_tail: 41 dev.nvme.0.ioq.3.sq_head: 41 dev.nvme.0.ioq.3.num_trackers: 128 dev.nvme.0.ioq.3.num_entries: 256 dev.nvme.0.ioq.2.dump_debug: 0 dev.nvme.0.ioq.2.recovery: 0 dev.nvme.0.ioq.2.num_recovery_nolock: 0 dev.nvme.0.ioq.2.num_ignored: 0 dev.nvme.0.ioq.2.num_failures: 0 dev.nvme.0.ioq.2.num_retries: 0 dev.nvme.0.ioq.2.num_intr_handler_calls: 5321 dev.nvme.0.ioq.2.num_cmds: 5410 dev.nvme.0.ioq.2.cq_head: 34 dev.nvme.0.ioq.2.sq_tail: 34 dev.nvme.0.ioq.2.sq_head: 34 dev.nvme.0.ioq.2.num_trackers: 128 dev.nvme.0.ioq.2.num_entries: 256 dev.nvme.0.ioq.1.dump_debug: 0 dev.nvme.0.ioq.1.recovery: 0 dev.nvme.0.ioq.1.num_recovery_nolock: 0 dev.nvme.0.ioq.1.num_ignored: 0 dev.nvme.0.ioq.1.num_failures: 0 dev.nvme.0.ioq.1.num_retries: 0 dev.nvme.0.ioq.1.num_intr_handler_calls: 10896 dev.nvme.0.ioq.1.num_cmds: 11014 dev.nvme.0.ioq.1.cq_head: 6 dev.nvme.0.ioq.1.sq_tail: 6 dev.nvme.0.ioq.1.sq_head: 6 dev.nvme.0.ioq.1.num_trackers: 128 dev.nvme.0.ioq.1.num_entries: 256 dev.nvme.0.ioq.0.dump_debug: 0 dev.nvme.0.ioq.0.recovery: 0 dev.nvme.0.ioq.0.num_recovery_nolock: 0 dev.nvme.0.ioq.0.num_ignored: 0 dev.nvme.0.ioq.0.num_failures: 0 dev.nvme.0.ioq.0.num_retries: 0 dev.nvme.0.ioq.0.num_intr_handler_calls: 10407 dev.nvme.0.ioq.0.num_cmds: 10503 dev.nvme.0.ioq.0.cq_head: 7 dev.nvme.0.ioq.0.sq_tail: 7 dev.nvme.0.ioq.0.sq_head: 7 dev.nvme.0.ioq.0.num_trackers: 128 dev.nvme.0.ioq.0.num_entries: 256 dev.nvme.0.adminq.dump_debug: 0 dev.nvme.0.adminq.recovery: 0 dev.nvme.0.adminq.num_recovery_nolock: 0 dev.nvme.0.adminq.num_ignored: 0 dev.nvme.0.adminq.num_failures: 0 dev.nvme.0.adminq.num_retries: 0 dev.nvme.0.adminq.num_intr_handler_calls: 225 dev.nvme.0.adminq.num_cmds: 229 dev.nvme.0.adminq.cq_head: 97 dev.nvme.0.adminq.sq_tail: 101 dev.nvme.0.adminq.sq_head: 101 dev.nvme.0.adminq.num_trackers: 16 dev.nvme.0.adminq.num_entries: 128 dev.nvme.0.fail_on_reset: 0 dev.nvme.0.cap_hi: 48 dev.nvme.0.cap_lo: 1006845951 dev.nvme.0.reset_stats: 0 dev.nvme.0.num_recovery_nolock: 0 dev.nvme.0.num_ignored: 0 dev.nvme.0.num_failures: 0 dev.nvme.0.num_retries: 0 dev.nvme.0.num_intr_handler_calls: 37764 dev.nvme.0.num_cmds: 38205 dev.nvme.0.timeout_period: 30 dev.nvme.0.admin_timeout_period: 60 dev.nvme.0.int_coal_threshold: 0 dev.nvme.0.int_coal_time: 0 dev.nvme.0.num_io_queues: 4 dev.nvme.0.wake: 0 dev.nvme.0.%iommu: rid=0x200 dev.nvme.0.%parent: pci1 dev.nvme.0.%pnpinfo: vendor=0x144d device=0xa809 subvendor=0x144d subdevice=0xa801 class=0x010802 dev.nvme.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PC00.PEG0.PEGP dev.nvme.0.%driver: nvme dev.nvme.0.%desc: Generic NVMe Device dev.nvme.%parent: ``` If there is more information I need to provide, let me know. I usually just wait for such things to work themselves out, and they usually do over time, but given the call for testers I thought this feedback might be useful... and it would be nice to have my power sipping setup back. Regards, Ross