Re: Call for testers with NVMe
- Reply: S. Ross Gohlke: "Re: Call for testers with NVMe"
- In reply to: S. Ross Gohlke: "Re: Call for testers with NVMe"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Mon, 14 Sep 2026 16:31:27 UTC
On Sun, Sep 13, 2026 at 9:41 AM S. Ross Gohlke <ross@bisd.ro> wrote: > > > On 9/13/26 08:10, Abdelkader Boudih wrote: > > Hi Again > > > > The last round endup with 80% of the Diffs merged and 50% already MFCed. > > > > imp@ still want to try one diff in a Netflix server... > > So if there is an outage that spoil your favorite show, I'm the one to > > blame. :) > > > > Marcin Cieslak reported a wedge in Apple S3x . (I still trying to > > reproduce it, but maybe this stack fix it) > > > > New round of NVME testing : > > > > D59637 nvme: do not touch INTMS/INTMC when configured for MSI-X > > D59636 nvme: enforce the Abort Command Limit > > D59634 nvme: do not complete a command when its Abort is not performed > > D59633 nvme: delete the I/O queues in the system shutdown path > > D59631 nvme: honor CRTO for the controller ready timeout > > D59628 nvme: derive CC.CSS from CAP.CSS instead of hardcoding the NVM set > > D59627 nvme: honor FLBAS Format Index Upper when selecting the LBA format > > D59626 nvme: bound the AER error log byte-swap by the fetched length > > D59625 nvme: reject namespaces formatted with metadata > > > > This all extracted from https://nvmexpress.org/specifications/ (Spec > > 2.4 , August 2026) > > Link to the top of the stack https://reviews.freebsd.org/D59636 > > > > Mostly Spec review, previous versions of the spec was either vague, or > > non-existant and nobody touched the code since then. > > > > I request more people to test it so we add Quirks to drives that are > > not spec compliant, but first we need to get the driver to spec > > complianciancy first. > > > > I tested this on my fleet for 10 days, and the machines are daily > > building kernels and running as CI. > > > > Best, > > Abdelkader > > > I upgraded CURRENT via pkgbase last night, from 1600021 to 1600025. > * FreeBSD-src-16.snap20260912223212 > * FreeBSD-utilities-16.snap20260912182422 > I run a custom kernel based on MINIMAL_MMCCAM, but nvme is in the kernel. > ``` > % sysctl kern.conftxt | grep nvm > device nvme > ``` > Also, I tested with GENERIC kernel and got the same result. > > System is ThinkPad L14 Gen 2 Tiger Lake. > > I have everything working as expected, but intr shows unusually high > utilization, around > 50% of a core, constantly, in top(1). Did you identify _which_ intr? It sounds like it could be a22eb754c4bb. > I tried everything I could think of, with no change. > * unload kernel modules > * stop all services > Then remembered the nvme upgrades. > I think this is the cause of the problem. > > I don't know anything substantial about storage hardware in general or > nvme in specific. > I tried different values in loader.conf(5) for some tunables mentioned > in nvme(4) that > seemed plausibly related. > * hw.nvme.apst_enable > * hw.nvme.force_intx > They do not even show up in sysctl(8) after reboot. > > ``` > % doas nvmecontrol devlist > nvme0: SAMSUNG MZALQ256HBJD-00BL2 > nvme0ns1 (244198MB) > > % doas pciconf -lv > ... > nvme0@pci0:2:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d > device=0xa809 subvendor=0x144d subdevice=0xa801 > vendor = 'Samsung Electronics Co Ltd' > device = 'NVMe SSD Controller 980 (DRAM-less)' > class = mass storage > subclass = NVM > ... > > % doas nvmecontrol identify nvme0 > Controller Capabilities/Features > ================================ > Vendor ID: 144d > Subsystem Vendor ID: 144d > Serial Number: S65FNE1R713832 > Model Number: SAMSUNG MZALQ256HBJD-00BL2 > Firmware Version: 7L2QFXM7 > Recommended Arb Burst: 2 > IEEE OUI Identifier: 00 25 38 > Multi-Path I/O Capabilities: Not Supported > Max Data Transfer Size: 2097152 bytes > Sanitize Crypto Erase: Not Supported > Sanitize Block Erase: Supported > Sanitize Overwrite: Not Supported > Sanitize NDI: Supported > Sanitize NODMMAS: No > Controller ID: 0x0005 > Version: 1.4.0 > Traffic Based Keep Alive: Not Supported > Controller Type: I/O Controller > Keep Alive Timer Not Supported > Maximum Outstanding Commands Not Specified > > Admin Command Set Attributes > ============================ > Security Send/Receive: Supported > Format NVM: Supported > Firmware Activate/Download: Supported > Namespace Management: Not Supported > Device Self-test: Supported > Directives: Not Supported > NVMe-MI Send/Receive: Not Supported > Virtualization Management: Not Supported > Doorbell Buffer Config: Not Supported > Get LBA Status: Not Supported > Sanitize: block, > Abort Command Limit: 8 > Async Event Request Limit: 4 > Number of Firmware Slots: 3 > Firmware Slot 1 Read-Only: No > Per-Namespace SMART Log: Yes > Error Log Page Entries: 64 > Number of Power States: 5 > Total NVM Capacity: 256060514304 bytes > Unallocated NVM Capacity: 0 bytes > Firmware Update Granularity: 04 (16384 bytes) > Host Buffer Preferred Size: 67108864 bytes > Host Buffer Minimum Size: 16777216 bytes > > NVM Command Set Attributes > ========================== > Submission Queue Entry Size > Max: 64 > Min: 64 > Completion Queue Entry Size > Max: 16 > Min: 16 > Number of Namespaces: 1 > Compare Command: Supported > Write Uncorrectable Command: Supported > Dataset Management Command: Supported > Write Zeroes Command: Not Supported > Save Features: Supported > Reservations: Not Supported > Timestamp feature: Supported > Verify feature: Not Supported > Fused Operation Support: Not Supported > Format NVM Attributes: Per-NS Erase, Per-NS Format > Volatile Write Cache: Present, flush all > > NVM Subsystem Name: nqn.1994-11.com.samsung:nvme:PM991a:M.2:S65FNE1R713832 > > % doas nvmecontrol identify nvme0ns1 > Size: 500118192 blocks > Capacity: 500118192 blocks > Utilization: 484394000 blocks > Thin Provisioning: Not Supported > Number of LBA Formats: 1 > Current LBA Format: LBA Format #00 > Metadata Capabilities > Extended: Not Supported > Separate: Not Supported > Data Protection Caps: Not Supported > Data Protection Settings: Not Enabled > Multi-Path I/O Capabilities: Not Supported > Reservation Capabilities: Not Supported > Format Progress Indicator: 0% remains > Deallocate Logical Block: Read Not Reported > Optimal I/O Boundary: 0 blocks > NVM Capacity: 256060514304 bytes > Preferred Write Granularity: 32 blocks > Preferred Write Alignment: 8 blocks > Preferred Deallocate Granul: 10368 blocks > Preferred Deallocate Align: 10368 blocks > Optimal Write Size: 256 blocks > Globally Unique Identifier: 00000000000000000000000000000000 > IEEE EUI64: 002538d711133242 > LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: Best > > % sysctl hw.nvme > hw.nvme.verbose_cmd_dump: 1 > hw.nvme.use_nvd: 0 > > % sysctl dev.nvme > dev.nvme.0.alignment_splits: 0 > dev.nvme.0.ioq.3.dump_debug: 0 > dev.nvme.0.ioq.3.recovery: 0 > dev.nvme.0.ioq.3.num_recovery_nolock: 0 > dev.nvme.0.ioq.3.num_ignored: 0 > dev.nvme.0.ioq.3.num_failures: 0 > dev.nvme.0.ioq.3.num_retries: 0 > dev.nvme.0.ioq.3.num_intr_handler_calls: 10915 > dev.nvme.0.ioq.3.num_cmds: 11049 > dev.nvme.0.ioq.3.cq_head: 41 > dev.nvme.0.ioq.3.sq_tail: 41 > dev.nvme.0.ioq.3.sq_head: 41 > dev.nvme.0.ioq.3.num_trackers: 128 > dev.nvme.0.ioq.3.num_entries: 256 > dev.nvme.0.ioq.2.dump_debug: 0 > dev.nvme.0.ioq.2.recovery: 0 > dev.nvme.0.ioq.2.num_recovery_nolock: 0 > dev.nvme.0.ioq.2.num_ignored: 0 > dev.nvme.0.ioq.2.num_failures: 0 > dev.nvme.0.ioq.2.num_retries: 0 > dev.nvme.0.ioq.2.num_intr_handler_calls: 5321 > dev.nvme.0.ioq.2.num_cmds: 5410 > dev.nvme.0.ioq.2.cq_head: 34 > dev.nvme.0.ioq.2.sq_tail: 34 > dev.nvme.0.ioq.2.sq_head: 34 > dev.nvme.0.ioq.2.num_trackers: 128 > dev.nvme.0.ioq.2.num_entries: 256 > dev.nvme.0.ioq.1.dump_debug: 0 > dev.nvme.0.ioq.1.recovery: 0 > dev.nvme.0.ioq.1.num_recovery_nolock: 0 > dev.nvme.0.ioq.1.num_ignored: 0 > dev.nvme.0.ioq.1.num_failures: 0 > dev.nvme.0.ioq.1.num_retries: 0 > dev.nvme.0.ioq.1.num_intr_handler_calls: 10896 > dev.nvme.0.ioq.1.num_cmds: 11014 > dev.nvme.0.ioq.1.cq_head: 6 > dev.nvme.0.ioq.1.sq_tail: 6 > dev.nvme.0.ioq.1.sq_head: 6 > dev.nvme.0.ioq.1.num_trackers: 128 > dev.nvme.0.ioq.1.num_entries: 256 > dev.nvme.0.ioq.0.dump_debug: 0 > dev.nvme.0.ioq.0.recovery: 0 > dev.nvme.0.ioq.0.num_recovery_nolock: 0 > dev.nvme.0.ioq.0.num_ignored: 0 > dev.nvme.0.ioq.0.num_failures: 0 > dev.nvme.0.ioq.0.num_retries: 0 > dev.nvme.0.ioq.0.num_intr_handler_calls: 10407 > dev.nvme.0.ioq.0.num_cmds: 10503 > dev.nvme.0.ioq.0.cq_head: 7 > dev.nvme.0.ioq.0.sq_tail: 7 > dev.nvme.0.ioq.0.sq_head: 7 > dev.nvme.0.ioq.0.num_trackers: 128 > dev.nvme.0.ioq.0.num_entries: 256 > dev.nvme.0.adminq.dump_debug: 0 > dev.nvme.0.adminq.recovery: 0 > dev.nvme.0.adminq.num_recovery_nolock: 0 > dev.nvme.0.adminq.num_ignored: 0 > dev.nvme.0.adminq.num_failures: 0 > dev.nvme.0.adminq.num_retries: 0 > dev.nvme.0.adminq.num_intr_handler_calls: 225 > dev.nvme.0.adminq.num_cmds: 229 > dev.nvme.0.adminq.cq_head: 97 > dev.nvme.0.adminq.sq_tail: 101 > dev.nvme.0.adminq.sq_head: 101 > dev.nvme.0.adminq.num_trackers: 16 > dev.nvme.0.adminq.num_entries: 128 > dev.nvme.0.fail_on_reset: 0 > dev.nvme.0.cap_hi: 48 > dev.nvme.0.cap_lo: 1006845951 > dev.nvme.0.reset_stats: 0 > dev.nvme.0.num_recovery_nolock: 0 > dev.nvme.0.num_ignored: 0 > dev.nvme.0.num_failures: 0 > dev.nvme.0.num_retries: 0 > dev.nvme.0.num_intr_handler_calls: 37764 > dev.nvme.0.num_cmds: 38205 > dev.nvme.0.timeout_period: 30 > dev.nvme.0.admin_timeout_period: 60 > dev.nvme.0.int_coal_threshold: 0 > dev.nvme.0.int_coal_time: 0 > dev.nvme.0.num_io_queues: 4 > dev.nvme.0.wake: 0 > dev.nvme.0.%iommu: rid=0x200 > dev.nvme.0.%parent: pci1 > dev.nvme.0.%pnpinfo: vendor=0x144d device=0xa809 subvendor=0x144d > subdevice=0xa801 class=0x010802 > dev.nvme.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 > handle=\_SB_.PC00.PEG0.PEGP > dev.nvme.0.%driver: nvme > dev.nvme.0.%desc: Generic NVMe Device > dev.nvme.%parent: > ``` > > If there is more information I need to provide, let me know. > > I usually just wait for such things to work themselves out, and they > usually do over time, but given the call for testers I thought this > feedback might be useful... and it would be nice to have my power > sipping setup back. > > > Regards, > > Ross > > >