Re: Call for testers with NVMe
- Reply: S. Ross Gohlke: "Re: Call for testers with NVMe"
- In reply to: S. Ross Gohlke: "Re: Call for testers with NVMe"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Mon, 21 Sep 2026 23:18:11 UTC
On 9/15/26 07:12, S. Ross Gohlke wrote: > > > On 9/14/26 11:31, Kevin Bowling wrote: >> On Sun, Sep 13, 2026 at 9:41 AM S. Ross Gohlke<ross@bisd.ro> wrote: >>> On 9/13/26 08:10, Abdelkader Boudih wrote: >>>> Hi Again >>>> >>>> The last round endup with 80% of the Diffs merged and 50% already MFCed. >>>> >>>> imp@ still want to try one diff in a Netflix server... >>>> So if there is an outage that spoil your favorite show, I'm the one to >>>> blame. :) >>>> >>>> Marcin Cieslak reported a wedge in Apple S3x . (I still trying to >>>> reproduce it, but maybe this stack fix it) >>>> >>>> New round of NVME testing : >>>> >>>> D59637 nvme: do not touch INTMS/INTMC when configured for MSI-X >>>> D59636 nvme: enforce the Abort Command Limit >>>> D59634 nvme: do not complete a command when its Abort is not performed >>>> D59633 nvme: delete the I/O queues in the system shutdown path >>>> D59631 nvme: honor CRTO for the controller ready timeout >>>> D59628 nvme: derive CC.CSS from CAP.CSS instead of hardcoding the NVM set >>>> D59627 nvme: honor FLBAS Format Index Upper when selecting the LBA format >>>> D59626 nvme: bound the AER error log byte-swap by the fetched length >>>> D59625 nvme: reject namespaces formatted with metadata >>>> >>>> This all extracted fromhttps://nvmexpress.org/specifications/ (Spec >>>> 2.4 , August 2026) >>>> Link to the top of the stackhttps://reviews.freebsd.org/D59636 >>>> >>>> Mostly Spec review, previous versions of the spec was either vague, or >>>> non-existant and nobody touched the code since then. >>>> >>>> I request more people to test it so we add Quirks to drives that are >>>> not spec compliant, but first we need to get the driver to spec >>>> complianciancy first. >>>> >>>> I tested this on my fleet for 10 days, and the machines are daily >>>> building kernels and running as CI. >>>> >>>> Best, >>>> Abdelkader >>>> >>> I upgraded CURRENT via pkgbase last night, from 1600021 to 1600025. >>> * FreeBSD-src-16.snap20260912223212 >>> * FreeBSD-utilities-16.snap20260912182422 >>> I run a custom kernel based on MINIMAL_MMCCAM, but nvme is in the kernel. >>> ``` >>> % sysctl kern.conftxt | grep nvm >>> device nvme >>> ``` >>> Also, I tested with GENERIC kernel and got the same result. >>> >>> System is ThinkPad L14 Gen 2 Tiger Lake. >>> >>> I have everything working as expected, but intr shows unusually high >>> utilization, around >>> 50% of a core, constantly, in top(1). >> Did you identify _which_ intr? It sounds like it could be a22eb754c4bb. > > I guess I don't know how to identify _which_ intr. > > I am seeing no output in the usual places -- dmesg, /var/log/messages. > > I tried enabling hw.nvme.verbose_cmd_dump: 1, but that makes no > difference. > > Maybe I am wrong about nvme being the cause, I just ran out of things > to check. > > All I know is, this is definitely new behavior and I have exhausted my > troubleshooting playbook. > > From "top -PCS": > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 11 root 8 199 ki31 0B 128K RUN 0 60.2H 788.76% > idle > 12 root 29 -54 - 0B 464K WAIT 0 290:43 60.54% intr > > If there is something else to check related to intr, please let me > know which manual page to start reading. > > Also, I will look for a22eb754c4bb. Thanks for the pointer. > >>> I tried everything I could think of, with no change. >>> * unload kernel modules >>> * stop all services >>> Then remembered the nvme upgrades. >>> I think this is the cause of the problem. >>> >>> I don't know anything substantial about storage hardware in general or >>> nvme in specific. >>> I tried different values in loader.conf(5) for some tunables mentioned >>> in nvme(4) that >>> seemed plausibly related. >>> * hw.nvme.apst_enable >>> * hw.nvme.force_intx >>> They do not even show up in sysctl(8) after reboot. >>> >>> ``` >>> % doas nvmecontrol devlist >>> nvme0: SAMSUNG MZALQ256HBJD-00BL2 >>> nvme0ns1 (244198MB) >>> >>> % doas pciconf -lv >>> ... >>> nvme0@pci0:2:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d >>> device=0xa809 subvendor=0x144d subdevice=0xa801 >>> vendor = 'Samsung Electronics Co Ltd' >>> device = 'NVMe SSD Controller 980 (DRAM-less)' >>> class = mass storage >>> subclass = NVM >>> ... >>> >>> % doas nvmecontrol identify nvme0 >>> Controller Capabilities/Features >>> ================================ >>> Vendor ID: 144d >>> Subsystem Vendor ID: 144d >>> Serial Number: S65FNE1R713832 >>> Model Number: SAMSUNG MZALQ256HBJD-00BL2 >>> Firmware Version: 7L2QFXM7 >>> Recommended Arb Burst: 2 >>> IEEE OUI Identifier: 00 25 38 >>> Multi-Path I/O Capabilities: Not Supported >>> Max Data Transfer Size: 2097152 bytes >>> Sanitize Crypto Erase: Not Supported >>> Sanitize Block Erase: Supported >>> Sanitize Overwrite: Not Supported >>> Sanitize NDI: Supported >>> Sanitize NODMMAS: No >>> Controller ID: 0x0005 >>> Version: 1.4.0 >>> Traffic Based Keep Alive: Not Supported >>> Controller Type: I/O Controller >>> Keep Alive Timer Not Supported >>> Maximum Outstanding Commands Not Specified >>> >>> Admin Command Set Attributes >>> ============================ >>> Security Send/Receive: Supported >>> Format NVM: Supported >>> Firmware Activate/Download: Supported >>> Namespace Management: Not Supported >>> Device Self-test: Supported >>> Directives: Not Supported >>> NVMe-MI Send/Receive: Not Supported >>> Virtualization Management: Not Supported >>> Doorbell Buffer Config: Not Supported >>> Get LBA Status: Not Supported >>> Sanitize: block, >>> Abort Command Limit: 8 >>> Async Event Request Limit: 4 >>> Number of Firmware Slots: 3 >>> Firmware Slot 1 Read-Only: No >>> Per-Namespace SMART Log: Yes >>> Error Log Page Entries: 64 >>> Number of Power States: 5 >>> Total NVM Capacity: 256060514304 bytes >>> Unallocated NVM Capacity: 0 bytes >>> Firmware Update Granularity: 04 (16384 bytes) >>> Host Buffer Preferred Size: 67108864 bytes >>> Host Buffer Minimum Size: 16777216 bytes >>> >>> NVM Command Set Attributes >>> ========================== >>> Submission Queue Entry Size >>> Max: 64 >>> Min: 64 >>> Completion Queue Entry Size >>> Max: 16 >>> Min: 16 >>> Number of Namespaces: 1 >>> Compare Command: Supported >>> Write Uncorrectable Command: Supported >>> Dataset Management Command: Supported >>> Write Zeroes Command: Not Supported >>> Save Features: Supported >>> Reservations: Not Supported >>> Timestamp feature: Supported >>> Verify feature: Not Supported >>> Fused Operation Support: Not Supported >>> Format NVM Attributes: Per-NS Erase, Per-NS Format >>> Volatile Write Cache: Present, flush all >>> >>> NVM Subsystem Name: nqn.1994-11.com.samsung:nvme:PM991a:M.2:S65FNE1R713832 >>> >>> % doas nvmecontrol identify nvme0ns1 >>> Size: 500118192 blocks >>> Capacity: 500118192 blocks >>> Utilization: 484394000 blocks >>> Thin Provisioning: Not Supported >>> Number of LBA Formats: 1 >>> Current LBA Format: LBA Format #00 >>> Metadata Capabilities >>> Extended: Not Supported >>> Separate: Not Supported >>> Data Protection Caps: Not Supported >>> Data Protection Settings: Not Enabled >>> Multi-Path I/O Capabilities: Not Supported >>> Reservation Capabilities: Not Supported >>> Format Progress Indicator: 0% remains >>> Deallocate Logical Block: Read Not Reported >>> Optimal I/O Boundary: 0 blocks >>> NVM Capacity: 256060514304 bytes >>> Preferred Write Granularity: 32 blocks >>> Preferred Write Alignment: 8 blocks >>> Preferred Deallocate Granul: 10368 blocks >>> Preferred Deallocate Align: 10368 blocks >>> Optimal Write Size: 256 blocks >>> Globally Unique Identifier: 00000000000000000000000000000000 >>> IEEE EUI64: 002538d711133242 >>> LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: Best >>> >>> % sysctl hw.nvme >>> hw.nvme.verbose_cmd_dump: 1 >>> hw.nvme.use_nvd: 0 >>> >>> % sysctl dev.nvme >>> dev.nvme.0.alignment_splits: 0 >>> dev.nvme.0.ioq.3.dump_debug: 0 >>> dev.nvme.0.ioq.3.recovery: 0 >>> dev.nvme.0.ioq.3.num_recovery_nolock: 0 >>> dev.nvme.0.ioq.3.num_ignored: 0 >>> dev.nvme.0.ioq.3.num_failures: 0 >>> dev.nvme.0.ioq.3.num_retries: 0 >>> dev.nvme.0.ioq.3.num_intr_handler_calls: 10915 >>> dev.nvme.0.ioq.3.num_cmds: 11049 >>> dev.nvme.0.ioq.3.cq_head: 41 >>> dev.nvme.0.ioq.3.sq_tail: 41 >>> dev.nvme.0.ioq.3.sq_head: 41 >>> dev.nvme.0.ioq.3.num_trackers: 128 >>> dev.nvme.0.ioq.3.num_entries: 256 >>> dev.nvme.0.ioq.2.dump_debug: 0 >>> dev.nvme.0.ioq.2.recovery: 0 >>> dev.nvme.0.ioq.2.num_recovery_nolock: 0 >>> dev.nvme.0.ioq.2.num_ignored: 0 >>> dev.nvme.0.ioq.2.num_failures: 0 >>> dev.nvme.0.ioq.2.num_retries: 0 >>> dev.nvme.0.ioq.2.num_intr_handler_calls: 5321 >>> dev.nvme.0.ioq.2.num_cmds: 5410 >>> dev.nvme.0.ioq.2.cq_head: 34 >>> dev.nvme.0.ioq.2.sq_tail: 34 >>> dev.nvme.0.ioq.2.sq_head: 34 >>> dev.nvme.0.ioq.2.num_trackers: 128 >>> dev.nvme.0.ioq.2.num_entries: 256 >>> dev.nvme.0.ioq.1.dump_debug: 0 >>> dev.nvme.0.ioq.1.recovery: 0 >>> dev.nvme.0.ioq.1.num_recovery_nolock: 0 >>> dev.nvme.0.ioq.1.num_ignored: 0 >>> dev.nvme.0.ioq.1.num_failures: 0 >>> dev.nvme.0.ioq.1.num_retries: 0 >>> dev.nvme.0.ioq.1.num_intr_handler_calls: 10896 >>> dev.nvme.0.ioq.1.num_cmds: 11014 >>> dev.nvme.0.ioq.1.cq_head: 6 >>> dev.nvme.0.ioq.1.sq_tail: 6 >>> dev.nvme.0.ioq.1.sq_head: 6 >>> dev.nvme.0.ioq.1.num_trackers: 128 >>> dev.nvme.0.ioq.1.num_entries: 256 >>> dev.nvme.0.ioq.0.dump_debug: 0 >>> dev.nvme.0.ioq.0.recovery: 0 >>> dev.nvme.0.ioq.0.num_recovery_nolock: 0 >>> dev.nvme.0.ioq.0.num_ignored: 0 >>> dev.nvme.0.ioq.0.num_failures: 0 >>> dev.nvme.0.ioq.0.num_retries: 0 >>> dev.nvme.0.ioq.0.num_intr_handler_calls: 10407 >>> dev.nvme.0.ioq.0.num_cmds: 10503 >>> dev.nvme.0.ioq.0.cq_head: 7 >>> dev.nvme.0.ioq.0.sq_tail: 7 >>> dev.nvme.0.ioq.0.sq_head: 7 >>> dev.nvme.0.ioq.0.num_trackers: 128 >>> dev.nvme.0.ioq.0.num_entries: 256 >>> dev.nvme.0.adminq.dump_debug: 0 >>> dev.nvme.0.adminq.recovery: 0 >>> dev.nvme.0.adminq.num_recovery_nolock: 0 >>> dev.nvme.0.adminq.num_ignored: 0 >>> dev.nvme.0.adminq.num_failures: 0 >>> dev.nvme.0.adminq.num_retries: 0 >>> dev.nvme.0.adminq.num_intr_handler_calls: 225 >>> dev.nvme.0.adminq.num_cmds: 229 >>> dev.nvme.0.adminq.cq_head: 97 >>> dev.nvme.0.adminq.sq_tail: 101 >>> dev.nvme.0.adminq.sq_head: 101 >>> dev.nvme.0.adminq.num_trackers: 16 >>> dev.nvme.0.adminq.num_entries: 128 >>> dev.nvme.0.fail_on_reset: 0 >>> dev.nvme.0.cap_hi: 48 >>> dev.nvme.0.cap_lo: 1006845951 >>> dev.nvme.0.reset_stats: 0 >>> dev.nvme.0.num_recovery_nolock: 0 >>> dev.nvme.0.num_ignored: 0 >>> dev.nvme.0.num_failures: 0 >>> dev.nvme.0.num_retries: 0 >>> dev.nvme.0.num_intr_handler_calls: 37764 >>> dev.nvme.0.num_cmds: 38205 >>> dev.nvme.0.timeout_period: 30 >>> dev.nvme.0.admin_timeout_period: 60 >>> dev.nvme.0.int_coal_threshold: 0 >>> dev.nvme.0.int_coal_time: 0 >>> dev.nvme.0.num_io_queues: 4 >>> dev.nvme.0.wake: 0 >>> dev.nvme.0.%iommu: rid=0x200 >>> dev.nvme.0.%parent: pci1 >>> dev.nvme.0.%pnpinfo: vendor=0x144d device=0xa809 subvendor=0x144d >>> subdevice=0xa801 class=0x010802 >>> dev.nvme.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 >>> handle=\_SB_.PC00.PEG0.PEGP >>> dev.nvme.0.%driver: nvme >>> dev.nvme.0.%desc: Generic NVMe Device >>> dev.nvme.%parent: >>> ``` >>> >>> If there is more information I need to provide, let me know. >>> >>> I usually just wait for such things to work themselves out, and they >>> usually do over time, but given the call for testers I thought this >>> feedback might be useful... and it would be nice to have my power >>> sipping setup back. >>> >>> >>> Regards, >>> >>> Ross >>> >>> >>> I upgraded pkgbase yesterday, from 1600025 to 1600026. * FreeBSD-src-16.snap20260920013406 * FreeBSD-utilities-16.snap20260919182608 intr utilization is down to 3-5% from 50-60%. Thanks. Regards, Ross