NVME aborting outstanding i/o

[Patrick M. Hausen]

Hi all,

I’m currently doing some load tests/burn in for two new servers.
These feature all NVME SSDs and run FreeNAS, i.e. FreeBSD 11.2-STABLE.

	pcib17: <ACPI PCI-PCI bridge> at device 3.2 numa-domain 1 on pci15
	pcib17: [GIANT-LOCKED]
	pci17: <ACPI PCI bus> numa-domain 1 on pcib17
	nvme7: <Generic NVMe Device> mem 0xeca10000-0xeca13fff at device 0.0 numa-domain 1 on pci17

When putting some moderate i/o load on the system, the log fills with these
messages:

	nvme7: aborting outstanding i/o
	nvme7: DATASET MANAGEMENT sqid:41 cid:91 nsid:1
	nvme7: ABORTED - BY REQUEST (00/07) sqid:41 cid:91 cdw0:0

There has been some discussion of this on on the iX Systems forum as well as various
FreeBSD media and one person suggested setting:

	hw.nvme.per_cpu_io_queues=0


This is where I need some help now. This is from the manpage for nvme(4):

----------
    To force a single I/O queue pair shared by all CPUs, set the following
    tunable value in loader.conf(5):

          hw.nvme.per_cpu_io_queues=0

    To assign more than one CPU per I/O queue pair, thereby reducing the
    number of MSI-X vectors consumed by the device, set the following tunable
    value in loader.conf(5):

          hw.nvme.min_cpus_per_ioq=X

    To force legacy interrupts for all nvme driver instances, set the
    following tunable value in loader.conf(5):

          hw.nvme.force_intx=1

    Note that use of INTx implies disabling of per-CPU I/O queue pairs.
----------

But:

	root at freenas01[~]# sysctl hw.nvme.per_cpu_io_queues
	sysctl: unknown oid 'hw.nvme.per_cpu_io_queues'
	root at freenas01[~]# sysctl hw.nvme.min_cpus_per_ioq
	sysctl: unknown oid 'hw.nvme.min_cpus_per_ioq'
	root at freenas01[~]# sysctl hw.nvme.force_intx
	sysctl: unknown oid 'hw.nvme.force_intx'


Where do I go from here?

Thanks!
Patrick
-- 
punkt.de GmbH			Internet - Dienstleistungen - Beratung
Kaiserallee 13a			Tel.: 0721 9109-0 Fax: -100
76133 Karlsruhe			info at punkt.de	http://punkt.de
AG Mannheim 108285		Gf: Juergen Egeling