Poor performance on Intel P3600 NVME driver

[Slawa Olhovchenkov]

On Fri, Jan 16, 2015 at 10:42:28AM +0200, Mihai Vintila wrote:

> Actually seems that i've was wrong vfs.zfs.top_maxinflight does not have 
> any influence and this is confirmed by the nvmecontrol perftests.
>   nvmecontrol perftest -n 32 -o read -s 512 -t30 nvme0ns1
> Threads: 32 Size:    512  READ Time:  30 IO/s:  270212 MB/s:  131
> nvmecontrol perftest -n 32 -o write -s 512 -t30 nvme0ns1
> Threads: 32 Size:    512 WRITE Time:  30 IO/s:   13658 MB/s:    6

I see datasheet specified performance metrics for 4K (not 512bytes) random r/w.
Can you try this? This is may be impact write IO/s at least.

> I was able to recover from the errors from previous message. They were 
> cause by the fact that i've commented hw.nvme.per_cpu_io_queues=0 in 
> loader.conf . After setting it back things got back to "slow normal"
> Performance is half of what it should be and  issue seems to be the nvme 
> driver. I'll try it on another OS to confirm it's not a hardware setting 
> issue.