Poor performance on Intel P3600 NVME driver

[Mihai Vintila]

There are clearly issues with the controller driver on FreeBSD. Not only 
zfs related:
I've did some testing under Centos 7 with ext4
         Command line used: iozone -Rb /root/output_iops.wks -O -i 0 -i 
1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -r 512K -s 8G
         Time Resolution = 0.000001 seconds.
         Processor cache size set to 1024 kBytes.
         Processor cache line size set to 32 bytes.
         File stride size set to 17 * record size.
random  random    bkwd   record   stride
               kB  reclen   write rewrite    read    reread read   
write    read  rewrite     read   fwrite frewrite   fread freread
          8388608       4  156982       0   859809        0 656572  208156
          8388608       8   82545       0   503549        0 435005  111282
          8388608      32   21620       0   129740        0 135738   31361
          8388608      64   11070       0    66027        0 71660   16175
          8388608     128    5558       0    31800        0 35361    8066
          8388608     512    1383       0     7520        0 8472    2068


Same test on BSD with ufs
         Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 
2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G
         Time Resolution = 0.000001 seconds.
         Processor cache size set to 1024 Kbytes.
         Processor cache line size set to 32 bytes.
         File stride size set to 17 * record size.
random  random    bkwd   record   stride
               KB  reclen   write rewrite    read    reread read   
write    read  rewrite     read   fwrite frewrite   fread freread
          1048576       4  120642       0   368188        0 252302   41650
          1048576       8   61871       0   253074        0 189851   35015
          1048576      32   15925       0    89190        0 80004   19357
          1048576      64    7958       0    47190        0 59120   10070
          1048576     128    4024       0    23619        0 22435    5542


ZFS test on BSD:
         Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 
2 -e -+n -r4K -s 1G
         Time Resolution = 0.000001 seconds.
         Processor cache size set to 1024 Kbytes.
         Processor cache line size set to 32 bytes.
         File stride size set to 17 * record size.
random  random    bkwd   record   stride
               KB  reclen   write rewrite    read    reread read   
write    read  rewrite     read   fwrite frewrite   fread freread
          1048576       4   69600       0    99750        0 92007   49556



Perftest while write is close to what it should be, read takes a 4x penalty
  nvmecontrol perftest -n 32 -o write -s 4096 -t30 nvme0ns1
Threads: 32 Size:   4096 WRITE Time:  30 IO/s:  211294 MB/s:  825
  nvmecontrol perftest -n 32 -o read -s 4096 -t30 nvme0ns1
Threads: 32 Size:   4096  READ Time:  30 IO/s:  221365 MB/s:  864



What i've tried for zfs:
vfs.zfs.vdev.sync_read_min_active: 32
vfs.zfs.vdev.sync_read_max_active: 32
vfs.zfs.vdev.sync_write_min_active: 32
vfs.zfs.vdev.sync_write_max_active: 32
vfs.zfs.vdev.async_read_min_active: 32
vfs.zfs.vdev.async_read_max_active: 32
vfs.zfs.vdev.async_write_min_active: 32
vfs.zfs.vdev.async_write_max_active: 32


No effect what so ever in tests.
increasing vfs.zfs.top_maxinflight  - > no effect


While i've managed to make it a little bit more stable so that 
controller doesn't just disappear by disabling ASPM in bios i still get :
nvme0: resetting controller
and after this i have to reboot.

Some other errors i had today which where fixed with ssd firmware update 
and by forcing PCIeX to 16x :

1511.530545] Buffer I/O error on device nvme0n1, logical block 0
[ 1511.530594] lost page write due to I/O error on nvme0n1
[ 1517.694014] Buffer I/O error on device nvme1n1, logical block 0
[ 1517.694064] lost page write due to I/O error on nvme1n1



Best regards,
Vintila Mihai Alexandru

On 1/15/2015 7:59 PM, Slawa Olhovchenkov wrote:
> On Thu, Jan 15, 2015 at 07:22:49PM +0200, Mihai Vintila wrote:
>
>> /etc/sysctl.conf
>>
>> net.inet.tcp.fast_finwait2_recycle=1
>> net.inet.ip.portrange.randomized=0
>> net.inet.ip.portrange.first=1024
>> net.inet.ip.portrange.last=65535
>> net.inet.tcp.recvbuf_max=16777216
>> net.inet.tcp.sendbuf_max=16777216
>> net.inet.tcp.recvbuf_inc=65536
>> vfs.zfs.vdev.trim_on_init=0
>> #close time_wait connections at 2*7500ms
>> net.inet.tcp.msl=7500
>> kern.ipc.somaxconn=4096
>> net.inet.icmp.icmplim=2000
>>
>> #zfs
>> vfs.zfs.txg.timeout=5
>> vfs.zfs.prefetch_disable=1
>
>> Any help to bring this device to proper speed will be welcomed.
> Do you try to increase vfs.zfs.top_maxinflight?