Poor performance on Intel P3600 NVME driver

[Barney Wolff]

Apology if I've missed a mention, but are you sure there aren't zfs snapshots
in existence, and things like access time updates differing between Linux
and FreeBSD?

On Fri, Jan 16, 2015 at 07:08:56PM +0200, Mihai Vintila wrote:
> There are clearly issues with the controller driver on FreeBSD. Not only 
> zfs related:
> I've did some testing under Centos 7 with ext4
>          Command line used: iozone -Rb /root/output_iops.wks -O -i 0 -i 
> 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -r 512K -s 8G
>          Time Resolution = 0.000001 seconds.
>          Processor cache size set to 1024 kBytes.
>          Processor cache line size set to 32 bytes.
>          File stride size set to 17 * record size.
> random  random    bkwd   record   stride
>                kB  reclen   write rewrite    read    reread read   
> write    read  rewrite     read   fwrite frewrite   fread freread
>           8388608       4  156982       0   859809        0 656572  208156
>           8388608       8   82545       0   503549        0 435005  111282
>           8388608      32   21620       0   129740        0 135738   31361
>           8388608      64   11070       0    66027        0 71660   16175
>           8388608     128    5558       0    31800        0 35361    8066
>           8388608     512    1383       0     7520        0 8472    2068
> 
> 
> Same test on BSD with ufs
>          Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 
> 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G
>          Time Resolution = 0.000001 seconds.
>          Processor cache size set to 1024 Kbytes.
>          Processor cache line size set to 32 bytes.
>          File stride size set to 17 * record size.
> random  random    bkwd   record   stride
>                KB  reclen   write rewrite    read    reread read   
> write    read  rewrite     read   fwrite frewrite   fread freread
>           1048576       4  120642       0   368188        0 252302   41650
>           1048576       8   61871       0   253074        0 189851   35015
>           1048576      32   15925       0    89190        0 80004   19357
>           1048576      64    7958       0    47190        0 59120   10070
>           1048576     128    4024       0    23619        0 22435    5542
> 
> 
> ZFS test on BSD:
>          Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 
> 2 -e -+n -r4K -s 1G
>          Time Resolution = 0.000001 seconds.
>          Processor cache size set to 1024 Kbytes.
>          Processor cache line size set to 32 bytes.
>          File stride size set to 17 * record size.
> random  random    bkwd   record   stride
>                KB  reclen   write rewrite    read    reread read   
> write    read  rewrite     read   fwrite frewrite   fread freread
>           1048576       4   69600       0    99750        0 92007   49556
> 
> 
> 
> Perftest while write is close to what it should be, read takes a 4x penalty
>   nvmecontrol perftest -n 32 -o write -s 4096 -t30 nvme0ns1
> Threads: 32 Size:   4096 WRITE Time:  30 IO/s:  211294 MB/s:  825
>   nvmecontrol perftest -n 32 -o read -s 4096 -t30 nvme0ns1
> Threads: 32 Size:   4096  READ Time:  30 IO/s:  221365 MB/s:  864