NVMe performance 4x slower than expected

[Tobias Oberstein]

>     > FreeBSD 11 Current with patches (DMAR and ZFS patches, otherwise the box
>     > doesn't boot at all .. because of 3TB RAM and the amount of periphery).
>
>     Do you still have WITNESS and INVARIANTS turned on in your kernel
>     config?  They're turned on by default for Current, but they do have
>     some performance impact.  To turn them off, just build a
>     GENERIC-NODEBUG kernel .

WITNESS is off, INVARIANTS is still on.

Here is complete config:

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_kernel_conf.md

This is the aggregated patch (work was done by Konstantin - thanks again 
btw!)

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_patch.md

> Could you also post full dmesg output as well as vmstat -i?

dmesg:

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_dmesg.md

vmstat:

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_vmstat.md

===

Here are results from FIO under FreeBSD:

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd.md

Here are results using _same_ FIO control file under Linux:

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/linux.md

===

The firmware for the P3700 cards was updated to the very latest as of 
today (using isdct under Linux).

/Tobias