NVMe performance 4x slower than expected

[Jim Harris]

On Wed, Apr 1, 2015 at 7:42 AM, Alan Somers <asomers at freebsd.org> wrote:

> On Wed, Apr 1, 2015 at 4:16 AM, Tobias Oberstein
> <tobias.oberstein at gmail.com> wrote:
> > Hi,
> >
> > I am testing performance of a NVMe device (Intel P3700) using FIO at the
> > block device level and get 4x slower performance than expected:
> >
> > 4kB Random Read
> >
> >              Intel Datasheet   FIO Measurement   Match
> > P3700        450,000           107,092           24%
> > DC S3700     75,000            67,186            90%
> >
> > The 2nd line are results for an Intel DC S3700 for comparison (with this
> > device, I do see the performance expected, but not for the P3700).
> >
> > Hardware:
> >
> > - 4 sockets, 48 core x86-64, 3TB RAM
> > - 8 x Intel P3700 2TB
> > - 12 x Intel DC S3700 800GB (via LSI HBAs)
> >
> > Software:
> >
> > FreeBSD 11 Current with patches (DMAR and ZFS patches, otherwise the box
> > doesn't boot at all .. because of 3TB RAM and the amount of periphery).
>
>
> Do you still have WITNESS and INVARIANTS turned on in your kernel
> config?  They're turned on by default for Current, but they do have
> some performance impact.  To turn them off, just build a
> GENERIC-NODEBUG kernel .
>
>
Could you also post full dmesg output as well as vmstat -i?


>
> >
> > Complete info and test logs are here:
> >
> >
> https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md
> >
> > Right now I am running Linux on the box (openSUSE 13.2). Using the exact
> > same FIO control file, the values for the DC S3700 are very close to
> > FreeBSD, but the values for the P3700 are much higher:
> >
> >
> https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md#more-numbers-linux
> >
> > I am looking for tuning hints or general advice for FreeBSD and NVMe.
> >
> > I would like to go with FreeBSD (a major aspect is ZFS), but the
> performance
> > issues with NVMe might be a deal breaker.
> >
> > Cheers,
> > /Tobias
> > _______________________________________________
> > freebsd-hackers at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> > To unsubscribe, send any mail to "
> freebsd-hackers-unsubscribe at freebsd.org"
>