Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

[Warner Losh]

On Fri, Feb 22, 2019 at 8:04 AM Alan Somers <asomers at freebsd.org> wrote:

> On Fri, Feb 22, 2019 at 5:29 AM Rebecca Cran via freebsd-hackers
> <freebsd-hackers at freebsd.org> wrote:
> >
> > On 2/22/19 1:51 AM, Rajesh Kumar wrote:
> > >     1. Should we use "posixaio" as the ioengine (or) something else?
> > >     2. Should we use single thread (or) multiple threads for test? If
> > >     multiple threads, how can we decide on the optimal thread count?
> > >     3. Should we use "raw device files" (Eg: nvme namespace file -
> > >     /dev/nvme0ns1) without filesystem (or) use a mounted filesystem
> with a
> > >     regular file (Eg: /mnt/nvme/test1). Looks like raw device files
> give better
> > >     numbers.
> > >     4. Should we use a shared file (or) one file per thread?
> > >     5. I believe 1Job should be fine for benchmarking. (or) should we
> try
> > >     multiple jobs?
> >
> >
> > I just ran a quick test on a filesystem on my machine which has an M.2
> > NVMe drive, and it seems posixaio performs pretty poorly compared to the
> > sync ioengine: around 700 MB/s vs. 1100 MB/s!
>
> When AIO is run on a filesystem, it uses an internal thread pool to
> process requests.  But if you run it on a bare drive, then the I/O is
> direct and should be faster than the sync ioengine.
>


Here's the script I typically use to get first pass raw numbers. The key to
getting good benchmark numbers is getting as many I/O requests on the
device as possible. At present, posixaio is the only thing outside of the
kernel that can do that. I usually get close to datasheet numbers using
this test:

; SSD testing: 128k I/O 64 jobs 32 deep queue

[global]
direct=1
rw=randread
refill_buffers
norandommap
randrepeat=0
bs=128k
ioengine=posixaio
iodepth=128
numjobs=64
runtime=30
group_reporting
thread

[nvme128k]

And I agree with Alan: to get the best numbers, you need to go to the raw
device. I've not cared about from userland I/O performance, so I've not
looked at making the UFS or ZFS cases work better in fio.

Warner

>
> > I _was_ going to suggest using posixaio and setting iodepth to something
> > like 32, but since it performs badly I'd suggest playing around with the
> > numjobs parameter and seeing where the best performance is achieved -
> > whether that's latency or throughput.
> >
> >
> > On my system, single-threaded achieves ~530 MB/s, 8 jobs/threads 1150
> > MB/s and 32 1840 MB/s with a 4 KB block size.
> >
> > Bumping the block size from 4 KB to 16 KB makes the throughput more
> > jumpy, but appears to average 2300 MB/s when used with 32 jobs.
> >
> >
> > --
> > Rebecca Cran
> >
> > _______________________________________________
> > freebsd-hackers at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> > To unsubscribe, send any mail to "
> freebsd-hackers-unsubscribe at freebsd.org"
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"
>