Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

[Rajesh Kumar]

Hi,

I am trying to run FIO benchmark test with NVMe devices and see how FreeBSD
performs. There are lot of variables and combination. So, can anyone
suggest a Ideal way to do FIO benchmarking in FreeBSD? My intent is to
check what is the maximum throughput and IOPS the device delivers in
FreeBSD.

Few questions regarding the same,

   1. Should we use "posixaio" as the ioengine (or) something else?
   2. Should we use single thread (or) multiple threads for test? If
   multiple threads, how can we decide on the optimal thread count?
   3. Should we use "raw device files" (Eg: nvme namespace file -
   /dev/nvme0ns1) without filesystem (or) use a mounted filesystem with a
   regular file (Eg: /mnt/nvme/test1). Looks like raw device files give better
   numbers.
   4. Should we use a shared file (or) one file per thread?
   5. I believe 1Job should be fine for benchmarking. (or) should we try
   multiple jobs?

Please let me know your suggestions. Also, please suggest performance
tuning methods for NVMe and storage devices in general.

Thanks,
Rajesh.