Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

[Rebecca Cran]

On 2/22/19 1:51 AM, Rajesh Kumar wrote:
>     1. Should we use "posixaio" as the ioengine (or) something else?
>     2. Should we use single thread (or) multiple threads for test? If
>     multiple threads, how can we decide on the optimal thread count?
>     3. Should we use "raw device files" (Eg: nvme namespace file -
>     /dev/nvme0ns1) without filesystem (or) use a mounted filesystem with a
>     regular file (Eg: /mnt/nvme/test1). Looks like raw device files give better
>     numbers.
>     4. Should we use a shared file (or) one file per thread?
>     5. I believe 1Job should be fine for benchmarking. (or) should we try
>     multiple jobs?


I just ran a quick test on a filesystem on my machine which has an M.2 
NVMe drive, and it seems posixaio performs pretty poorly compared to the 
sync ioengine: around 700 MB/s vs. 1100 MB/s!

I _was_ going to suggest using posixaio and setting iodepth to something 
like 32, but since it performs badly I'd suggest playing around with the 
numjobs parameter and seeing where the best performance is achieved - 
whether that's latency or throughput.


On my system, single-threaded achieves ~530 MB/s, 8 jobs/threads 1150 
MB/s and 32 1840 MB/s with a 4 KB block size.

Bumping the block size from 4 KB to 16 KB makes the throughput more 
jumpy, but appears to average 2300 MB/s when used with 32 jobs.


-- 
Rebecca Cran