NVMe performance 4x slower than expected

[Tobias Oberstein]

Hi,

I am testing performance of a NVMe device (Intel P3700) using FIO at the 
block device level and get 4x slower performance than expected:

4kB Random Read

              Intel Datasheet   FIO Measurement   Match
P3700        450,000           107,092           24%
DC S3700     75,000            67,186            90%

The 2nd line are results for an Intel DC S3700 for comparison (with this 
device, I do see the performance expected, but not for the P3700).

Hardware:

- 4 sockets, 48 core x86-64, 3TB RAM
- 8 x Intel P3700 2TB
- 12 x Intel DC S3700 800GB (via LSI HBAs)

Software:

FreeBSD 11 Current with patches (DMAR and ZFS patches, otherwise the box 
doesn't boot at all .. because of 3TB RAM and the amount of periphery).

Complete info and test logs are here:

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md

Right now I am running Linux on the box (openSUSE 13.2). Using the exact 
same FIO control file, the values for the DC S3700 are very close to 
FreeBSD, but the values for the P3700 are much higher:

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md#more-numbers-linux

I am looking for tuning hints or general advice for FreeBSD and NVMe.

I would like to go with FreeBSD (a major aspect is ZFS), but the 
performance issues with NVMe might be a deal breaker.

Cheers,
/Tobias