NVMe performance 4x slower than expected

[Tobias Oberstein]

>     Is this vmstat after the test ?

No, it wasn't (I ran vmstat hours after the test).

Here is right after test (shortened test duration, otherwise exactly the 
same FIO config):

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_vmstat.md#nvd7

>     Somewhat funny is that nvme does not use MSI(X).
>
>
> Yes - this is exactly the problem.
>
> nvme does use MSI-X if it can allocate the vectors (one per core).  With
> 48 cores,
> I suspect we are quickly running out of vectors, so NVMe is reverting to
> INTx.
>
> Could you actually send vmstat -ia (I left off the 'a' previously) -
> just so we can
> see all allocated interrupt vectors.
>
> As an experiment, can you try disabling hyperthreading - this will
> reduce the

The CPUs in this box

root at s4l-zfs:~/src/sys/amd64/conf # sysctl hw.model
hw.model: Intel(R) Xeon(R) CPU E7-8857 v2 @ 3.00GHz

don't have hyperthreading (we deliberately selected CPU model for max. 
clock rather than HT)

http://ark.intel.com/products/75254/Intel-Xeon-Processor-E7-8857-v2-30M-Cache-3_00-GHz

> number of cores and should let you get MSI-X vectors allocated for at least
> the first couple of NVMe controllers.  Then please re-run your performance
> test on one of those controllers.
>

You mean I should run against nvdN where N is a controller that still 
got MSI-X while other controllers did not?

How would I find out which controller N? I don't know which nvdN is 
mounted in a PCIe slot directly assigned to which CPU socket, and I 
don't know which one's still got MSI-X and which not.

I could arrange for disabling all but 1 CPU and retest. Would that help?

===

Right after running against nvd7

irq56: nvme0                        6440          0
...
irq106: nvme7                     145056          3


Then, immediately thereafter, running against nvd0

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_vmstat.md#nvd0

irq56: nvme0                        9233          0
...
irq106: nvme7                     145056          3

===

Earlier this day, I ran multiple longer tests .. all against nvd7. So if 
these are cumulative numbers since last boot, that would make sense.