NVMe performance 4x slower than expected

[Tobias Oberstein]

Hi Adrian,

 > Dell has graciously loaned me a bunch of hardware to continue doing

FWIW, Dell has a roughly comparable system: Dell R920. But they don't 
have Intel NVMe's on their menu, only Samsung (and FusionIO, but that's 
not NVMe).

> NUMA development on, but I have no NVMe hardware. I'm hoping people at

The 8 NVMe PCIe SSDs in the box we're deploying are a key feature of 
this system (will be a data-warehouse). A single NVMe probably won't 
have triggered (all) issues we experienced.

We are using the largest model (2TB), and this amounts to 50k bucks for 
all eight. The smallest model (400GB) is 1.5k, so 12k in total.

> Intel can continue kicking along any desires for NUMA that they
> require. (Which they have, fwiw.)

It's already awesome that Intel has senior engineers working on FreeBSD 
driver code! And it would underline Intel's Open-source commitment and 
tech leadership if they donated a couple of these beefy NVMes.

The specs are incredible, but extracing all the performance is 
non-trivial ..

Cheers,
/Tobias