NVMe performance 4x slower than expected

[Tobias Oberstein]

Hi Jim,

I have now done extensive tests under Linux (SLES12) at the block device
level.

8kB Random IO results:
http://tavendo.com.s3.amazonaws.com/scratch/fio_p3700_8kB_random.pdf

All results:
http://tavendo.com.s3.amazonaws.com/scratch/fio_p3700.pdf

What becomes apparent is:

1) IOPS is scaling nicely "linear" for (software) RAID-0

It scales up to roughly 2.2 Mio. 8kB random reads, and 750k 8kB random
writes. Extrapolating Intel's datasheet would give: 2.36 Mio / 720k

Awesome!

2) It does not scale for RAID-1.

In fact, the write performance fully collapses for more than 4 devices.

Note: I don't know which NVMe is wired to which CPU socket, and which
block device - IOW: I did not "handplace" the devices into RAID sets or
anything.

==

I am currently running the same set of tests against 10 DC S3700 via SAS.

This should reveal if it's a general mdadm thing, or NVMe related.

==

For now, we likely will use the NVMes in a RAID-0 setup to leverage the
maximum performance.

Cheers,
/Tobias


Am 10.04.2015 um 18:58 schrieb Jim Harris:
> On Fri, Apr 10, 2015 at 9:07 AM, Tobias Oberstein <
> tobias.oberstein at gmail.com> wrote:
> 
>> Hi Adrian,
>>
>>> Dell has graciously loaned me a bunch of hardware to continue doing
>>
>> FWIW, Dell has a roughly comparable system: Dell R920. But they don't have
>> Intel NVMe's on their menu, only Samsung (and FusionIO, but that's not
>> NVMe).
>>
>>  NUMA development on, but I have no NVMe hardware. I'm hoping people at
>>>
>>
>> The 8 NVMe PCIe SSDs in the box we're deploying are a key feature of this
>> system (will be a data-warehouse). A single NVMe probably won't have
>> triggered (all) issues we experienced.
>>
>> We are using the largest model (2TB), and this amounts to 50k bucks for
>> all eight. The smallest model (400GB) is 1.5k, so 12k in total.
>>
>>  Intel can continue kicking along any desires for NUMA that they
>>> require. (Which they have, fwiw.)
>>>
>>
>> It's already awesome that Intel has senior engineers working on FreeBSD
>> driver code! And it would underline Intel's Open-source commitment and tech
>> leadership if they donated a couple of these beefy NVMes.
>>
> 
> Intel has agreed to send DC P3700 samples to the FreeBSD Foundation to put
> in the cluster for this kind of work - we are working on getting these
> through the internal sample distribution process at the moment.
> 
> -Jim
>