NVMe performance 4x slower than expected

[Konstantin Belousov]

On Sat, Apr 04, 2015 at 01:14:59PM +0200, Tobias Oberstein wrote:
> >> I was advised (off list) to run tests against a pure ramdisk.
> >>
> >> Here are results from a single socket E3:
> >>
> >> https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#xeon-e3-machine
> >>
> >> and here are results for the 48 core box
> >>
> >> https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#48-core-big-machine
> >>
> >> Performance with this box is 1/10 on this test compared to single socket E3!
> >>
> >> Something is severely wrong. It seems, there might be multiple issues
> >> (not only NVMe). And this is after already running with 3 patches to
> >> make it even boot.
> > The speed of dd already looks wrong.
> 
> Yes.
Try some simple checks for the performance anomalies.  E.g., verify the
raw memory bandwidth, check that it is in the expected range.  Quick search
popped up the following tool, e.g.:
https://zsmith.co/bandwidth.html

> 
> >
> > Check the CPU frequency settings in BIOS, and check what sysctl dev.cpu
> > reports.  Ensure that cpufreq.ko is loaded from loader.conf.
> 
> It's loaded now, and CPU clock is at maximum: dev.cpu.0.freq: 3000
> 
> Unfortuanetly, performance (randomio/fio) did not change ..