40 cores, 48 NVMe disks, feel free to take over

[Christoph Pilka]

Hi,

the server we got to experiment with is the SuperMicro 2028R-NR48N (https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm <https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm>), the board itself is a X10DSC+

//Chris

> On 09 Sep 2016, at 23:14, Dennis Glatting <freebsd at pki2.com> wrote:
> 
> On Fri, 2016-09-09 at 22:51 +0200, Christoph Pilka wrote:
>> Hi,
>> 
>> we've just been granted a short-term loan of a server from Supermicro
>> with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of
>> mucking about, we managed to get 11-RC running. A couple of things
>> are preventing the system from being terribly useful:
>> 
>> - We have to use hw.nvme.force_intx=1 for the server to boot
>> If we don't, it panics around the 9th NVMe drive with "panic:
>> couldn't find an APIC vector for IRQ...". Increasing
>> hw.nvme.min_cpus_per_ioq brings it further, but it still panics later
>> in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it
>> to panic later (I suspect during ixl init - the box has 4x10gb
>> ethernet ports).
>> 
>> - zfskern seems to be the limiting factor when doing ~40 parallel "dd
>> if=/dev/zer of=<file> bs=1m" on a zpool stripe of all 48 drives. Each
>> drive shows ~30% utilization (gstat), I can do ~14GB/sec write and 16
>> read.
>> 
>> - direct writing to the NVMe devices (dd from /dev/zero) gives about
>> 550MB/sec and ~91% utilization per device 
>> 
>> Obviously, the first item is the most troublesome. The rest is based
>> on entirely synthetic testing and may have little or no actual impact
>> on the server's usability or fitness for our purposes. 
>> 
>> There is nothing but sshd running on the server, and if anyone wants
>> to play around you'll have IPMI access (remote kvm, virtual media,
>> power) and root.
>> 
>> Any takers?
>> 
> 
> 
> I'm curious to know what board you have. I have had FreeBSD, including
> release 11 candidates, running on SM boards without any trouble
> although some of them are older boards. I haven't looked at ZFS
> performance because mine are typically low disk use. That said, my
> virtual server (also a SM) IOPs suck but so do its disks.
> 
> I recently found the Intel RAID chip on one SM isn't real RAID, rather
> it's pseudo RAID but for a few dollars more it could be real RAID. :(
> It was killing IOPs so I popped in an old LSI board, routed the cables
> from the Intel chip, and the server is now a happy camper. I then
> replaced 11-RC with Ubuntu 16.10 due to a specific application but I am
> also running RAIDz2 under Ubuntu on three trash 2.5T disks (I didn't do
> this for any reason other than fun). 
> 
> root at Tuck3r:/opt/bin# zpool status
>   pool: opt
>  state: ONLINE
>   scan: none requested
> config:
> 
> 	NAME        STATE     READ WRITE CKSUM
> 	opt         ONLINE       0     0     0
> 	  raidz2-0  ONLINE       0     0     0
> 	    sda     ONLINE       0     0     0
> 	    sdb     ONLINE       0     0     0
> 	    sdc     ONLINE       0     0     0
> 
> 
> 
>> Wbr
>> Christoph Pilka
>> Modirum MDpay
>> 
>> Sent from my iPhone
>> _______________________________________________
>> freebsd-questions at freebsd.org <mailto:freebsd-questions at freebsd.org> mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-questions <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>
>> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freeb
>> sd.org <http://sd.org/>"
> _______________________________________________
> freebsd-questions at freebsd.org <mailto:freebsd-questions at freebsd.org> mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org <mailto:freebsd-questions-unsubscribe at freebsd.org>"