Disappointing packets-per-second performance results on a Dell, PE R530

[Olivier Cochard-Labbé]

On Tue, Jan 3, 2017 at 5:35 PM, Jordan Caraballo <
jordancaraballo87 at gmail.com> wrote:

> We recently tested a Dell R530 with a Chelsio T580 card, under FreeBSD
> 10.3, 11.0, -STABLE and -CURRENT, and Centos 7.
>
> Based on our research, including netmap-fwd and with the routing
> improvements project (https://wiki.freebsd.org/ProjectsRoutingProposal),
> we hoped for packets-per-second (pps) in the 5+ million range, or even
> higher.
>
> Based on prior testing (http://marc.info/?t=140604252400002&r=1&w=2), we
> expected 3-4 Million to be easily obtainable.
>
> Unfortunately, our current results top out at no more than 1.5 M (64 bytes
> length packets) with FreeBSD, and
> surprisingly around 3.2 M (128 bytes length packets) with Centos 7, and we
> are at a loss as to why.
>
>
> ​
How do you generate your IP traffic ?
You need to take care to generate multiple flows for correctly using all
queues of your NIC.
If you've got 18 cores per each CPU, cxbge drivers will create 16 NIC
queues: Check they are all bound to cores belonging to the the same package.
How did you bind your NIC queues ?

Did you disable Ethernet Flow-control and Ethernet entropy harvest ?

On my HP ProLiant DL360p Gen8 with 8 cores (Intel Xeon E5-2650 @ 2.60GHz)
and Chelsio NIC, I obtain about 5Mpps on a recent head:
https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650-8Cores-Chelsio_T540-CR/fastforwarding-pf-ipfw/results/fbsd12-head.r299288.D8526/README.md
​

​My lab setup and tools used is described here:
https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr

For generating my traffic, I'm using a patched version of netmap pkt-gen,
patch here:
https://github.com/ocochard/BSDRP/blob/master/BSDRPcur/patches/freebsd.pkt-gen.ae-ipv6.patch

Regards,

Olivier