Strange network performance on Intel Rangeley (8 cores Atom)

[Olivier Cochard-Labbé]

Hi all,

I've got a new toy in my network bench lab: a SuperMicro SuperServer
5018A-FTN4.
But I've got a problem for understanding and obtaining good throughput for
"routing" or "firewalling" usages.

I'm using only the embedded 4 gigabit ports of the Atom C2758 SoC.
With the default igb(4) parameters which is to create 8 queues (because
there is 8 cores) this server is not able to receive more than 585K
packet-per-seconds into one port which is far from the gigabit line-rate
(1.48Mpps): I was expecting better throughput with 8 cores.
Then I did a bunch of new benchmarks by measuring the impact of number of
queue and the results are here:
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4#graph

=> I've got better results with only 4 queues than 8... but still low
throughput with only 938Kpps.

Then I decided to measure the impact of pf and ipfw on the throughput with
4 and 8 queues.
And the results are annoying:
http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4#graph1

=> With 8 queues, enabling pf or ipfw improve the input throughput of the
igb(4) port.

Why so low throughput with 8 queues ?
Why better throughput with pf or ipfw enabled than without ?

Thanks,

Olivier