40Gbps http client benchmark

[Eugene Grosbein]

07.08.2016 1:03, Eugene Grosbein пишет:
> Hi!
>
> Is there any high performance benchmark acting as http client for outer http server
> capable to receive 40Gbps without overwhelming CPU with insane number of syscalls?
>
> I've tried benchmarks/wrk version 4.0.2 and it works just fine upto 20Gbps
> for my hardware: two 6-core (HT disabled) Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
> with two dual-port ix(4) 82599ES 10-Gigabit SFI/SFP+ Network Connection
> combined to single lagg interface (lagghash l4).
>
> But each worker pthread of wrk generates too many kqueue() system calls
> polling for incoming data and eats 100% of its CPU core and cannot receive more.
> Or, it may be some kqueue() kernel level lock contention, I do not know.
> More worker threads, more overloaded CPU cores, no increase of transfer over about 20Gbps.

Hmm, it seems that's not number of system calls that hurts me but some kernel-level problem.
This is NUMA system running under 10.3-STABLE r303291. It has two NUMA domains
with first physical CPU (cores 0-7) and first dual-port ix adapter belonging to one domain
and second CPU (cores 8-11) and second dual-port adatper belonging to another domain:

http://www.grosbein.net/img/r4.svg

Just prepend of "cpuset -l 0-7" to wrk invocation boots transfer upto nearly 30Gbps
without any other changes. That's strange.