Polling with multiqueue support

[Fabien Thomas]

Ryan post on new polling modification remembered me to post a quick note about polling with multiqueue support i've done some month ago.

The code is more intrusive and add a new handler for the queue. 
The handling of the network is nearly the same as deferred taskqueue in the drivers.
There is now two pass one for the receive and one for transmit (to flush pending transmit when all receive pass done).

The main gain is for packet forwarding with more than one interface.
The CPU can be easily reserved for application by binding a specific number of core to the network.
Performance is on par with interrupt on 10Gb or 1Gb interface and latency can be reduced by using higher HZ.
Most of the time using less core achieve higher global efficiency of the system by freeing CPU cycle and reducing contention.

Ex setup:

6 cores CPU, 2 ixgbe with 3 queue, 4 igb with 3 queue

with 3 cores for polling:
CPU0 will handle ixgbe0 queue 0, ixgbe1 queue 0, igb0 queue0, ...
CPU1 will handle ixgbe0 queue 1, ...
...

For those interested a test branch can be found here based on 8.x with ixgbe / igb and em modification:
http://www.gitorious.org/~fabient/freebsd/fabient-sandbox/commits/work/pollng_mq_stable_8
Extracted patchset here:
http://people.freebsd.org/~fabient/patch-poll_mq-20110202-stable_8
http://people.freebsd.org/~fabient/kern_poll.c-20110202 -> put to kern/kern_poll.c

--
Fabien Thomas