igb(4) at peak in big purple

[Juli Mallett]

On Thu, Apr 26, 2012 at 11:07, Sean Bruno <seanbru at yahoo-inc.com> wrote:
> I note form top that igb0 queue 0 is always "more busy" than any other
> queue, there appears to be a second kernel igb0 "queue" process/thread
> that appears to be moderately busy and 3 kernel igb0 "queue"
> processes/threads that appear to be doing nothing in particular.

Queue splitting in Intel cards is done using a hash of protocol
headers, so this is expected behavior.  This also helps with TCP and
UDP performance, in terms of keeping packets for the same protocol
control block on the same core, but for other applications it's not
ideal.  If your application does not require that kind of locality,
there are things that can be done in the driver to make it easier to
balance packets between all queues about-evenly.

I've also seen it be the case that some cores will use more CPU time
for reasons of unusually-slow memory access from that core, i.e. with
non-uniform memory, or with some CPU threads, rather than because more
packets are going to a queue.  It would certainly be very nice at some
point for netstat -I to grow awareness of multiple queues (which is a
bit non-trivial because of how those statistics are gathered from
struct ifnet) so that one could more easily watch in real time the
packet rate to each queue.  Probably someone with more patience than
me could do this using the sysctl nodes the drivers export.

Juli.