ixgbe & if_igb RX ring locking

[Adrian Chadd]

The reason why I've started moving net80211 and ath _away_ from using
direct dispatch (for now) and to using a taskqueue for TX (and RX) is
because it's too freaking annoying right now to deal with all the
crazy long-held locks to guarantee consistency between multiple
transmitting threads.

Considering that the driver and net80211 stack:

* sometimes is PCI, sometimes is USB (with all the differing thread
models that exist there);
* sometimes bridge traffic, sometimes route traffic, sometimes source
or terminate TCP/UDP connections;
* sometimes has one sender, sometimes has multiple senders, with some
other modules in between (bridge, pf, ipfw, etc) with locks being held
here and there;
* since the stack(s) like doing direct dispatch, RX very often causes
TX to occur, which for some drivers will block on a long-held driver
lock (with all the LORs that occur) - and drivers that do this (eg
iwn) will simply drop the lock before passing the packet up. Dropping
the lock before passing net80211_input*() .. is just plain silly.

Now, I'd _like_ to eventually make net80211/ath support direct
dispatch, but that also requires making sure only -one- transmitter is
working at once. I'd like to not have the extra context switch
overhead, but I haven't seen a better way of doing it yet.

It's fun to see the gige/10ge driver have lots of long held locks with
lots of concurrent sender processes possibly blocking until TX
completes.. so I wonder if that has scaling issues for lots of
connections/sending processes.



Adrian