ixgbe & if_igb RX ring locking

[Adrian Chadd]

On 16 October 2012 05:38, John Baldwin <jhb at freebsd.org> wrote:

> I don't follow how this is related to this thread at all (which has more to do
> with ixgbe scheduling duplicate work).  However, is your issue that the stack
> locks (e.g. socket and protocol layer locks) are held across
> if_start/if_transmit?

It's a comment on the larger scale architectural problem. Since
if_transmit and if_start are called from multiple thread contexts, the
current ways drivers implement this are:

* support direct dispatch to hardware, but wrap the whole sending
process in one enormous lock, to prevent packet reordering issues; or
* drop TX and TX completion into a TX taskqueue (or multiple, one per
hardware send queue) and push frames into that taskqueue via some
queue and then wake said taskqueue up; or
* some bastardised version of both.

For the intel drivers, the locks are held for a (potentially) very
long time. Both igb and ixgb both hold the locks for the entirety of
the TX process. It's not protecting something like a queue operation,
it's effectively serialising the entirety of the TX and TX completion
process.

That works ok-ish for ethernet drivers which are "send and ignore",
but for wireless drivers where the stack implements a lot more state,
it really does quite suck. And since wireless drivers have a top level
idea of sequence and encryption (ie, it's not per-TCP stream, it's
across multiple sending streams to a given node), I can't model the
locking and serialisation on what the TCP/UDP code does.

I wish we had a better way of implementing "serialisation without
long, long held locks" but short of stuffing everything into a
taskqueue and only locking the send queue involved, I can't really
think of anything.



Adrian