ixgbe & if_igb RX ring locking

[Gleb Smirnoff]

On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote:
A> Packets receiving code for both ixgbe and if_igb looks like the following:
A> ixgbe_msix_que
A> 
A> -- ixgbe_rxeof()
A>     {
A>        IXGBE_RX_LOCK(rxr);
A>          while
A>          {
A>             get_packet;
A> 
A>             -- ixgbe_rx_input()
A>                {
A>                   ++ IXGBE_RX_UNLOCK(rxr);
A>                   if_input(packet);
A>                   ++ IXGBE_RX_LOCK(rxr);
A>                }
A> 
A>          }
A>        IXGBE_RX_UNLOCK(rxr);
A>      }
A> 
A> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe).
A> 
A> These lines probably do LORs masking (if any) well.
A> However, such change introduce quite significant performance drop:
A> 
A> On my routing setup (nearly the same from previous -Intel 10G thread in 
A> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is 
A> nearly 20%.
A> 
A> So my questions are:
A> 
A> Can any real LORs happen in some complex setup? (I can't imagine any).
A> If so: maybe we can somehow avoid/workaround such cases? (and consider 
A> removing those locks).

To me this unlock/lock looks like a legacy from times, when the driver
had a single mutex for both TX and RX parts.

And removing this re-locking in foo_rxeof() was one of the aims for separate
TX/RX locking.

Really, lurking through history shows that once driver had split its locking
to separate RX and TX part, these unlock/lock was removed. However, later
this unlock/lock was added back:

http://svnweb.freebsd.org/base/head/sys/dev/e1000/if_igb.c?revision=209068&view=markup

, without any comments for the reason it is added back.

-- 
Totus tuus, Glebius.