ixgbe & if_igb RX ring locking

[Alexander V. Chernikov]

On 16.10.2012 17:20, Ryan Stone wrote:
> On Tue, Oct 16, 2012 at 8:47 AM, Gleb Smirnoff<glebius at freebsd.org>  wrote:
>> Can you please provide hints how can SIOCADDMULTI lead to obtaining RX
>> lock in the stock driver?
>
> It doesn't.  But it does acquire the core lock, and the core lock is
> acquired before the RX lock (in ixgbe_init, for instance).

I still can't get how any of these can happen.

We basically has 2 related places where ixgbe driver interact with OS:

1) forwarding path (ixgbe_rxeof via ithread / taskq)
2) ioctl path

I'm currently talking about multicast only.

So, first can lead us to:

ixgbe_rxeof() (rxr->rx_mtx "ix:rx")
   ether_input()
     ...
     udp_input() (V_udbinfo "udp" READ lock)
       udp_input() (inp->inp_lock "udpinp" READ lock)
>

2) ioctl part can lead to the following locks to be held:
   IXGBE_CORE_LOCK() in every ioctl ("ix?:core" mtx)
     ixgbe_init_locked()
       ixgbe_setup_receive_structures()
         ixgbe_setup_receive_ring() (rxr->rx_mtx "ix:rx")

.. with any possibly locks prepended (like in_multi_mtx or any other)


 > Last time I checked(FreeBSD 8.2), this is not true.  The problematic
 > (and convoluted) ordering is:
 >
 > ix:rx ->  udp ->  udpinp ->  in_multi_mtx ->  ix:core ->  ix:rx
 >
 > udp ->  udpinp ->  in_multi_mtx is defined in subr_witness.c.
Actually subr_witness defines the following:
1) udp -> udpinp -> so_snd
2) udpinp -> in_multi_mtx -> .. -> ..

 >
 > ix:core ->  ix:rx is fairly obvious, and happens in places like 
ixgbe_init.
 >
Yes. But how can I trigger reverted order (e.g. ixgbe_init[_locked]() 
(actually, ioctl) on packet input ?

(And if such place really exists we should consider fixing it, but not 
the driver side)

 > ix:rx ->  udp is also fairly obvious (here's one backtrace):
The same question, where "udp" -> "ix:rx" can happen ?