ixgbe rx & tx locks

[Ryan Stone]

On Wed, Sep 26, 2012 at 9:55 AM, John Baldwin <jhb at freebsd.org> wrote:
> You only have to drop the RX lock around if_input() if you use the same lock
> for both TX and RX (as if_transmit() / if_start() can be invoked while locks
> in the network stack are held).

Last time I checked(FreeBSD 8.2), this is not true.  The problematic
(and convoluted) ordering is:

ix:rx -> udp -> udpinp -> in_multi_mtx -> ix:core -> ix:rx

udp -> udpinp -> in_multi_mtx is defined in subr_witness.c.

ix:core -> ix:rx is fairly obvious, and happens in places like ixgbe_init.

ix:rx -> udp is also fairly obvious (here's one backtrace):

lock order reversal:^M^M
 1st 0xffffff800153c138 ix:rx (ix:rx) @ src/sys/dev/ixgbe/ixgbe.c:7113^M^M
 2nd 0xffffffff80af9c48 udp (udp) @ src/sys/netinet/udp_usrreq.c:471^M^M
KDB: stack backtrace:^M^M
db_trace_self_wrapper() at 0xffffffff801dd5aa = db_trace_self_wrapper+0x2a^M^M
_witness_debugger() at 0xffffffff8044411e = _witness_debugger+0x2e^M^M
witness_checkorder() at 0xffffffff804453c7 = witness_checkorder+0x807^M^M
_rw_rlock() at 0xffffffff803fb61a = _rw_rlock+0x7a^M^M
udp_input() at 0xffffffff80517d1c = udp_input+0x1bc^M^M
ip_input() at 0xffffffff804f6b32 = ip_input+0x1e2^M^M
netisr_dispatch_src() at 0xffffffff804bfc38 = netisr_dispatch_src+0xb8^M^M
ether_demux() at 0xffffffff804b0fca = ether_demux+0x1aa^M^M
ether_input() at 0xffffffff804b141a = ether_input+0x1ca^M^M
ixgbe_rxeof() at 0xffffffff802d8ba3 = ixgbe_rxeof+0x203^M^M
ixgbe_msix_que() at 0xffffffff802e1790 = ixgbe_msix_que+0xf0^M^M
intr_event_execute_handlers() at 0xffffffff803d4096 = intr_event_execute_handler
s+0x66^M^M
ithread_loop() at 0xffffffff803d4e12 = ithread_loop+0xb2^M^M
fork_exit() at 0xffffffff803d1fba = fork_exit+0x12a^M^M
fork_trampoline() at 0xffffffff805f582e = fork_trampoline+0xe^M^M
--- trap 0, rip = 0, rsp = 0xffffff8000148d00, rbp = 0 ---^M^M

in_multi_mtx -> ix:core comes from the following backtrace:

lock order reversal:
 1st 0xffffffff80ae2440 in_multi_mtx (in_multi_mtx) @
src/sys/netinet/in_mcast.c:1095
 2nd 0xffffff8001539400 ixgbe0 (IXGBE Core Lock) @
src/sys/dev/ixgbe/ixgbe.c:1725
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff801dd5aa = db_trace_self_wrapper+0x2a
_witness_debugger() at 0xffffffff8044411e = _witness_debugger+0x2e
witness_checkorder() at 0xffffffff804453c7 = witness_checkorder+0x807
_mtx_lock_flags() at 0xffffffff803ec3ba = _mtx_lock_flags+0x8a
ixgbe_ioctl() at 0xffffffff802e07ae = ixgbe_ioctl+0x60e
if_addmulti() at 0xffffffff804a9f7b = if_addmulti+0x19b
in_joingroup_locked() at 0xffffffff804db8ec = in_joingroup_locked+0x1bc
in_joingroup() at 0xffffffff804dd5a2 = in_joingroup+0x52
in_control() at 0xffffffff804d7a70 = in_control+0x1160
ifioctl() at 0xffffffff804adec6 = ifioctl+0x5b6
nfs_mountroot() at 0xffffffff80567244 = nfs_mountroot+0x94
nfs_mount() at 0xffffffff80567b7b = nfs_mount+0x4db
vfs_donmount() at 0xffffffff8048919e = vfs_donmount+0xcde
kernel_mount() at 0xffffffff80489a71 = kernel_mount+0xa1
vfs_mountroot_try() at 0xffffffff80489f9d = vfs_mountroot_try+0x17d
vfs_mountroot() at 0xffffffff8048ab7d = vfs_mountroot+0x4fd
start_init() at 0xffffffff803b2932 = start_init+0x62
fork_exit() at 0xffffffff803d1fba = fork_exit+0x12a
fork_trampoline() at 0xffffffff805f582e = fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000042d00, rbp = 0 ---