[Bug 253587] iflib (?): reproducible mbuf-related crashes

Sun Feb 21 18:06:27 UTC 2021

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253587

--- Comment #5 from Kamigishi Rei <spambox at haruhiism.net> ---
Update: this happens with maxthreads=1 as well. Does not happen inside a VM.

With an INVARIANTS kernel I can reproduce this reliably by initiating a zfs
send over SSH through this host acting as a router (4 crashes out of 4 send
attempts). Out of these 4 crashes, three were the same KASSERT:

panic: Assertion m->m_nextpkt == NULL failed at /usr/src/sys/net/iflib.c:3638
cpuid = 2
time = 1613930234
KDB: stack backtrace:
#0 0xffffffff807fcfe5 at kdb_backtrace+0x65
#1 0xffffffff807b2cd1 at vpanic+0x181
#2 0xffffffff807b2aa3 at panic+0x43
#3 0xffffffff808ec3a1 at iflib_completed_tx_reclaim+0x2d1
#4 0xffffffff808eb780 at iflib_txq_drain+0x60
#5 0xffffffff808f2dfe at drain_ring_lockless+0x9e
#6 0xffffffff808f2b93 at ifmp_ring_enqueue+0x313
#7 0xffffffff808f1520 at iflib_if_transmit+0xa0
#8 0xffffffff808d0418 at bridge_enqueue+0xc8
#9 0xffffffff808d26c4 at bridge_output+0x134
#10 0xffffffff808d73af at ether_output+0x63f
#11 0xffffffff8097480b at ip6_forward+0x95b
#12 0xffffffff80976084 at ip6_input+0xf04
#13 0xffffffff808f4491 at netisr_dispatch_src+0xb1
#14 0xffffffff808d76be at ether_demux+0x17e
#15 0xffffffff808d8d4c at ether_nh_input+0x40c
#16 0xffffffff808f4491 at netisr_dispatch_src+0xb1
#17 0xffffffff808d7bb1 at ether_input+0xa1
Uptime: 1m36s
Dumping 402 out of 4051 MB:..4%..12%..24%..32%..44%..52%..64%..72%..84%..92%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct
pcpu,
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:399
#2  0xffffffff807b28fb in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:486
#3  0xffffffff807b2d40 in vpanic (fmt=<optimized out>, ap=<optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:919
#4  0xffffffff807b2aa3 in panic (fmt=<unavailable>) at
/usr/src/sys/kern/kern_shutdown.c:843
#5  0xffffffff808ec3a1 in iflib_tx_desc_free (txq=<optimized out>, n=<optimized
out>) at /usr/src/sys/net/iflib.c:3638
#6  iflib_completed_tx_reclaim (txq=<optimized out>,
txq at entry=0xfffffe0063088000, thresh=<optimized out>) at
/usr/src/sys/net/iflib.c:3680
#7  0xffffffff808eb780 in iflib_txq_drain (r=0xfffffe0063094000, r at entry=<error
reading variable: value is not available>, cidx=718, cidx at entry=<error reading
variable: value is not available>, pidx=719,
    pidx at entry=<error reading variable: value is not available>) at
/usr/src/sys/net/iflib.c:3744
#8  0xffffffff808f2dfe in drain_ring_lockless (r=<optimized out>, os=...,
prev=0, budget=<optimized out>) at /usr/src/sys/net/mp_ring.c:187
#9  0xffffffff808f2b93 in ifmp_ring_enqueue (r=0xfffffe0063094000,
items=<optimized out>, items at entry=0xfffffe0007f924e8, n=<optimized out>,
n at entry=1, budget=<optimized out>, budget at entry=32, abdicate=<optimized out>,
    abdicate at entry=0) at /usr/src/sys/net/mp_ring.c:470
#10 0xffffffff808f1520 in iflib_if_transmit (ifp=<optimized out>,
m=0xfffff80015f48000) at /usr/src/sys/net/iflib.c:4135
#11 0xffffffff808d0418 in bridge_enqueue (sc=sc at entry=0xfffff80015aa0c00,
dst_ifp=dst_ifp at entry=0xfffff80002647800, m=<unavailable>,
m at entry=0xfffff80015f48000) at /usr/src/sys/net/if_bridge.c:1983
#12 0xffffffff808d26c4 in bridge_output (ifp=<optimized out>, ifp at entry=<error
reading variable: value is not available>, m=0xfffff80015f48000, m at entry=<error
reading variable: value is not available>, sa=<unavailable>,
    sa at entry=<error reading variable: value is not available>,
rt=<unavailable>, rt at entry=<error reading variable: value is not available>) at
/usr/src/sys/net/if_bridge.c:2145
#13 0xffffffff808d73af in ether_output (ifp=0xfffff80002647800,
m=<unavailable>, dst=0xfffffe0007f92670, ro=<optimized out>) at
/usr/src/sys/net/if_ethersubr.c:414
#14 0xffffffff8097480b in ip6_forward (m=<unavailable>, srcrt=srcrt at entry=0) at
/usr/src/sys/netinet6/ip6_forward.c:387
#15 0xffffffff80976084 in ip6_input (m=<unavailable>, m at entry=<error reading
variable: value is not available>) at /usr/src/sys/netinet6/ip6_input.c:896
#16 0xffffffff808f4491 in netisr_dispatch_src (proto=6, source=source at entry=0,
m=0xfffff80023e49900) at /usr/src/sys/net/netisr.c:1143
#17 0xffffffff808f47df in netisr_dispatch (proto=<unavailable>,
m=<unavailable>) at /usr/src/sys/net/netisr.c:1234
#18 0xffffffff808d76be in ether_demux (ifp=ifp at entry=0xfffff800026cb800,
m=<unavailable>) at /usr/src/sys/net/if_ethersubr.c:923
#19 0xffffffff808d8d4c in ether_input_internal (ifp=0xfffff800026cb800,
m=<unavailable>) at /usr/src/sys/net/if_ethersubr.c:709
#20 ether_nh_input (m=<optimized out>, m at entry=<error reading variable: value
is not available>) at /usr/src/sys/net/if_ethersubr.c:739
#21 0xffffffff808f4491 in netisr_dispatch_src (proto=proto at entry=5,
source=source at entry=0, m=m at entry=0xfffff80023e49900) at
/usr/src/sys/net/netisr.c:1143
#22 0xffffffff808f47df in netisr_dispatch (proto=<unavailable>, proto at entry=5,
m=<unavailable>, m at entry=0xfffff80023e49900) at /usr/src/sys/net/netisr.c:1234
#23 0xffffffff808d7bb1 in ether_input (ifp=0xfffff800026cb800,
m=0xfffff80023e49900) at /usr/src/sys/net/if_ethersubr.c:830
#24 0xffffffff808f0556 in iflib_rxeof (rxq=<optimized out>,
rxq at entry=0xfffff800026cb000, budget=<optimized out>) at
/usr/src/sys/net/iflib.c:3008
#25 0xffffffff808ea0ca in _task_fn_rx (context=0xfffff800026cb000) at
/usr/src/sys/net/iflib.c:3951
#26 0xffffffff807fb977 in gtaskqueue_run_locked
(queue=queue at entry=0xfffff80002423300) at
/usr/src/sys/kern/subr_gtaskqueue.c:371
#27 0xffffffff807fb774 in gtaskqueue_thread_loop
(arg=arg at entry=0xfffffe0008d54038) at /usr/src/sys/kern/subr_gtaskqueue.c:547
#28 0xffffffff8076efb0 in fork_exit (callout=0xffffffff807fb6e0
<gtaskqueue_thread_loop>, arg=0xfffffe0008d54038, frame=0xfffffe0007f92c00) at
/usr/src/sys/kern/kern_fork.c:1069
#29 <signal handler called>

4th crash:

panic: m_dup: no mbuf packet header!
cpuid = 1
time = 1613919472
KDB: stack backtrace:
#0 0xffffffff807fcfe5 at kdb_backtrace+0x65
#1 0xffffffff807b2cd1 at vpanic+0x181
#2 0xffffffff807b2aa3 at panic+0x43
#3 0xffffffff80842981 at m_dup+0x351
#4 0xffffffff808ec610 at iflib_encap+0x210
#5 0xffffffff808ebb39 at iflib_txq_drain+0x419
#6 0xffffffff808f2dfe at drain_ring_lockless+0x9e
#7 0xffffffff808f2b93 at ifmp_ring_enqueue+0x313
#8 0xffffffff808f1520 at iflib_if_transmit+0xa0
#9 0xffffffff808d0418 at bridge_enqueue+0xc8
#10 0xffffffff808d26c4 at bridge_output+0x134
#11 0xffffffff808d73af at ether_output+0x63f
#12 0xffffffff8097480b at ip6_forward+0x95b
#13 0xffffffff80976084 at ip6_input+0xf04
#14 0xffffffff808f4491 at netisr_dispatch_src+0xb1
#15 0xffffffff808d76be at ether_demux+0x17e
#16 0xffffffff808d8d4c at ether_nh_input+0x40c
#17 0xffffffff808f4491 at netisr_dispatch_src+0xb1
Uptime: 3m59s
Dumping 409 out of 4051 MB:..4%..12%..24%..32%..43%..51%..63%..71%..83%..94%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct
pcpu,
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:399
#2  0xffffffff807b28fb in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:486
#3  0xffffffff807b2d40 in vpanic (fmt=<optimized out>, ap=<optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:919
#4  0xffffffff807b2aa3 in panic (fmt=<unavailable>) at
/usr/src/sys/kern/kern_shutdown.c:843
#5  0xffffffff80842981 in m_dup (m=<optimized out>, how=1) at
/usr/src/sys/kern/uipc_mbuf.c:733
#6  0xffffffff808ec610 in iflib_parse_header (txq=0xfffffe006302ea40,
pi=0xfffffe0007f47338, mp=0xfffffe006304f7f8) at /usr/src/sys/net/iflib.c:3138
#7  iflib_encap (txq=txq at entry=0xfffffe006302ea40,
m_headp=m_headp at entry=0xfffffe006304f7f8) at /usr/src/sys/net/iflib.c:3464
#8  0xffffffff808ebb39 in iflib_txq_drain (r=<optimized out>, r at entry=<error
reading variable: value is not available>, cidx=<optimized out>,
cidx at entry=<error reading variable: value is not available>, pidx=0,
    pidx at entry=<error reading variable: value is not available>) at
/usr/src/sys/net/iflib.c:3801
#9  0xffffffff808f2dfe in drain_ring_lockless (r=<optimized out>, os=...,
prev=0, budget=<optimized out>) at /usr/src/sys/net/mp_ring.c:187
#10 0xffffffff808f2b93 in ifmp_ring_enqueue (r=0xfffffe006304c000,
items=<optimized out>, items at entry=0xfffffe0007f474e8, n=<optimized out>,
n at entry=1, budget=<optimized out>, budget at entry=32, abdicate=<optimized out>,
    abdicate at entry=0) at /usr/src/sys/net/mp_ring.c:470
#11 0xffffffff808f1520 in iflib_if_transmit (ifp=<optimized out>,
m=0xfffff800586f9000) at /usr/src/sys/net/iflib.c:4135
#12 0xffffffff808d0418 in bridge_enqueue (sc=sc at entry=0xfffff80016b54c00,
dst_ifp=dst_ifp at entry=0xfffff80002456800, m=<unavailable>,
m at entry=0xfffff800586f9000) at /usr/src/sys/net/if_bridge.c:1983
#13 0xffffffff808d26c4 in bridge_output (ifp=<optimized out>, ifp at entry=<error
reading variable: value is not available>, m=0xfffff800586f9000, m at entry=<error
reading variable: value is not available>, sa=<unavailable>,
    sa at entry=<error reading variable: value is not available>,
rt=<unavailable>, rt at entry=<error reading variable: value is not available>) at
/usr/src/sys/net/if_bridge.c:2145
#14 0xffffffff808d73af in ether_output (ifp=0xfffff80002456800,
m=<unavailable>, dst=0xfffffe0007f47670, ro=<optimized out>) at
/usr/src/sys/net/if_ethersubr.c:414
#15 0xffffffff8097480b in ip6_forward (m=<unavailable>, srcrt=srcrt at entry=0) at
/usr/src/sys/netinet6/ip6_forward.c:387
#16 0xffffffff80976084 in ip6_input (m=<unavailable>, m at entry=<error reading
variable: value is not available>) at /usr/src/sys/netinet6/ip6_input.c:896
#17 0xffffffff808f4491 in netisr_dispatch_src (proto=6, source=source at entry=0,
m=0xfffff80016ed7600) at /usr/src/sys/net/netisr.c:1143
#18 0xffffffff808f47df in netisr_dispatch (proto=<unavailable>,
m=<unavailable>) at /usr/src/sys/net/netisr.c:1234
#19 0xffffffff808d76be in ether_demux (ifp=ifp at entry=0xfffff80002480800,
m=<unavailable>) at /usr/src/sys/net/if_ethersubr.c:923
#20 0xffffffff808d8d4c in ether_input_internal (ifp=0xfffff80002480800,
m=<unavailable>) at /usr/src/sys/net/if_ethersubr.c:709
#21 ether_nh_input (m=<optimized out>, m at entry=<error reading variable: value
is not available>) at /usr/src/sys/net/if_ethersubr.c:739
#22 0xffffffff808f4491 in netisr_dispatch_src (proto=proto at entry=5,
source=source at entry=0, m=m at entry=0xfffff80016ed7600) at
/usr/src/sys/net/netisr.c:1143
#23 0xffffffff808f47df in netisr_dispatch (proto=<unavailable>, proto at entry=5,
m=<unavailable>, m at entry=0xfffff80016ed7600) at /usr/src/sys/net/netisr.c:1234
#24 0xffffffff808d7bb1 in ether_input (ifp=0xfffff80002480800,
m=0xfffff80016ed7600) at /usr/src/sys/net/if_ethersubr.c:830
#25 0xffffffff808f0556 in iflib_rxeof (rxq=<optimized out>,
rxq at entry=0xfffff80002480300, budget=<optimized out>) at
/usr/src/sys/net/iflib.c:3008
#26 0xffffffff808ea0ca in _task_fn_rx (context=0xfffff80002480300) at
/usr/src/sys/net/iflib.c:3951
#27 0xffffffff807fb977 in gtaskqueue_run_locked
(queue=queue at entry=0xfffff80002422500) at
/usr/src/sys/kern/subr_gtaskqueue.c:371
#28 0xffffffff807fb774 in gtaskqueue_thread_loop
(arg=arg at entry=0xfffffe0008d54020) at /usr/src/sys/kern/subr_gtaskqueue.c:547
#29 0xffffffff8076efb0 in fork_exit (callout=0xffffffff807fb6e0
<gtaskqueue_thread_loop>, arg=0xfffffe0008d54020, frame=0xfffffe0007f47c00) at
/usr/src/sys/kern/kern_fork.c:1069
#30 <signal handler called>

-- 
You are receiving this mail because:
You are the assignee for the bug.