Multiple cores/race conditions in IPv6 RA
Jason Wolfe
j at nitrology.com
Tue Dec 8 23:29:39 UTC 2015
I believe I've found a way to reproduce this, simply an rtsol against 2
interfaces in isolated loops, which elicits a broadcasted RA. I am able
to deadlock the system fairly quickly, which finally results in a core
after __rw_wunlock_hard steps in. We have seen this deadlock in one
other case, but the code path prior is the same as most of the others,
so hopefully it's keeping me on the right track. I tested backing out
r243148 without success.
load: 51.27 cmd: rtsol 7678 [*Giant] 832.44r 0.00u 0.00s 0% 1868k
(prior to core)
https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L654
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address = 0x30
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff807ada7c
stack pointer = 0x28:0xfffffe201edcff60
frame pointer = 0x28:0xfffffe201edcff90
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 12 (irq279: ix0:que 1)
trap number = 12
panic: page fault
cpuid = 1
(kgdb) bt
#0 doadump (textdump=1) at pcpu.h:219
#1 0xffffffff8075fa07 in kern_reboot (howto=16644) at
/usr/src/sys/kern/kern_shutdown.c:451
#2 0xffffffff8075fe05 in vpanic (fmt=<value optimized out>, ap=<value
optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:758
#3 0xffffffff8075fc93 in panic (fmt=0x0) at
/usr/src/sys/kern/kern_shutdown.c:687
#4 0xffffffff80ace3cb in trap_fatal (frame=<value optimized out>,
eva=<value optimized out>)
at /usr/src/sys/amd64/amd64/trap.c:851
#5 0xffffffff80ace6cd in trap_pfault (frame=0xfffffe201edcfeb0,
usermode=<value optimized out>)
at /usr/src/sys/amd64/amd64/trap.c:674
#6 0xffffffff80acdd6a in trap (frame=0xfffffe201edcfeb0) at
/usr/src/sys/amd64/amd64/trap.c:440
#7 0xffffffff80ab3d62 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:236
#8 0xffffffff807ada7c in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:838
#9 0xffffffff8075e080 in __rw_wunlock_hard (c=0xfffffe1f29f2f3f8,
tid=1, file=0x1 <Address 0x1 out of bounds>, line=1)
at /usr/src/sys/kern/kern_rwlock.c:988
#10 0xffffffff808a7136 in defrouter_select () at
/usr/src/sys/netinet6/nd6_rtr.c:654
#11 0xffffffff808a5a58 in nd6_ra_input (m=<value optimized out>,
off=<value optimized out>, icmp6len=<value optimized out>)
at /usr/src/sys/netinet6/nd6_rtr.c:804
#12 0xffffffff8087f31f in icmp6_input (mp=<value optimized out>,
offp=0xfffffe201edd064c, proto=<value optimized out>)
at /usr/src/sys/netinet6/icmp6.c:808
#13 0xffffffff80894adc in ip6_input (m=0xfffff8038cab7000) at
/usr/src/sys/netinet6/ip6_input.c:1019
#14 0xffffffff80832f02 in netisr_dispatch_src (proto=<value optimized
out>, source=<value optimized out>, m=0x10)
at /usr/src/sys/net/netisr.c:976
#15 0xffffffff8082a226 in ether_demux (ifp=<value optimized out>,
m=0xfffff8038cab7000)
at /usr/src/sys/net/if_ethersubr.c:851
#16 0xffffffff8082aece in ether_nh_input (m=<value optimized out>) at
/usr/src/sys/net/if_ethersubr.c:646
#17 0xffffffff80832f02 in netisr_dispatch_src (proto=<value optimized
out>, source=<value optimized out>, m=0x10)
at /usr/src/sys/net/netisr.c:976
#18 0xffffffff804c7e89 in ixgbe_rxeof (que=0xfffff8011680d470) at
/usr/src/sys/dev/ixgbe/ix_txrx.c:1681
#19 0xffffffff804c1e0a in ixgbe_msix_que (arg=0xfffff8011680d470) at
/usr/src/sys/dev/ixgbe/if_ix.c:1391
#20 0xffffffff8072ff9b in intr_event_execute_handlers (p=<value
optimized out>, ie=0xfffff80116806600)
at /usr/src/sys/kern/kern_intr.c:1264
#21 0xffffffff80730936 in ithread_loop (arg=0xfffff8011681e480) at
/usr/src/sys/kern/kern_intr.c:1277
#22 0xffffffff8072dbba in fork_exit (callout=0xffffffff807308a0
<ithread_loop>, arg=0xfffff8011681e480,
frame=0xfffffe201edd0ac0) at /usr/src/sys/kern/kern_fork.c:1018
#23 0xffffffff80ab429e in fork_trampoline () at
/usr/src/sys/amd64/amd64/exception.S:611
#24 0x0000000000000000 in ?? ()
(kgdb) frame 10
#10 0xffffffff808a7136 in defrouter_select () at
/usr/src/sys/netinet6/nd6_rtr.c:654
654 IF_AFDATA_UNLOCK(dr->ifp);
(kgdb) list
649 if (selected_dr == NULL &&
650 (ln = nd6_lookup(&dr->rtaddr, 0, dr->ifp))
&&
651 ND6_IS_LLINFO_PROBREACH(ln)) {
652 selected_dr = dr;
653 }
654 IF_AFDATA_UNLOCK(dr->ifp);
655 if (ln != NULL) {
656 LLE_RUNLOCK(ln);
657 ln = NULL;
658 }
(kgdb) print *dr
Cannot access memory at address 0xa00000001
(kgdb) print dr->ifp
Cannot access memory at address 0xa00000031
Thanks again,
Jason
On 2015-12-07 22:32, Jason wrote:
> Hi,
>
> It appears the IPv6 router advertisement code paths were written
> fairly lockless, assuming you would never process multiples
> concurrently. We are seeing multiple page faults in various places
> processing the messages and modifying the routing table. We have
> multiple L3 devices and multiple v6 blocks broadcasting these messages
> to hardware with dual uplinks in the same VLAN, which I believe is
> making us susceptible to this. Though I believe the dual uplink is
> all that's required for this, as it can be seen in configurations with
> a single v6 block.
>
> We are running stable/10 @ r285800, and it doesn't appear anything
> relevant has changed since then. Our other widely deployed version is
> 8.3-RELEASE, which does not see this issue. Upon bumping a machine
> from 8.3 -> 10 we can see it start to exhibit this behavior. The only
> change I see that might be relevant is r243148, but these cores are
> relatively rare, so testing is tough without a considerable
> deployment. So basically I'm hoping someone with a trained eye can
> send us in the right direction before we go down that road.
>
> Every backtrace looks pretty much like this, with the location in
> nd6_rtr differing:
>
> panic: page fault
> #0 doadump (textdump=1) at pcpu.h:219
> #1 0xffffffff8075fa07 in kern_reboot (howto=260) at
> /usr/src/sys/kern/kern_shutdown.c:451
> #2 0xffffffff8075fe05 in vpanic (fmt=<value optimized out>, ap=<value
> optimized out>) at /usr/src/sys/kern/kern_shutdown.c:758
> #3 0xffffffff8075fc93 in panic (fmt=0x0) at
> /usr/src/sys/kern/kern_shutdown.c:687
> #4 0xffffffff80acdf9b in trap_fatal (frame=<value optimized out>,
> eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:851
> #5 0xffffffff80ace29d in trap_pfault (frame=0xfffffe0f959b0ff0,
> usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:674
> #6 0xffffffff80acd93a in trap (frame=0xfffffe0f959b0ff0) at
> /usr/src/sys/amd64/amd64/trap.c:440
> #7 0xffffffff80ab3932 in calltrap () at
> /usr/src/sys/amd64/amd64/exception.S:236
> #8 0xffffffff808a5550 in nd6_ra_input (m=<value optimized out>,
> off=<value optimized out>, icmp6len=<value optimized out>)
> at /usr/src/sys/netinet6/nd6_rtr.c:739
> #9 0xffffffff8087f31f in icmp6_input (mp=<value optimized out>,
> offp=0xfffffe0f959b167c, proto=<value optimized out>)
> at /usr/src/sys/netinet6/icmp6.c:808
> #10 0xffffffff808949fc in ip6_input (m=0xfffff8002e743200) at
> /usr/src/sys/netinet6/ip6_input.c:1019
> #11 0xffffffff80832f02 in netisr_dispatch_src (proto=<value optimized
> out>, source=<value optimized out>, m=0x1)
> at /usr/src/sys/net/netisr.c:976
> #12 0xffffffff8082a226 in ether_demux (ifp=<value optimized out>,
> m=0xfffff8002e743200) at /usr/src/sys/net/if_ethersubr.c:851
> #13 0xffffffff8082aece in ether_nh_input (m=<value optimized out>) at
> /usr/src/sys/net/if_ethersubr.c:646
> #14 0xffffffff80832f02 in netisr_dispatch_src (proto=<value optimized
> out>, source=<value optimized out>, m=0x1)
> at /usr/src/sys/net/netisr.c:976
>
> I'll link to GH for the various relevant bits, because I know everyone
> can agree it's the superior RCS. It appears to be that most of these
> are caused by the dr struct being freed by concurrent processing:
>
> https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L578
> https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L654
> https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L728
> https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L739
> https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L800
> https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L1312
>
> Thanks for any assistance,
> Jason
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
More information about the freebsd-net
mailing list