fib6_lookup() returning deleted struct ifnet

Hi,

Several pfSense users report IPv6-related panics when an interface is 
deleted.
The relevant bug reports are https://redmine.pfsense.org/issues/14164 
and https://redmine.pfsense.org/issues/14431.
The latest report is for a build that includes commits up to 
1a18383a52bc373e316d224cef1298debf6f7e25 (“libcrypto: link engines and 
the legacy provider to libcrypto”, September 15th).

I believe all reports are for users running PPPoE, via netgraph, but 
that might be coincidental, as that’s the most likely way for 
interfaces to be destroyed (when PPP disconnects and reconnects).

There are a few different backtraces, but they appear to have the same 
root cause, so I’ll focus on one of them:

	db:1:pfs> bt
	Tracing pid 2 tid 100041 td 0xfffffe0085264560
	kdb_enter() at kdb_enter+0x32/frame 0xfffffe00850ad910
	vpanic() at vpanic+0x183/frame 0xfffffe00850ad960
	panic() at panic+0x43/frame 0xfffffe00850ad9c0
	trap_fatal() at trap_fatal+0x409/frame 0xfffffe00850ada20
	trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00850ada80
	calltrap() at calltrap+0x8/frame 0xfffffe00850ada80
	--- trap 0xc, rip = 0xffffffff80f5a036, rsp = 0xfffffe00850adb50, rbp = 
0xfffffe00850adb80 ---
	in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00850adb80
	tcp_default_output() at tcp_default_output+0x1ded/frame 
0xfffffe00850add70
	tcp_timer_rexmt() at tcp_timer_rexmt+0x514/frame 0xfffffe00850addd0
	tcp_timer_enter() at tcp_timer_enter+0x102/frame 0xfffffe00850ade10
	softclock_call_cc() at softclock_call_cc+0x13c/frame 0xfffffe00850adec0
	softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00850adef0
	fork_exit() at fork_exit+0x7d/frame 0xfffffe00850adf30
	fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00850adf30
	--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

This happens in the TCP output path, where we look up the hop limit for 
a specific destination. I’ve obtained a core dump for such a crash, 
and I believe the panic happens on line 
https://cgit.freebsd.org/src/tree/sys/netinet6/in6_src.c#n861

The call in tcp_default_output() is `in6_selecthlim(int, NULL);`, so we 
don’t get an ifp from the caller, but instead perform a route lookup, 
and try to obtain the hop limit through `ND_IFINFO(nh->nh_ifp)`. This 
panics because the afdata[AF_INET6] pointer is NULL. The core dump shows 
a deleted structure ifnet:

	(kgdb) p *(struct ifnet *)0xfffff80203712800
	$3 = {
	  if_link = {
	    cstqe_next = 0x0
	  },
	  if_clones = {
	    le_next = 0x0,
	    le_prev = 0x0
	  },
	  if_groups = {
	    cstqh_first = 0x0,
	    cstqh_last = 0xfffff80203712818
	  },
	  if_alloctype = 53 '5',
	  if_numa_domain = 255 '\377',
	  if_softc = 0xfffff80103447a00,
	  if_llsoftc = 0x0,
	  if_l2com = 0x0,
	  if_dname = 0xffffffff81492f70 "ng",
	  if_dunit = 0,
	  if_index = 14,
	  if_idxgen = 2,
	  if_xname = "pppoe0\000\000\000\000\000\000\000\000\000",
	  if_description = 0xfffff8003a5f83d0 "WAN",
	  if_flags = 2132112,
	  if_drv_flags = 0,
	  if_capabilities = 0,
	  if_capabilities2 = 0,
	…
       if_afdata = {0x0 <repeats 44 times>},
	…
	  if_output = 0xffffffff80e29c60 <ifdead_output>,
	  if_input = 0xffffffff80e29c80 <ifdead_input>,
	  if_bridge_input = 0x0,
	  if_bridge_output = 0x0,
	  if_bridge_linkstate = 0x0,
	  if_start = 0xffffffff80e29c90 <ifdead_start>,
	  if_ioctl = 0xffffffff80e29ca0 <ifdead_ioctl>,
	…

My understanding is that the fib table should get updated whenever we 
change the routing table (such as during interface cleanup in 
`if_detach_internal()`). Some quick experimentation with epair and 
dtrace also shows:

	 20  20388           sync_algo_end_cb:entry Stage 1
	              kernel`setup_fd_instance+0x41f
	              kernel`rebuild_fd_flm+0x99
	              kernel`rebuild_fd+0x136
	              kernel`rib_notify+0x50
	              kernel`rt_delete_conditional+0xf1
	              kernel`rib_del_route+0x1fc
	              kernel`rib_handle_ifaddr_info+0xd9
	              kernel`nd6_prefix_offlink+0x1ce
	              kernel`nd6_prefix_del+0x94
	              kernel`if_purgeaddrs+0x148
	              kernel`if_detach_internal+0x1e8
	              kernel`if_detach+0x71
	              if_epair.ko`epair_clone_destroy+0x62
	              kernel`if_clone_destroyif_flags+0x6a
	              kernel`if_clone_destroy+0x100
	              kernel`ifioctl+0x8a5
	              kernel`kern_ioctl+0x286
	              kernel`sys_ioctl+0x152
	              kernel`amd64_syscall+0x153
	              kernel`0xffffffff8102315b

In other words, when we delete the interface `if_detach_internal()` 
purges the interface addresses, which ends up rebuilding the fib 
(`rebuild_fd()`) via `rib_del_route()`.
That ought to ensure that we cannot end up finding this struct ifnet 
through `fib6_lookup()`, as the purging of the addresses (and thus the 
rebuilding of the fib) is done before we `if_domdetach()` at the end of 
`if_detach_internal()`, and the NULL afdata[AF_INET6] demonstrates that 
we’ve gotten there.

We’ve also gone through `if_free()`, as the ifindex_table no longer 
contains the struct ifnet pointer for the relevant interface.
We appear to have not yet called `if_free_deferred()` (and indeed, 
ifp->if_refcount is 4, so we wouldn’t have called that yet).

I’m confused as to how this can happen, and would appreciate hints.

Thanks,
Kristof