Kernel Panic

[Joe Jones]

Hi,

we have a kernel panic after compiling with witness and invariant

Feb 27 13:49:33 sovapn1 kernel: lock order reversal:
Feb 27 13:49:33 sovapn1 kernel: 1st 0xfffffe000fed78b8 pf_idhash (pf_idhash) @ /usr/src/sys/netpfil/pf/pf.c:1078
Feb 27 13:49:33 sovapn1 kernel: 2nd 0xfffff8001e0474a8 pfsync (pfsync) @ /usr/src/sys/netpfil/pf/if_pfsync.c:1667
Feb 27 13:49:33 sovapn1 kernel: stack backtrace:
Feb 27 13:49:33 sovapn1 kernel: #0 0xffffffff80a9efe0 at witness_debugger+0x70
Feb 27 13:49:33 sovapn1 kernel: #1 0xffffffff80a9eed3 at witness_checkorder+0xe23
Feb 27 13:49:33 sovapn1 kernel: #2 0xffffffff80a20aad at __mtx_lock_flags+0x9d
Feb 27 13:49:33 sovapn1 kernel: #3 0xffffffff826d9f9d at pfsync_insert_state+0x7d
Feb 27 13:49:33 sovapn1 kernel: #4 0xffffffff826ea729 at pf_state_insert+0x8b9
Feb 27 13:49:33 sovapn1 kernel: #5 0xffffffff826f444c at pf_test_rule+0x2bbc
Feb 27 13:49:33 sovapn1 kernel: #6 0xffffffff826eec58 at pf_test+0x1598
Feb 27 13:49:33 sovapn1 kernel: #7 0xffffffff826ffb2d at pf_check_in+0x1d
Feb 27 13:49:33 sovapn1 kernel: #8 0xffffffff80b488c4 at pfil_run_hooks+0xa4
Feb 27 13:49:33 sovapn1 kernel: #9 0xffffffff80ba5757 at ip_tryforward+0x157
Feb 27 13:49:33 sovapn1 kernel: #10 0xffffffff80ba7a8a at ip_input+0x38a
Feb 27 13:49:33 sovapn1 kernel: #11 0xffffffff80b47a00 at netisr_dispatch_src+0x80
Feb 27 13:49:33 sovapn1 kernel: #12 0xffffffff80b309ce at ether_demux+0x15e
Feb 27 13:49:33 sovapn1 kernel: #13 0xffffffff80b317ee at ether_nh_input+0x31e
Feb 27 13:49:33 sovapn1 kernel: #14 0xffffffff80b47a00 at netisr_dispatch_src+0x80
Feb 27 13:49:33 sovapn1 kernel: #15 0xffffffff80b30d62 at ether_input+0x62
Feb 27 13:49:33 sovapn1 kernel: #16 0xffffffff82671427 at igb_rxeof+0x6e7
Feb 27 13:49:33 sovapn1 kernel: #17 0xffffffff826708fe at igb_msix_que+0xee


we compiled with

options         INVARIANTS
options         INVARIANT_SUPPORT
options         WITNESSoptions         WITNESS_SKIPSPIN
options         WITNESS_KDB

we went back to the 11.1 release as the relenge branch didn't compile 
for us (probably our fault).


Regards

Joe Jones

On 27/02/18 04:40, Kristof Provost wrote:
>
> On 26 Feb 2018, at 17:06, Joe Jones wrote:
>
>     Hi Kristof,
>
>     we are not updating rules during the test although in production
>     we will reload the rule set from time to time. We are constantly
>     adding and removing from tables though, using the DIOCRADDADDRS
>     and DIOCRDELADDRS ioctl, also DIOCKILLSTATES is being called a
>     lot. These are all in response to RADIUS events. We tried using
>     pfctl shell command rather than calling ioctl directly, to check
>     that it wasn't a problem with how we are calling the ioctl.
>
> That’s interesting.
>
> The panic leads me to suspect something’s wrong with the 
> kt->pfrkt_ipv4->rt, which would explain why we get the unexpected NULL 
> result.
> My first guess at the cause would be a race condition, where it’s 
> being modified (through one of the ioctls you do) while the 
> pfr_pool_get() is walking it.
>
> I don’t immediately see where that’d happen though, because both 
> DIOCRADDADDRS and DIOCRDELADDRS take the rules lock (and 
> pfr_pool_get() takes it too).
>
> It might be interesting to run this with these extra asserts (and be 
> sure to enable INVARIANTS).
>
> |diff --git a/sys/netpfil/pf/pf_table.c b/sys/netpfil/pf/pf_table.c 
> index 18342a94073..cad9b4ea89f 100644 --- a/sys/netpfil/pf/pf_table.c 
> +++ b/sys/netpfil/pf/pf_table.c @@ -962,6 +962,8 @@ 
> pfr_unroute_kentry(struct pfr_ktable *kt, struct pfr_kentry *ke) 
> struct radix_node *rn; struct radix_head *head = NULL; + 
> PF_RULES_WASSERT(); + if (ke->pfrke_af == AF_INET) head = 
> &kt->pfrkt_ip4->rh; else if (ke->pfrke_af == AF_INET6) @@ -1855,6 
> +1859,8 @@ pfr_destroy_ktable(struct pfr_ktable *kt, int flushaddr) { 
> struct pfr_kentryworkq addrq; + PF_RULES_WASSERT(); + if (flushaddr) { 
> pfr_enqueue_addrs(kt, &addrq, NULL, 0); pfr_clean_node_mask(kt, &addrq); |
>
> Regards,
> Kristof
>