[rfc] TCP timewait and credential handling - why would we get a TW with no credential?

[Adrian Chadd]

Hi,

I've been seeing this panic under high very short-term connection TCP
throughput tests (30,000 connections a second) with SO_REUSEPORT:

Current language:  auto; currently minimal

(kgdb) frame 11

#11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0)
at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644

644             crfree(tw->tw_cred);

(kgdb) frame 11

#11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0)
at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644

644             crfree(tw->tw_cred);

(kgdb) print tw

$1 = (struct tcptw *) 0xfffff801348b5780

(kgdb) print tw->tw_cred

$2 = (struct ucred *) 0x0

(kgdb)

and:

#10 0xffffffff808d017e in crfree (cr=0x0) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/kern/kern_prot.c:1837

#11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0)
at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644

#12 0xffffffff80a6bc53 in tcp_twcheck (inp=<value optimized out>,
to=<value optimized out>, th=<value optimized out>, m=<value optimized
out>, tlen=<value optimized out>) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:425

#13 0xffffffff80a5cf3f in tcp_input (m=<value optimized out>,
off0=<value optimized out>) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_input.c:949

#14 0xffffffff809fa4e7 in ip_input (m=0xfffff8004831e000) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/ip_input.c:729

#15 0xffffffff80998441 in netisr_dispatch_src (proto=<value optimized
out>, source=<value optimized out>, m=0xffffffff) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/net/netisr.c:972

#16 0xffffffff80990173 in ether_demux (ifp=<value optimized out>,
m=0xfffff8004831e000) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/net/if_ethersubr.c:775

#17 0xffffffff80990dce in ether_nh_input (m=<value optimized out>) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/net/if_ethersubr.c:582

#18 0xffffffff80998441 in netisr_dispatch_src (proto=<value optimized
out>, source=<value optimized out>, m=0xffffffff) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/net/netisr.c:972

#19 0xffffffff809903c6 in ether_input (ifp=<value optimized out>,
m=0x0) at /usr/home/adrian/git/github/erikarn/freebsd/sys/net/if_ethersubr.c:683

#20 0xffffffff804e9d67 in igb_rxeof (count=99) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/dev/e1000/if_igb.c:4882

#21 0xffffffff804ea47f in igb_msix_que (arg=0xfffff8000d47e670) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/dev/e1000/if_igb.c:1626

#22 0xffffffff808aef03 in intr_event_execute_handlers (p=<value
optimized out>, ie=0xfffff8000f76cb00) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/kern/kern_intr.c:1263

#23 0xffffffff808af866 in ithread_loop (arg=0xfffff8000f908d20) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/kern/kern_intr.c:1276

#24 0xffffffff808acc41 in fork_exit (callout=0xffffffff808af7d0
<ithread_loop>, arg=0xfffff8000f908d20, frame=0xfffffe1046c2cac0) at
/usr/home/adrian/git/github/erikarn/freebsd/sys/kern/kern_fork.c:977

#25 0xffffffff80c939ae in fork_trampoline () at
/usr/home/adrian/git/github/erikarn/freebsd/sys/amd64/amd64/exception.S:605

#26 0x0000000000000000 in ?? ()


... there's only one path to deleting the credentials and that's via
tcp_twclose() -> tcp_tw_2msl_stop().

Has anyone seen this kind of problem before?


-a