panic: in_pcblookup_local (?)

[John Baldwin]

On Tuesday, April 30, 2013 5:19:08 pm Glen Barber wrote:
> On Tue, Apr 30, 2013 at 04:53:13PM -0400, John Baldwin wrote:
> > Try 'p phd' to start.  INP_PCBPORTHASH is a macro, so you will
> > have to do it by hand:
> > 
> > 'p pcbinfo->ipi_porthashbase[lport & pcbinfo->ipi_porthashmask]'
> > 
> > (That should be what 'porthash' is.)
> > 
> 
> Thanks for the pointers.  (Hah!)
> 
> Hopefully this is the info you are looking for:
> 
> Script started on Tue Apr 30 17:16:07 2013
> root at orion:/usr/obj/usr/src/sys/ORION # kgdb ./kernel.debug 
/var/crash/vmcore.4
> [...]
> #0  doadump (textdump=<value optimized out>) at pcpu.h:231
> 231		__asm("movq %%gs:%1,%0" : "=r" (td)
> (kgdb) frame 6
> #6  0xffffffff80736cec in in_pcblookup_local (pcbinfo=0xffffffff80dc9180, 
laddr=
>       {s_addr = 50374848}, lport=339, lookupflags=1, 
cred=0xfffffe016cdad100)
>     at /usr/src/sys/netinet/in_pcb.c:1438
> 1438			LIST_FOREACH(phd, porthash, phd_hash) {
> (kgdb) p phd
> $1 = (struct inpcbport *) 0x9e17b100fffffe00

That is odd, that looks word-swapped, as if it should be
0xfffffe009e17b100 (which would be a more normal pointer in the kernel on 
amd64).

> (kgdb) p pcbinfo->ipi_porthashbase[lport & pcbinfo->ipi_porthashmask]
> $2 = {lh_first = 0x0}

So the list is now empty. :(

This feels like the list was updated out from under the pcbinfo.  Looking at
your earlier e-mail:

(kgdb) p *pcbinfo
$1 = {ipi_lock = {lock_object = {lo_name = 0xffffffff809d4d82 "udp", lo_flags 
= 69926912, 
      lo_data = 0, lo_witness = 0x0}, rw_lock = 1}, ipi_listhead = 
0xffffffff80dc9108, 
  ipi_count = 28, ipi_gencnt = 535501, ipi_lastport = 21249, ipi_lastlow = 0, 
  ipi_lasthi = 0, ipi_zone = 0xfffffe0017b60380, ipi_pcbgroups = 0x0, 
ipi_npcbgroups = 0, 
  ipi_hashfields = 0, ipi_hash_lock = {lock_object = {
      lo_name = 0xffffffff80a03d80 "pcbinfohash", lo_flags = 69402624, lo_data 
= 0, 
      lo_witness = 0x0}, rw_lock = 18446741877615517696}, ipi_hashbase = 
0xfffffe00120f6000, 
  ipi_hashmask = 127, ipi_porthashbase = 0xfffffe00120f5c04, ipi_porthashmask 
= 127, 
  ipi_wildbase = 0x0, ipi_wildmask = 0, ipi_vnet = 0x0, ipi_pspare = {0x0, 
0x0}}

It looks like the ipi_hash_lock is locked (and udp_connect() locks it), so I 
think the offending code is somewhere else.  Also, I can't find anything that
removes an inp without hold the correct pcbinfo lock.  Only thing I can think
of is if the pcbinfo pointer for an inp could change, so we could maybe
lock the wrong one while removing it?

Hmmmmmm, you know.  In in_pcbremlists() and in_pcbdrop(), we read inp_phd 
without holding the hash lock. I think that probably don't actaully break
anything, but this feels like a locking issue of some sort.

-- 
John Baldwin