panic: in_pcblookup_local (?)

[John Baldwin]

On Wednesday, May 01, 2013 2:08:57 pm Robert N. M. Watson wrote:
> 
> On 1 May 2013, at 19:03, Glen Barber wrote:
> 
> >> I'll need to catch up on this thread later, but a few questions:
> >> 
> >> Do we know if the application in question is multithreaded, and
> >> if so, might it be attempting concurrent operations on this socket?
> > 
> > I do not know if zabbix-agent is multithreaded, but cf-agent is.
> 
> If in DDB, it would be useful to do a "ps" so we can identify threads in the 
process, and in particular, whether they might be in the kernel around the 
moment of the panic.
> 
> > I will follow up with this information as soon as possible.
> 
> Thanks. Do keep around as much information as you can from DDB, crashdumps, 
etc. A useful set of things to keep from DDB includes the initial panic 
information and trap frame, "show pcpu", "show allpcpu", "trace", "alltrace", 
"ps", and if WITNESS is compiled in, "show locks" and "show alllocks". On busy 
systems, all the backtraces add up to a lot of space, so you might hold onto 
that rather than e-mail it, but contain useful information. Often, debugging 
this sort of race condition involves looking at what other network-centred 
threads are doing -- e.g., device-driver ithreads, netisr, other involved user 
threads. You may be able to extract much of that information using ps on the 
crashdump (not sure if procstat is there yet for crashdumps) -- if so, be sure 
to use -H (or whatever the argument is to print thread, not just process, 
information).

You can also grab my kgdb scripts from www.freebsd.org/~jhb/gdb/

Put those in a dir and do 'source gdb6'.  You can then run 'ps' to get a good 
ps listing that includes threads.  You can also use 'thread apply all bt' to 
get stacktraces of all threads in kgdb.  I believe there is an 'allpcpu' 
command that is similar to 'show allpcpu' in DDB.

Robert, in this case he has a full crashdump, so we can get quite a bit of 
information from it.

-- 
John Baldwin