freebsd-5.4-stable panics (fwd)

[Robert Watson]

The attached message and debugging information suggests that a thread in 
an incompletely initialized state (td->td_ksegrp == NULL) is being hooked 
up to a process, and that fill_kinfo_thread() is pretty sure this will 
never happen (it assumes that td_ksegrp != NULL).  I've suggested a 
work-around of skipping threads with td_ksegrp == NULL in the calling 
function, but this raises a general question about the invariants of the 
KSE/thread/... code during thread initialization.  Should a thread hooked 
up to a process thread list ever have a td_ksegrp of NULL?  Once a thread 
has a KSE group that is non-NULL, will it ever change such that monitoring 
code needs to worry?  Is kern_proc.c making assumptions that are too 
strong, or do we just have a nasty SMP race here and some locking work is 
needed?  The remainder of the thread can be found on the -hackers list, 
but the key points are:

- Shows up on quad-processor systems running 6.x
- Multi-threaded apps are doing lots of network I/O
- Occurs while running top under high load

Thanks,

Robert N M Watson

---------- Forwarded message ----------
Date: Thu, 29 Sep 2005 16:17:57 -0400 (EDT)
From: Rob Watt <rob at hudson-trading.com>
To: Robert Watson <rwatson at FreeBSD.org>
Cc: Rob Watt <rob at hudson-trading.com>, Jason Carroll <jason at hudson-trading.com>,
     freebsd-hackers at FreeBSD.org, freebsd-amd64 at FreeBSD.org,
     mikep at hudson-trading.com
Subject: Re: freebsd-5.4-stable panics

On Thu, 29 Sep 2005, Robert Watson wrote:

> Could you dump the contents of *td and *td->td_proc for me?  I'm quite
> interested to know what the value in td->td_proc->p_state is, among other
> things.  If I could also have you generate a dump of the KSE group
> structures in td->td_proc->p_ksegrps and the threads in
> td->td_proc->p_threads.

I've attached a file with many of the values you have asked for. We
looked at some of the threads referenced by td->td_proc->p_threads, but we
weren't sure we were walking the list correctly. Do you have any tips for
walking those thread lists?

>
> Could you tell me if the program named by p->p_comm is linked against a
> threading library?  If it's a custom app, you may already know, and if
> not, you can run ldd on the application to see what it is linked against.
>

The programs named by p->p_comm is linked against the pthreads library.

> Depending on how much time you have available, it might make sense for me
> to grab from you a copy of your source tree, compiled kernel with debug
> symbols, and core dump.

We can upload the source, kernel etc somewhere, but uncompressed that is
about 5G of data. What is the best way to get that to you?

Thanks.

-
Rob Watt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 6.0-BETA5.kgdb.out
Type: application/octet-stream
Size: 15282 bytes
Desc: 
Url : http://lists.freebsd.org/pipermail/freebsd-threads/attachments/20050929/d73d13d7/6.0-BETA5.kgdb.obj