More on MySQL -- Fatal trap 12

[Kris Gale]

>> What I seem to be seeing is a bogging down of MySQL
>> when new threads are being created in bursts.  This
>> causes MySQL to temporarily become unresponsive,
>> and will sometimes crash the whole system.
>
> My  first question is:
> "In what state are these threads waiting for work?
> are they in the kernel, or are they in userland?
> is there a single thread that listens on a socket and then hands the
> work to worker threads using some userland synchronisation, or do the
> threads enter the kernel and wait on the sockets themselves?
> (i.e. what does ps -H show?) "

I'm not really sure how to answer this.  During the startup
of my test script, when I'm seeing all of the connect errors,
"ps auxwH" shows a growing number of mysqld threads.
All but one have "SL" listed in the STAT column.  The other
one has "RL" listed.

> what happens if you have 1 process with X*Y threads?
> (just curious).

No problems.  1800 threads run fine.  The problem
seems to be with the child processes trying to initialize
connections simultaneously.  I can also "solve" the
problem by putting a sleep in so the child processes
aren't forked at the same time.  However, this is
not a workable real-world solution, since the application
I'm trying to simulate is a theoretical burst of connections
from a web cluster.

> is the server threaded process you are debugging, or is the
> test program what is being debugged?

The test program is just meant to simulate the problems I
saw in production.  I'm trying to test MySQL.

> use vmstat -z to check the values for:

Here's a snapshot from the system when mysql first starts:

UPCALL:           44,        0,      7,   2151,    11723
KSE:              84,        0,    384,   2158,    12096
KSEGRP:          128,        0,    381,   2099,    12081
THREAD:          328,        0,    389,   1915,    17090
PROC:            440,        0,    182,    196,     1611

Here's a snapshot during startup of my test script, while
I'm seeing lots of connect errors:

UPCALL:           44,        0,    346,   1812,    11725
KSE:              84,        0,    723,   1819,    12227
KSEGRP:          128,        0,    720,   1760,    12253
THREAD:          328,        0,    727,   1577,    17323
PROC:            440,        0,    273,    105,     1642

I wanted to include a snapshot of the system after
everything stabalized, but I got the trap 12 panic
just after running the second vmstat.

> Also, are they system scope threds or process scope?

What can I do to figure this out?

> what happens with libthr?

I haven't done very much testing with libthr, but the
little I've done shows that I still see the problems
handling that many connections, although it "catches
up" faster if I let it run.  I did also see the trap 12
crash with libthr.

The perl script I've written to do all of this is
totally self-contained, creates the table it needs
for testing, and should work on any system.

It needs p5-DBI, p5-DBD-mysql and p5-Digest-MD5
from ports.

http://hutta.com/test-threads.pl

I'd be interested to see if people have the same
problems I'm having on different hardware, with
different threading configurations, etc.

> of course it's fixable.. we just needd more info :-)

I appreciate everyone's help on this, and I'd be glad
to provide any info I can.  Just let me know what's
needed.

Kris Gale