kernel panic with swap zone exhausted, increase kern.maxswzone

[John Baldwin]

On Tuesday, September 27, 2011 1:46:19 am Eitan Adler wrote:
> My computer recently paniced and broke into ddb after spamming my
> console with "swap zone exhausted, increase kern.maxswzone"
> Immediately prior to the panic X was killed and I was able to switch
> to vty1 and log in as root (I planned on killing runaway programs)
> 
> I called doadump and have the saved textdump, vmcore.0 and kernel
> available to provide any relevant information. They are fairly large
> so I'll provide them upon request.
> 
> I'm running FreeBSD radar 9.0-BETA2 FreeBSD 9.0-BETA2 #0 r225471M: Mon
> Sep 12 20:43:44 EDT 2011
> eitan at radar:/usr/obj/usr/src/head/sys/EADLER  amd64
> 
> This is the diff between my kernel and GENERIC:
> http://people.freebsd.org/~eadler/files/diff-my-kernel-and-generic.diff
> 
> gdb> bt
> #0  doadump (textdump=0x31d622d0) at 
/usr/src/head/sys/kern/kern_shutdown.c:260
> #1  0xffffffff802ea54c in db_fncall (dummy1=Variable "dummy1" is not 
available.
> ) at /usr/src/head/sys/ddb/db_command.c:572
> #2  0xffffffff802ea881 in db_command (last_cmdp=0xffffffff80e5a6c0,
> cmd_table=Variable "cmd_table" is not available.
> )
>     at /usr/src/head/sys/ddb/db_command.c:448
> #3  0xffffffff802eaad0 in db_command_loop () at
> /usr/src/head/sys/ddb/db_command.c:501
> #4  0xffffffff802ecc19 in db_trap (type=Variable "type" is not available.
> ) at /usr/src/head/sys/ddb/db_main.c:229
> #5  0xffffffff807731f1 in kdb_trap (type=0x3, code=0x0, 
tf=0xffffff8231d62500)
>     at /usr/src/head/sys/kern/subr_kdb.c:631
> #6  0xffffffff809c0e36 in trap (frame=0xffffff8231d62500) at
> /usr/src/head/sys/amd64/amd64/trap.c:590
> #7  0xffffffff809ab5df in calltrap () at
> /usr/src/head/sys/amd64/amd64/exception.S:228
> #8  0xffffffff80772f9b in kdb_enter (why=0xffffffff80aec7a0 "panic",
> msg=0x80 <Address 0x80 out of bounds>)
>     at cpufunc.h:63
> #9  0xffffffff8073db00 in panic (fmt=Variable "fmt" is not available.
> ) at /usr/src/head/sys/kern/kern_shutdown.c:599
> #10 0xffffffff807836f8 in propagate_priority (td=0xfffffe011de2a000)
> at /usr/src/head/sys/kern/subr_turnstile.c:222

The actual panic is probably a "sleeping thread holding a mutex".  The 
messages in dmesg should have identified what thread was holding the mutex and 
output a backtrace.  If so, that is the first thing we need.

Alternatively, if that doesn't work, grab my gdb scripts from 
www.freebsd.org/~jhb/gdb and from within kgdb (on the crashdump) do this:

cd /path/to/gdb/scripts
source gdb6
frame 11
lockchain td->td_tid

Find the pid/tid of the last thread mentioned (should be sleeping on 
something) and switch to to it using 'proc <pid>' or 'tid <tid>' and get a 
backtrace.

-- 
John Baldwin