FreeBSD 6.3 deadlock (vm_map?) with DDB output
Stef Walter
stef-list at memberwebs.com
Sun Jun 15 11:46:33 UTC 2008
I've been trying to track down a deadlock on some newish production
servers running FreeBSD 6.3-RELEASE-p2. The deadlock occurs on a
specific (although mundane) hardware configuration, and each of several
servers running this hardware deadlock about once per week.
Although I suspect that this is not hardware related, from a (naive)
perusal of the attached stack traces.
Forgive me if my interpretation of this is all wrong, but I'm pretty
desperate for help. So here's my basic understanding of the deadlock:
These processes seem to be waiting on the page queue mutex:
sendmail (in vm_mmap > vm_map_find > vm_map_insert > vm_map_pmap_enter)
bsnmpd (in malloc, uma_large_malloc > page_alloc > kmem_malloc)
httpd (in trap > trap_pfault > vm_fault)
[g_up] (in g_vfs_done > bufdone)
The page queue mutex is held by rsync process:
rsync (in trap > trap_pfault > vm_fault > pmap_enter)
Rsync kernel process (in pmap_enter) was interrupted while holding the
page queue lock?
Giant is enabled in loader.conf due to the needs of the pf firewall when
dealing with user credentials lookups. I do not believe that Giant plays
into this deadlock. Kernel config attached.
Any and all help or info is welcome. Thanks in advance.
Stef Walter
-------------- next part --------------
# --------------------------------------------------------------------
# FREEBSD BUILD
FreeBSD m2.npubs.com 6.3-RELEASE-p2 FreeBSD 6.3-RELEASE-p2 #0: Tue Jun 10 02:06:41 UTC 2008 nate at m2.npubs.com:/usr/obj/usr/src/sys/RACK2DBG i386
# --------------------------------------------------------------------
#
# --------------------------------------------------------------------
# PROCESS LIST
db> ps
pid ppid pgrp uid state wmesg wchan cmd
35638 35637 4448 80 LJ *vm page 0xc8d5f780 sendmail
35637 40546 4448 80 SJ wait 0xc9e79218 sh
35620 1801 1801 80 SJ lockf 0xcceb4d00 httpd
35593 1801 1801 80 SJ lockf 0xcce46900 httpd
35566 1801 1801 80 SJ kqread 0xc8f8e000 httpd
35438 15070 15063 70 SJ sbwait 0xd017220c postgres
35401 4527 4527 10219 SJ select 0xc080b7e4 cleanup
35337 4527 4527 10219 SJ select 0xc080b7e4 pickup
35144 4413 4413 80 SJ select 0xc080b7e4 httpd
34797 8611 8611 0 SJ select 0xc080b7e4 virtual
34784 8611 8611 125 SJ select 0xc080b7e4 smtpd
34781 8611 8611 125 SJ select 0xc080b7e4 smtp
34761 8611 8611 125 SJ select 0xc080b7e4 proxymap
34760 8611 8611 125 SJ select 0xc080b7e4 smtpd
34757 8611 8611 125 SJ select 0xc080b7e4 smtp
34750 8611 8611 125 SJ select 0xc080b7e4 cleanup
34087 6435 6435 125 SJ kqread 0xcae9f900 anvil
34075 6435 6435 125 SJ kqread 0xcfb5b600 smtpd
33801 15070 15063 70 SJ sbwait 0xd01a8bc8 postgres
33796 4527 4527 10219 SJ select 0xc080b7e4 bounce
33423 1801 1801 80 SJ select 0xc080b7e4 httpd
33417 15070 15063 70 SJ sbwait 0xd087f638 postgres
33392 1801 1801 80 SJ select 0xc080b7e4 httpd
33207 4527 4527 10219 SJ select 0xc080b7e4 flush
33200 4527 4527 10219 SJ select 0xc080b7e4 bounce
33125 4527 4527 10219 SJ kqread 0xcb2ff900 smtp
33118 4527 4527 10219 SJ select 0xc080b7e4 smtp
33113 4527 4527 10219 SJ select 0xc080b7e4 smtp
33108 4527 4527 10219 SJ select 0xc080b7e4 smtp
33101 4527 4527 10219 SJ select 0xc080b7e4 smtp
33094 4527 4527 10219 SJ select 0xc080b7e4 smtp
33090 4527 4527 10219 SJ select 0xc080b7e4 smtp
33070 33067 33067 0 LJ *Giant 0xcaaf97c0 lynx
33067 33066 33067 0 SsJ wait 0xc9fd9648 sh
33066 4564 4564 0 SJ piperd 0xd0ab7000 cron
33056 4527 4527 10219 SJ select 0xc080b7e4 smtp
33055 4527 4527 10219 SJ lockf 0xc992f480 bounce
32994 15070 15063 70 SJ sbwait 0xc9cdc370 postgres
32976 15070 15063 70 SJ sbwait 0xd0f8e4d4 postgres
32971 4527 4527 10219 SJ select 0xc080b7e4 proxymap
32970 4527 4527 10219 SJ select 0xc080b7e4 smtpd
32869 15070 15063 70 SJ sbwait 0xc8fcc370 postgres
32852 9298 9298 10102 SJ select 0xc080b7e4 cleanup
32837 32836 32836 0 S select 0xc080b7e4 rsync
32836 7802 32836 0 RLs rsync
32774 4527 4527 0 SJ select 0xc080b7e4 local
32754 1801 1801 80 LLJ *vm page 0xc8d5f780 httpd
32753 15070 15063 70 SJ sbwait 0xc9c8ce90 postgres
32747 15070 15063 70 SJ sbwait 0xd01a8370 postgres
32739 15070 15063 70 SJ sbwait 0xccdd3d2c postgres
32710 1 32710 0 Ss biord 0xdcb814d4 jail-measure
32633 1801 1801 80 SJ select 0xc080b7e4 httpd
32623 1801 1801 80 SJ select 0xc080b7e4 httpd
32599 1801 1801 80 SJ select 0xc080b7e4 httpd
32598 1801 1801 80 SJ select 0xc080b7e4 httpd
32597 1801 1801 80 SJ select 0xc080b7e4 httpd
32596 1801 1801 80 SJ lockf 0xcceb43c0 httpd
32595 1801 1801 80 SJ select 0xc080b7e4 httpd
32301 4413 4413 80 SJ lockf 0xcce44680 httpd
32255 4413 4413 80 SJ lockf 0xcce44340 httpd
32206 15070 15063 70 SJ sbwait 0xc9cdf638 postgres
32163 1801 1801 80 SJ select 0xc080b7e4 httpd
32147 4527 4527 10219 SJ select 0xc080b7e4 trivial-rewrite
31910 8611 8611 125 SJ select 0xc080b7e4 smtpd
31740 8611 8611 125 SJ select 0xc080b7e4 trivial-rewrite
30378 4413 4413 80 SJ lockf 0xcd551b40 httpd
30355 4413 4413 80 SJ lockf 0xcceb41c0 httpd
29801 4413 4413 80 SJ lockf 0xcb331340 httpd
29375 9298 9298 10102 SJ select 0xc080b7e4 pickup
28283 4413 4413 80 SJ lockf 0xcb3317c0 httpd
28247 4413 4413 80 SJ kqread 0xcb092200 httpd
28113 9298 9298 10102 SJ select 0xc080b7e4 smtpd
28067 4413 4413 80 SJ lockf 0xd0ff87c0 httpd
26846 4413 4413 80 SJ select 0xc080b7e4 httpd
26581 9298 9298 10102 SJ select 0xc080b7e4 smtpd
26529 26528 8216 125 SJ select 0xc080b7e4 imapd
26528 8216 8216 0 SJ select 0xc080b7e4 couriertls
26424 4413 4413 80 SJ lockf 0xc92fe900 httpd
23902 15070 15063 70 SJ sbwait 0xccdcb0a8 postgres
23891 1801 1801 80 SJ select 0xc080b7e4 httpd
23721 8611 8611 125 SJ select 0xc080b7e4 pickup
22472 6435 6435 125 SJ kqread 0xcfb0cb00 pickup
21951 8958 8958 125 SJ kqread 0xcfb5b200 pickup
19724 3711 3711 125 SJ kqread 0xcfbee800 pickup
17335 9298 9298 10102 SJ select 0xc080b7e4 proxymap
17334 9298 9298 10102 SJ select 0xc080b7e4 trivial-rewrite
11733 40103 40103 80 SJ accept 0xd087d03a httpd
11732 40103 40103 80 SJ accept 0xd087d03a httpd
11731 40103 40103 80 SJ accept 0xd087d03a httpd
11730 40103 40103 80 SJ accept 0xd087d03a httpd
11728 40103 40103 80 SJ accept 0xd087d03a httpd
11727 40103 40103 80 SJ accept 0xd087d03a httpd
11726 40103 40103 80 SJ accept 0xd087d03a httpd
2181 1 2181 181 SsJ (threaded) nagios
100858 S select 0xc080b7e4 nagios
100434 S nanslp 0xc07be46c nagios
87785 7530 7530 80 SJ accept 0xca004b5a httpd
61004 8697 8697 80 SJ accept 0xca96519e httpd
56340 1917 1917 80 SJ lockf 0xcb3311c0 httpd
53973 8611 8611 125 SJ select 0xc080b7e4 anvil
52531 17166 52531 0 R+J CPU 3 ee
45059 7530 7530 80 SJ accept 0xca004b5a httpd
41862 4448 4448 80 SJ accept 0xc9c7d5ca httpd
41181 4448 4448 80 SJ accept 0xc9c7d5ca httpd
41180 4448 4448 80 SJ accept 0xc9c7d5ca httpd
41179 4448 4448 80 SJ accept 0xc9c7d5ca httpd
41178 7530 7530 80 SJ accept 0xca004b5a httpd
41175 4448 4448 80 SJ accept 0xc9c7d5ca httpd
40748 9036 9036 80 SJ accept 0xca965466 httpd
40742 9036 9036 80 SJ accept 0xca965466 httpd
40739 9036 9036 80 SJ accept 0xca965466 httpd
40726 9036 9036 80 SJ accept 0xca965466 httpd
40546 4448 4448 80 SJ wait 0xd024c000 httpd
40444 1917 1917 80 SJ lockf 0xcaa00580 httpd
40443 1917 1917 80 SJ kqread 0xcf271100 httpd
40442 1917 1917 80 SJ lockf 0xcce44500 httpd
40441 1917 1917 80 SJ lockf 0xcd551480 httpd
40440 1917 1917 80 SJ lockf 0xc8dba6c0 httpd
40428 8697 8697 80 SJ accept 0xca96519e httpd
40427 8697 8697 80 SJ accept 0xca96519e httpd
40426 8697 8697 80 SJ accept 0xca96519e httpd
40425 8697 8697 80 SJ accept 0xca96519e httpd
40424 8697 8697 80 SJ accept 0xca96519e httpd
40384 9036 9036 80 SJ accept 0xca965466 httpd
40378 7530 7530 80 SJ accept 0xca004b5a httpd
40377 7530 7530 80 SJ accept 0xca004b5a httpd
40376 7530 7530 80 SJ accept 0xca004b5a httpd
40375 7530 7530 80 SJ accept 0xca004b5a httpd
40374 7530 7530 80 SJ accept 0xca004b5a httpd
40372 9036 9036 80 SJ accept 0xca965466 httpd
40371 9036 9036 80 SJ accept 0xca965466 httpd
40370 9036 9036 80 SJ accept 0xca965466 httpd
40369 9036 9036 80 SJ accept 0xca965466 httpd
40368 9036 9036 80 SJ accept 0xca965466 httpd
40334 40332 4448 0 SJ piperd 0xcca2e198 cronolog
40333 4448 4448 80 SJ accept 0xc9c7d5ca httpd
40332 4448 4448 0 SJ wait 0xcb2cf218 sh
40310 40309 40309 0 SJ piperd 0xc9257cc0 sockin
40309 8742 40309 0 SsJ wait 0xd0171430 sh
40294 40291 40291 0 SJ piperd 0xc8fe7cc0 sockin
40291 6919 40291 0 SsJ wait 0xcdccba78 sh
36263 1 36263 0 SsJ (threaded) rrdbotd
100452 S kserel 0xce155d54 rrdbotd
100188 S select 0xc080b7e4 rrdbotd
100984 S kserel 0xce155d54 rrdbotd
100311 S kserel 0xce155d54 rrdbotd
100271 S select 0xc080b7e4 rrdbotd
100268 S kserel 0xce155d54 rrdbotd
100624 S ksesigwa 0xcaa27780 rrdbotd
17166 17165 17166 0 S+J wait 0xc994d000 bash
17165 17155 17152 0 S+J wait 0xd0d11218 su
17155 17152 17152 0 S+ wait 0xc9ec0a78 sh
17152 1980 17152 0 S+ wait 0xc8a42648 sh
15079 15078 15063 70 S+J select 0xc080b7e4 postgres
15078 15070 15063 70 S+J select 0xc080b7e4 postgres
15077 15070 15063 70 S+J select 0xc080b7e4 postgres
15074 15070 15063 70 S+J select 0xc080b7e4 postgres
15070 1 15063 70 S+J select 0xc080b7e4 postgres
1980 1 1980 0 S+ wait 0xcaa27430 bash
31591 2354 2354 80 SJ lockf 0xc8c8ee80 httpd
31578 2354 2354 80 SJ lockf 0xc8c8e640 httpd
40103 1 40103 0 SsJ nanslp 0xc07be46c httpd
38293 1 38293 0 SsJ nanslp 0xc07be46c cron
38286 1 38286 0 SsJ select 0xc080b7e4 sshd
38224 1 38224 0 SsJ select 0xc080b7e4 syslogd
91887 1 91886 0 SJ select 0xc080b7e4 ruby18
91876 1 91875 0 SJ select 0xc080b7e4 ruby18
61971 61968 8645 65534 SJ piperd 0xcd2d3000 cat
61970 61967 8645 65534 SJ piperd 0xcca51cc0 cat
61968 1 8645 65534 SJ wait 0xce938c90 sh
61967 1 8645 65534 SJ wait 0xc9d2c860 sh
64190 2354 2354 80 SJ kqread 0xcbe8e500 httpd
19433 19430 8645 65534 SJ piperd 0xcffe74c8 cat
19432 19431 8645 65534 SJ piperd 0xcad1d000 cat
19431 1 8645 65534 SJ wait 0xca5b1860 sh
19430 1 8645 65534 SJ wait 0xd0195430 sh
72427 3606 3606 80 SJ select 0xc080b7e4 httpd
95713 3606 3606 80 SJ lockf 0xcd551800 httpd
79740 3606 3606 80 SJ lockf 0xc8c8e7c0 httpd
36833 1 36833 0 SsJ select 0xc080b7e4 ruby
15574 3606 3606 80 SJ lockf 0xcaa00680 httpd
13381 6677 6677 80 SJ accept 0xc9df4466 httpd
11234 9367 9367 80 SJ lockf 0xc8d16a80 httpd
10994 9367 9367 80 SJ lockf 0xd00c2900 httpd
7909 6752 6677 80 SJ select 0xc080b7e4 ruby
73129 6677 6677 80 SJ accept 0xc9df4466 httpd
58056 6752 6677 80 SJ select 0xc080b7e4 ruby
41698 3606 3606 80 SJ lockf 0xc8a53740 httpd
41694 8569 8569 80 SJ accept 0xca965b5a httpd
37070 3266 3266 80 SJ kqread 0xcadb3700 httpd
37069 3266 3266 80 SJ lockf 0xcb7f4280 httpd
37068 3266 3266 80 SJ lockf 0xcec83700 httpd
30486 9367 9367 80 SJ lockf 0xd0ff8800 httpd
30262 9367 9367 80 SJ lockf 0xca6e3d40 httpd
11692 8743 8743 80 SJ lockf 0xcb7f4b80 httpd
11690 8743 8743 80 SJ select 0xc080b7e4 httpd
11599 8743 8743 80 SJ lockf 0xcb331bc0 httpd
11214 8743 8743 80 SJ lockf 0xcb0df340 httpd
11169 8743 8743 80 SJ lockf 0xd0043c00 httpd
9660 6677 6677 80 SJ accept 0xc9df4466 httpd
9659 6752 6677 80 SJ select 0xc080b7e4 ruby
9445 3266 3266 80 SJ lockf 0xcb0df900 httpd
9407 8611 8611 125 SJ select 0xc080b7e4 tlsmgr
9404 2354 2354 80 SJ lockf 0xc95798c0 httpd
9401 9367 9367 80 SJ lockf 0xcb7f4ac0 httpd
9400 9367 9367 80 SJ lockf 0xc95628c0 httpd
9399 9367 9367 80 SJ lockf 0xd05bd080 httpd
9398 9367 9367 80 SJ kqread 0xca619400 httpd
9397 9367 9367 80 SJ lockf 0xc9b5b440 httpd
9382 1 9382 0 SsJ nanslp 0xc07be46c cron
9375 1 9375 0 SsJ select 0xc080b7e4 sshd
9367 1 9367 0 SsJ nanslp 0xc07be46c httpd
9359 3266 3266 80 SJ lockf 0xc957a0c0 httpd
9334 9333 8760 70 SJ select 0xc080b7e4 postgres
9333 8760 8760 70 SJ select 0xc080b7e4 postgres
9332 8760 8760 70 SJ select 0xc080b7e4 postgres
9328 1 9328 0 SsJ nanslp 0xc07be46c cron
9320 1 9320 0 SsJ select 0xc080b7e4 inetd
9315 1 9315 0 SsJ accept 0xcaec119e saslauthd1
9304 9298 9298 10102 SJ select 0xc080b7e4 qmgr
9298 1 9298 0 SsJ select 0xc080b7e4 master
9268 2354 2354 80 SJ lockf 0xcec83ac0 httpd
9241 1 9241 0 SsJ select 0xc080b7e4 bsnmpd
9214 1 9214 0 SsJ nanslp 0xc07be46c cron
9202 1 9202 0 SsJ select 0xc080b7e4 sshd
9139 9107 9090 88 SJ (threaded) mysqld
100427 S kserel 0xc9a0b4b4 mysqld
100308 S kserel 0xc9a0b4b4 mysqld
100276 S kserel 0xc9a0b4b4 mysqld
100582 S kserel 0xc9a0b4b4 mysqld
100572 S select 0xc080b7e4 mysqld
100568 S kserel 0xcadf8394 mysqld
100571 S sigwait 0xeca04c14 mysqld
100564 S ksesigwa 0xc9b7f780 mysqld
9107 1 9090 88 SJ wait 0xcac42a78 sh
9093 1 9093 225 SsJ select 0xc080b7e4 perl5.8.8
9068 1 9068 0 SsJ nanslp 0xc07be46c cron
9058 1 9058 0 SsJ select 0xc080b7e4 sshd
9042 1 41 0 SJ piperd 0xcad1dcc0 logger
9041 1 41 0 SJ fifoor 0xca9e6c58 tail
9036 1 9036 0 SsJ select 0xc080b7e4 httpd
9035 8996 8980 88 SJ (threaded) mysqld
100855 S kserel 0xc8cee394 mysqld
101020 S sbwait 0xc950fd2c mysqld
100994 S select 0xc080b7e4 mysqld
101012 S sbwait 0xcce6320c mysqld
101021 S sbwait 0xd08750a8 mysqld
100393 S kserel 0xc8cee394 mysqld
100999 S sbwait 0xd0eab370 mysqld
101189 S sbwait 0xd0aa1bc8 mysqld
100578 S kserel 0xc8cee394 mysqld
101017 S kserel 0xc8cee394 mysqld
100573 S kserel 0xcadf82d4 mysqld
100558 S sigwait 0xec83bc14 mysqld
100555 S ksesigwa 0xc97f4780 mysqld
8996 1 8980 88 SJ wait 0xca0c4860 sh
8963 8958 8958 125 SJ kqread 0xcabf6400 qmgr
8958 1 8958 0 SsJ kqread 0xc8d71500 master
8893 1 8893 0 SsJ nanslp 0xc07be46c cron
8874 1 8874 0 SsJ select 0xc080b7e4 sshd
8844 1 8844 0 SsJ select 0xc080b7e4 sshd
8811 1 8811 0 SsJ nanslp 0xc07be46c cron
8804 8569 8569 80 SJ accept 0xca965b5a httpd
8802 8569 8569 80 SJ accept 0xca965b5a httpd
8801 8569 8569 80 SJ accept 0xca965b5a httpd
8799 8569 8569 80 SJ accept 0xca965b5a httpd
8798 1 8798 0 SsJ select 0xc080b7e4 sshd
8797 8569 8569 80 SJ accept 0xca965b5a httpd
8791 1 8791 0 SsJ select 0xc080b7e4 proftpd
8769 8743 8743 80 SJ lockf 0xcd6aeb00 httpd
8768 8743 8743 80 SJ lockf 0xcb7f4880 httpd
8767 8743 8743 80 SJ lockf 0xcb06a580 httpd
8766 8743 8743 80 SJ lockf 0xcb0df140 httpd
8765 8743 8743 80 SJ lockf 0xc8fb1700 httpd
8760 1 8760 70 SsJ select 0xc080b7e4 postgres
8743 1 8743 0 SsJ select 0xc080b7e4 httpd
8742 1 8742 0 SsJ select 0xc080b7e4 syslogd
8715 1 8715 0 SsJ nanslp 0xc07be46c cron
8706 1 8706 0 SsJ select 0xc080b7e4 sshd
8697 1 8697 0 SsJ nanslp 0xc07be46c httpd
8645 1 8645 65534 SsJ (threaded) proxsmtpd
100309 S kserel 0xca663154 proxsmtpd
100975 S kserel 0xca663154 proxsmtpd
101018 S kserel 0xca663154 proxsmtpd
100649 S kserel 0xca663154 proxsmtpd
101013 S accept 0xca9655ca proxsmtpd
100581 S ksesigwa 0xcaaf7138 proxsmtpd
8617 8611 8611 125 SJ select 0xc080b7e4 qmgr
8611 1 8611 0 SsJ select 0xc080b7e4 master
8569 1 8569 0 SsJ nanslp 0xc07be46c httpd
8496 1 8496 0 SsJ select 0xc080b7e4 syslogd
8437 1 8437 0 SsJ select 0xc080b7e4 syslogd
8300 8170 8169 0 SJ select 0xc080b7e4 authdaemond
8299 8170 8169 0 SJ select 0xc080b7e4 authdaemond
8298 8170 8169 0 SJ select 0xc080b7e4 authdaemond
8295 1 8295 0 SsJ select 0xc080b7e4 syslogd
8272 8271 8272 0 SJ select 0xc080b7e4 couriertcpd
8271 1 8271 0 SJ piperd 0xca366b28 courierlogger
8253 8252 8253 0 SJ select 0xc080b7e4 couriertcpd
8252 1 8252 0 SJ piperd 0xc904f990 courierlogger
8229 8228 8229 0 SJ select 0xc080b7e4 couriertcpd
8228 1 8228 0 SJ piperd 0xc9e1cb28 courierlogger
8216 8215 8216 0 SJ select 0xc080b7e4 couriertcpd
8215 1 8215 0 SJ piperd 0xca610330 courierlogger
8170 8169 8169 0 SJ select 0xc080b7e4 authdaemond
8169 1 8169 0 SJ piperd 0xc9dc1cc0 courierlogger
8121 1 8121 0 SsJ nanslp 0xc07be46c cron
8105 1 8105 0 SsJ select 0xc080b7e4 sshd
8082 1 8082 0 SsJ select 0xc080b7e4 syslogd
8081 1 8081 0 SsJ select 0xc080b7e4 syslogd
7979 1 7979 0 SsJ nanslp 0xc07be46c cron
7972 1 7972 0 SsJ select 0xc080b7e4 sshd
7848 1 7848 0 Ss+ ttyin 0xc8a6e410 getty
7847 1 7847 0 Ss+ ttyin 0xc8a74810 getty
7846 1 7846 0 Ss+ ttyin 0xc8a73010 getty
7845 1 7845 0 Ss+ ttyin 0xc8a73410 getty
7844 1 7844 0 Ss+ ttyin 0xc8a70c10 getty
7843 1 7843 0 Ss+ ttyin 0xc8a73c10 getty
7842 1 7842 0 Ss+ ttyin 0xc8a76010 getty
7841 1 7841 0 Ss+ ttyin 0xc8a75010 getty
7840 1 7840 0 Ss+ ttyin 0xc8a74410 getty
7829 1 7829 0 Ls *vm page 0xc8d5f780 bsnmpd
7802 1 7802 0 Ss select 0xc080b7e4 inetd
7695 1 7695 0 SsJ select 0xc080b7e4 syslogd
7530 1 7530 0 SsJ nanslp 0xc07be46c httpd
7302 1 7302 0 SsJ select 0xc080b7e4 bsnmpd
6919 1 6919 0 SsJ select 0xc080b7e4 syslogd
6754 6677 6677 80 SJ accept 0xc9df4466 httpd
6753 6677 6677 80 SJ accept 0xc9df4466 httpd
6752 6677 6677 80 SJ select 0xc080b7e4 httpd
6694 1 6694 0 SsJ nanslp 0xc07be46c cron
6687 1 6687 0 SsJ select 0xc080b7e4 sshd
6677 1 6677 0 SsJ nanslp 0xc07be46c httpd
6639 6578 6577 88 S+J piperd 0xca3517f8 logger
6638 6578 6577 88 S+J (threaded) mysqld
100566 S kserel 0xc8e765d4 mysqld
100383 S kserel 0xc8e765d4 mysqld
100988 S kserel 0xc8e765d4 mysqld
101056 S kserel 0xc8e765d4 mysqld
101005 S select 0xc080b7e4 mysqld
100392 S sbwait 0xcceda370 mysqld
100387 S sigwait 0xec6adc14 mysqld
100381 S ksesigwa 0xc9a09568 mysqld
6578 1 6577 88 S+J wait 0xca0c6430 sh
6568 6567 6568 0 S+J select 0xc080b7e4 couriertcpd
6567 1 6567 0 S+J piperd 0xca365660 courierlogger
6509 6508 6509 0 S+J select 0xc080b7e4 couriertcpd
6508 1 6508 0 S+J piperd 0xc99fe990 courierlogger
6505 1 6505 0 SsJ select 0xc080b7e4 syslogd
6454 6435 6435 125 SJ kqread 0xca2a1400 qmgr
6435 1 6435 0 SsJ kqread 0xca122500 master
5653 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5652 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5651 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5650 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5649 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5529 5521 5521 0 S+J select 0xc080b7e4 authdaemond
5521 1 5521 0 S+J piperd 0xc8ee7cc0 courierlogger
5337 1 5337 0 SsJ select 0xc080b7e4 syslogd
5166 1 5166 0 SsJ nanslp 0xc07be46c cron
5153 1 5153 0 SsJ select 0xc080b7e4 sshd
5117 1 5117 0 SsJ (threaded) httpauthd
100563 S kserel 0xc8cee2d4 httpauthd
100652 S kserel 0xc8cee2d4 httpauthd
100468 S kserel 0xc8cee2d4 httpauthd
100444 S kserel 0xc8cee2d4 httpauthd
100183 S sbwait 0xc9df4bc8 httpauthd
100277 S sbwait 0xcb305370 httpauthd
100382 S sbwait 0xcafa0638 httpauthd
100314 S sbwait 0xcb1054d4 httpauthd
100499 S sbwait 0xcae5aa64 httpauthd
100223 S sbwait 0xcae4d4d4 httpauthd
100310 S sbwait 0xcb0bfd2c httpauthd
100973 S sbwait 0xcebd34d4 httpauthd
100388 S sbwait 0xcae4d20c httpauthd
100646 S sbwait 0xd0aa0900 httpauthd
100658 S sbwait 0xc8d3779c httpauthd
100471 S sbwait 0xcae4ca64 httpauthd
100985 S sbwait 0xc9ce420c httpauthd
100175 S sbwait 0xd0cdae90 httpauthd
100317 S sbwait 0xcebce4d4 httpauthd
100580 S sbwait 0xcb105638 httpauthd
100587 S accept 0xc9e0319e httpauthd
100272 S sbwait 0xcb0aa0a8 httpauthd
100556 S sbwait 0xcae59e90 httpauthd
100562 S sbwait 0xcb154e90 httpauthd
100306 S sbwait 0xd0173e90 httpauthd
100221 S sbwait 0xcebc4370 httpauthd
100448 S sbwait 0xd01744d4 httpauthd
100380 S sbwait 0xca004638 httpauthd
100318 S sbwait 0xc9c7da64 httpauthd
100560 S sbwait 0xcaec0370 httpauthd
100205 S sbwait 0xc9dd9e90 httpauthd
100394 S sbwait 0xcb0aaa64 httpauthd
100251 S sbwait 0xcabc120c httpauthd
101072 S sbwait 0xd07560a8 httpauthd
100575 S sbwait 0xccdd7bc8 httpauthd
101207 S sbwait 0xcfd28370 httpauthd
101211 S sbwait 0xd087c370 httpauthd
100469 S sbwait 0xd0177d2c httpauthd
101004 S sbwait 0xc8fa0638 httpauthd
100253 S sbwait 0xc9e04d2c httpauthd
100585 S sbwait 0xca265a64 httpauthd
100270 S sbwait 0xcaf8f20c httpauthd
100313 S sbwait 0xd01780a8 httpauthd
100673 S sbwait 0xcaf8ce90 httpauthd
100651 S sbwait 0xcae6f638 httpauthd
100324 S sbwait 0xcfd27370 httpauthd
100259 S sbwait 0xc9ce0638 httpauthd
100391 S sbwait 0xc9dd9370 httpauthd
100321 S sbwait 0xd01a8900 httpauthd
100476 S sbwait 0xcafa0a64 httpauthd
101190 S sbwait 0xcaf8fbc8 httpauthd
100316 S sbwait 0xc9ce379c httpauthd
100657 S sbwait 0xcb090638 httpauthd
100594 S sbwait 0xd01720a8 httpauthd
100559 S sbwait 0xd01a9638 httpauthd
100583 S sbwait 0xcaf8b900 httpauthd
100574 S sbwait 0xcb154a64 httpauthd
100364 S sbwait 0xc9cd8bc8 httpauthd
100567 S sbwait 0xcf485e90 httpauthd
100971 S sbwait 0xca254900 httpauthd
101016 S sbwait 0xccdf2d2c httpauthd
100977 S sbwait 0xcb0aa638 httpauthd
100447 S sbwait 0xc9cd9d2c httpauthd
100856 S sbwait 0xd0176d2c httpauthd
100586 S sbwait 0xcebcfe90 httpauthd
100249 S sbwait 0xc8d3b370 httpauthd
100972 S sbwait 0xc8d3ba64 httpauthd
100450 S sbwait 0xcaa86370 httpauthd
100384 S sbwait 0xcebcebc8 httpauthd
100981 S sbwait 0xccdcba64 httpauthd
100591 S sbwait 0xd01760a8 httpauthd
100998 S sbwait 0xccdf279c httpauthd
100320 S sbwait 0xcab88d2c httpauthd
101031 S sbwait 0xcdb35a64 httpauthd
100349 S sbwait 0xcf4a7bc8 httpauthd
100589 S sbwait 0xcaf8b79c httpauthd
100386 S sbwait 0xd0cd60a8 httpauthd
100979 S sbwait 0xcae4de90 httpauthd
100590 S sbwait 0xccdcbd2c httpauthd
100569 S sbwait 0xcacd2bc8 httpauthd
101008 S sbwait 0xcebd3638 httpauthd
100672 S sbwait 0xcb153900 httpauthd
100557 S sbwait 0xcb153a64 httpauthd
100541 S sbwait 0xcae59bc8 httpauthd
100389 S sbwait 0xca265900 httpauthd
100415 S sbwait 0xcaf8e0a8 httpauthd
100577 S sbwait 0xce18cbc8 httpauthd
101025 S sbwait 0xcf4a7d2c httpauthd
100561 S sbwait 0xcebd0d2c httpauthd
100385 S sbwait 0xd01774d4 httpauthd
100256 S sbwait 0xcceb74d4 httpauthd
100284 S sbwait 0xcaf8cd2c httpauthd
101040 S sbwait 0xc9cdcbc8 httpauthd
100319 S sbwait 0xd0172d2c httpauthd
100274 S sbwait 0xcb0bfe90 httpauthd
100269 S sbwait 0xd0cd84d4 httpauthd
101011 S sbwait 0xcb3294d4 httpauthd
101069 S sbwait 0xd0cdb79c httpauthd
100449 S sbwait 0xcce6379c httpauthd
100970 S sbwait 0xc8f75a64 httpauthd
101002 S sbwait 0xcb2de0a8 httpauthd
100983 S sbwait 0xc8d3b638 httpauthd
100312 S ksesigwa 0xc97f4bb0 httpauthd
5000 1 5000 389 SsJ (threaded) slapd
100993 S kserel 0xc97f6c34 slapd
100445 S kserel 0xc97f6c34 slapd
100694 S kserel 0xc97f6c34 slapd
100247 S kserel 0xc97f6c34 slapd
100857 S select 0xc080b7e4 slapd
100305 S ksesigwa 0xc9e24138 slapd
4867 1 4867 0 SsJ select 0xc080b7e4 inetd
4847 1 4847 0 SsJ accept 0xc9c7de22 svnserve
4835 1 4834 0 SJ select 0xc080b7e4 ruby18
4796 1 4796 0 SsJ select 0xc080b7e4 syslogd
4712 1 4711 0 SJ select 0xc080b7e4 ruby18
4707 1 4706 0 SJ select 0xc080b7e4 ruby18
4701 1 4700 0 SJ select 0xc080b7e4 ruby18
4648 1 4647 0 SJ select 0xc080b7e4 ruby18
4564 1 4564 0 SsJ nanslp 0xc07be46c cron
4555 1 4555 0 SsJ select 0xc080b7e4 inetd
4550 1 4550 0 SsJ accept 0xc9c7dcbe saslauthd1
4542 1 4541 0 SJ select 0xc080b7e4 ruby18
4535 4527 4527 10219 SJ select 0xc080b7e4 qmgr
4527 1 4527 0 SsJ select 0xc080b7e4 master
4454 1 41 0 S+J piperd 0xc8ee07f8 logger
4453 1 41 0 S+J fifoor 0xc8d04a38 tail
4448 1 4448 0 SsJ select 0xc080b7e4 httpd
4428 1 4428 0 SsJ nanslp 0xc07be46c cron
4421 1 4421 0 SsJ select 0xc080b7e4 sshd
4413 1 4413 0 SsJ nanslp 0xc07be46c httpd
4396 1 4396 0 SsJ select 0xc080b7e4 sshd
4335 1 4335 0 SsJ select 0xc080b7e4 syslogd
4174 4106 4105 88 S+J (threaded) mysqld
100565 S kserel 0xc9a0b394 mysqld
100519 S kserel 0xc9a0b394 mysqld
100976 S kserel 0xc9a0b394 mysqld
100980 S kserel 0xc9a0b394 mysqld
100859 S select 0xc080b7e4 mysqld
101014 S sbwait 0xc950fe90 mysqld
100991 S sbwait 0xd0cd7638 mysqld
101003 S sbwait 0xcdb354d4 mysqld
101001 S sbwait 0xd074820c mysqld
100588 S sbwait 0xccdcb370 mysqld
100454 S sbwait 0xcaf8c79c mysqld
100273 S sigwait 0xec480c14 mysqld
100267 S ksesigwa 0xc9b7fdc8 mysqld
4106 1 4105 88 S+J wait 0xc97f4218 sh
4033 1 4033 0 SsJ select 0xc080b7e4 syslogd
3879 1 3879 0 SsJ select 0xc080b7e4 inetd
3865 1 3865 0 SsJ nanslp 0xc07be46c cron
3858 1 3858 0 SsJ select 0xc080b7e4 sshd
3857 3820 3817 88 S+J (threaded) mysqld
100252 S kserel 0xc8cee4b4 mysqld
100365 S kserel 0xc8cee4b4 mysqld
100245 S select 0xc080b7e4 mysqld
100323 S kserel 0xc8cee4b4 mysqld
100650 S kserel 0xc8cee4b4 mysqld
100250 S kserel 0xc9a0b634 mysqld
100244 S sigwait 0xe9ef5c14 mysqld
100242 S ksesigwa 0xc97f4138 mysqld
3820 1 3817 88 S+J wait 0xc994d430 sh
3818 3606 3606 80 SJ lockf 0xc97e7180 httpd
3816 3606 3606 80 SJ lockf 0xccecb780 httpd
3815 3606 3606 80 SJ lockf 0xc9579c80 httpd
3814 3606 3606 80 SJ lockf 0xcb06a880 httpd
3813 3606 3606 80 SJ lockf 0xc9ec7900 httpd
3729 3711 3711 125 SJ kqread 0xc9920300 qmgr
3711 1 3711 0 SsJ kqread 0xc9914200 master
3613 1 3613 0 SsJ select 0xc080b7e4 proftpd
3606 1 3606 0 SsJ select 0xc080b7e4 httpd
3472 1 3472 0 SsJ select 0xc080b7e4 syslogd
3436 3266 3266 80 SJ lockf 0xd00437c0 httpd
3435 3266 3266 80 SJ lockf 0xcec83d80 httpd
3434 3266 3266 80 SJ lockf 0xc8dba200 httpd
3433 3266 3266 80 SJ lockf 0xd0210540 httpd
3432 3266 3266 80 SJ lockf 0xc8fd8b80 httpd
3398 1 3398 0 SsJ select 0xc080b7e4 inetd
3379 1 3379 0 SsJ nanslp 0xc07be46c cron
3344 1 3344 0 SsJ select 0xc080b7e4 sshd
3288 3279 3266 0 SJ piperd 0xc927ab28 cronolog
3287 3276 3266 0 SJ piperd 0xc8d54330 cronolog
3284 3275 3266 0 SJ piperd 0xc927a990 cronolog
3283 3272 3266 0 SJ piperd 0xc8d55330 cronolog
3279 3266 3266 0 SJ wait 0xc90c7430 sh
3276 3266 3266 0 SJ wait 0xc95a5c90 sh
3275 3266 3266 0 SJ wait 0xc97ef000 sh
3272 3266 3266 0 SJ wait 0xc97ef218 sh
3266 1 3266 0 SsJ nanslp 0xc07be46c httpd
3128 3073 3072 88 S+J (threaded) mysqld
100198 S kserel 0xc8950e74 mysqld
100204 S kserel 0xc8950e74 mysqld
100206 S kserel 0xc8950e74 mysqld
100202 S select 0xc080b7e4 mysqld
100203 S kserel 0xc8950e74 mysqld
100200 S sigwait 0xec3dac14 mysqld
100197 S ksesigwa 0xc8cc0780 mysqld
3073 1 3072 88 S+J wait 0xc90c6a78 sh
3022 1 3022 0 SsJ select 0xc080b7e4 syslogd
2545 2354 2354 80 SJ lockf 0xc8c8e700 httpd
2542 2354 2354 80 SJ lockf 0xc8c8e340 httpd
2541 2354 2354 80 SJ lockf 0xccebf4c0 httpd
2539 2354 2354 80 SJ lockf 0xc8c8e740 httpd
2538 2354 2354 80 SJ lockf 0xc8c8eb40 httpd
2369 1 2369 0 SsJ nanslp 0xc07be46c cron
2362 1 2362 0 SsJ select 0xc080b7e4 sshd
2354 1 2354 0 SsJ nanslp 0xc07be46c httpd
2259 2258 2255 70 SJ select 0xc080b7e4 postgres
2258 2255 2255 70 SJ select 0xc080b7e4 postgres
2257 2255 2255 70 SJ select 0xc080b7e4 postgres
2255 1 2255 70 SsJ select 0xc080b7e4 postgres
2199 1 2199 0 SsJ select 0xc080b7e4 syslogd
1917 1 1917 0 SsJ nanslp 0xc07be46c httpd
1892 1 1892 0 SsJ nanslp 0xc07be46c cron
1885 1 1885 0 SsJ select 0xc080b7e4 sshd
1842 1 1842 0 SsJ nanslp 0xc07be46c cron
1833 1 1833 0 SsJ select 0xc080b7e4 sshd
1801 1 1801 0 SsJ nanslp 0xc07be46c httpd
1800 1 1800 0 SsJ select 0xc080b7e4 syslogd
1476 1 1476 0 SsJ select 0xc080b7e4 syslogd
1161 1 41 0 S+J piperd 0xc9257660 logger
1160 1 41 0 S+J fifoor 0xc8c9ca58 tail
1152 1 1152 0 SsJ nanslp 0xc07be46c cron
1090 1 1090 0 SsJ select 0xc080b7e4 syslogd
680 1 680 0 Ss nanslp 0xc07be46c cron
673 1 673 0 Ss select 0xc080b7e4 sshd
644 1 643 0 S nanslp 0xc07be46c python
613 1 613 0 Ss select 0xc080b7e4 ntpd
524 1 524 0 Ss select 0xc080b7e4 syslogd
469 1 469 0 Ss select 0xc080b7e4 devd
40 0 0 0 SL - 0xec126d04 [schedcpu]
39 0 0 0 SL sdflush 0xc081caf4 [softdepflush]
38 0 0 0 SL syncer 0xc07be1dc [syncer]
37 0 0 0 SL vlruwt 0xc8a58000 [vnlru]
36 0 0 0 SL psleep 0xc080bd60 [bufdaemon]
35 0 0 0 SL pgzero 0xc081dac4 [pagezero]
34 0 0 0 SL psleep 0xc081d5cc [vmdaemon]
33 0 0 0 SL psleep 0xc081d588 [pagedaemon]
32 0 0 0 WL [swi0: sio]
31 0 0 0 WL [irq1: atkbd0]
30 0 0 0 WL [irq19: atapci1]
29 0 0 0 WL [irq15: ata1]
28 0 0 0 WL [irq14: ata0]
27 0 0 0 LL *Giant 0xcaaf97c0 [irq17: em1]
26 0 0 0 LL *Giant 0xcaaf97c0 [irq16: em0]
25 0 0 0 WL [irq9: acpi0]
24 0 0 0 SL - 0xc89a3c80 [acpi_task_2]
23 0 0 0 SL - 0xc89a3c80 [acpi_task_1]
22 0 0 0 SL - 0xc89a3c80 [acpi_task_0]
21 0 0 0 WL [swi2: cambio]
9 0 0 0 SL ccb_scan 0xc07b8b84 [xpt_thrd]
8 0 0 0 SL - 0xc89a3e00 [kqueue taskq]
20 0 0 0 WL [swi5: +]
7 0 0 0 SL - 0xc89d7080 [thread taskq]
19 0 0 0 WL [swi6: Giant taskq]
18 0 0 0 WL [swi6: task queue]
17 0 0 0 SL - 0xc07bae00 [yarrow]
6 0 0 0 SL crypto_r 0xc081c7c4 [crypto returns]
5 0 0 0 SL crypto_w 0xc081c784 [crypto]
4 0 0 0 SL - 0xc07bb928 [g_down]
3 0 0 0 LL *vm page 0xc8d5f780 [g_up]
2 0 0 0 SL - 0xc07bb91c [g_event]
16 0 0 0 WL [swi1: net]
15 0 0 0 WL [swi3: vm]
14 0 0 0 LL *Giant 0xcaaf97c0 [swi4: clock sio]
13 0 0 0 RL CPU 0 [idle: cpu0]
12 0 0 0 RL CPU 1 [idle: cpu1]
11 0 0 0 RL CPU 2 [idle: cpu2]
10 0 0 0 RL [idle: cpu3]
1 0 1 0 SLs wait 0xc8951000 [init]
0 0 0 0 WLs [swapper]
# --------------------------------------------------------------------
# CPU LISTING
db> show pcpu
cpuid = 1
curthread = 0xc894c900: pid 12 "idle: cpu1"
curpcb = 0xe81f2d90
fpcurthread = none
idlethread = 0xc894c900: pid 12 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
spin locks held:
db> show allpcpu
Current CPU: 1
cpuid = 0
curthread = 0xc894ca80: pid 13 "idle: cpu0"
curpcb = 0xe81f5d90
fpcurthread = none
idlethread = 0xc894ca80: pid 13 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
spin locks held:
cpuid = 1
curthread = 0xc894c900: pid 12 "idle: cpu1"
curpcb = 0xe81f2d90
fpcurthread = none
idlethread = 0xc894c900: pid 12 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
spin locks held:
cpuid = 2
curthread = 0xc894c780: pid 11 "idle: cpu2"
curpcb = 0xe81efd90
fpcurthread = none
idlethread = 0xc894c780: pid 11 "idle: cpu2"
APIC ID = 2
currentldt = 0x50
spin locks held:
cpuid = 3
curthread = 0xca268900: pid 52531 "ee"
curpcb = 0xec710d90
fpcurthread = none
idlethread = 0xc894c600: pid 10 "idle: cpu3"
APIC ID = 3
currentldt = 0x50
spin locks held:
# --------------------------------------------------------------------
# LOCK LISTING
db> show locks
db> show alllocks
Process 35638 (sendmail) thread 0xd0530600 (101044)
exclusive sleep mutex vm object (standard object) r = 0 (0xc9c01318) locked @ /usr/src/sys/vm/vm_map.c:1472
exclusive sx user map r = 0 (0xc8fb7734) locked @ /usr/src/sys/vm/vm_map.c:1208
Process 32836 (rsync) thread 0xc9ec1900 (100360)
exclusive sleep mutex pmap r = 0 (0xca89f560) locked @ /usr/src/sys/i386/i386/pmap.c:2041
exclusive sleep mutex vm page queue mutex r = 0 (0xc081d540) locked @ /usr/src/sys/i386/i386/pmap.c:2040
exclusive sx user map r = 0 (0xca89f4e4) locked @ /usr/src/sys/vm/vm_map.c:3105
Process 32754 (httpd) thread 0xcfcc5600 (101170)
exclusive sleep mutex vm object (standard object) r = 0 (0xcc8378c4) locked @ /usr/src/sys/vm/vm_fault.c:297
exclusive sx user map r = 0 (0xc984a4e4) locked @ /usr/src/sys/vm/vm_map.c:3105
Process 7829 (bsnmpd) thread 0xca196180 (100409)
exclusive sleep mutex vm object (kmem object) r = 0 (0xc081d440) locked @ /usr/src/sys/vm/vm_kern.c:325
exclusive sleep mutex system map r = 0 (0xc1068144) locked @ /usr/src/sys/vm/vm_kern.c:295
exclusive sx sysctl lock r = 0 (0xc07be240) locked @ /usr/src/sys/kern/kern_sysctl.c:1375
exclusive sleep mutex Giant r = 0 (0xc07bdb80) locked @ /usr/src/sys/kern/kern_sysctl.c:1313
Process 3 (g_up) thread 0xc894ec00 (100015)
exclusive sleep mutex vm object (standard object) r = 0 (0xcaf6e8c4) locked @ /usr/src/sys/kern/vfs_bio.c:3120
db> show lockedvnods
Locked vnodes
0xcdcce6cc: tag ufs, type VDIR
usecount 2, writecount 0, refcount 4 mountedhere 0
flags ()
v_object 0xcaf6e8c4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc97f1000 (pid 32710)#0 0xc054d7f1 at lockmgr+0x4ed
#1 0xc06a0f96 at ffs_lock+0x76
#2 0xc07170ab at VOP_LOCK_APV+0x87
#3 0xc05bd370 at vn_lock+0xac
#4 0xc05b9f0b at getdirentries+0xff
#5 0xc07046b3 at syscall+0x25b
#6 0xc06efccf at Xint0x80_syscall+0x1f
ino 20398955, on dev ad8s1e
# --------------------------------------------------------------------
# PROCESS STACK TRACES
# SENDMAIL
db> trace 35638
Tracing pid 35638 tid 101044 td 0xd0530600
sched_switch(d0530600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,d0530600,0,c0768214,5f8) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c0768214,5f8) at _mtx_lock_flags+0xa2
vm_map_pmap_enter(c8fb76f0,28095000,5,c9c01318,0,0,a000,10) at vm_map_pmap_enter+0x1df
vm_map_insert(c8fb76f0,c9c01318,0,0,28095000,2809f000,5,7,112) at vm_map_insert+0x28c
vm_map_find(c8fb76f0,c9c01318,0,0,ed810cc8,a000,1,5,7,112) at vm_map_find+0x86
vm_mmap(c8fb76f0,ed810cc8,a000,5,7,20002,2,c9cd0ae0,0,0) at vm_mmap+0x266
mmap(d0530600,ed810d04) at mmap+0x31f
syscall(2807003b,2806003b,bfbf003b,28082060,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (198, FreeBSD ELF32, nosys), eip = 0x2806cd63, esp = 0xbfbfea7c, ebp = 0xbfbfeab8 ---
# RSYNC
db> trace 32836
Tracing pid 32836 tid 100360 td 0xc9ec1900
sched_switch(c9ec1900,0,2) at sched_switch+0x177
mi_switch(2,0,c07bdb40,0,c074c42d,...) at mi_switch+0x270
critical_exit(1,c9ec1900,0,9d6f000,bfc275bc,...) at critical_exit+0x8b
intr_execute_handlers(c893f6e4,ec552b6c,13,ec552bd4,c06f0033,...) at intr_execute_handlers+0x129
lapic_handle_intr(35) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc0700897, esp = 0xec552bb0, ebp = 0xec552bd4 ---
pmap_enter(ca89f560,9d6f000,c332e380,7,0,ced476b4,0,c0767c8b,383) at pmap_enter+0x1bf
vm_fault(ca89f4a0,9d6f000,2,8,c9ec1900,...) at vm_fault+0x10b8
trap_pfault(ec552d38,1,9d6ffb8,9d6ffb8,0,...) at trap_pfault+0xce
trap(bfbf003b,3b,bfbf003b,9d6ffb8,0,...) at trap+0x1eb
calltrap() at calltrap+0x5
--- trap 0xc, eip = 0x2818527f, esp = 0xbfbf9670, ebp = 0xbfbf9b08 ---
# HTTPD
db> trace 32754
Tracing pid 32754 tid 101170 td 0xcfcc5600
sched_switch(cfcc5600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,cfcc5600,0,c0767c8b,356) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c0767c8b,356,c07bdb40,...) at _mtx_lock_flags+0xa2
vm_fault(c984a4a0,88d1000,2,8,cfcc5600,...) at vm_fault+0xfa7
trap_pfault(ed840d38,1,88d1d90,88d1d90,0,...) at trap_pfault+0xce
trap(80a003b,875003b,bfbf003b,2360,8382000,...) at trap+0x1eb
calltrap() at calltrap+0x5
--- trap 0xc, eip = 0x286a34c1, esp = 0xbfbfb860, ebp = 0xbfbfb898 ---
# BSNMPD
db> trace 7829
Tracing pid 7829 tid 100409 td 0xca196180
sched_switch(ca196180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,ca196180,0,c076806c,16f) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c076806c,16f,10928000,...) at _mtx_lock_flags+0xa2
kmem_malloc(c10680c0,7000,102,ec649b64,c06b330f,...) at kmem_malloc+0x282
page_alloc(0,7000,ec649b57,102) at page_alloc+0x1a
uma_large_malloc(7000,102,64f0,c05469d6,ec649bf4,...) at uma_large_malloc+0x3b
malloc(64f0,c078fb40,102,c057de5a,ec649bf4,...) at malloc+0xf3
sysctl_jail_list(c078f220,0,0,ec649bf4,c078f220,...) at sysctl_jail_list+0x8e
sysctl_root(0,ec649c74,3,ec649bf4) at sysctl_root+0x11b
userland_sysctl(ca196180,ec649c74,3,0,bfbfc46c,0,0,0,ec649c70,0,c07bdb80,0,c074b5cd,521) at userland_sysctl+0xf4
__sysctl(ca196180,ec649d04) at __sysctl+0x77
syscall(bfbe003b,2818003b,bfbe003b,3,bfbfc46c,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (202, FreeBSD ELF32, __sysctl), eip = 0x2814e77b, esp = 0xbfbfc34c, ebp = 0xbfbfc388 ---
# G_UP
db> trace 3
Tracing pid 3 tid 100015 td 0xc894ec00
sched_switch(c894ec00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,c894ec00,0,c0752afd,c43) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c0752afd,c43,c06b29a0,...) at _mtx_lock_flags+0xa2
bufdone(dcb814d4) at bufdone+0x176
g_vfs_done(cc0806b4) at g_vfs_done+0x8a
biodone(cc0806b4) at biodone+0x58
g_io_schedule_up(c894ec00) at g_io_schedule_up+0xcb
g_up_procbody(0,e8216d38,0,c0523700,0,...) at g_up_procbody+0x5a
fork_exit(c0523700,0,e8216d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8216d6c, ebp = 0 ---
# EE
db> trace 52531
Tracing pid 52531 tid 100453 td 0xca268900
sched_switch(3dc,c074f76f,ec710b90,c0550d8e,c080b7c0,...) at sched_switch+0x177
sellock(c055154c,c07497db,a,c055154c,0,...) at sellock
(null)(786574,50414d4b,544e4520,75005952,20726573,...) at 0x9
# SWI4: CLOCK SIO
db> trace 14
Tracing pid 14 tid 100002 td 0xc894cc00
sched_switch(c894cc00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,c894cc00,0,c074bcf0,102) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c074bcf0,102) at _mtx_lock_flags+0xa2
softclock(0) at softclock+0x18f
ithread_execute_handlers(c894b430,c89a2b80) at ithread_execute_handlers+0xe6
ithread_loop(c89288e0,e81f8d38,c89288e0,c05456e0,0,...) at ithread_loop+0x67
fork_exit(c05456e0,c89288e0,e81f8d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe81f8d6c, ebp = 0 ---
# EM0
db> trace 27
Tracing pid 27 tid 100017 td 0xc894e900
sched_switch(c894e900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,c894e900,0,c0747942,2a9) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c0747942,2a9) at _mtx_lock_flags+0xa2
ithread_execute_handlers(c89a4648,c89a1480) at ithread_execute_handlers+0xde
ithread_loop(c8a438d0,e8210d38,c8a438d0,c05456e0,0,...) at ithread_loop+0x67
fork_exit(c05456e0,c8a438d0,e8210d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8210d6c, ebp = 0 ---
# em1
db> trace 26
Tracing pid 26 tid 100018 td 0xc894e780
sched_switch(c894e780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,c894e780,0,c0747942,2a9) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c0747942,2a9) at _mtx_lock_flags+0xa2
ithread_execute_handlers(c89a4860,c89a1500) at ithread_execute_handlers+0xde
ithread_loop(c8a3f6a0,e820dd38,c8a3f6a0,c05456e0,0,...) at ithread_loop+0x67
fork_exit(c05456e0,c8a3f6a0,e820dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe820dd6c, ebp = 0 ---
# LYNX
db> trace 33070
Tracing pid 33070 tid 100668 td 0xca02f600
sched_switch(ca02f600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,ca02f600,0,c074fb3c,ec) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c074fb3c,ec) at _mtx_lock_flags+0xa2
soo_poll(d015b000,40,d06e1b80,ca02f600) at soo_poll+0x2a
selscan(ca02f600,ec5d5b94,ec5d5b84,14,c080b7c0,0,c074f76f,300) at selscan+0x1a5
kern_select(ca02f600,14,bfbfe304,0,0,...) at kern_select+0x33b
select(ca02f600,ec5d5d04) at select+0x44
syscall(3b,bfbf003b,bfbf003b,0,bfbfe304,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x281e6be4, esp = 0xbfbfe2a8, ebp = 0xbfbfe384 ---
-------------- next part --------------
#
# RACK2 -- M2 config file
#
machine i386
cpu I686_CPU
ident RACK2
# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for devices.
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
options SCHED_4BSD # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options MD_ROOT # MD is a potential root device
options NFSCLIENT # Network Filesystem Client
options NFSSERVER # Network Filesystem Server
options NFS_ROOT # NFS usable as /, requires NFSCLIENT
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_GPT # GUID Partition Tables.
options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!]
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive.
options SMP
options FAST_IPSEC
options IPFIREWALL
options IPFIREWALL_DEFAULT_TO_ACCEPT
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_CDNR
options ALTQ_PRIQ
options KDB
options KDB_TRACE
options KDB_UNATTENDED
options DDB
options INVARIANTS
options INVARIANT_SUPPORT
options WITNESS
options WITNESS_SKIPSPIN
options DIAGNOSTIC
options BREAK_TO_DEBUGGER
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS
device crypto
device apic # I/O APIC
# Bus support.
device eisa
device pci
# Floppy drives
device fdc
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID # Static device numbering
# SCSI Controllers
#device ahb # EISA AHA1742 family
device ahc # AHA2940 and onboard AIC7xxx devices
#options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~128k to driver.
device ahd # AHA39320/29320 and onboard AIC79xx devices
#options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~215k to driver.
#device amd # AMD 53C974 (Tekram DC-390(T))
#device isp # Qlogic family
#device ispfw # Firmware for QLogic HBAs- normally a module
#device mpt # LSI-Logic MPT-Fusion
#device ncr # NCR/Symbios Logic
#device sym # NCR/Symbios Logic (newer chipsets + those of `ncr')
#device trm # Tekram DC395U/UW/F DC315U adapters
#device adv # Advansys SCSI adapters
#device adw # Advansys wide SCSI adapters
device aha # Adaptec 154x SCSI adapters
device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
#device bt # Buslogic/Mylex MultiMaster SCSI adapters
#device ncv # NCR 53C500
#device nsp # Workbit Ninja SCSI-3
#device stg # TMC 18C30/18C50
# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
#device ch # SCSI media changers
device da # Direct Access (disks)
#device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
#device ses # SCSI Environmental Services (and SAF-TE)
# RAID controllers interfaced to the SCSI subsystem
#device amr # AMI MegaRAID
#device arcmsr # Areca SATA II RAID
#device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID
#device ciss # Compaq Smart RAID 5*
#device dpt # DPT Smartcache III, IV - See NOTES for options
#device hptmv # Highpoint RocketRAID 182x
#device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx
#device rr232x # Highpoint RocketRAID 232x
#device iir # Intel Integrated RAID
#device ips # IBM (Adaptec) ServeRAID
#device mly # Mylex AcceleRAID/eXtremeRAID
#device twa # 3ware 9000 series PATA/SATA RAID
# RAID controllers
device aac # Adaptec FSA RAID
device aacp # SCSI passthrough for aac (requires CAM)
#device ida # Compaq Smart RAID
#device mfi # LSI MegaRAID SAS
#device mlx # Mylex DAC960 family
#device pst # Promise Supertrak SX6000
#device twe # 3ware ATA RAID
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
# Enable this for the pcvt (VT220 compatible) console driver
#device vt
#options XSERVER # support for X server on a vt console
#options FAT_CURSOR # start with block cursor
device agp # support several AGP chipsets
# Power management support (see NOTES for more options)
#device apm
# Add suspend/resume support for the i8254.
device pmtimer
# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
#device cbb # cardbus (yenta) bridge
#device pccard # PC Card (16-bit) bus
#device cardbus # CardBus (32-bit) bus
# Serial (COM) ports
device sio # 8250, 16[45]50 based serial ports
# Parallel port
#device ppc
#device ppbus # Parallel port bus (required)
#device lpt # Printer
#device plip # TCP/IP over parallel
#device ppi # Parallel port interface device
#device vpo # Requires scbus and da
# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to the sio and/or ppc drivers):
#device puc
# PCI Ethernet NICs.
#device de # DEC/Intel DC21x4x (``Tulip'')
device em # Intel PRO/1000 adapter Gigabit Ethernet Card
device ixgb # Intel PRO/10GbE Ethernet Card
#device txp # 3Com 3cR990 (``Typhoon'')
#device vx # 3Com 3c590, 3c595 (``Vortex'')
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus # MII bus support
#device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet
#device bfe # Broadcom BCM440x 10/100 Ethernet
device bge # Broadcom BCM570xx Gigabit Ethernet
device dc # DEC/Intel 21143 and various workalikes
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
#device lge # Level 1 LXT1001 gigabit Ethernet
#device msk # Marvell/SysKonnect Yukon II Gigabit Ethernet
#device nge # NatSemi DP83820 gigabit Ethernet
#device nve # nVidia nForce MCP on-board Ethernet Networking
#device pcn # AMD Am79C97x PCI 10/100(precedence over 'lnc')
device re # RealTek 8139C+/8169/8169S/8110S
device rl # RealTek 8129/8139
#device sf # Adaptec AIC-6915 (``Starfire'')
device sis # Silicon Integrated Systems SiS 900/SiS 7016
#device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet
#device ste # Sundance ST201 (D-Link DFE-550TX)
#device stge # Sundance/Tamarack TC9021 gigabit Ethernet
#device ti # Alteon Networks Tigon I/II gigabit Ethernet
#device tl # Texas Instruments ThunderLAN
#device tx # SMC EtherPower II (83c170 ``EPIC'')
#device vge # VIA VT612x gigabit Ethernet
device vr # VIA Rhine, Rhine II
#device wb # Winbond W89C840F
device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')
# ISA Ethernet NICs. pccard NICs included.
#device cs # Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
#device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards
#device ex # Intel EtherExpress Pro/10 and Pro/10+
#device ep # Etherlink III based cards
#device fe # Fujitsu MB8696x based cards
#device ie # EtherExpress 8/16, 3C507, StarLAN 10 etc.
device lnc # NE2100, NE32-VL Lance Ethernet cards
#device sn # SMC's 9000 series of Ethernet chips
#device xe # Xircom pccard Ethernet
# Wireless NIC cards
#device wlan # 802.11 support
#device wlan_wep # 802.11 WEP support
#device wlan_ccmp # 802.11 CCMP support
#device wlan_tkip # 802.11 TKIP support
#device an # Aironet 4500/4800 802.11 wireless NICs.
#device ath # Atheros pci/cardbus NIC's
#device ath_hal # Atheros HAL (Hardware Access Layer)
#device ath_rate_sample # SampleRate tx rate control for ath
#device awi # BayStack 660 and others
#device ral # Ralink Technology RT2500 wireless NICs.
#device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
#device wl # Older non 802.11 Wavelan wireless NIC.
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device sl # Kernel SLIP
device ppp # Kernel PPP
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device faith # IPv6-to-IPv4 relaying (translation)
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
# USB support
#device uhci # UHCI PCI->USB interface
#device ohci # OHCI PCI->USB interface
#device ehci # EHCI PCI->USB interface (USB 2.0)
#device usb # USB Bus (required)
#device udbp # USB Double Bulk Pipe devices
#device ugen # Generic
#device uhid # "Human Interface Devices"
#device ukbd # Keyboard
#device ulpt # Printer
#device umass # Disks/Mass storage - Requires scbus and da
#device ums # Mouse
#device ural # Ralink Technology RT2500USB wireless NICs
#device urio # Diamond Rio 500 MP3 player
#device uscanner # Scanners
# USB Ethernet, requires miibus
#device aue # ADMtek USB Ethernet
#device axe # ASIX Electronics USB Ethernet
#device cdce # Generic USB over Ethernet
#device cue # CATC USB Ethernet
#device kue # Kawasaki LSI USB Ethernet
#device rue # RealTek RTL8150 USB Ethernet
# FireWire support
#device firewire # FireWire bus code
#device sbp # SCSI over FireWire (Requires scbus and da)
#device fwe # Ethernet over FireWire (non-standard!)
More information about the freebsd-stable
mailing list