7.1-STABLE Sun Mar 29 01:06:46 ADT 2009 Locks up ...

[Marc G. Fournier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi ...

  Over the past little while, two of my servers have suddenly started to hang
... servers that up until this started, have been reasonably rock solid ...
they are generally within a day of each other for source code, and the hardware
on both are pretty much identical (HP Proliant DL360 Servers) ...

  I have serial console configured on both so that I can do CR ~ ^b to get to
DDB ... except, when it hangs, all I get is:

"KDB: enter: Break sequence on console"

  And it hangs there, no prompt.

  I setup a simple script (see attached) to run every 5 minutes that gathers
various pieces of info that I think are pertinent, but most likely don't cover
everything ...

  Whenever this happens, on either machine, vmstat show data *like* (notice the 
high procs -> w values?):

 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr da0 pa0   in   sy   cs us sy 
id
165 106 2 12699168   33840  3080  38   2   2  3082 1623   0   0  337 36961 4731 
18  7 75
64 75 4 12761744   23084 46809 623  65  43 19307 116 334   0 1189 83674 11708 
70 20 10
 1 68 25 12773980   23068 11036 3003   9  36  4055 116 282   0 1336 78346 14869 
56 16 28
 0 71 25 12774236   23084   186 769   1   5    18  80 249   0  609 9298 5894  5 
5 91
 5 90 31 12747296   23352   626 2546   5 104  1147 368 281   0 1536 40945 19980 
6  5 90

  Where procs -> w just seems to keep rising ... note that the output for
vmstat *5 minutes before* shows:

 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr da0 pa0   in   sy   cs us sy 
id
35 121 0 12414692   90552  3080  32   2   1  3090 1403   0   0  337 37022 4730 
18  7 75
31 93 0 12314408   62024 36550 414  46   6 34285  27 563   0  916 94851 8813 67 
33  0
43 179 0 12270932   23080 24035 101  41  12 13887  36 375   0  766 61969 6945 
69 23  7
92 44 0 12265524  119804  2122 2028   1  32 13051 1096092 205   0  558 19460 
4561 19 50 32
38 34 0 12330068   89140 30758 103  39 119 37037 2837365 165   0  773 92041 
7111 47 53  0


  I have one QEMU VPS running on this box, with kqemu running the latest kernel
module ... but the other machine experiencing the same issue is only running
FreeBSD jails ...

  Both servers are running SCHED_4BSD, if that matters any ... ?

  I'm at a loss as to what to look at / for next ... pointers would be greatly
appreciated ...

  I have the various output files that the script generates available if anyone 
thinks they would be useful ...

thank you ...


Marc G. Fournier        Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . scrappy at hub.org                              MSN . scrappy at hub.org
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (FreeBSD)

iEYEARECAAYFAknlRcMACgkQ4QvfyHIvDvNmIgCfSWdT9gug6VCjYM1VVMuv1UkN
K28AoK298b6mxEeiddu4BAH0+IpkRsti
=q6lD
-----END PGP SIGNATURE-----
-------------- next part --------------
Skipped content of type multipart/mixed