Debugging a panic (was: )

[Phil Schulz]

Dan Mahoney, System Admin wrote:
> After recently upgrading to 4.10, on a machine that's known for getting 
> 100+ days uptime, I got the following error on an unexpected reboot:
> 
[Panic]
> 
> What could cause this?
> 
> -Dan Mahoney
> 

Hi Dan!

  I'm no expert at all, but I'll give it a try...

  If you cannot reproduce the panic, you might face a hardware problem. 
Memory, harddisk, power supply, etc. are the usual suspects. If you, 
however, can reproduce the panic, please continue to read.

  Please note that the steps outlined below are things I did on a 
5.3-BETA5 system. I might have forgotten some things while I have 
included other, unneeded steps.

  There are a few requirements to really debug a panic:

  * Build a kernel w/ debug symbols. Add "makeoptions DEBUG=-g" to your
    kernel configuration file.
  * Set up the "dumpdev" and "dumpdir" variables in rc.conf - mine are
    set to:
      dumpdev="/dev/ad0s2b"
      dumpdir="/var/crash"
  * If the machine doesn't aim at minimum downtime, you might want to
    build a kernel debugger into your kernel, so the kernel can drop into
    the debugger in case of a panic. At the debugger prompt, you can type
    "backtrace" to get a useful trace on how the kernel ended up in the
    place where it crashed. I've also found that you need to type "call
    doadump" to get a crash dump before you can "reset" the machine. This
    might or might not apply to 4.x, however.
  * If you're aiming at minimum downtime and have set the machine to
    automatically reset itself in case of a panic, you'll have to analyze
    the core dump to get a trace.

  It is my understanding that the instruction pointer listed in the 
panic message points to the place where things blew up. You can use the 
address to point to the line in the source code, provided you have built 
in debug symbols and you have a core dump. However, the address might 
point to different places with different kernel configurations, i.e. the 
adress you gave us only applies to your kernel.
  I also think that debug symbols do not have any negative impact on 
performance, so it's a good idea to keep them around.
  Someone please correct me if I'm wrong.

   The link below [1] and a quick Google search for "Debugging Kernel 
Problems" will point you to documentation I've found very useful.

HTH,

Phil.

[1]http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.html

-- 
Did you know...

If you play a Windows 2000 CD backwards, you hear satanic messages,
but what's worse is when you play it forward....
                                      ...it installs Windows 2000

   -- Alfred Perlstein on chat at freebsd.org