ZFS Crash

[Larry Rosenman]

On Fri, 29 May 2009, Larry Rosenman wrote:

> On Fri, 29 May 2009, Larry Rosenman wrote:
>> 
>> Ok, it just crashed.  Unfortunately, I'm at work and the box is at home.
>> 
>> I did have my script running every minute of that entire boot.
>> 
>> What I saw was a full backup running, and then we started paging, and then
>> the backup jobs got pager errors, and were killed.
>> 
>> I'm not sure what else went on, so I restarted the bacula daemons that
>> got killed, and was in the bacula console when it died.
>> 
>> I'll see if I can get a cell-phone camera shot of the console.
>> 
>> I'll also tar up the vmstat outputs and put them on my web server.
>> 
>> What other forensics should I get?  Bear in mind the system is probably
>> locked up with no dump taken :(
> One other "interesting" thing is the IPMI card seems to also be locked up. 
> I.E. if I try to login to it, it just hangs after giving id/pw.
>
Ok, I let the IPMI sit, and it eventually showed me the console.

I took a screenshot, and then reset the box.  I did get a textdump
but it didn't run my scripts for the ddb stuff.

Here is links to what I do have:
http://www.lerctr.org/~ler/ZFS_CRASH/

$ ls -l
total 5017
-rw-r--r--  1 ler  ler  5051465 May 29 13:14 crash.stats.tar.gz
-rw-r--r--  1 ler  ler      253 May 29 13:18 index.html
-rw-r--r--  1 ler  ler    77004 May 29 13:16 ipmiconsole.png
-rw-r--r--  1 ler  ler    70656 May 29 13:14 textdump.tar.6
$

What else can I supply?

The crash.stats.tar.gz contains the minute by minute output of the followuing
script for the entire boot:
-----
#!/bin/sh
DATE=`date +%Y%m%d.%H%M%S`
(echo "Uptime:";uptime;echo "vmstat -m:";vmstat -m
  echo "vmstat -z:";vmstat -z) >/home/ler/stats/${DATE}.stats
-----


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 512-248-2683                 E-Mail: ler at lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893