zpool import reboots computer

Fri Jan 20 09:09:42 UTC 2012

On 2012-01-19 21:10, Andriy Gapon wrote: 
>on 19/01/2012 21:58 Martin Ranne said the following:
>>On 2012-01-19 18:55, Andriy Gapon wrote: 
>>on 19/01/2012 19:36 Martin Ranne said the following:
>>>On 2012-01-19 17:32, Andriy Gapon wrote: 
>>>on 19/01/2012 17:36 Martin Ranne said the following:
>>>>>>I had a failure in one server where i try to determine if it is memory or cpu. It shows up as memory failure in memtest86. >>The result is that it managed to damage the zpool which is a raidz2 with 6 disks.

>>>>>>If I boot from a FreeBSD 9.0-RELEASE usb stick and import it with zpool -f -R /mnt/zroot zroot it will reboot the computer. >>I have also tried to import it in another computer which is running 9-STABLE with the same result. On the second computer I >>used zpool -f -R /mnt/zroot "zpool-id" serv06zroot 

>>>>>>Can I get some help on how to be able to debug this and in the end be able to import it to repair it.

>>>>>>Data for the second computer can be found attached. The disks in question are da0 to da5 in this.

>>>>>And the panic message is?

>>>>I am trying to get a crash dump but it hangs when dumping.

>>>Alternatives:
>>>- serial console
>>>- digital camera
>>>- eyes plus pen and paper

>>Finally here it is. Is there anything i can do in the debugger to make it possible to find what is crashing in there?

>>Fatal trap 12: page fault while in kernel mode
>>Fatal trap 12: page fault while in kernel mode
>>cpuid = 0; cpuid = 2; apic id = 00
>>apic id = 02
>>fault virtual address        = 0x88
>>fault virtual address        = 0x38
>>fault code                    = supervisor read data, page not present
>>fault code                    = supervisor read data, page not present
>>instruction pointer            = 0x20:0xffffffff814a7ef5
>>instruction pointer            = 0x20:0xffffffff814872a1
>>stack pointer                = 0x28:0xffffff8c10252ad0
>>stack pointer                = 0x28:0xffffff8c0d564f00
>>frame pointer                = 0x28:0xffffff8c10252b40
>>frame pointer                = 0x28:0xffffff8c0d564f30
>>code segment                = base 0x0, limit 0xfffff, type 0x1b
>>code segment                = base 0x0, limit 0xfffff, type 0x1b
>>                            = DPL 0, pres 1, long 1, def32 0, gran 1
>>                            = DPL 0, pres 1, long 1, def32 0, gran 1
>>processor eflags            = processor eflags        = interrupt enabled, interrupt enabled, resume, resume, IOPL = 0
>>IOPL = 0
>>current process                = current process                = 2659 (zpool)
>>0 [ thread pid 2659 tid 100592 ]

>Hmm, two traps running almost perfectly in parallel...

>stopped at        zio_vdev_child_io+0x25: cmpq    $0,0x88(%r10)
>db>

>At least the 'bt' command.

>It could be that the panic is caused by corrupted vdev label, but not sure...

I tried again to get into the debugger. It will not always work as it freezes before i get to the prompt most of the times but here it is. Any other commands to run in the debugger to get better information to help solve this?

I used the command zpool import -F -f -o readonly=on -R /mnt/serv06 zroot

Result is the following
Fatal trap 12: page fault while in kernel mode
Fatal trap 12: page fault while in kernel mode
cpuid = 0; cpuid = 5; apic id = 00
apic id = 05
fault virtual address   = 0x38
fault virtual address   = 0x88
fault code              = supervisor read data, page not present
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff814872a1
instruction pointer     = 0x20:0xffffffff814a7ef5
stack pointer           = 0x28:0xffffff8c0d564f00
stack pointer           = 0x28:0xffffff8c0ffd7ad0
frame pointer           = 0x28:0xffffff8c0d564f30
frame pointer           = 0x28:0xffffff8c0ffd7b40
code segment            = base 0x0, limit 0xfffff, type 0x1b
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = processor eflags      = interrupt enabled, interrupt enabled, resume, resume, IOPL = 0
IOPL = 0
current process         = current process               = 0 (system_task1_3)
26[ thread pid 0 tid 100099 ]
Stopped at      vdev_is_dead+0x1:       cmpq    $0x5,0x28(%rdi)
db> bt
Tracing pid 0 tid 100099 td 0xfffffe000e546460
vdev_is_dead() at vdev_is_dead+0x1
vdev_mirror_child_select() at vdev_mirror_child_select+0x67
vdev_mirror_io_start() at vdev_mirror_io_start+0x24c
zio_vdev_io_start() at zio_vdev_io_start+0x232
zio_execute() at zio_execute+0xc3
zio_gang_assemble() at zio_gang_assemble+0x1b
zio_execute() at zio_execute+0xc3
arc_read_nolock() at arc_read_nolock+0x6d1
arc_read() at arc_read+0x93
traverse_prefetcher() at traverse_prefetcher+0x103
traverse_visitbp() at traverse_visitbp+0x21c
traverse_dnode() at traverse_dnode+0x7c
traverse_visitbp() at traverse_visitbp+0x3ff
traverse_visitbp() at traverse_visitbp+0x316
traverse_visitbp() at traverse_visitbp+0x316
traverse_visitbp() at traverse_visitbp+0x316
traverse_visitbp() at traverse_visitbp+0x316
traverse_visitbp() at traverse_visitbp+0x316
traverse_visitbp() at traverse_visitbp+0x316
traverse_dnode() at traverse_dnode+0x7c
traverse_visitbp() at traverse_visitbp+0x48c
traverse_prefetch_thread() at traverse_prefetch_thread+0x78
taskq_run() at taskq_run+0x13
taskqueue_run_locked() at taskqueue_run_locked+0x85
taskqueue_thread_loop() at taskqueue_thread_loop+0x46
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8c0d565d00, rbp = 0 ---
db>

//Martin Ranne
________________________________________
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1901 / Virus Database: 2109/4754 - Release Date: 01/19/12