Re: ZFS + FreeBSD XEN dom0 panic

On Sun, Apr 03, 2022 at 09:54:24AM +0300, Ze Dupsys wrote:
> On 2022.03.27. 12:13, Roger Pau Monné wrote:
> >> ==== COUNT: 1
> >> Fatal trap 12: page fault while in kernel mode
> >> cpuid = 1; apic id = 02
> >> fault virtual address = 0x148
> >> fault code    = supervisor read data, page not present
> >> instruction pointer = 0x20:0xffffffff8248cef4
> >> stack pointer         = 0x28:0xfffffe009941d9a0
> >> frame pointer         = 0x28:0xfffffe009941d9a0
> >> code segment    = base 0x0, limit 0xfffff, type 0x1b
> >>        = DPL 0, pres 1, long 1, def32 0, gran 1
> >> processor eflags  = interrupt enabled, resume, IOPL = 0
> >> current process   = 0 (xbbd1 taskq)
> >> trap number   = 12
> >> panic: page fault
> >> cpuid = 1
> >> time = 1646129773
> >> KDB: stack backtrace:
> >> #0 0xffffffff80c57525 at kdb_backtrace+0x65
> >> #1 0xffffffff80c09f01 at vpanic+0x181
> >> #2 0xffffffff80c09d73 at panic+0x43
> >> #3 0xffffffff8108b1a7 at trap+0xbc7
> >> #4 0xffffffff8108b1ff at trap+0xc1f
> >> #5 0xffffffff8108a85d at trap+0x27d
> >> #6 0xffffffff81061b18 at calltrap+0x8
> >> #7 0xffffffff825cb76e at zil_open+0xe
> >> #8 0xffffffff82456d02 at zvol_ensure_zilog+0xb2
> >> #9 0xffffffff82456818 at zvol_geom_bio_strategy+0x88
> >> #10 0xffffffff80a7f214 at xbd_instance_create+0xa394
> >> #11 0xffffffff80a7b1ea at xbd_instance_create+0x636a
> >> #12 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
> >> #13 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
> >> #14 0xffffffff80bc7e3e at fork_exit+0x7e
> >> #15 0xffffffff81062b9e at fork_trampoline+0xe
> >
> > Hm, those last ones are in ZFS code, can you try to get the line
> > numbers for those?
> 
> The thing with these ZFS cases is that, luckily i had saved copy of 13.0-p7
> partition, it was official build, but the trace for me seems un unsable for
> unknown reasons. It feels that kernel.debug file is the does not match
> kernel, but i did manage to install 13-p7 in VirtualBox as well and md5 sum
> for that one is the same as in my backup, so i don't know what to think,
> because i expected that it would point to blkfront.c and not random lines in
> blkback.c.
> 
> cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line
> -e /mnt/gpt_root_13_p7/usr/lib/debug/boot/kernel/kernel.debug
> /usr/src/sys/kern/subr_kdb.c:443
> /usr/src/sys/kern/kern_shutdown.c:0
> /usr/src/sys/kern/kern_shutdown.c:843
> /usr/src/sys/amd64/amd64/trap.c:915
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/exception.S:292
> ??:0
> ??:0
> ??:0
> /usr/src/sys/dev/xen/blkback/blkback.c:2183
> /usr/src/sys/dev/xen/blkback/blkback.c:1772
> /usr/src/sys/kern/subr_taskqueue.c:478
> /usr/src/sys/kern/subr_taskqueue.c:799
> /usr/src/sys/kern/kern_fork.c:1075
> /usr/src/sys/amd64/amd64/exception.S:1093
> 
> From official update servers kern.debug for 13.0-p7 can be downloaded:
> fetch http://update2.freebsd.org/13.0-RELEASE/amd64/f/f96dbe024c25febf8a798b5b3d6ca717c2e18f79cc6060cb714c3b3681404608.gz

Let's see if you can get another trace of this panic, because I don't
seem to be able to make sense out of this. I've tried looking at the
RELEASE-13.0 sources (there's no blkback related changed in p7 or the
ones before), but the file:line doesn't seem to make much sense.

I would assume the call is from the:

(*dev_data->csw->d_strategy)(bios[bio_idx]);

Call in xbb_dispatch_dev(), but that's not exactly where gdb (or
addr2line) points me to.

Thanks, Roger.