Panic in zfs_blkptr_verify()

[Peter Jeremy]

I'm trying to upgrade my main (amd64) server from 10-stable r276177 to
r287251 but the new kernel consistently panics:

panic: Solaris(panic): blkptr at 0xfffff80015961848 DVA 0 has invalid OFFSET 15724224479232
cpuid = 2
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe086027aff0
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe086027b0a0
vpanic() at vpanic+0x126/frame 0xfffffe086027b0e0
panic() at panic+0x43/frame 0xfffffe086027b140
vcmn_err() at vcmn_err+0xcf/frame 0xfffffe086027b270
zfs_panic_recover() at zfs_panic_recover+0x60/frame 0xfffffe086027b2d0
zfs_blkptr_verify() at zfs_blkptr_verify+0x297/frame 0xfffffe086027b310
zio_read() at zio_read+0x2f/frame 0xfffffe086027b3a0
arc_read() at arc_read+0xb1e/frame 0xfffffe086027b450
dmu_objset_open_impl() at dmu_objset_open_impl+0x196/frame 0xfffffe086027b4e0
dsl_pool_init() at dsl_pool_init+0x2a/frame 0xfffffe086027b510
spa_load() at spa_load+0xa20/frame 0xfffffe086027b650
spa_load_best() at spa_load_best+0x6f/frame 0xfffffe086027b6c0
spa_open_common() at spa_open_common+0x102/frame 0xfffffe086027b730
pool_status_check() at pool_status_check+0x4e/frame 0xfffffe086027b760
zfsdev_ioctl() at zfsdev_ioctl+0x52e/frame 0xfffffe086027b800
devfs_ioctl_f() at devfs_ioctl_f+0x121/frame 0xfffffe086027b860
kern_ioctl() at kern_ioctl+0x160/frame 0xfffffe086027b8c0
sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe086027b9a0
amd64_syscall() at amd64_syscall+0x22e/frame 0xfffffe086027bab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe086027bab0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801a1dc5a, rsp = 0x7fffffffd028, rbp = 0x7fffffffd0a0 ---

Unfortunately, I'm not sure how to resolve this.  zfs_blkptr_verify() was
MFH in r277582 so the checks don't exist in my old kernel.  This means it
could be a problem with one of my pools, rather than a software bug.  But
there's no information in the panic that would let me identify where the
dodgy offset was found other than it's DVA 0 of some undefined blkptr_t.

I've tried setting dumpon prior to this point (and I can see that
dumpdev is set) but I don't get a crashdump.  Since this is my primary
server, I'd prefer not to have it down for an extended period whilst I
rummaged around with ddb.  Any suggestions as to how to proceed?

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 949 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20150829/f82db929/attachment.bin>