Panic on BETA1 in the ZFS subsystem
Karl Denninger
karl at denninger.net
Wed Jul 20 15:56:37 UTC 2016
An attempted repeat of the transfer panic'd the machine again
immediately. I removed all of the snapshots on the root zfs and the
copy ran to completion.
This appears to be identical to the problem that was previously present
where snapshots on the root filesystem sometimes are corrupted, with the
corruption apparently occurring at the time when the snapshot is taken
(race condition in the snapshot code?) and when touched later you panic
the machine. Since zfs send "touches" everything in the incremental
path (or all snapshots if taken as a full send) once that bad snapshot
is there on the disk you're screwed on any attempt to copy that
filesystem elsewhere (e.g. for backup purposes.)
Note that when this situation arose in the past a scrub succeeds without
errors, and the same was true here -- since the root filesystem is small
a scrub is fast, and I ran one -- it said there were no problems.
There was a bug against this (which I referenced) but it is marked
closed/resolved and I *was* running that patch on 10.2 without a panic
for quite a long time (before the patch my average time-before-boom-boom
was about a week) -- so the root cause may be different in this case on
11-BETA1 but the outcome is the same.
On 7/20/2016 10:50, Steven Hartland wrote:
> The panic was due to stack exhaustion, why it was so deep not looked.
>
> On 20/07/2016 15:32, Karl Denninger wrote:
>> The panic occurred during a zfs send/receive operation for system
>> backup..... I've seen this one before, unfortunately, and it appears
>> that it's still there -- may be related to
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207464
>>
>> On 7/20/2016 09:26, Karl Denninger wrote:
>>> Came out of the ZFS system; this machine is running *unpatched*
>>> (none of
>>> my changes are in the kernel); I have a good core dump if further trace
>>> information will help.
>>>
>>> Dump header from device: /dev/gpt/dump
>>> Architecture: amd64
>>> Architecture Version: 2
>>> Dump Length: 9582022656
>>> Blocksize: 512
>>> Dumptime: Wed Jul 20 09:10:31 2016
>>> Hostname: NewFS.denninger.net
>>> Magic: FreeBSD Kernel Dump
>>> Version String: FreeBSD 11.0-BETA1 #0 r302489: Sat Jul 9
>>> 10:15:24 CDT
>>> 2016
>>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP
>>> Panic String: double fault
>>> Dump Parity: 445173880
>>> Bounds: 4
>>> Dump Status: good
>>>
>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.4
>>>
>>> Wed Jul 20 09:18:15 CDT 2016
>>>
>>> FreeBSD NewFS.denninger.net 11.0-BETA1 FreeBSD 11.0-BETA1 #0 r302489:
>>> Sat Jul 9 10:15:24 CDT 2016
>>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64
>>>
>>> panic: double fault
>>>
>>> GNU gdb 6.1.1 [FreeBSD]
>>> Copyright 2004 Free Software Foundation, Inc.
>>> GDB is free software, covered by the GNU General Public License, and
>>> you are
>>> welcome to change it and/or distribute copies of it under certain
>>> conditions.
>>> Type "show copying" to see the conditions.
>>> There is absolutely no warranty for GDB. Type "show warranty" for
>>> details.
>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>
>>> Unread portion of the kernel message buffer:
>>>
>>> Fatal double fault
>>> rip = 0xffffffff8220bc92
>>> rsp = 0xfffffe0667f0ef30
>>> rbp = 0xfffffe0667f0f3c0
>>> cpuid = 14; apic id = 34
>>> panic: double fault
>>> cpuid = 14
>>> KDB: stack backtrace:
>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>>> 0xfffffe0649db8e30
>>> vpanic() at vpanic+0x182/frame 0xfffffe0649db8eb0
>>> panic() at panic+0x43/frame 0xfffffe0649db8f10
>>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649db8f30
>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649db8f30
>>> --- trap 0x17, rip = 0xffffffff8220bc92, rsp = 0xfffffe0667f0ef30,
>>> rbp =
>>> 0xfffffe0667f0f3c0 ---
>>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x22/frame
>>> 0xfffffe0667f0f3c0
>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
>>> 0xfffffe0667f0f400
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f430
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f480
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f4e0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f530
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f0f570
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f5a0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f5f0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f650
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f6a0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f0f6e0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f710
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f760
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f7c0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f810
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f0f850
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f880
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f8d0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f930
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f980
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f0f9c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f9f0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fa40
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0faa0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0faf0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f0fb30
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fb60
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fbb0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fc10
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fc60
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f0fca0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fcd0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fd20
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fd80
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fdd0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f0fe10
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fe40
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fe90
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fef0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0ff40
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f0ff80
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0ffb0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10000
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10060
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f100b0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f100f0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10120
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10170
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f101d0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10220
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f10260
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10290
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f102e0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10340
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10390
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f103d0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10400
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10450
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f104b0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10500
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f10540
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10570
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f105c0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10620
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10670
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f106b0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f106e0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10730
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10790
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f107e0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f10820
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10850
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f108a0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10900
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10950
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f10990
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f109c0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10a10
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10a70
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10ac0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f10b00
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10b30
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10b80
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10be0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10c30
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f10c70
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10ca0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10cf0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10d50
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10da0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f10de0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10e10
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10e60
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10ec0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10f10
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f10f50
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10f80
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10fd0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11030
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11080
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f110c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f110f0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11140
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f111a0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f111f0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f11230
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11260
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f112b0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11310
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11360
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f113a0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f113d0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11420
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11480
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f114d0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f11510
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11540
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11590
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f115f0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11640
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f11680
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f116b0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11700
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11760
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f117b0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f117f0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11820
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11870
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f118d0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11920
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f11960
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11990
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f119e0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11a40
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11a90
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f11ad0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11b00
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11b50
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11bb0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11c00
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f11c40
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11c70
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11cc0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11d20
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11d70
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f11db0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11de0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11e30
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11e90
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11ee0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f11f20
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11f50
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11fa0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12000
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12050
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f12090
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f120c0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12110
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12170
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f121c0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f12200
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12230
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12280
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f122e0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12330
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f12370
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f123a0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f123f0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12450
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f124a0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f124e0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12510
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12560
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f125c0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12610
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f12650
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12680
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f126d0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12730
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12780
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f127c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f127f0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12840
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f128a0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f128f0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f12930
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12960
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f129b0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12a10
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12a60
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>> 0xfffffe0667f12aa0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12ad0
>>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12b20
>>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
>>> 0xfffffe0667f12b80
>>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
>>> 0xfffffe0667f12bb0
>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0667f12bf0
>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0667f12bf0
>>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>>> Uptime: 10d21h36m8s
>>> Dumping 9138 out of 24531
>>> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>>
>>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/zfs.ko
>>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/opensolaris.ko
>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/geom_eli.ko
>>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/geom_mirror.ko
>>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/aesni.ko
>>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/uhid.ko
>>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ums.ko
>>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ipfw.ko
>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ipfw_nat.ko
>>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/libalias.ko
>>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols
>>> from
>>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/daemon_saver.ko
>>> #0 doadump (textdump=1) at pcpu.h:221
>>> 221 pcpu.h: No such file or directory.
>>> in pcpu.h
>>> (kgdb) #0 doadump (textdump=1) at pcpu.h:221
>>> #1 0xffffffff80ae548e in kern_reboot (howto=260)
>>> at /usr/src/sys/kern/kern_shutdown.c:366
>>> #2 0xffffffff80ae5a5b in vpanic (fmt=<value optimized out>,
>>> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
>>> #3 0xffffffff80ae5893 in panic (fmt=0x0)
>>> at /usr/src/sys/kern/kern_shutdown.c:690
>>> #4 0xffffffff80fc9692 in dblfault_handler (frame=<value optimized
>>> out>)
>>> at /usr/src/sys/amd64/amd64/trap.c:868
>>> #5 0xffffffff80fac15c in Xdblfault ()
>>> at /usr/src/sys/amd64/amd64/exception.S:292
>>> #6 0xffffffff8220bc92 in vdev_queue_io_to_issue
>>> (vq=0xfffff804d86833e0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:585
>>>
>>> #7 0xffffffff8220c6e3 in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:881
>>>
>>> #8 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8018859c000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #9 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #10 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8018859c000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #11 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #12 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #13 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801df1f43b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #14 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #15 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801df1f43b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #16 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #17 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #18 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8024583e770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #19 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #20 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8024583e770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #21 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #22 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #23 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802e12df770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #24 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #25 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802e12df770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #26 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #27 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #28 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004f81f3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #29 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #30 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004f81f3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #31 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #32 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #33 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8021bdf7b28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #34 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #35 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8021bdf7b28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #36 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #37 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #38 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80165851770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #39 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #40 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80165851770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #41 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #42 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #43 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8060de643b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #44 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #45 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8060de643b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #46 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #47 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #48 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800793c0770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #49 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #50 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800793c0770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #51 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #52 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #53 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8026917cb28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #54 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #55 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8026917cb28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #56 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #57 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #58 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8012408d3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #59 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #60 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8012408d3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #61 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #62 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #63 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802bfc9ab28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #64 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #65 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802bfc9ab28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #66 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #67 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #68 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802dda5db28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #69 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #70 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802dda5db28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #71 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #72 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #73 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800ae13d000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #74 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #75 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800ae13d000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #76 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #77 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #78 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804a70efb28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #79 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #80 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804a70efb28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #81 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #82 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #83 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80485f5f000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #84 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #85 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80485f5f000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #86 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #87 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #88 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800666fc3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #89 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #90 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800666fc3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #91 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #92 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #93 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8010da3ab28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #94 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #95 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8010da3ab28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #96 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #97 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #98 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80315f73770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #99 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #100 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80315f73770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #101 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #102 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #103 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff806011703b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #104 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #105 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff806011703b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #106 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #107 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #108 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805964f1770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #109 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #110 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff805964f1770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #111 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #112 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #113 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8023902f770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #114 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #115 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8023902f770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #116 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #117 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #118 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff803bc7e2b28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #119 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #120 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff803bc7e2b28)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #121 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #122 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #123 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804dfbf43b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #124 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #125 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804dfbf43b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #126 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #127 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #128 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8005be4a3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #129 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #130 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8005be4a3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #131 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #132 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #133 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004ff79000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #134 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #135 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004ff79000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #136 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #137 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #138 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801aa2a2000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #139 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #140 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801aa2a2000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #141 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #142 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #143 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800592003b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #144 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #145 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800592003b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #146 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #147 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #148 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80202d11770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #149 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #150 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80202d11770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #151 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #152 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #153 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80392d3c3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #154 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #155 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80392d3c3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #156 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #157 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #158 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80550fdd000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #159 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #160 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80550fdd000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #161 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #162 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #163 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8055fd15770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #164 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #165 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8055fd15770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #166 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #167 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #168 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80576f99770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #169 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #170 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80576f99770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #171 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #172 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #173 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802269bf000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #174 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #175 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802269bf000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #176 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #177 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #178 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8003d7973b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #179 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #180 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8003d7973b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #181 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #182 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #183 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80108754770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #184 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #185 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80108754770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #186 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #188 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804cfdf7770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #189 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #190 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804cfdf7770)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #191 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #193 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801469a13b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #194 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>>> #196 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>>>
>>> #198 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805dcd9d3b8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>>> #199 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>>> #200 0xffffffff80b415aa in taskqueue_run_locked (queue=<value optimized
>>> out>)
>>> at /usr/src/sys/kern/subr_taskqueue.c:465
>>> #201 0xffffffff80b424b8 in taskqueue_thread_loop (arg=<value
>>> optimized out>)
>>> at /usr/src/sys/kern/subr_taskqueue.c:719
>>> #202 0xffffffff80a9ad85 in fork_exit (
>>> callout=0xffffffff80b423d0 <taskqueue_thread_loop>,
>>> arg=0xfffff8037db02580, frame=0xfffffe0667f12c00)
>>> at /usr/src/sys/kern/kern_fork.c:1038
>>> #203 0xffffffff80fac59e in fork_trampoline ()
>>> at /usr/src/sys/amd64/amd64/exception.S:611
>>> #204 0x0000000000000000 in ?? ()
>>> Current language: auto; currently minimal
>>> (kgdb)
>>>
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
--
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2996 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160720/f5a784be/attachment.bin>
More information about the freebsd-stable
mailing list