Panic on BETA1 in the ZFS subsystem
Karl Denninger
karl at denninger.net
Wed Jul 20 14:27:07 UTC 2016
Came out of the ZFS system; this machine is running *unpatched* (none of
my changes are in the kernel); I have a good core dump if further trace
information will help.
Dump header from device: /dev/gpt/dump
Architecture: amd64
Architecture Version: 2
Dump Length: 9582022656
Blocksize: 512
Dumptime: Wed Jul 20 09:10:31 2016
Hostname: NewFS.denninger.net
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 11.0-BETA1 #0 r302489: Sat Jul 9 10:15:24 CDT
2016
karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP
Panic String: double fault
Dump Parity: 445173880
Bounds: 4
Dump Status: good
NewFS.denninger.net dumped core - see /var/crash/vmcore.4
Wed Jul 20 09:18:15 CDT 2016
FreeBSD NewFS.denninger.net 11.0-BETA1 FreeBSD 11.0-BETA1 #0 r302489:
Sat Jul 9 10:15:24 CDT 2016
karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64
panic: double fault
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal double fault
rip = 0xffffffff8220bc92
rsp = 0xfffffe0667f0ef30
rbp = 0xfffffe0667f0f3c0
cpuid = 14; apic id = 34
panic: double fault
cpuid = 14
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe0649db8e30
vpanic() at vpanic+0x182/frame 0xfffffe0649db8eb0
panic() at panic+0x43/frame 0xfffffe0649db8f10
dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649db8f30
Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649db8f30
--- trap 0x17, rip = 0xffffffff8220bc92, rsp = 0xfffffe0667f0ef30, rbp =
0xfffffe0667f0f3c0 ---
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x22/frame
0xfffffe0667f0f3c0
vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe0667f0f400
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f430
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f480
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f4e0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f530
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f570
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f5a0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f5f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f650
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f6a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f6e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f710
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f760
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f7c0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f810
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f850
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f880
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f8d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f930
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f980
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f9c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f9f0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fa40
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0faa0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0faf0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fb30
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fb60
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fbb0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fc10
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fc60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fca0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fcd0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fd20
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fd80
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fdd0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fe10
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fe40
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fe90
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fef0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0ff40
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0ff80
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0ffb0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10000
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10060
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f100b0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f100f0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10120
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10170
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f101d0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10220
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10260
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10290
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f102e0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10340
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10390
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f103d0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10400
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10450
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f104b0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10500
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10540
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10570
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f105c0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10620
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10670
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f106b0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f106e0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10730
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10790
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f107e0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10820
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10850
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f108a0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10900
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10950
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10990
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f109c0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10a10
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10a70
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10ac0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10b00
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10b30
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10b80
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10be0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10c30
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10c70
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10ca0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10cf0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10d50
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10da0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10de0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10e10
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10e60
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10ec0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10f10
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10f50
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10f80
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10fd0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11030
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11080
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f110c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f110f0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11140
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f111a0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f111f0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11230
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11260
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f112b0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11310
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11360
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f113a0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f113d0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11420
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11480
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f114d0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11510
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11540
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11590
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f115f0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11640
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11680
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f116b0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11700
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11760
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f117b0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f117f0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11820
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11870
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f118d0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11920
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11960
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11990
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f119e0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11a40
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11a90
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11ad0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11b00
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11b50
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11bb0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11c00
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11c40
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11c70
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11cc0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11d20
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11d70
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11db0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11de0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11e30
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11e90
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11ee0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11f20
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11f50
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11fa0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12000
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12050
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12090
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f120c0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12110
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12170
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f121c0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12200
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12230
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12280
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f122e0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12330
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12370
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f123a0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f123f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12450
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f124a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f124e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12510
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12560
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f125c0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12610
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12650
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12680
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f126d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12730
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12780
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f127c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f127f0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12840
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f128a0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f128f0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12930
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12960
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f129b0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12a10
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12a60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12aa0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12ad0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12b20
taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
0xfffffe0667f12b80
taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
0xfffffe0667f12bb0
fork_exit() at fork_exit+0x85/frame 0xfffffe0667f12bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0667f12bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 10d21h36m8s
Dumping 9138 out of 24531
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
/usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_eli.ko
Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
/usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
/usr/lib/debug//boot/kernel/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel/aesni.ko
Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
/usr/lib/debug//boot/kernel/uhid.ko.debug...done.
done.
Loaded symbols for /boot/kernel/uhid.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ipfw_nat.ko
Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
/usr/lib/debug//boot/kernel/libalias.ko.debug...done.
done.
Loaded symbols for /boot/kernel/libalias.ko
Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
/usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
done.
Loaded symbols for /boot/kernel/daemon_saver.ko
#0 doadump (textdump=1) at pcpu.h:221
221 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump (textdump=1) at pcpu.h:221
#1 0xffffffff80ae548e in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:366
#2 0xffffffff80ae5a5b in vpanic (fmt=<value optimized out>,
ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80ae5893 in panic (fmt=0x0)
at /usr/src/sys/kern/kern_shutdown.c:690
#4 0xffffffff80fc9692 in dblfault_handler (frame=<value optimized out>)
at /usr/src/sys/amd64/amd64/trap.c:868
#5 0xffffffff80fac15c in Xdblfault ()
at /usr/src/sys/amd64/amd64/exception.S:292
#6 0xffffffff8220bc92 in vdev_queue_io_to_issue (vq=0xfffff804d86833e0)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:585
#7 0xffffffff8220c6e3 in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:881
#8 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8018859c000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#9 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#10 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8018859c000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#11 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#12 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#13 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801df1f43b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#14 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#15 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801df1f43b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#16 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#17 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#18 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8024583e770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#19 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#20 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8024583e770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#21 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#22 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#23 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802e12df770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#24 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#25 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802e12df770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#26 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#27 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#28 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004f81f3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#29 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#30 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004f81f3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#31 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#32 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#33 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8021bdf7b28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#34 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#35 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8021bdf7b28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#36 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#37 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#38 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80165851770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#39 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#40 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80165851770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#41 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#42 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#43 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8060de643b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#44 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#45 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8060de643b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#46 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#47 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#48 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800793c0770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#49 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#50 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800793c0770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#51 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#52 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#53 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8026917cb28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#54 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#55 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8026917cb28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#56 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#57 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#58 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8012408d3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#59 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#60 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8012408d3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#61 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#62 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#63 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802bfc9ab28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#64 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#65 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802bfc9ab28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#66 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#67 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#68 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802dda5db28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#69 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#70 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802dda5db28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#71 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#72 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#73 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800ae13d000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#74 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#75 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800ae13d000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#76 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#77 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#78 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804a70efb28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#79 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#80 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804a70efb28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#81 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#82 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#83 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80485f5f000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#84 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#85 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80485f5f000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#86 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#87 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#88 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800666fc3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#89 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#90 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800666fc3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#91 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#92 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#93 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8010da3ab28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#94 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#95 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8010da3ab28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#96 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#97 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#98 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80315f73770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#99 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#100 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80315f73770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#101 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#102 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#103 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff806011703b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#104 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#105 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff806011703b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#106 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#107 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#108 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805964f1770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#109 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#110 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff805964f1770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#111 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#112 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#113 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8023902f770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#114 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#115 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8023902f770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#116 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#117 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#118 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff803bc7e2b28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#119 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#120 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff803bc7e2b28)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#121 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#122 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#123 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804dfbf43b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#124 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#125 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804dfbf43b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#126 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#127 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#128 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8005be4a3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#129 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#130 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8005be4a3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#131 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#132 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#133 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004ff79000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#134 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#135 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004ff79000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#136 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#137 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#138 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801aa2a2000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#139 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#140 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801aa2a2000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#141 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#142 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#143 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800592003b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#144 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#145 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800592003b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#146 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#147 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#148 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80202d11770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#149 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#150 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80202d11770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#151 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#152 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#153 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80392d3c3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#154 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#155 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80392d3c3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#156 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#157 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#158 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80550fdd000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#159 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#160 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80550fdd000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#161 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#162 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#163 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8055fd15770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#164 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#165 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8055fd15770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#166 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#167 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#168 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80576f99770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#169 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#170 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80576f99770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#171 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#172 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#173 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802269bf000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#174 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#175 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802269bf000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#176 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#177 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#178 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8003d7973b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#179 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#180 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8003d7973b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#181 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#182 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#183 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80108754770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#184 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#185 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80108754770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#186 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#187 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#188 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804cfdf7770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#189 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#190 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804cfdf7770)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#191 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#192 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#193 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801469a13b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#194 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#196 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#197 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#198 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805dcd9d3b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#199 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#200 0xffffffff80b415aa in taskqueue_run_locked (queue=<value optimized
out>)
at /usr/src/sys/kern/subr_taskqueue.c:465
#201 0xffffffff80b424b8 in taskqueue_thread_loop (arg=<value optimized out>)
at /usr/src/sys/kern/subr_taskqueue.c:719
#202 0xffffffff80a9ad85 in fork_exit (
callout=0xffffffff80b423d0 <taskqueue_thread_loop>,
arg=0xfffff8037db02580, frame=0xfffffe0667f12c00)
at /usr/src/sys/kern/kern_fork.c:1038
#203 0xffffffff80fac59e in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:611
#204 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb)
--
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2996 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160720/05bdadfb/attachment.bin>
More information about the freebsd-stable
mailing list