Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE
Steven Hartland
killing at multiplay.co.uk
Mon Oct 17 20:16:38 UTC 2016
Be good to confirm its not an infinite loop by giving it a good bump first.
On 17/10/2016 19:58, Karl Denninger wrote:
> I can certainly attempt setting that higher.... but is that not just
> hiding the problem rather than addressing it?
>
>
> On 10/17/2016 13:54, Steven Hartland wrote:
>> You're hitting stack exhaustion, have you tried increasing the kernel
>> stack pages?
>> It can be changed from /boot/loader.conf
>> kern.kstack_pages="6"
>>
>> Default on amd64 is 4 IIRC
>>
>> On 17/10/2016 19:08, Karl Denninger wrote:
>>> The target (and devices that trigger this) are a pair of 4Gb 7200RPM
>>> SATA rotating rust drives (zmirror) with each provider geli-encrypted
>>> (that is, the actual devices used for the pool create are the .eli's)
>>>
>>> The machine generating the problem has both rotating rust devices *and*
>>> SSDs, so I can't simply shut TRIM off system-wide and call it a day as
>>> TRIM itself is heavily-used; both the boot/root pools and a Postgresql
>>> database pool are on SSDs, while several terabytes of lesser-used data
>>> is on a pool of Raidz2 that is made up of spinning rust.
>> snip...
>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>>
>>> Mon Oct 17 09:02:33 CDT 2016
>>>
>>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64
>>>
>>> panic: double fault
>>>
>>> GNU gdb 6.1.1 [FreeBSD]
>>> Copyright 2004 Free Software Foundation, Inc.
>>> GDB is free software, covered by the GNU General Public License, and
>>> you are
>>> welcome to change it and/or distribute copies of it under certain
>>> conditions.
>>> Type "show copying" to see the conditions.
>>> There is absolutely no warranty for GDB. Type "show warranty" for
>>> details.
>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>
>>> Unread portion of the kernel message buffer:
>>>
>>> Fatal double fault
>>> rip = 0xffffffff8220d9ec
>>> rsp = 0xfffffe066821f000
>>> rbp = 0xfffffe066821f020
>>> cpuid = 6; apic id = 14
>>> panic: double fault
>>> cpuid = 6
>>> KDB: stack backtrace:
>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>>> 0xfffffe0649d78e30
>>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0
>>> panic() at panic+0x43/frame 0xfffffe0649d78f10
>>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30
>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30
>>> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000, rbp =
>>> 0xfffffe066821f020 ---
>>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020
>>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070
>>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
>>> 0xfffffe066821f530
>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe066821f570
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f650
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f6e0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f850
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f930
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f9c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821faa0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fb30
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc10
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fca0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd80
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fe10
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fef0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821ff80
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220060
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682200f0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201d0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220260
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220340
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682203d0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204b0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220540
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220620
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682206b0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220790
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220820
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220900
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220990
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a70
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220b00
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220be0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220c70
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d50
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220de0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220ec0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220f50
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221030
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682210c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221230
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221310
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682213a0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221480
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221510
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221680
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221760
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682217f0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218d0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221960
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a40
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221ad0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221bb0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221c40
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d20
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221db0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e90
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221f20
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222000
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222090
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222170
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222200
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222e0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222370
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222450
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682224e0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222650
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222730
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682227c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222930
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a10
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222aa0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20
>>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
>>> 0xfffffe0668222b80
>>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
>>> 0xfffffe0668222bb0
>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0
>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0
>>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>>> Uptime: 4m29s
>>> Dumping 1497 out of 24531
>>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>>
>>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/zfs.ko
>>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/opensolaris.ko
>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/geom_eli.ko
>>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/geom_mirror.ko
>>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/aesni.ko
>>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/uhid.ko
>>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ums.ko
>>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ipfw.ko
>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ipfw_nat.ko
>>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/libalias.ko
>>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/daemon_saver.ko
>>> #0 doadump (textdump=1) at pcpu.h:221
>>> 221 pcpu.h: No such file or directory.
>>> in pcpu.h
>>> (kgdb) #0 doadump (textdump=1) at pcpu.h:221
>>> #1 0xffffffff80ae9b6e in kern_reboot (howto=260)
>>> at /usr/src/sys/kern/kern_shutdown.c:366
>>> #2 0xffffffff80aea13b in vpanic (fmt=<value optimized out>,
>>> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
>>> #3 0xffffffff80ae9f73 in panic (fmt=0x0)
>>> at /usr/src/sys/kern/kern_shutdown.c:690
>>> #4 0xffffffff80fd18f2 in dblfault_handler (frame=<value optimized out>)
>>> at /usr/src/sys/amd64/amd64/trap.c:868
>>> #5 0xffffffff80fb406c in Xdblfault ()
>>> at /usr/src/sys/amd64/amd64/exception.S:292
>>> #6 0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510,
>>> node=0xfffff8056e45cdd0, balance=2)
>>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304
>>> #7 0xffffffff8220de58 in avl_remove (tree=<value optimized out>,
>>> data=<value optimized out>)
>>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818
>>> #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439
>>> #9 0xffffffff822a2123 in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907
>>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #11 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e3f33d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #13 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #16 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e578000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #18 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #21 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ec3d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #23 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #26 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e48a000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #28 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #31 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e57ab88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #33 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #36 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69a3000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #38 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #41 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5c8000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #43 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #46 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23f9000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #48 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #51 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5147b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #53 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #56 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e977000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #58 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #61 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90b3d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #63 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #66 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b1160b88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #68 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #71 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4f53d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #73 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #76 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8ebb88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #78 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #81 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8d9b88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #83 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #86 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21a03d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #88 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #91 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e6903d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #93 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #96 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a22dc7b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #98 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5fb7b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #101 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5fb7b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #103 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ae3d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #106 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ae3d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #108 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a2131b88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #111 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a2131b88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #113 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e80b7b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #116 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e80b7b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #118 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b14b9000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #121 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b14b9000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #123 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21fa3d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #126 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21fa3d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #128 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5333d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #131 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5333d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #133 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90ab88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #136 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90ab88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #138 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4047b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #141 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4047b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #143 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e7083d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #146 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e7083d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #148 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23917b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #151 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23917b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #153 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e636000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #156 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e636000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #158 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5afb88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #161 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5afb88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #163 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69523d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #166 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69523d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #168 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e47c000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #171 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e47c000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #173 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b17b23d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #176 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b17b23d8)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #178 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d367b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #181 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a6d367b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #183 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e99fb88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #186 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e99fb88)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #188 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5227b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #191 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5227b0)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #193 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>>> out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d3c000)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #196 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>> at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=<value optimized
>>> out>)
>>> at /usr/src/sys/kern/subr_taskqueue.c:454
>>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value
>>> optimized out>)
>>> at /usr/src/sys/kern/subr_taskqueue.c:724
>>> #199 0xffffffff80a9f255 in fork_exit (
>>> callout=0xffffffff80b49a70 <taskqueue_thread_loop>,
>>> arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00)
>>> at /usr/src/sys/kern/kern_fork.c:1040
>>> #200 0xffffffff80fb44ae in fork_trampoline ()
>>> at /usr/src/sys/amd64/amd64/exception.S:611
>>> #201 0x0000000000000000 in ?? ()
>>> Current language: auto; currently minimal
>>> (kgdb)
>>>
>>>
>>>>> --
>>>>> Karl Denninger
>>>>> karl at denninger.net <mailto:karl at denninger.net>
>>>>> /The Market Ticker/
>>>>> /[S/MIME encrypted email preferred]/
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
More information about the freebsd-stable
mailing list