Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE
Karl Denninger
karl at denninger.net
Mon Oct 17 16:02:16 UTC 2016
Update from my test system:
Setting vfs.zfs.vdev_trim_max_active to 10 (from default 64) does *not*
stop the panics.
Setting vfs.zfs.vdev.trim.enabled = 0 (which requires a reboot) DOES
stop the panics.
I am going to run a scrub on the pack, but I suspect the pack itself
(now that I can actually mount it without the machine blowing up!) is fine.
THIS (OBVIOUSLY) NEEDS ATTENTION!
On 10/17/2016 09:17, Karl Denninger wrote:
> This is a situation I've had happen before, and reported -- it appeared
> to be a kernel stack overflow, and it has gotten materially worse on
> 11.0-STABLE.
>
> The issue occurs after some period of time (normally a week or so.) The
> system has a mirrored pair of large drives used for backup purposes to
> which ZFS snapshots are written using a script that iterates over the
> system.
>
> The panic /only /happens when the root filesystem is being sent, and it
> appears that the panic itself is being triggered by an I/O pattern on
> the /backup /drive -- not the source drives. Zpool scrubs on the source
> are clean; I am going to run one now on the backup, but in the past that
> has been clean as well.
>
> I now have a *repeatable* panic in that if I attempt a "zfs list -rt all
> backup" on the backup volume I get the below panic. A "zfs list"
> does*__*not panic the system.
>
> The operating theory previously (after digging through the passed
> structures in the dump) was that the ZFS system was attempting to issue
> TRIMs on a device that can't do them before the ZFS system realizes this
> and stops asking (the backup volume is comprised of spinning rust) but
> the appearance of the panic now on the volume when I simply do a "zfs
> list -rt all backup" appears to negate that theory since no writes are
> performed by that operation, and thus no TRIM calls should be issued.
>
> I can leave the backup volume in the state that causes this for a short
> period of time in an attempt to find and fix this.
>
>
> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>
> Mon Oct 17 09:02:33 CDT 2016
>
> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
> r307318M: Fri Oct 14 09:23:46 CDT 2016
> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64
>
> panic: double fault
>
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
>
> Fatal double fault
> rip = 0xffffffff8220d9ec
> rsp = 0xfffffe066821f000
> rbp = 0xfffffe066821f020
> cpuid = 6; apic id = 14
> panic: double fault
> cpuid = 6
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfffffe0649d78e30
> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0
> panic() at panic+0x43/frame 0xfffffe0649d78f10
> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30
> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30
> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000, rbp =
> 0xfffffe066821f020 ---
> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020
> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070
> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
> 0xfffffe066821f530
> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe066821f570
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f650
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f6e0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7c0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f850
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f930
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f9c0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821faa0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fb30
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc10
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fca0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd80
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fe10
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fef0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821ff80
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220060
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682200f0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201d0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220260
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220340
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682203d0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204b0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220540
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220620
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682206b0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220790
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220820
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220900
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220990
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a70
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220b00
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220be0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220c70
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d50
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220de0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220ec0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220f50
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221030
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682210c0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211a0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221230
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221310
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682213a0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221480
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221510
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215f0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221680
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221760
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682217f0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218d0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221960
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a40
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221ad0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221bb0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221c40
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d20
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221db0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e90
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221f20
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222000
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222090
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222170
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222200
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222e0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222370
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222450
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682224e0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225c0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222650
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222730
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682227c0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228a0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222930
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960
> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a10
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222aa0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0
> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20
> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
> 0xfffffe0668222b80
> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
> 0xfffffe0668222bb0
> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0
> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> Uptime: 4m29s
> Dumping 1497 out of 24531
> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>
> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/zfs.ko
> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/opensolaris.ko
> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/geom_eli.ko
> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/geom_mirror.ko
> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/aesni.ko
> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/uhid.ko
> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/ums.ko
> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/ipfw.ko
> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/ipfw_nat.ko
> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/libalias.ko
> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/daemon_saver.ko
> #0 doadump (textdump=1) at pcpu.h:221
> 221 pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb) #0 doadump (textdump=1) at pcpu.h:221
> #1 0xffffffff80ae9b6e in kern_reboot (howto=260)
> at /usr/src/sys/kern/kern_shutdown.c:366
> #2 0xffffffff80aea13b in vpanic (fmt=<value optimized out>,
> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
> #3 0xffffffff80ae9f73 in panic (fmt=0x0)
> at /usr/src/sys/kern/kern_shutdown.c:690
> #4 0xffffffff80fd18f2 in dblfault_handler (frame=<value optimized out>)
> at /usr/src/sys/amd64/amd64/trap.c:868
> #5 0xffffffff80fb406c in Xdblfault ()
> at /usr/src/sys/amd64/amd64/exception.S:292
> #6 0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510,
> node=0xfffff8056e45cdd0, balance=2)
> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304
> #7 0xffffffff8220de58 in avl_remove (tree=<value optimized out>,
> data=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818
> #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439
> #9 0xffffffff822a2123 in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907
> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #11 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e3f33d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #13 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #16 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e578000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #18 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #21 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ec3d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #23 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #26 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e48a000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #28 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #31 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e57ab88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #33 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #36 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69a3000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #38 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #41 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5c8000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #43 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #46 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23f9000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #48 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #51 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5147b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #53 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #56 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e977000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #58 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #61 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90b3d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #63 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #66 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b1160b88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #68 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #71 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4f53d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #73 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #76 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8ebb88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #78 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #81 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8d9b88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #83 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #86 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21a03d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #88 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #91 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e6903d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #93 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #96 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a22dc7b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #98 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5fb7b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #101 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5fb7b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #103 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ae3d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #106 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ae3d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #108 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a2131b88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #111 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a2131b88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #113 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e80b7b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #116 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e80b7b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #118 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b14b9000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #121 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b14b9000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #123 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21fa3d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #126 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21fa3d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #128 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5333d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #131 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5333d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #133 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90ab88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #136 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90ab88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #138 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4047b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #141 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4047b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #143 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e7083d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #146 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e7083d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #148 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23917b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #151 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23917b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #153 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e636000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #156 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e636000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #158 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5afb88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #161 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5afb88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #163 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69523d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #166 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69523d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #168 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e47c000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #171 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e47c000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #173 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b17b23d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #176 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b17b23d8)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #178 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d367b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #181 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a6d367b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #183 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e99fb88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #186 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e99fb88)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #188 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5227b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #191 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5227b0)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
> #193 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d3c000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
> #196 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=<value optimized
> out>)
> at /usr/src/sys/kern/subr_taskqueue.c:454
> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value optimized out>)
> at /usr/src/sys/kern/subr_taskqueue.c:724
> #199 0xffffffff80a9f255 in fork_exit (
> callout=0xffffffff80b49a70 <taskqueue_thread_loop>,
> arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00)
> at /usr/src/sys/kern/kern_fork.c:1040
> #200 0xffffffff80fb44ae in fork_trampoline ()
> at /usr/src/sys/amd64/amd64/exception.S:611
> #201 0x0000000000000000 in ?? ()
> Current language: auto; currently minimal
> (kgdb)
>
>
--
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2996 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20161017/b7efb90a/attachment.bin>
More information about the freebsd-stable
mailing list