panic on zpool mirror removal
Roger Hammerstein
cheeky.m at live.com
Wed May 23 15:22:43 UTC 2018
11.2-beta2
steps to reproduce:
zpool create test mirror da1 da2 mirror da3 d4 mirror da5 da6
(sometimes copy data to it, but that doesn't seem necessary)
zpool remove test mirror-1
panics there.
it also seems to be panic after reboot when trying to import it,
or scrub it, or destroy it.
# kgdb /boot/kernel/kernel /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
panic: page fault
cpuid = 8
KDB: stack backtrace:
#0 0xffffffff80b3d407 at kdb_backtrace+0x67
#1 0xffffffff80af6a77 at vpanic+0x177
#2 0xffffffff80af68f3 at panic+0x43
#3 0xffffffff80f77f6f at trap_fatal+0x35f
#4 0xffffffff80f77fc9 at trap_pfault+0x49
#5 0xffffffff80f77797 at trap+0x2c7
#6 0xffffffff80f5744c at calltrap+0x8
#7 0xffffffff824f01d7 at vdev_indirect_io_start_cb+0x37
#8 0xffffffff824efe58 at vdev_indirect_remap+0x2f8
#9 0xffffffff824efb3d at vdev_indirect_io_start+0x2d
#10 0xffffffff8251ac9e at zio_vdev_io_start+0x2ae
#11 0xffffffff8251774c at zio_execute+0xac
#12 0xffffffff8251706b at zio_nowait+0xcb
#13 0xffffffff824f38ef at vdev_mirror_io_start+0x3ff
#14 0xffffffff8251ab52 at zio_vdev_io_start+0x162
#15 0xffffffff8251774c at zio_execute+0xac
#16 0xffffffff80b4ec14 at taskqueue_run_locked+0x154
#17 0xffffffff80b4fd78 at taskqueue_thread_loop+0x98
Uptime: 1m46s
Dumping 1886 out of 49109 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
#0 doadump (textdump=<value optimized out>) at pcpu.h:229
229 pcpu.h: No such file or directory.
in pcpu.h
(kgdb)
(kgdb) bt
#0 doadump (textdump=<value optimized out>) at pcpu.h:229
#1 0xffffffff80af668b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:383
#2 0xffffffff80af6ab1 in vpanic (fmt=<value optimized out>, ap=<value optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:776
#3 0xffffffff80af68f3 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:707
#4 0xffffffff80f77f6f in trap_fatal (frame=0xfffffe0c58d30720, eva=0) at /usr/src/sys/amd64/amd64/trap.c:875
#5 0xffffffff80f77fc9 in trap_pfault (frame=0xfffffe0c58d30720, usermode=0) at pcpu.h:229
#6 0xffffffff80f77797 in trap (frame=0xfffffe0c58d30720) at /usr/src/sys/amd64/amd64/trap.c:415
#7 0xffffffff80f5744c in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231
#8 0xffffffff82476994 in abd_get_offset (sabd=0x0, off=0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c:443
#9 0xffffffff824f01d7 in vdev_indirect_io_start_cb (split_offset=<value optimized out>, vd=0xfffff800237fd000,
offset=1258659328, size=3584, arg=0xfffff8006ac49000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_indirect.c:1082
#10 0xffffffff824efe58 in vdev_indirect_remap (vd=<value optimized out>, offset=<value optimized out>,
asize=<value optimized out>, func=0xffffffff824f01a0 <vdev_indirect_io_start_cb>, arg=0xfffff8006ac49000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_indirect.c:1041
#11 0xffffffff824efb3d in vdev_indirect_io_start (zio=0xfffff8006ac49000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_indirect.c:1099
#12 0xffffffff8251ac9e in zio_vdev_io_start (zio=0xfffff8006ac49000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3297
#13 0xffffffff8251774c in zio_execute (zio=0xfffff8006ac49000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1768
#14 0xffffffff8251706b in zio_nowait (zio=0xfffff8006ac49000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1826
#15 0xffffffff824f38ef in vdev_mirror_io_start (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:557
#16 0xffffffff8251ab52 in zio_vdev_io_start (zio=0xfffff8006ad1c000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3166
#17 0xffffffff8251774c in zio_execute (zio=0xfffff8006ad1c000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1768
#18 0xffffffff80b4ec14 in taskqueue_run_locked (queue=0xfffff8006a839900) at /usr/src/sys/kern/subr_taskqueue.c:463
#19 0xffffffff80b4fd78 in taskqueue_thread_loop (arg=<value optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:755
#20 0xffffffff80aba0b3 in fork_exit (callout=0xffffffff80b4fce0 <taskqueue_thread_loop>, arg=0xfffff8006a7b81f0,
frame=0xfffffe0c58d30c00) at /usr/src/sys/kern/kern_fork.c:1054
#21 0xffffffff80f5836e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:957
#22 0x0000000000000000 in ?? ()
(kgdb)
--------------------------------------
after rebooting and trying to mount it to see what happened
kgdb /boot/kernel/kernel /var/crash/vmcore.1
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
current process = 0 (zio_free_issue_6_1)
trap number = 12
fault code = supervisor read data, page not present
= DPL 0, pres 1, long 1, def32 0, gran 1
instruction pointer = 0x20:0xffffffff82476994
stack pointer = 0x28:0xfffffe0c58b217e0
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (zio_free_issue_3_1)
frame pointer = 0x28:0xfffffe0c58b21810
panic: page fault
cpuid = 5
KDB: stack backtrace:
#0 0xffffffff80b3d407 at kdb_backtrace+0x67
#1 0xffffffff80af6a77 at vpanic+0x177
#2 0xffffffff80af68f3 at panic+0x43
#3 0xffffffff80f77f6f at trap_fatal+0x35f
#4 0xffffffff80f77fc9 at trap_pfault+0x49
#5 0xffffffff80f77797 at trap+0x2c7
#6 0xffffffff80f5744c at calltrap+0x8
#7 0xffffffff824f01d7 at vdev_indirect_io_start_cb+0x37
#8 0xffffffff824efe58 at vdev_indirect_remap+0x2f8
#9 0xffffffff824efb3d at vdev_indirect_io_start+0x2d
#10 0xffffffff8251ac9e at zio_vdev_io_start+0x2ae
#11 0xffffffff8251774c at zio_execute+0xac
#12 0xffffffff8251706b at zio_nowait+0xcb
#13 0xffffffff824f38ef at vdev_mirror_io_start+0x3ff
#14 0xffffffff8251ab52 at zio_vdev_io_start+0x162
#15 0xffffffff8251774c at zio_execute+0xac
#16 0xffffffff80b4ec14 at taskqueue_run_locked+0x154
#17 0xffffffff80b4fd78 at taskqueue_thread_loop+0x98
Uptime: 16m41s
Dumping 1894 out of 49109 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
#0 doadump (textdump=<value optimized out>) at pcpu.h:229
229 pcpu.h: No such file or directory.
in pcpu.h
(kgdb)
(kgdb) bt
#0 doadump (textdump=<value optimized out>) at pcpu.h:229
#1 0xffffffff80af668b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:383
#2 0xffffffff80af6ab1 in vpanic (fmt=<value optimized out>, ap=<value optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:776
#3 0xffffffff80af68f3 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:707
#4 0xffffffff80f77f6f in trap_fatal (frame=0xfffffe0c58adb720, eva=0) at /usr/src/sys/amd64/amd64/trap.c:875
#5 0xffffffff80f77fc9 in trap_pfault (frame=0xfffffe0c58adb720, usermode=0) at pcpu.h:229
#6 0xffffffff80f77797 in trap (frame=0xfffffe0c58adb720) at /usr/src/sys/amd64/amd64/trap.c:415
#7 0xffffffff80f5744c in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231
#8 0xffffffff82476994 in abd_get_offset (sabd=0x0, off=0)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c:443
#9 0xffffffff824f01d7 in vdev_indirect_io_start_cb (split_offset=<value optimized out>, vd=0xfffff8002373f800,
offset=1225929216, size=2560, arg=0xfffff801c43c9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_indirect.c:1082
#10 0xffffffff824efe58 in vdev_indirect_remap (vd=<value optimized out>, offset=<value optimized out>,
asize=<value optimized out>, func=0xffffffff824f01a0 <vdev_indirect_io_start_cb>, arg=0xfffff801c43c9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_indirect.c:1041
#11 0xffffffff824efb3d in vdev_indirect_io_start (zio=0xfffff801c43c9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_indirect.c:1099
#12 0xffffffff8251ac9e in zio_vdev_io_start (zio=0xfffff801c43c9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3297
#13 0xffffffff8251774c in zio_execute (zio=0xfffff801c43c9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1768
#14 0xffffffff8251706b in zio_nowait (zio=0xfffff801c43c9000)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1826
#15 0xffffffff824f38ef in vdev_mirror_io_start (zio=<value optimized out>)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:557
#16 0xffffffff8251ab52 in zio_vdev_io_start (zio=0xfffff801c4419820)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3166
#17 0xffffffff8251774c in zio_execute (zio=0xfffff801c4419820)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1768
#18 0xffffffff80b4ec14 in taskqueue_run_locked (queue=0xfffff80168168100) at /usr/src/sys/kern/subr_taskqueue.c:463
#19 0xffffffff80b4fd78 in taskqueue_thread_loop (arg=<value optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:755
#20 0xffffffff80aba0b3 in fork_exit (callout=0xffffffff80b4fce0 <taskqueue_thread_loop>, arg=0xfffff800220f6f60,
frame=0xfffffe0c58adbc00) at /usr/src/sys/kern/kern_fork.c:1054
#21 0xffffffff80f5836e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:957
#22 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb)
# zpool import
pool: test
id: 632784374722369342
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
test ONLINE
mirror-0 ONLINE
da1 ONLINE
da2 ONLINE
indirect-1 ONLINE
mirror-2 ONLINE
da5 ONLINE
da6 ONLINE
and then it panics when trying to import.
More information about the freebsd-fs
mailing list