solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: ...zfs/vdev_queue.c, line: 302

[Bryan Drewery]

On 8/10/15 12:45 PM, Bryan Drewery wrote:
> On 8/10/15 12:41 PM, Kristof Provost wrote:
>> Hi,
>>
>> Booting CURRENT (r286582) earlier I saw the following panic:
>> panic: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c, line: 302
>> cpuid = 1
>> Uptime: 41s
>>
>> ...
>> (kgdb) #0  doadump (textdump=1) at pcpu.h:221
>> #1  0xffffffff80a19e65 in kern_reboot (howto=260)
>>     at /usr/src/sys/kern/kern_shutdown.c:329
>> #2  0xffffffff80a1a458 in vpanic (fmt=<value optimized out>,
>>     ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:626
>> #3  0xffffffff80a1a4a3 in panic (fmt=0x0)
>>     at /usr/src/sys/kern/kern_shutdown.c:557
>> #4  0xffffffff8242622a in assfail (a=<value optimized out>,
>>     f=<value optimized out>, l=<value optimized out>)
>>     at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
>> #5  0xffffffff8213f8be in vdev_queue_io (zio=0xfffff8000ede1000)
>>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:302
>> #6  0xffffffff82163406 in zio_vdev_io_start (zio=0xfffff8000ede1000)
>>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2774
>> #7  0xffffffff8215eaa9 in zio_execute (zio=0xfffff8000ede1000)
>>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1496
>> #8  0xffffffff8215de5b in zio_nowait (zio=0xfffff8000ede1000)
>>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1550
>> #9  0xffffffff8219f23c in trim_map_commit (spa=0xfffffe00032b7000,
>>     zio=0xfffff8000eda6760, vd=0xfffff8000e7dd800)
>>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:476
>> #10 0xffffffff8219f046 in trim_map_commit (spa=0xfffffe00032b7000,
>>     zio=0xfffff8000eda6760, vd=0xfffff8000e9b4800)
>>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:532
>> #11 0xffffffff8219f046 in trim_map_commit (spa=0xfffffe00032b7000,
>>     zio=0xfffff8000eda6760, vd=0xfffff8000e9b4000)
>>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:532
>> #12 0xffffffff8219ed9f in trim_thread (arg=0xfffffe00032b7000)
>>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:579
>> #13 0xffffffff809e0374 in fork_exit (
>>     callout=0xffffffff8219ecb0 <trim_thread>, arg=0xfffffe00032b7000,
>>     frame=0xfffffe0239c09c00) at /usr/src/sys/kern/kern_fork.c:1006
>> #14 0xffffffff80e5e7ae in fork_trampoline ()
>>     at /usr/src/sys/amd64/amd64/exception.S:610
>> #15 0x0000000000000000 in ?? ()
>>
>> This is a regular amd64 machine with 4 disks in raidz. I'd pulled one drive and
>> replaced it with an empty one. It's possible that's related to the panic.
>>
> 
> 
> Same here...
> 
> panic: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file:
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c,
> line: 302
> cpuid = 1
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfffffe35545c6780
> vpanic() at vpanic+0x189/frame 0xfffffe35545c6800
> panic() at panic+0x43/frame 0xfffffe35545c6860
> assfail() at assfail+0x1a/frame 0xfffffe35545c6870
> vdev_queue_io() at vdev_queue_io+0x17e/frame 0xfffffe35545c68c0
> zio_vdev_io_start() at zio_vdev_io_start+0x446/frame 0xfffffe35545c6920
> zio_execute() at zio_execute+0x2e9/frame 0xfffffe35545c6970
> zio_nowait() at zio_nowait+0x6b/frame 0xfffffe35545c6990
> trim_map_commit() at trim_map_commit+0x2dc/frame 0xfffffe35545c6a30
> trim_map_commit() at trim_map_commit+0xe6/frame 0xfffffe35545c6ad0
> trim_map_commit() at trim_map_commit+0xe6/frame 0xfffffe35545c6b70
> trim_thread() at trim_thread+0xef/frame 0xfffffe35545c6bb0
> fork_exit() at fork_exit+0x84/frame 0xfffffe35545c6bf0
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe35545c6bf0
> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> KDB: enter: panic
> [ thread pid 6 tid 100684 ]
> Stopped at      kdb_enter+0x3e: movq    $0,kdb_why
> 
> 
> 2 disk mirror + an SSD log/cache.
> 
> I have no way to debug it currently, rolling back is hard enough on this
> system.
> 
> 

Booting with vfs.zfs.trim.enabled=0 seems to work fine as a workaround
until mav@ has a fix.

-- 
Regards,
Bryan Drewery