Fatal trap 9 triggered by zfs?

[Stefan Bethke]

Am 04.12.2009 um 17:52 schrieb Stefan Bethke:

> I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems.
> 
> FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec  1 14:30:54 UTC 2009     root at XXX.hanse.de:/usr/obj/usr/src/sys/EISENBOOT  amd64
> 
> # zpool status
>  pool: tank
> state: ONLINE
> scrub: none requested
> config:
> 
> 	NAME        STATE     READ WRITE CKSUM
> 	tank        ONLINE       0     0     0
> 	  ad4s1d    ONLINE       0     0     0
> # cat /boot/loader.conf
> vfs.zfs.arc_max="512M"
> vfs.zfs.prefetch_disable="1"
> vfs.zfs.zil_disable="1"

Got another, different one.  Any tuning suggestions or similar?

#0  doadump () at pcpu.h:223
223	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump () at pcpu.h:223
#1  0xffffffff80337bd9 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:416
#2  0xffffffff8033802c in panic (fmt=Variable "fmt" is not available.
)
    at /usr/src/sys/kern/kern_shutdown.c:579
#3  0xffffffff805cc2ad in trap_fatal (frame=0x9, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff805cce12 in trap (frame=0xffffff80625db030)
    at /usr/src/sys/amd64/amd64/trap.c:644
#5  0xffffffff805b2943 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:224
#6  0xffffffff80586c7a in vm_map_entry_splay (addr=Variable "addr" is not available.
)
    at /usr/src/sys/vm/vm_map.c:771
#7  0xffffffff80587f37 in vm_map_lookup_entry (map=0xffffff00010000e8, 
    address=18446743523979624448, entry=0xffffff80625db170)
    at /usr/src/sys/vm/vm_map.c:1021
#8  0xffffffff80588aa3 in vm_map_delete (map=0xffffff00010000e8, 
    start=18446743523979624448, end=18446743523979689984)
    at /usr/src/sys/vm/vm_map.c:2685
#9  0xffffffff80588e61 in vm_map_remove (map=0xffffff00010000e8, 
    start=18446743523979624448, end=18446743523979689984)
    at /usr/src/sys/vm/vm_map.c:2774
#10 0xffffffff8057db85 in uma_large_free (slab=0xffffff005fcc7000)
    at /usr/src/sys/vm/uma_core.c:3021
#11 0xffffffff80325987 in free (addr=0xffffff80018b0000, 
    mtp=0xffffffff80ac61e0) at /usr/src/sys/kern/kern_malloc.c:471
#12 0xffffffff80a36d03 in vdev_cache_evict (vc=0xffffff0001723ce0, 
    ve=0xffffff003dd52200)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:151
#13 0xffffffff80a372ad in vdev_cache_read (zio=0xffffff005f5ca2d0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:182
#14 0xffffffff80a4a954 in zio_vdev_io_start (zio=0xffffff005f5ca2d0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1814
#15 0xffffffff80a4ae87 in zio_execute (zio=0xffffff005f5ca2d0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996
#16 0xffffffff80a3a080 in vdev_mirror_io_start (zio=0xffffff005f811b40)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:303
#17 0xffffffff80a4ae87 in zio_execute (zio=0xffffff005f811b40)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996
#18 0xffffffff809ff45a in arc_read_nolock (pio=0xffffff005f66d5a0, 
    spa=0xffffff000150a000, bp=0xffffff800a91c440, 
    done=0xffffffff80a02630 <dbuf_read_done>, private=Variable "private" is not available.
)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2763
#19 0xffffffff809ff8ec in arc_read (pio=0xffffff005f66d5a0, 
    spa=0xffffff000150a000, bp=0xffffff800a91c440, pbuf=0xffffff0042a3ca20, 
    done=0xffffffff80a02630 <dbuf_read_done>, private=0xffffff005fbfc620, 
    priority=0, zio_flags=1, arc_flags=0xffffff80625db5ec, 
    zb=0xffffff80625db5c0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2508
#20 0xffffffff80a02aba in dbuf_read (db=0xffffff005fbfc620, 
    zio=0xffffff005f66d5a0, flags=2)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:521
#21 0xffffffff80a0602c in dmu_buf_hold (os=Variable "os" is not available.
)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:106
#22 0xffffffff80a40db5 in zap_lockdir (os=0xffffff005f937610, obj=247890, 
    tx=0x0, lti=RW_READER, fatreader=1, adding=0, zapp=0xffffff80625db888)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:388
#23 0xffffffff80a41724 in zap_cursor_retrieve (zc=0xffffff80625db880, 
    za=0xffffff80625db8c0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1004
#24 0xffffffff80a61b66 in zfs_freebsd_readdir (ap=Variable "ap" is not available.
)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2157
#25 0xffffffff803cfde9 in kern_getdirentries (td=0xffffff0057bfe000, fd=Variable "fd" is not available.
)
    at vnode_if.h:758
#26 0xffffffff803d0093 in getdirentries (td=Variable "td" is not available.
)
    at /usr/src/sys/kern/vfs_syscalls.c:4051
#27 0xffffffff805cc906 in syscall (frame=0xffffff80625dbc80)
    at /usr/src/sys/amd64/amd64/trap.c:989
#28 0xffffffff805b2c21 in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:373
#29 0x0000000800724cdc in ?? ()
Previous frame inner to this frame (corrupt stack?)


-- 
Stefan Bethke <stb at lassitu.de>   Fon +49 151 14070811