Fatal trap 9 triggered by zfs?

[Stefan Bethke]

I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems.

FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec  1 14:30:54 UTC 2009     root at XXX.hanse.de:/usr/obj/usr/src/sys/EISENBOOT  amd64

# zpool status
  pool: tank
 state: ONLINE
 scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  ad4s1d    ONLINE       0     0     0
# cat /boot/loader.conf
vfs.zfs.arc_max="512M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.zil_disable="1"

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer	= 0x20:0xffffffff80a39900
stack pointer	        = 0x28:0xffffff80622ddae0
frame pointer	        = 0x28:0xffffff80622ddb10
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (spa_zio)
trap number		= 9
panic: general protection fault
cpuid = 0
Uptime: 17h44m5s
Physical memory: 3313 MB
Dumping 1843 MB: 1828 1812 1796 1780 1764 1748 1732 1716 1700 1684 1668 1652 1636 1620 1604 1588 1572 1556 1540 1524 1508 1492 1476 1460 1444 1428 1412 1396 1380 1364 1348 1332 1316 1300 1284 1268 1252 1236 1220 1204 1188 1172 1156 1140 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964 948 932 916 900 884 868 852 836 820 804 788 772 756 740 724 708 692 676 660 644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388 372 356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100 84 68 52 36 20 4

#0  doadump () at pcpu.h:223
223	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump () at pcpu.h:223
#1  0xffffffff803374b9 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:416
#2  0xffffffff8033790c in panic (fmt=Variable "fmt" is not available.
)
    at /usr/src/sys/kern/kern_shutdown.c:579
#3  0xffffffff805cbb8d in trap_fatal (frame=0x9, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff805cc6f2 in trap (frame=0xffffff80622dda30)
    at /usr/src/sys/amd64/amd64/trap.c:644
#5  0xffffffff805b2223 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:224
#6  0xffffffff80a39900 in vdev_queue_agg_io_done (aio=0xffffff00374562d0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:174
#7  0xffffffff80a4be6f in zio_done (zio=0xffffff00374562d0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2243
#8  0xffffffff80a49e87 in zio_execute (zio=0xffffff00374562d0)
    at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996
#9  0xffffffff809ed603 in taskq_run (arg=0xffffff008d8d0420, pending=Variable "pending" is not available.
)
    at /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_taskq.c:108
#10 0xffffffff80373533 in taskqueue_run (queue=0xffffff00017e1400)
    at /usr/src/sys/kern/subr_taskqueue.c:239
#11 0xffffffff803737b6 in taskqueue_thread_loop (arg=Variable "arg" is not available.
)
    at /usr/src/sys/kern/subr_taskqueue.c:360
#12 0xffffffff8030e0b8 in fork_exit (
    callout=0xffffffff80373770 <taskqueue_thread_loop>, 
    arg=0xffffff00016434e0, frame=0xffffff80622ddc80)
    at /usr/src/sys/kern/kern_fork.c:843
#13 0xffffffff805b26fe in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:561
#14 0x0000000000000000 in ?? ()
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000c6c000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x000000000000000b in ?? ()
#41 0xffffffff80832500 in affinity ()
#42 0xffffff000173c390 in ?? ()
#43 0xffffff80622dd240 in ?? ()
#44 0xffffff80622dd1f8 in ?? ()
#45 0xffffff00015ecab0 in ?? ()
#46 0xffffffff8035aa48 in sched_switch (td=0xffffffff80373770, 
    newtd=0xffffff00016434e0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1858
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

-- 
Stefan Bethke <stb at lassitu.de>   Fon +49 151 14070811