Reliably trigger-able ZFS panic

Sun Mar 2 11:49:28 UTC 2008

Hi,

The following iozone test case on ZFS would reliably trigger panic:

/usr/local/bin/iozone -M -e -+u -T -t 128 -S 4096 -L 64 -R -r 4k -s 30g 
-i 0 -i 1 -i 2 -i 8 -+p 70 -C

Unfortunately the kgdb can not reveal useful backtrace.  I have tried 
KDB_TRACE, but have not yet be able to further investigate it.

fs12# kgdb /boot/kernel/kernel.symbols vmcore.0
[GDB will not be able to debug user-mode threads: 
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:

Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 05
fault virtual address   = 0x18
fault code              = supervisor read data, page not present
instruction pointer     = 0x8:0xffffffff80763d16
stack pointer           = 0x10:0xffffffffd94798f0
frame pointer           = 0x10:0xffffffffd9479920
code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 340 (txg_thread_enter)
trap number             = 12
panic: page fault
cpuid = 5
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x17a
trap_fatal() at trap_fatal+0x29f
trap_pfault() at trap_pfault+0x294
trap() at trap+0x2ea
calltrap() at calltrap+0x8
--- trap 0xc, rip = 0xffffffff80763d16, rsp = 0xffffffffd94798f0, rbp = 
0xffffffffd9479920 ---
dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x26
dmu_objset_sync() at dmu_objset_sync+0x12d
dsl_pool_sync() at dsl_pool_sync+0x72
spa_sync() at spa_sync+0x390
txg_sync_thread() at txg_sync_thread+0x12f
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffd9479d30, rbp = 0 ---
Uptime: 25m7s
Physical memory: 4081 MB
Dumping 1139 MB: 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964 948 
932 916 900 884 868 852 836 820 804 788 772 756 740 724 708 692 676 660 
644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388 372 
356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100 84 
68 52 36 20 4

#0  doadump () at pcpu.h:194
194     pcpu.h: No such file or directory.
         in pcpu.h
(kgdb) add-symbol-file /boot/kernel/zfs.ko.symbols
add symbol table from file "/boot/kernel/zfs.ko.symbols" at
(y or n) y
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
(kgdb) where
#0  doadump () at pcpu.h:194
#1  0xffffffff80277aa8 in boot (howto=260) at 
/usr/src/sys/kern/kern_shutdown.c:409
#2  0xffffffff80277f07 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xffffffff80465a1f in trap_fatal (frame=0xc, eva=Variable "eva" is 
not available.
) at /usr/src/sys/amd64/amd64/trap.c:724
#4  0xffffffff80465e04 in trap_pfault (frame=0xffffffffd9479840, usermode=0)
     at /usr/src/sys/amd64/amd64/trap.c:641
#5  0xffffffff8046677a in trap (frame=0xffffffffd9479840) at 
/usr/src/sys/amd64/amd64/trap.c:410
#6  0xffffffff8044babe in calltrap () at 
/usr/src/sys/amd64/amd64/exception.S:169
#7  0xffffffff80763d16 in ?? ()
#8  0x0000000000000004 in adjust_ace_pair ()
#9  0x0000000000000004 in adjust_ace_pair ()
#10 0xffffffffd94799e0 in ?? ()
#11 0xffffffff80763e7d in ?? ()
#12 0xffffff0004275a80 in ?? ()
#13 0xffffff00045a1190 in ?? ()
#14 0xffffffff807639b0 in ?? ()
#15 0xffffffff80763f20 in ?? ()
#16 0xffffff00042dc800 in ?? ()
#17 0x0000000000000004 in adjust_ace_pair ()
#18 0xffffffffd9479990 in ?? ()
#19 0x000000000000b55d in z_deflateInit2_ (strm=0xffffff00042dc8e0, 
level=70109184, method=68351768,
     windowBits=68351600, memLevel=76231808, strategy=76231808, 
version=Cannot access memory at address 0xffffffff00040010
)
     at 
/usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/zmod/deflate.c:318
Previous frame inner to this frame (corrupt stack?)
-- 
Xin LI <delphij at delphij.net>	http://www.delphij.net/
FreeBSD - The Power to Serve!

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20080302/449f639a/signature.pgp