Fresh 7.0 Install: Fatal Trap 12 panic when put under load

[Kostik Belousov]

On Tue, Jul 15, 2008 at 08:19:15PM +0100, john at basicnets.co.uk wrote:
> 
> 
>   > Please collect kgdb/ddb backtraces.
> 
>   kgdb backtrace:
> 
>   server251# kgdb -c /var/crash/vmcore.0
> kgdb: couldn't find a suitable kernel image
> server251# kgdb /boot/kernel/kernel /var/crash/vmcore.0
> kgdb: kvm_read: invalid address (0xffffff00010e5468)
> [GDB will not be able to debug user-mode threads:  
> /usr/lib/libthread_db.so: Unde
> fined symbol "ps_pglobal_lookup"]
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain 
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd".
> 
>   Unread portion of the kernel message buffer:
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address   = 0x6400000000
> fault code              = supervisor read instruction, page not present
> instruction pointer     = 0x8:0x6400000000
> stack pointer           = 0x10:0xffffffffb1d7f590
> frame pointer           = 0x10:0xffffff0035d2dcc0
> code segment            = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 88622 (make)
> trap number             = 12
> panic: page fault
> cpuid = 0
> Uptime: 5h57m22s
> Physical memory: 4082 MB
> Dumping 444 MB: 429 413 397 381 365 349 333 317 301 285 269 253 237  
> 221 205 189
> 173 157 141 125 109 93 77 61 45 29 13
> 
>   #0  doadump () at pcpu.h:194
> 194     pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb)
> (kgdb) list *0x6400000000
> No source file for address 0x6400000000.
> (kgdb) backtrace
> #0  doadump () at pcpu.h:194
> #1  0xffffff0004742440 in ?? ()
> #2  0xffffffff80477699 in boot (howto=260)
> at /usr/src/sys/kern/kern_shutdown.c:409
> #3  0xffffffff80477a9d in panic (fmt=0x104 <Address 0x104 out of bounds>)
> at /usr/src/sys/kern/kern_shutdown.c:563
> #4  0xffffffff8072ed44 in trap_fatal (frame=0xffffff00048ee000,
> eva=18446742974275512528) at /usr/src/sys/amd64/amd64/trap.c:724
> #5  0xffffffff8072f115 in trap_pfault (frame=0xffffffffb1d7f4e0, usermode=0)
> at /usr/src/sys/amd64/amd64/trap.c:641
> #6  0xffffffff8072fa58 in trap (frame=0xffffffffb1d7f4e0)
> at /usr/src/sys/amd64/amd64/trap.c:410
> #7  0xffffffff807156be in calltrap ()
> at /usr/src/sys/amd64/amd64/exception.S:169
> #8  0x0000006400000000 in ?? ()
> #9  0xffffffff8067d3ee in uma_zalloc_arg (zone=0xffffff00bfed07e0, 
> udata=0x0,
> flags=-256) at /usr/src/sys/vm/uma_core.c:1835
From the frame #9, please do
p *zone
I am esp. interested in the value of the uz_ctor member.

It seems that it becomes corrupted, it value should be 0, as this seems
to be ffs inode zone.  I suspect that gdb would show 0x6400000000 instead.

That may be kernel memory corruption, but might be a bad memory
as well (double bit inversion ?).

> #10 0xffffffff80661ecf in ffs_vget (mp=0xffffff00047f4978, ino=47884512,
> flags=2, vpp=0xffffffffb1d7f728) at uma.h:277
> #11 0xffffffff8066d010 in ufs_lookup (ap=0xffffffffb1d7f780)
> at /usr/src/sys/ufs/ufs/ufs_lookup.c:573
> #12 0xffffffff804dfa89 in vfs_cache_lookup (ap=Variable "ap" is not 
> available.
> ) at vnode_if.h:83
> #13 0xffffffff8077235f in VOP_LOOKUP_APV (vop=0xffffffff809e7de0,
> a=0xffffffffb1d7f840) at vnode_if.c:99
> ---Type <return> to continue, or q <return> to quit---
> #14 0xffffffff804e6394 in lookup (ndp=0xffffffffb1d7f950) at vnode_if.h:57
> #15 0xffffffff804e7228 in namei (ndp=0xffffffffb1d7f950)
> at /usr/src/sys/kern/vfs_lookup.c:219
> #16 0xffffffff804f4717 in kern_stat (td=0xffffff00048ee000,
> path=0x8006f7040 <Address 0x8006f7040 out of bounds>,  
> pathseg=Variable "path
> seg" is not available.
> )
> at /usr/src/sys/kern/vfs_syscalls.c:2109
> #17 0xffffffff804f4987 in stat (td=Variable "td" is not available.
> ) at /usr/src/sys/kern/vfs_syscalls.c:2093
> #18 0xffffffff8072f397 in syscall (frame=0xffffffffb1d7fc70)
> at /usr/src/sys/amd64/amd64/trap.c:852
> #19 0xffffffff807158cb in Xfast_syscall ()
> at /usr/src/sys/amd64/amd64/exception.S:290
> #20 0x000000000043127c in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> 
>   I really don't understand this -any advice you can give would  
> really be appreciated.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20080715/4fe7c4ac/attachment.pgp