kern/127699: [panic] Fatal trap 12: page fault while in kernel mode

Sat Oct 4 01:20:07 UTC 2008

The following reply was made to PR kern/127699; it has been noted by GNATS.

From: Volker Werth <vwe at freebsd.org>
To: Wouter Snels <nospam at ofloo.net>
Cc: bug-followup at freebsd.org
Subject: Re: kern/127699: [panic] Fatal trap 12: page fault while in kernel
 mode
Date: Sat, 04 Oct 2008 02:42:58 +0200

 On 10/03/08 23:37, Wouter Snels wrote:
 > I'm glad someone is anwering, I did however do make update just today in
 > /usr/src i was planning to update the kernel since i've noticed there is
 > a new IPv6 security issue, .. incase it matters, ..

 Have you already recompiled and installed a new kernel? This would make
 our debugging much harder.

 > Also thank you for the for the information on compiling the kernel.
 > 
 > vmcore.3
 > 
 > kgdb /boot/kernel/kernel /var/crash/vmcore.3
 > [GDB will not be able to debug user-mode threads:
 > /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 > GNU gdb 6.1.1 [FreeBSD]
 > Copyright 2004 Free Software Foundation, Inc.
 > GDB is free software, covered by the GNU General Public License, and you are
 > welcome to change it and/or distribute copies of it under certain
 > conditions.
 > Type "show copying" to see the conditions.
 > There is absolutely no warranty for GDB.  Type "show warranty" for details.
 > This GDB was configured as "i386-marcel-freebsd".
 > bt
 > 
 > Unread portion of the kernel message buffer:
 > 
 > 
 > Fatal trap 12: page fault while in kernel mode
 > cpuid = 3; apic id = 02
 > fault virtual address    = 0x4c
 > fault code        = supervisor read, page not present
 > instruction pointer    = 0x20:0xc577373d
 > stack pointer            = 0x28:0xe8edd808
 > frame pointer            = 0x28:0xe8edd84c
 > code segment        = base 0x0, limit 0xfffff, type 0x1b
 >             = DPL 0, pres 1, def32 1, gran 1
 > processor eflags    = interrupt enabled, resume, IOPL = 0
 > current process        = 7569 (irssi)
 > trap number        = 12
 > panic: page fault
 > cpuid = 3
 > Uptime: 14h39m53s
 > Physical memory: 2035 MB
 > Dumping 182 MB: 167 151 135 119 103 87 71 55 39 23 7
 > 
 > #0  doadump () at pcpu.h:195
 > 195    pcpu.h: No such file or directory.
 >     in pcpu.h
 > (kgdb) bt
 > #0  doadump () at pcpu.h:195
 > #1  0xc0754507 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
 > #2  0xc07547c9 in panic (fmt=Variable "fmt" is not available.
 > ) at /usr/src/sys/kern/kern_shutdown.c:563
 > #3  0xc0a5632c in trap_fatal (frame=0xe8edd7c8, eva=76)
 >     at /usr/src/sys/i386/i386/trap.c:899
 > #4  0xc0a56590 in trap_pfault (frame=0xe8edd7c8, usermode=0, eva=76)
 >     at /usr/src/sys/i386/i386/trap.c:812
 > #5  0xc0a56f12 in trap (frame=0xe8edd7c8) at
 > /usr/src/sys/i386/i386/trap.c:490
 > #6  0xc0a3d88b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
 > #7  0xc577373d in ?? ()
 > Previous frame inner to this frame (corrupt stack?)
 > (kgdb)

 Ok, this is from within if_stf.ko which has not been loaded while you
 were running kgdb.

 Assuming the kernel hasn't been recompiled since your last crash, can
 you please try:

 ``kgdb -q''
 (kgdb) list *0xc577373d

 This will get a code fragment from your currently running kernel and
 list us the source of the lines causing the crash.

 From your output, I'm pretty sure the fault is caused by if_stf. We just
 need to figure out what's causing this so we need a hint which function
 has called what.

 > vmcore.1
 > 
 > kgdb /boot/kernel/kernel /var/crash/vmcore.1
 > [GDB will not be able to debug user-mode threads:
 > /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 > GNU gdb 6.1.1 [FreeBSD]
 > Copyright 2004 Free Software Foundation, Inc.
 > GDB is free software, covered by the GNU General Public License, and you are
 > welcome to change it and/or distribute copies of it under certain
 > conditions.
 > Type "show copying" to see the conditions.
 > There is absolutely no warranty for GDB.  Type "show warranty" for details.
 > This GDB was configured as "i386-marcel-freebsd".
 > bt
 > 
 > Unread portion of the kernel message buffer:
 > <118>Aug 31 15:32:26  syslogd: exiting on signal 15
 > Waiting (max 60 seconds) for system process `vnlru' to stop...done
 > Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
 > Waiting (max 60 seconds) for system process `syncer' to stop...
 > Syncing disks, vnodes remaining...3 2 1 0 0 0 done
 > All buffers synced.
 > panic: dqflush: stray dquot
 > cpuid = 0
 > Uptime: 21h5m59s
 > Physical memory: 2035 MB
 > Dumping 230 MB: 215 199 183 167 151 135 119 103 87 71 55 39 23 7
 > 
 > #0  doadump () at pcpu.h:195
 > 195    pcpu.h: No such file or directory.
 >     in pcpu.h
 > (kgdb) bt
 > #0  doadump () at pcpu.h:195
 > #1  0xc0754507 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
 > #2  0xc07547c9 in panic (fmt=Variable "fmt" is not available.
 > ) at /usr/src/sys/kern/kern_shutdown.c:563
 > #3  0xc0969a1c in quotaoff1 (td=0xc6786880, mp=Variable "mp" is not
 > available.
 > ) at /usr/src/sys/ufs/ufs/ufs_quota.c:1443
 > #4  0xc0969d08 in quotaoff_inchange (td=Variable "td" is not available.
 > ) at /usr/src/sys/ufs/ufs/ufs_quota.c:721
 > #5  0xc0969f81 in quotaoff (td=0xc6786880, mp=0xc551c7d4, type=0) at
 > /usr/src/sys/ufs/ufs/ufs_quota.c:763
 > #6  0xc095afb1 in ffs_flushfiles (mp=0xc551c7d4, flags=2, td=0xc6786880)
 > at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1096
 > #7  0xc09506f7 in softdep_flushfiles (oldmnt=0xc551c7d4, flags=2,
 > td=0xc6786880) at /usr/src/sys/ufs/ffs/ffs_softdep.c:1095
 > #8  0xc095b10d in ffs_unmount (mp=0xc551c7d4, mntflags=524288,
 > td=0xc6786880) at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1031
 > #9  0xc07c72df in dounmount (mp=0xc551c7d4, flags=524288, td=0xc6786880)
 > at /usr/src/sys/kern/vfs_mount.c:1286
 > #10 0xc07cd4c3 in vfs_unmountall () at /usr/src/sys/kern/vfs_subr.c:2936
 > #11 0xc0754294 in boot (howto=0) at /usr/src/sys/kern/kern_shutdown.c:391
 > #12 0xc07546b7 in reboot (td=0xc6786880, uap=0xe8e34cfc) at
 > /usr/src/sys/kern/kern_shutdown.c:169
 > #13 0xc0a568e5 in syscall (frame=0xe8e34d38) at
 > /usr/src/sys/i386/i386/trap.c:1035
 > #14 0xc0a3d8f0 in Xint0x80_syscall () at
 > /usr/src/sys/i386/i386/exception.s:196
 > #15 0x00000033 in ?? ()
 > Previous frame inner to this frame (corrupt stack?)
 > (kgdb)

 This one is different. It's caused by quotas. Can you reproduce this?

 Volker