Crashes (Fatal trap 12: page fault while in kernel mode)

Tue Feb 24 12:59:36 PST 2004

I have a 4.9-RELEASE-p1 system that crashes a few times per week. The 
invalid page fault seems to be triggered by nfs, as shown by the 
backtrace, though with my (very) limited kernel debugging experience 
I've been unable to diagnose further. Perhaps someone has had similar 
problems and knows how to fix it, or can at least give me some ideas to 
try? Any help would be very well appreciated.

Thanks,
Dave.

Below is the output from gdb:

SMP 4 cpus
IdlePTD at phsyical address 0x00572000
initial pcb at physical address 0x0049bfe0
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
mp_lock = 00000002; cpuid = 0; lapic.id = 00000000
fault virtual address   = 0x21
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc0253050
stack pointer           = 0x10:0xf1e9bd34
frame pointer           = 0x10:0xf1e9bd70
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 4274 (imapd)
interrupt mask          = bio  <- SMP: XXX
trap number             = 12
panic: page fault
mp_lock = 00000002; cpuid = 0; lapic.id = 00000000
boot() called on cpu#0

syncing disks... 2118 2101 2101 2100 2100 2100 2100 2100 2100 2100 2109 
2098 2098 2098 2098 2098 2098 2098 2101 2098 2098 2098 2098 2098 2098 
2098 2098 2098 2098 2098 2098 2098 2098 2098 2098 2098 2098 2098 2098
done
Uptime: 1d0h8m27s

dumping to dev #aacd/0x20001, offset 4194432
dump 2047 2046 2045 2044 2043 2042 2041

------------snip!-------------

 4 3 2 1 0 succeeded
aac0: shutting down controller...
---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487             if (dumping++) {
(kgdb) l *0xc0253050
0xc0253050 is in brelse (/usr/src/sys/kern/vfs_bio.c:993).
988
989             KASSERT(!(bp->b_flags & (B_CLUSTER|B_PAGING)), ("brelse: 
inappropriate B_PAGING or B_CLUSTER bp %p", bp));
990
991             s = splbio();
992
993             if (bp->b_flags & B_LOCKED)
994                     bp->b_flags &= ~B_ERROR;
995
996             if ((bp->b_flags & (B_READ | B_ERROR | B_INVAL)) == 
B_ERROR) {
997                     /*
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc022c71b in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc022cb74 in poweroff_wait (junk=0xc042e4d9, howto=-1069359217) at 
/usr/src/sys/kern/kern_shutdown.c:595
#3  0xc03abda4 in trap_fatal (frame=0xf1e9bcf4, eva=33) at 
/usr/src/sys/i386/i386/trap.c:974
#4  0xc03aba35 in trap_pfault (frame=0xf1e9bcf4, usermode=0, eva=33) at 
/usr/src/sys/i386/i386/trap.c:867
#5  0xc03ab5a7 in trap (frame={tf_fs = 24, tf_es = 16, tf_ds = 16, 
tf_edi = 0, tf_esi = 0, tf_ebp = -236339856, tf_isp = -236339936, tf_ebx 
= 0,
      tf_edx = 1745141856, tf_ecx = 32, tf_eax = 0, tf_trapno = 12, 
tf_err = 0, tf_eip = -1071304624, tf_cs = 8, tf_eflags = 66054, tf_esp = 0,
      tf_ss = -227323200}) at /usr/src/sys/i386/i386/trap.c:466
#6  0xc0253050 in brelse (bp=0x0) at /usr/src/sys/kern/vfs_bio.c:991
#7  0xc02b1c29 in nfs_bioread (vp=0xf1a32dc0, uio=0xf1e9be78, ioflag=0, 
cred=0xc50b4800) at /usr/src/sys/nfs/nfs_bio.c:710
#8  0xc02da315 in nfs_readlink (ap=0xf1e9be68) at 
/usr/src/sys/nfs/nfs_vnops.c:1037
#9  0xc025931e in namei (ndp=0xf1e9bedc) at vnode_if.h:794
#10 0xc025e149 in change_dir (ndp=0xf1e9bedc, p=0xf1313ee0) at 
/usr/src/sys/kern/vfs_syscalls.c:969
#11 0xc025dffc in chdir (p=0xf1313ee0, uap=0xf1e9bf80) at 
/usr/src/sys/kern/vfs_syscalls.c:870
#12 0xc03ac0d5 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, 
tf_edi = 1, tf_esi = 674056736, tf_ebp = -1077938508, tf_isp = -236339244,
      tf_ebx = 0, tf_edx = 134886594, tf_ecx = 135323464, tf_eax = 12, 
tf_trapno = 12, tf_err = 2, tf_eip = 673714528, tf_cs = 31, tf_eflags = 518,
      tf_esp = -1077938552, tf_ss = 47}) at 
/usr/src/sys/i386/i386/trap.c:1175
#13 0xc03964eb in Xint0x80_syscall ()
#14 0x8056737 in ?? ()
#15 0x8055b8a in ?? ()
#16 0x8068693 in ?? ()
#17 0x804b2ef in ?? ()
#18 0x804a89a in ?? ()

-----------------------------------------------

 From dmesg:

Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.9-RELEASE-p1 #0: Mon Feb  2 16:34:57 EST 2004
Timecounter "i8254"  frequency 1193182 Hz
CPU: Intel(R) Xeon(TM) CPU 1.80GHz (1789.10-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping = 7

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Hyperthreading: 2 logical CPUs
real memory  = 2147418112 (2097088K bytes)
avail memory = 2086187008 (2037292K bytes)
Changing APIC ID for IO APIC #0 from 0 to 4 on chip
Changing APIC ID for IO APIC #1 from 0 to 5 on chip
Changing APIC ID for IO APIC #2 from 0 to 6 on chip
Programming 16 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
Programming 16 pins in IOAPIC #1
Programming 16 pins in IOAPIC #2
FreeBSD/SMP: Multiprocessor motherboard: 4 CPUs
 cpu0 (BSP): apic id:  0, version: 0x00050014, at 0xfee00000
 cpu1 (AP):  apic id:  1, version: 0x00050014, at 0xfee00000
 cpu2 (AP):  apic id:  2, version: 0x00050014, at 0xfee00000
 cpu3 (AP):  apic id:  3, version: 0x00050014, at 0xfee00000
 io0 (APIC): apic id:  4, version: 0x000f0011, at 0xfec00000
 io1 (APIC): apic id:  5, version: 0x000f0011, at 0xfec01000
 io2 (APIC): apic id:  6, version: 0x000f0011, at 0xfec02000
Preloaded elf kernel "kernel" at 0xc0553000.
Warning: Pentium 4 CPU: PSE disabled
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 9 entries at 0xc00fc490
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
IOAPIC #1 intpin 3 -> irq 2
IOAPIC #1 intpin 7 -> irq 7
IOAPIC #1 intpin 11 -> irq 10
pci0: <PCI bus> on pcib0
pci0: <unknown card> (vendor=0x1028, dev=0x000c) at 4.0 irq 2
pci0: <unknown card> (vendor=0x1028, dev=0x0008) at 4.1 irq 7
pci0: <unknown card> (vendor=0x1028, dev=0x000d) at 4.2 irq 10
pci0: <ATI Mach64-GR graphics accelerator> at 14.0
atapci0: <ServerWorks CSB5 ATA100 controller> port 
0x8b0-0x8bf,0x8d8-0x8db,0x8d0-0x8d7,0x8c8-0x8cb,0x8c0-0x8c7 at device 
15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
ohci0: <OHCI (generic) USB controller> mem 0xfe100000-0xfe100fff irq 5 
at device 15.2 on pci0
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
isab0: <PCI to ISA bridge (vendor=1166 device=0225)> at device 15.3 on pci0
isa0: <ISA bus> on isab0
pcib1: <Host to PCI bridge> on motherboard
pci1: <PCI bus> on pcib1
pcib8: <PCI to PCI bridge (vendor=8086 device=b152)> at device 8.0 on pci1
IOAPIC #1 intpin 4 -> irq 11
IOAPIC #1 intpin 5 -> irq 13
pci2: <PCI bus> on pcib8
fxp0: <Intel 82558 Pro/100 Ethernet> port 0xdce0-0xdcff mem 
0xfce00000-0xfcefffff,0xfcbff000-0xfcbfffff irq 11 at device 4.0 on pci2
fxp0: Ethernet address 00:02:b3:be:30:33
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: <Intel 82558 Pro/100 Ethernet> port 0xdcc0-0xdcdf mem 
0xfcd00000-0xfcdfffff,0xfcbfe000-0xfcbfefff irq 13 at device 5.0 on pci2
fxp1: Ethernet address 00:02:b3:be:30:34
inphy1: <i82555 10/100 media interface> on miibus1
inphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pcib2: <Host to PCI bridge> on motherboard
pci3: <PCI bus> on pcib2
pcib3: <Host to PCI bridge> on motherboard
IOAPIC #1 intpin 12 -> irq 16
IOAPIC #1 intpin 13 -> irq 17
pci4: <PCI bus> on pcib3
bge0: <Broadcom BCM5701 Gigabit Ethernet, ASIC rev. 0x105> mem 
0xfca10000-0xfca1ffff irq 16 at device 6.0 on pci4
bge0: Ethernet address: 00:06:5b:8f:f2:f7
miibus2: <MII bus> on bge0
brgphy0: <BCM5701 10/100/1000baseTX PHY> on miibus2
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 
1000baseTX-FDX, auto
bge1: <Broadcom BCM5701 Gigabit Ethernet, ASIC rev. 0x105> mem 
0xfca00000-0xfca0ffff irq 17 at device 8.0 on pci4
bge1: Ethernet address: 00:06:5b:8f:f2:f8
miibus3: <MII bus> on bge1
brgphy1: <BCM5701 10/100/1000baseTX PHY> on miibus3
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 
1000baseTX-FDX, auto
pcib4: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
IOAPIC #1 intpin 14 -> irq 18
pci5: <PCI bus> on pcib4
pcib9: <PCI to PCI bridge (vendor=8086 device=0309)> at device 8.0 on pci5
IOAPIC #1 intpin 15 -> irq 19
pci6: <PCI bus> on pcib9
pci6: <unknown card> (vendor=0x9005, dev=0x00c5) at 6.0 irq 18
pci6: <unknown card> (vendor=0x9005, dev=0x00c5) at 6.1 irq 19
aac0: <Dell PERC 3/Di> mem 0xf0000000-0xf7ffffff irq 18 at device 8.1 on 
pci5
aac0: i960RX 100MHz, 118MB cache memory, optional battery present
aac0: Kernel 2.7-1, Build 3170, S/N  441d3
aac0: Supported 
Options=75c<WCACHE,DATA64,HOSTTIME,WINDOW4GB,SOFTERR,NORECOND,SGMAP64>
pcib5: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
pci7: <PCI bus> on pcib5
pcib6: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
pci8: <PCI bus> on pcib6
pcib7: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
pci9: <PCI bus> on pcib7
orm0: <Option ROMs> at iomem 
0xc0000-0xc7fff,0xc8000-0xcbfff,0xec000-0xeffff on isa0
pmtimer0 on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model IntelliMouse, device ID 3
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: parallel port not found.
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 
intpin 2
APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0
IP packet filtering initialized, divert disabled, rule-based forwarding 
enabled, default to accept, logging limited to 100 packets/entry by default
IP Filter: v3.4.31 initialized.  Default = pass all, Logging = enabled
ata0-slave: ATAPI identify retries exceeded
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
acd0: CDROM <TEAC CD-ROM CD-224E> at ata0-master PIO4
aacd0: <RAID 5> on aac0
aacd0: 279994MB (573428736 sectors)
Mounting root from ufs:/dev/aacd0s1a
WARNING: / was not properly dismounted