Frequent Page Faults

Sun Mar 16 01:00:09 UTC 2014

I have a system running:

FreeBSD zoon.lafn.org 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #3: Thu Aug 19 20:09:11 PDT 2010

This morning it started crashing frequently.  The system had no issues prior to today.  Sometimes it auto reboots, others it just hangs.  The console messages are always very similar.  I do have core dumps for 3 of them.  At first it appeared that the problem was being caused by an attack on port 110.  While the attack was in process, the system would stay up for only a few minutes.  After discovering and blocking the attack, the system remained up a couple hours but crashed while I was watching it.  I am beginning to suspect a HW issue that was worsened by the load of the attack, but not caused directly by the attack.

I have been in the process of building a new set of disks for this system using 9.2, but thats not complete yet.  In addition it will take a couple days to get the disks on site.  Its a remote facility.  I found several references to issues with this problem and 7.2 that have apparently been fixed.  However, I have been running 7.2 since it first came out on this system without any similar issues.  Actually, I don't recall any issues at all with 7.2.  The source for this system no longer exists.  My development system has been upgraded to 9.2.  Its a modified kernel with some of the older processors commented out and includes QUOTA and ALTQ.  I don't recall any other changes.

I can hurry up the setup of the 9.2 system, but it would be at least until Wed before it could be installed at tried.  If that would correct the problem that would be great.  However, I have a concern that there is also a HW issue here and am not sure how to identify such.  My review of the dumps shows that acpi is always involved.  Don't know for sure what that implies though.  I don't believe the os has degraded and it has not been touched since 2010.  That pretty much leaves me with a HW issue.  Any ideas where the problem is will be appreciated.

Here are a couple of the dumps.  The first one was while the attack was in progress.  The second after it was terminated.

--------------------------------------------------------------

zoon# kgdb /boot/kernel/kernel vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
3)
trap number		= 12
panic: page fault
cpuid = 3
Uptime: 3m48s
Physical memory: 1993 MB
Dumping 187 MB: 172 156 140 124

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address	= 0x4
fault code		= supervisor write, page not present
instruction pointer	= 0x20:0xc0c72c00
stack pointer	        = 0x28:0xe566faf4
frame pointer	        = 0x28:0xe566fb14
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 4 (g_down)
trap number		= 12
 108 92 76 60 44 28 12

Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot/kernel/fdescfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fdescfs.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /boot/kernel/pflog.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
#0  doadump () at pcpu.h:196
196	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) where
#0  doadump () at pcpu.h:196
#1  0xc07a5e27 in boot (howto=260) at /usr2/src/sys/kern/kern_shutdown.c:418
#2  0xc07a60f9 in panic (fmt=Variable "fmt" is not available.
) at /usr2/src/sys/kern/kern_shutdown.c:574
#3  0xc0aa792c in trap_fatal (frame=0xc4ff8c48, eva=1361334589)
    at /usr2/src/sys/i386/i386/trap.c:939
#4  0xc0aa7b90 in trap_pfault (frame=0xc4ff8c48, usermode=0, eva=1361334589)
    at /usr2/src/sys/i386/i386/trap.c:852
#5  0xc0aa8512 in trap (frame=0xc4ff8c48) at /usr2/src/sys/i386/i386/trap.c:530
#6  0xc0a8d62b in calltrap () at /usr2/src/sys/i386/i386/exception.s:159
#7  0xc0e21715 in acpi_cpu_c1 ()
    at /usr2/src/sys/modules/acpi/acpi/../../../i386/acpica/acpi_machdep.c:550
#8  0xc0e1a594 in acpi_cpu_idle ()
    at /usr2/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:943
#9  0xc0a97f78 in cpu_idle () at /usr2/src/sys/i386/i386/machdep.c:1183
#10 0xc07c7904 in sched_idletd (dummy=0x0)
    at /usr2/src/sys/kern/sched_ule.c:2681
#11 0xc07808d9 in fork_exit (callout=0xc07c7640 <sched_idletd>, arg=0x0, 
    frame=0xc4ff8d38) at /usr2/src/sys/kern/kern_fork.c:810
#12 0xc0a8d6a0 in fork_trampoline () at /usr2/src/sys/i386/i386/exception.s:264

--------------------------------------------------------------

zoon# kgdb /boot/kernel/kernel vmcore.2
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault:
eip = 0xc0e21715
esp = 0xc4ff8d80
ebp = 0xc4ff8c88
cpuid = 3; apic id = 03
panic: double fault
cpuid = 3
Uptime: 2h19m49s
Physical memory: 1993 MB
Dumping 187 MB: 172 156 140 124 108 92 76 60 44 28 12

Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot/kernel/fdescfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fdescfs.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /boot/kernel/pflog.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
#0  doadump () at pcpu.h:196
196	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) up
#1  0xc07a5e27 in boot (howto=260) at /usr2/src/sys/kern/kern_shutdown.c:418
418	/usr2/src/sys/kern/kern_shutdown.c: No such file or directory.
	in /usr2/src/sys/kern/kern_shutdown.c
(kgdb) down
#0  doadump () at pcpu.h:196
196	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) where
#0  doadump () at pcpu.h:196
#1  0xc07a5e27 in boot (howto=260) at /usr2/src/sys/kern/kern_shutdown.c:418
#2  0xc07a60f9 in panic (fmt=Variable "fmt" is not available.
) at /usr2/src/sys/kern/kern_shutdown.c:574
#3  0xc0aa763b in dblfault_handler () at /usr2/src/sys/i386/i386/trap.c:972
#4  0xc0e21715 in acpi_cpu_c1 ()
    at /usr2/src/sys/modules/acpi/acpi/../../../i386/acpica/acpi_machdep.c:550
#5  0xc0e1a594 in acpi_cpu_idle ()
    at /usr2/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:943
#6  0xc0a97f78 in cpu_idle () at /usr2/src/sys/i386/i386/machdep.c:1183
#7  0xc07c7904 in sched_idletd (dummy=0x0)
    at /usr2/src/sys/kern/sched_ule.c:2681
#8  0xc07808d9 in fork_exit (callout=0xc07c7640 <sched_idletd>, arg=0x0, 
    frame=0xc4ff8d38) at /usr2/src/sys/kern/kern_fork.c:810
#9  0xc0a8d6a0 in fork_trampoline () at /usr2/src/sys/i386/i386/exception.s:264
(kgdb) 