repeteable panics on a 7.2-RELEASE box

[Fernando Gleiser]

Hi. we're using a 7.2 box as a firewall/nat/ipsec machine with ipfilter/ipnat and ipsec-tools

This box is crashing a lot, and all the panics are identical:

root at firewall /var/crash]# uname -a
FreeBSD firewall. 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #3: Mon Dec 28
16:45:47 ART 2009     root at firewall.:/usr/obj/usr/src/sys/DTRACE  i386

[root at firewall /var/crash]# kgdb /boot/kernel/kernel.symbols /var/crash/vmcore.16
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x4
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc33ac8ab
stack pointer           = 0x28:0xc2f909bc
frame pointer           = 0x28:0xc2f90a38
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 21 (irq17: xl1)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 11d22h28m55s
Physical memory: 499 MB
Dumping 115 MB: 100 84 68 52 36 20 4

Reading symbols from /boot/kernel/ipl.ko...Reading symbols from /boot/kernel/ipl.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipl.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/ng_iface.ko...Reading symbols from /boot/kernel/ng_iface.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_iface.ko
Reading symbols from /boot/kernel/ng_ppp.ko...Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ppp.ko
Reading symbols from /boot/kernel/ng_bpf.ko...Reading symbols from /boot/kernel/ng_bpf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_bpf.ko
Reading symbols from /boot/kernel/ng_vjc.ko...Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_vjc.ko
Reading symbols from /boot/kernel/logo_saver.ko...Reading symbols from /boot/kernel/logo_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/logo_saver.ko
Reading symbols from /boot/kernel/ng_pptpgre.ko...Reading symbols from /boot/kernel/ng_pptpgre.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_pptpgre.ko
Reading symbols from /boot/kernel/ng_ksocket.ko...Reading symbols from /boot/kernel/ng_ksocket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ksocket.ko
Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_mppc.ko
Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /boot/kernel/rc4.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/rc4.ko
#0  doadump () at pcpu.h:196
196     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc07ec1f7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc07ec4c9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc0b18f2c in trap_fatal (frame=0xc2f9097c, eva=4) at /usr/src/sys/i386/i386/trap.c:939
#4  0xc0b191b0 in trap_pfault (frame=0xc2f9097c, usermode=0, eva=4) at /usr/src/sys/i386/i386/trap.c:852
#5  0xc0b19c2c in trap (frame=0xc2f9097c) at /usr/src/sys/i386/i386/trap.c:530
#6  0xc0afe20b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
#7  0xc33ac8ab in nat_new () from /boot/kernel/ipl.ko
#8  0xc33b0574 in fr_checknatin () from /boot/kernel/ipl.ko
#9  0xc33c9723 in fr_check () from /boot/kernel/ipl.ko
#10 0xc33c170e in fr_check_wrapper () from /boot/kernel/ipl.ko
#11 0xc0897418 in pfil_run_hooks (ph=0xc0d03100, mp=0xc2f90be8, ifp=0xc3191400, dir=1, inp=0x0)
    at /usr/src/sys/net/pfil.c:78
#12 0xc08d76e2 in ip_input (m=0xc34ef200) at /usr/src/sys/netinet/ip_input.c:416
#13 0xc0895bb5 in netisr_dispatch (num=2, m=0xc34ef200) at /usr/src/sys/net/netisr.c:185
#14 0xc088bb51 in ether_demux (ifp=0xc3191400, m=0xc34ef200) at /usr/src/sys/net/if_ethersubr.c:834
#15 0xc088bf43 in ether_input (ifp=0xc3191400, m=0xc34ef200) at /usr/src/sys/net/if_ethersubr.c:692
#16 0xc09ec818 in xl_rxeof (sc=0xc3199000) at /usr/src/sys/pci/if_xl.c:2022
#17 0xc09eed24 in xl_intr (arg=0xc3199000) at /usr/src/sys/pci/if_xl.c:2257
#18 0xc07c9f5b in ithread_loop (arg=0xc3190300) at /usr/src/sys/kern/kern_intr.c:1088
#19 0xc07c6a59 in fork_exit (callout=0xc07c9da0 <ithread_loop>, arg=0xc3190300, frame=0xc2f90d38)
    at /usr/src/sys/kern/kern_fork.c:810
#20 0xc0afe2b0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:264

the panic always happens inside nat_new. 


Most of the "fatal trap 12"s I saw were hardware-related, but they
happened in random places, This one always happens in the same place,
the stack traces are identical in all the dumps.


any pointers on what can be the cause of the panic, would be greatly apreciated.


thanks in advance, Fer