6.3-RELEASE Page Fault in ipfilter on heavy load

[Eric Bellotti]

Hi,
 
6.3 RELEASE SMP.  We have a firewall with ipfilter and 418 in rules, 16
out rules, and roughly 700 nat rules.
We had previously installed 7.0 RELEASE, which also page faulted about 5
min after being plugged in.
The box has is a Dell PE1950 with two on-board BCM75008, and a quad
Intel PRO/1000 PCIX card.

What do you experts make of this?  What else can I provide to assist in
debugging?

(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc06a7872 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:409
#2  0xc06a7b99 in panic (fmt=0xc0975003 "%s") at
/usr/src/sys/kern/kern_shutdown.c:565
#3  0xc0915e9c in trap_fatal (frame=0xe38d0a38, eva=4) at
/usr/src/sys/i386/i386/trap.c:838
#4  0xc0915bdb in trap_pfault (frame=0xe38d0a38, usermode=0, eva=4) at
/usr/src/sys/i386/i386/trap.c:745
#5  0xc0915815 in trap (frame=
      {tf_fs = 8, tf_es = 40, tf_ds = -477298648, tf_edi = 167772191,
tf_esi = 0, tf_ebp = -477295884, tf_isp = -477296028, tf_ebx =
-977793024, tf_edx = 6, tf_ecx = -965245440, tf_eax = 0, tf_trapno = 12,
tf_err = 0, tf_eip = -986197785, tf_cs = 32, tf_eflags = 66118, tf_esp =
1, tf_ss = -477295788}) at /usr/src/sys/i386/i386/trap.c:435
#6  0xc09006ca in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc537d0e7 in nat_new (fin=0xe38d0b68, np=0xc5b81000, natsave=0x0,
flags=0, direction=0) at endian.h:144
#8  0xc537d7de in fr_checknatin (fin=0xe38d0b68, passp=0xe38d0b64) at
/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c:41
40
#9  0xc5392155 in fr_check (ip=0xc539e4cc, hlen=-477295772, ifp=0x0,
out=0, mp=0xe38d0c50) at
/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/fil.c:2572
#10 0xc538d985 in fr_check_wrapper (arg=0x0, mp=0x6, ifp=0xc500e800,
dir=1) at
/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_fil_free
bsd.c:178
#11 0xc072455f in pfil_run_hooks (ph=0xc0a80ca0, mp=0xe38d0ca8,
ifp=0xc500e800, dir=1, inp=0x0) at /usr/src/sys/net/pfil.c:139
#12 0xc0749cf5 in ip_input (m=0xc7912c00) at
/usr/src/sys/netinet/ip_input.c:468
#13 0xc07230d3 in netisr_processqueue (ni=0xc0a80278) at
/usr/src/sys/net/netisr.c:236
#14 0xc07232d2 in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349
#15 0xc0690ff5 in ithread_execute_handlers (p=0xc4ede430, ie=0xc4f2e180)
at /usr/src/sys/kern/kern_intr.c:682
#16 0xc0691115 in ithread_loop (arg=0xc4ebd8b0) at
/usr/src/sys/kern/kern_intr.c:766
#17 0xc068fda9 in fork_exit (callout=0xc06910c0 <ithread_loop>,
arg=0xc4ebd8b0, frame=0xe38d0d38) at /usr/src/sys/kern/kern_fork.c:788
#18 0xc090072c in fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:208
(kgdb)

# sysctl  -a | grep ipf
net.inet.ipf.fr_minttl: 4
net.inet.ipf.fr_chksrc: 0
net.inet.ipf.fr_defaultauthage: 600
net.inet.ipf.fr_authused: 0
net.inet.ipf.fr_authsize: 32
net.inet.ipf.ipf_hostmap_sz: 2047
net.inet.ipf.ipf_rdrrules_sz: 1009
net.inet.ipf.ipf_natrules_sz: 1009
net.inet.ipf.ipf_nattable_sz: 16889
net.inet.ipf.fr_statemax: 7079
net.inet.ipf.fr_statesize: 10163
net.inet.ipf.fr_running: 1
net.inet.ipf.fr_ipfrttl: 120
net.inet.ipf.fr_defnatage: 1200
net.inet.ipf.fr_icmptimeout: 120
net.inet.ipf.fr_udpacktimeout: 24
net.inet.ipf.fr_udptimeout: 240
net.inet.ipf.fr_tcpclosed: 60
net.inet.ipf.fr_tcptimeout: 480
net.inet.ipf.fr_tcplastack: 60
net.inet.ipf.fr_tcpclosewait: 480
net.inet.ipf.fr_tcphalfclosed: 7200
net.inet.ipf.fr_tcpidletimeout: 172800
net.inet.ipf.fr_active: 0
net.inet.ipf.fr_pass: 134217730
net.inet.ipf.fr_flags: 0
net.link.ether.ipfw: 0

BR

Eric