kern/182557: Frequent reboots due to kernel trap happening in pf_test_rule

Tue Oct 1 19:50:00 UTC 2013

>Number:         182557
>Category:       kern
>Synopsis:       Frequent reboots due to kernel trap happening in pf_test_rule
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Oct 01 19:50:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     Berend de Boer
>Release:        9.2-RELEASE
>Organization:
Xplain Hosting
>Environment:
FreeBSD bmach.nederware.nl 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255962: Wed Oct  2 07:24:02 NZDT 2013     root at bmach.nederware.nl:/usr/obj/usr/src/9.2/sys/BMACH  amd64

>Description:
Oct  2 08:34:09 bmach syslogd: kernel boot file is /boot/kernel/kernel
Oct  2 08:34:09 bmach kernel: 
Oct  2 08:34:09 bmach kernel: 
Oct  2 08:34:09 bmach kernel: Fatal trap 12: page fault while in kernel mode
Oct  2 08:34:09 bmach kernel: cpuid = 1; apic id = 01
Oct  2 08:34:09 bmach kernel: fault virtual address	= 0x10
Oct  2 08:34:09 bmach kernel: fault code		= supervisor read data, page not present
Oct  2 08:34:09 bmach kernel: instruction pointer	= 0x20:0xffffffff81a1b8b8
Oct  2 08:34:09 bmach kernel: stack pointer	        = 0x28:0xffffff81281f2380
Oct  2 08:34:09 bmach kernel: frame pointer	        = 0x28:0xffffff81281f2390
Oct  2 08:34:09 bmach kernel: code segment		= base 0x0, limit 0xfffff, type 0x1b
Oct  2 08:34:09 bmach kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Oct  2 08:34:09 bmach kernel: processor eflags	= interrupt enabled, resume, IOPL = 0
Oct  2 08:34:09 bmach kernel: current process		= 1843 (named)
Oct  2 08:34:09 bmach kernel: trap number		= 12
Oct  2 08:34:09 bmach kernel: panic: page fault
Oct  2 08:34:09 bmach kernel: cpuid = 1
Oct  2 08:34:09 bmach kernel: KDB: stack backtrace:
Oct  2 08:34:09 bmach kernel: #0 0xffffffff80950086 at kdb_backtrace+0x66
Oct  2 08:34:09 bmach kernel: #1 0xffffffff809161fd at panic+0x1cd
Oct  2 08:34:09 bmach kernel: #2 0xffffffff80d00fb0 at trap_fatal+0x290
Oct  2 08:34:09 bmach kernel: #3 0xffffffff80d01311 at trap_pfault+0x211
Oct  2 08:34:09 bmach kernel: #4 0xffffffff80d018c4 at trap+0x344
Oct  2 08:34:09 bmach kernel: #5 0xffffffff80ceac53 at calltrap+0x8
Oct  2 08:34:09 bmach kernel: #6 0xffffffff81a25e7c at pf_test_rule+0xf3c
Oct  2 08:34:09 bmach kernel: #7 0xffffffff81a29634 at pf_test+0xe34
Oct  2 08:34:09 bmach kernel: #8 0xffffffff81a311c1 at pf_check_out+0x41
Oct  2 08:34:09 bmach kernel: #9 0xffffffff809e249e at pfil_run_hooks+0x9e
Oct  2 08:34:09 bmach kernel: #10 0xffffffff80a46373 at ip_output+0x3a3
Oct  2 08:34:09 bmach kernel: #11 0xffffffff80ac3588 at udp_send+0x508
Oct  2 08:34:09 bmach kernel: #12 0xffffffff80985c4f at sosend_dgram+0x2cf
Oct  2 08:34:09 bmach kernel: #13 0xffffffff80989e93 at kern_sendit+0x1a3
Oct  2 08:34:09 bmach kernel: #14 0xffffffff8098a14c at sendit+0xdc
Oct  2 08:34:09 bmach kernel: #15 0xffffffff8098a1d7 at sys_sendmsg+0x87
Oct  2 08:34:09 bmach kernel: #16 0xffffffff80d0075a at amd64_syscall+0x5ea
Oct  2 08:34:09 bmach kernel: #17 0xffffffff80ceaf37 at Xfast_syscall+0xf7
Oct  2 08:34:09 bmach kernel: Uptime: 46m10s

>How-To-Repeat:
Just wait...

Custom compiled kernel, GENERIC + the following options:

options         MROUTING                # Multicast routing

# ALTQ
options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_CDNR
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)

Options don't matter btw, reboot happens also with generic.

Have simplified my pf.conf to this:

set limit { frags 30000, states 30000 }
set optimization conservative

scrub in all

nat on egress proto udp from { lan:network, $vpn_if:network, 192.168.233.21, 192.168.233.25 } port $voip_ports to any -> (egress) static-port
nat on egress from any to any -> (egress) round-robin sticky-address

block log
pass on lan
pass on public_wifi
pass on $vpn_if
set skip on lo0
pass out on egress

But the kind of firewall doesn't seem to be matter.

I also described the problem here with experiments on FreeBSD-9.1: http://www.freebsd.org/cgi/query-pr.cgi?pr=182141

Trap happens on i386 or amd64, so doesn't look like a hardware problem (and hardware has been fine with FreeBSD 8-STABLE).
>Fix:
Have tried everything. Never had an issue like this in the 18 years I've been using FreeBSD.

>Release-Note:
>Audit-Trail:
>Unformatted: