kern/134519: Kernel painc (page fault in kernel)

Wed May 13 19:50:02 UTC 2009

>Number:         134519
>Category:       kern
>Synopsis:       Kernel painc (page fault in kernel)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed May 13 19:50:01 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     Alexey Kovalenko
>Release:        7.2
>Organization:
>Environment:
FreeBSD gate.kovalhome.local 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Tue May 12 19:36:54 YEKST 2009     root at gate.kovalhome.local:/usr/obj/usr/src/sys/gate-koval-home  i386

>Description:
System purpose is home router, proxy and storage.
Before updating to 7.2, uptime was about 80 days, so I sure - hardware is ok.

After updating 7.1 to 7.2, I've moved pppoe from dsl modem to pc and maked some experiments with ipfw, ipfw nat and port forwarding. IPFW config was unoptimal and sometimes very buggy :(

I've got two kernel panics during 12 hours. 
One I've missed, one I saw myself. On 2nd case, network load was very low.

There are two core dumps.

Here is some kgdb info:

(kgdb) core-file vmcore.1

Unread portion of the kernel message buffer:
<110>ipfw: 250 SkipTo 30000 UDP 192.168.22.1:138 192.168.22.255:138 in via lan
<110>ipfw: 30000 Count UDP 192.168.22.1:138 192.168.22.255:138 in via lan
<110>ipfw: 30250 Nat UDP 192.168.22.1:138 192.168.22.255:138 in via lan

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x7d78
fault code              = supervisor write, page not present
instruction pointer     = 0x20:0xc0634086
stack pointer           = 0x28:0xe110f760
frame pointer           = 0x28:0xe110f77c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 34 (irq22: vr0)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 8h48m12s
Physical memory: 502 MB
Dumping 260 MB: (CTRL-C to abort)  (CTRL-C to abort)  (CTRL-C to abort)  (CTRL-C to abort)  (CTRL-C to abort)  (CTRL-C to abort)  245 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5

<symbols loading skipped>

#0  doadump () at pcpu.h:196
196     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc054ed3f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc054f004 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc06fa060 in trap_fatal (frame=0xe110f720, eva=32120) at /usr/src/sys/i386/i386/trap.c:939
#4  0xc06fa2b0 in trap_pfault (frame=0xe110f720, usermode=0, eva=32120)
    at /usr/src/sys/i386/i386/trap.c:852
#5  0xc06fabfa in trap (frame=0xe110f720) at /usr/src/sys/i386/i386/trap.c:530
#6  0xc06e0c2b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
#7  0xc0634086 in LibAliasIn (la=0x0, ptr=0xc5481816 "E", maxpacketsize=2026) at atomic.h:149
#8  0xc0619587 in ipfw_nat (args=0xe110fa8c, t=0xc37358c0, m=0xc5439b00)
    at /usr/src/sys/netinet/ip_fw_nat.c:320
#9  0xc0617bb1 in ipfw_chk (args=0xe110fa8c) at /usr/src/sys/netinet/ip_fw2.c:3331
#10 0xc061831c in ipfw_check_in (arg=0x0, m0=0xe110fb90, ifp=0xc2d6f000, dir=1, inp=0x0)
    at /usr/src/sys/netinet/ip_fw_pfil.c:125
#11 0xc05f308a in pfil_run_hooks (ph=0xc07c9020, mp=0xe110fbec, ifp=0xc2d6f000, dir=1, inp=0x0)
    at /usr/src/sys/net/pfil.c:78
#12 0xc061bf04 in ip_input (m=0xc5439b00) at /usr/src/sys/netinet/ip_input.c:416
#13 0xc05f1909 in netisr_dispatch (num=2, m=0xc5439b00) at /usr/src/sys/net/netisr.c:185
#14 0xc05e9789 in ether_demux (ifp=0xc2d6f000, m=0xc5439b00)
    at /usr/src/sys/net/if_ethersubr.c:834
#15 0xc05e9b5d in ether_input (ifp=0xc2d6f000, m=0xc5439b00)
    at /usr/src/sys/net/if_ethersubr.c:692
#16 0xc0936d69 in vr_rxeof (sc=0xc2d83000) at /usr/src/sys/modules/vr/../../dev/vr/if_vr.c:1415
#17 0xc093a13c in vr_intr (arg=0xc2d83000) at /usr/src/sys/modules/vr/../../dev/vr/if_vr.c:1702
#18 0xc052d7d8 in ithread_loop (arg=0xc2d9a290) at /usr/src/sys/kern/kern_intr.c:1088
#19 0xc052a441 in fork_exit (callout=0xc052d624 <ithread_loop>, arg=0xc2d9a290, frame=0xe110fd38)
    at /usr/src/sys/kern/kern_fork.c:810
#20 0xc06e0ca0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:264
(kgdb)

and other dump:

(kgdb) core-file vmcore.0

Unread portion of the kernel message buffer:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x33e9d47
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc5e38dc8
stack pointer           = 0x28:0xe1127aac
frame pointer           = 0x28:0xe1127ad0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 35 (irq21: rl0)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 16h27m24s
Physical memory: 502 MB
Dumping 314 MB: 299 283 267 251 235 219 203 187 171 155 139 123 107 91 75 59 43 27 11

<symbols loading skipped>

#0  doadump () at pcpu.h:196
196     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc054ed3f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc054f004 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc06fa060 in trap_fatal (frame=0xe1127a6c, eva=54435143) at /usr/src/sys/i386/i386/trap.c:939
#4  0xc06fa2b0 in trap_pfault (frame=0xe1127a6c, usermode=0, eva=54435143)
    at /usr/src/sys/i386/i386/trap.c:852
#5  0xc06fabfa in trap (frame=0xe1127a6c) at /usr/src/sys/i386/i386/trap.c:530
#6  0xc06e0c2b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
#7  0xc5e38dc8 in pppoe_findsession (privp=Variable "privp" is not available.
)
    at /usr/src/sys/modules/netgraph/pppoe/../../../netgraph/ng_pppoe.c:566
#8  0xc5e3950a in ng_pppoe_rcvdata_ether (hook=0xc57b7200, item=0xc5ebb2d0)
    at /usr/src/sys/modules/netgraph/pppoe/../../../netgraph/ng_pppoe.c:1612
#9  0xc05fe0ce in ng_apply_item (node=0xc57b7100, item=0xc5ebb2d0, rw=0)
    at /usr/src/sys/netgraph/ng_base.c:2336
#10 0xc05fd2f0 in ng_snd_item (item=0xc5ebb2d0, flags=Variable "flags" is not available.
) at /usr/src/sys/netgraph/ng_base.c:2254
#11 0xc05e9825 in ether_demux (ifp=0xc2d95000, m=0xc2ea2b00)
    at /usr/src/sys/net/if_ethersubr.c:851
#12 0xc05e9b5d in ether_input (ifp=0xc2d95000, m=0xc2ea2b00)
    at /usr/src/sys/net/if_ethersubr.c:692
#13 0xc0644e57 in rl_rxeof (sc=0xc2da6000) at /usr/src/sys/pci/if_rl.c:1320
#14 0xc06460cc in rl_intr (arg=0xc2da6000) at /usr/src/sys/pci/if_rl.c:1477
#15 0xc052d7d8 in ithread_loop (arg=0xc2da8310) at /usr/src/sys/kern/kern_intr.c:1088
#16 0xc052a441 in fork_exit (callout=0xc052d624 <ithread_loop>, arg=0xc2da8310, frame=0xe1127d38)
    at /usr/src/sys/kern/kern_fork.c:810
#17 0xc06e0ca0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:264
(kgdb)

>How-To-Repeat:

>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: