kern/152113: page fault on 8.1-RELEASE caused by certain amount of
traffic through router
Dmitry Selivanov
sd at mostnet.ru
Wed Nov 10 17:10:10 UTC 2010
>Number: 152113
>Category: kern
>Synopsis: page fault on 8.1-RELEASE caused by certain amount of traffic through router
>Confidential: no
>Severity: serious
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Nov 10 17:10:08 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator: Dmitry Selivanov
>Release: 8.1-RELEASE i386
>Organization:
>Environment:
FreeBSD r-1 8.1-RELEASE-p1 FreeBSD 8.1-RELEASE-p1 #0: Fri Oct 8 17:11:39 MSD 2010 root at r-1:/usr/src/sys/i386/compile/DI i386
>Description:
Router is the first gateway and shaper (ipfw+dummynet) for clients. It has many VLAN interfaces with many IP addresses (gateways for clients). vlan0 interface is connected to the next router (default gateway). The average load is 80-100 Mbit/s. On average every 5 days it hangs. Sometimes it restarts automaticly (there are core dumps after it), sometimes reset only helps (and no core dumps). The more load the more often it hangs.
Added kernel options to GENERIC:
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFIREWALL_FORWARD
options IPDIVERT
options DUMMYNET
$pciconf -l
hostb0 at pci0:0:0:0: class=0x060000 card=0x34d08086 chip=0x29f08086 rev=0x00 hdr=0x00
pcib1 at pci0:0:6:0: class=0x060400 card=0x00008086 chip=0x29f98086 rev=0x00 hdr=0x01
em0 at pci0:0:25:0: class=0x020000 card=0x34d08086 chip=0x10bd8086 rev=0x02 hdr=0x00
uhci0 at pci0:0:26:0: class=0x0c0300 card=0x34d08086 chip=0x29378086 rev=0x02 hdr=0x00
uhci1 at pci0:0:26:1: class=0x0c0300 card=0x34d08086 chip=0x29388086 rev=0x02 hdr=0x00
ehci0 at pci0:0:26:7: class=0x0c0320 card=0x34d08086 chip=0x293c8086 rev=0x02 hdr=0x00
pcib2 at pci0:0:28:0: class=0x060400 card=0x29408086 chip=0x29408086 rev=0x02 hdr=0x01
pcib3 at pci0:0:28:4: class=0x060400 card=0x29488086 chip=0x29488086 rev=0x02 hdr=0x01
uhci2 at pci0:0:29:0: class=0x0c0300 card=0x34d08086 chip=0x29348086 rev=0x02 hdr=0x00
uhci3 at pci0:0:29:1: class=0x0c0300 card=0x34d08086 chip=0x29358086 rev=0x02 hdr=0x00
uhci4 at pci0:0:29:2: class=0x0c0300 card=0x34d08086 chip=0x29368086 rev=0x02 hdr=0x00
ehci1 at pci0:0:29:7: class=0x0c0320 card=0x34d08086 chip=0x293a8086 rev=0x02 hdr=0x00
pcib4 at pci0:0:30:0: class=0x060401 card=0x34d08086 chip=0x244e8086 rev=0x92 hdr=0x01
isab0 at pci0:0:31:0: class=0x060100 card=0x34d08086 chip=0x29168086 rev=0x02 hdr=0x00
atapci0 at pci0:0:31:2: class=0x010601 card=0x34d08086 chip=0x29228086 rev=0x02 hdr=0x00
none0 at pci0:0:31:3: class=0x0c0500 card=0x34d08086 chip=0x29308086 rev=0x02 hdr=0x00
igb0 at pci0:1:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086 rev=0x01 hdr=0x00
igb1 at pci0:1:0:1: class=0x020000 card=0xa03c8086 chip=0x10c98086 rev=0x01 hdr=0x00
vgapci0 at pci0:3:0:0: class=0x030000 card=0x01018086 chip=0x0522102b rev=0x02 hdr=0x00
em1 at pci0:4:2:0: class=0x020000 card=0x34d08086 chip=0x10768086 rev=0x05 hdr=0x00
igb0 is parent for vlan0, igb1 is parent for clients' vlans. em0 and em1 are not used.
Core dump info:
Fatal trap 12: page fault while in kernel mode
Fatal trap 12: page fault while in kernel mode
cpuid = 3;
cpuid = 0;
Fatal trap 12: page fault while in kernel mode
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 00apic id = 02
cpuid = 1;
fault virtual address = 0x1apic id = 01
fault code = supervisor read, page not present
fault virtual address = 0x1apic id = 03
fault virtual address = 0x1
instruction pointer = 0x20:0xc09bd89c
fault code = supervisor read, page not present
stack pointer = 0x28:0xc52a9adc
instruction pointer = 0x20:0xc099e945
fault virtual address = 0x1
fault code = supervisor read, page not present
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc099e945
frame pointer = 0x28:0xc52a9b0c
stack pointer = 0x28:0xc53059d8
code segment = base 0x0, limit 0xfffff, type 0x1b
stack pointer = 0x28:0xc52ff9d8
instruction pointer = 0x20:0xc099e945
frame pointer = 0x28:0xc52ff9dc
stack pointer = 0x28:0xc53119d8
code segment = base 0x0, limit 0xfffff, type 0x1b
frame pointer = 0x28:0xc53119dc
frame pointer = 0x28:0xc53059dc
= DPL 0, pres 1, def32 1, gran 1
= DPL 0, pres 1, def32 1, gran 1
code segment = base 0x0, limit 0xfffff, type 0x1b
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags =
processor eflags = interrupt enabled,
processor eflags = resume, interrupt enabled, IOPL = 0interrupt enabled,
current process = resume, 12 (irq261: igb1)resume,
= DPL 0, pres 1, def32 1, gran 1IOPL = 0
trap number = 12IOPL = 0
panic: page fault
current process =
cpuid = 012 (irq258: igb0)
Uptime:
current process =
processor eflags = 12 (irq264: igb1)interrupt enabled,
trap number = 12resume,
4d
trap number = 12IOPL = 0
current process = 16h12 (irq262: igb1)46m
trap number = 1248s
Physical memory: 2031 MB
Dumping 244 MB: 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5
#0 doadump () at pcpu.h:246
246 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump () at pcpu.h:246
#1 0xc089f5e7 in boot (howto=260) at ../../../kern/kern_shutdown.c:416
#2 0xc089f849 in panic (fmt=Variable "fmt" is not available.
) at ../../../kern/kern_shutdown.c:590
#3 0xc0be6f1c in trap_fatal (frame=0xc52ff998, eva=1)
at ../../../i386/i386/trap.c:938
#4 0xc0be71a0 in trap_pfault (frame=0xc52ff998, usermode=0, eva=1)
at ../../../i386/i386/trap.c:851
#5 0xc0be7ae5 in trap (frame=0xc52ff998) at ../../../i386/i386/trap.c:533
#6 0xc0bc9bfb in calltrap () at ../../../i386/i386/exception.s:165
#7 0xc099e945 in in_broadcast (in={s_addr = 241182556}, ifp=0xc5e9e400)
at ../../../netinet/in.c:1221
#8 0xc09bfcee in ip_output (m=0xc87f9900, opt=0x0, ro=0xc52ffaac, flags=1,
imo=0x0, inp=0x0) at ../../../netinet/ip_output.c:309
#9 0xc09bc399 in ip_forward (m=0xc87f9900, srcrt=1)
at ../../../netinet/ip_input.c:1531
#10 0xc09bd727 in ip_input (m=0xc87f9900) at ../../../netinet/ip_input.c:555
#11 0xc09566a9 in netisr_dispatch_src (proto=1, source=0, m=0xc87f9900)
at ../../../net/netisr.c:917
#12 0xc0956940 in netisr_dispatch (proto=1, m=0xc87f9900)
at ../../../net/netisr.c:1004
#13 0xc094c8b1 in ether_demux (ifp=0xc78ce400, m=0xc87f9900)
at ../../../net/if_ethersubr.c:901
#14 0xc094ce03 in ether_input (ifp=0xc78ce400, m=0xc87f9900)
at ../../../net/if_ethersubr.c:760
#15 0xc0955400 in vlan_input (ifp=0xc5798400, m=0xc87f9900)
at ../../../net/if_vlan.c:1005
#16 0xc094c7ff in ether_demux (ifp=0xc5798400, m=0xc87f9900)
at ../../../net/if_ethersubr.c:810
#17 0xc094ce03 in ether_input (ifp=0xc5798400, m=0xc87f9900)
at ../../../net/if_ethersubr.c:760
#18 0xc0633772 in igb_rxeof (que=0xc575b900, count=99)
at ../../../dev/e1000/if_igb.c:4200
#19 0xc0633920 in igb_msix_que (arg=0xc575b900)
at ../../../dev/e1000/if_igb.c:1424
#20 0xc0876ceb in intr_event_execute_handlers (p=0xc554c7f8, ie=0xc5749200)
at ../../../kern/kern_intr.c:1220
#21 0xc08783fb in ithread_loop (arg=0xc57608d0)
at ../../../kern/kern_intr.c:1233
#22 0xc0874a21 in fork_exit (callout=0xc0878390 <ithread_loop>,
arg=0xc57608d0, frame=0xc52ffd38) at ../../../kern/kern_fork.c:844
#23 0xc0bc9c70 in fork_trampoline () at ../../../i386/i386/exception.s:270
(kgdb)
There is another one error in core.txt.* :
netstat -id
Segmentation fault
Other commands are OK.
One of the last ipfw rules is
fwd 0.0.0.0 ip from any to any tagged 11
which forwards to the default gateway all incoming packets from clients (bypassing routing table sometimes).
On FreeBSD-8.1-STABLE-201010 or identical hardware problem persists.
On 7.3-RELEASE there is no such problem.
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list