kern/162558: [panic] seldom dummynet panics

[Eugene Grosbein]

>Number:         162558
>Category:       kern
>Synopsis:       [panic] seldom dummynet panics
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Nov 14 10:00:23 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Eugene Grosbein
>Release:        FreeBSD 8.2-STABLE amd64
>Organization:
RDTC JSC
>Environment:
System: FreeBSD k-45-pc-2.sd.rdtc.ru 8.2-STABLE FreeBSD 8.2-STABLE #2: Tue Oct 18 16:20:05 NOVT 2011 root at k-45-pc-1.sd.rdtc.ru:/usr/local/obj/nanobsd.pppoe/home/src/sys/PPPOE8 amd64

>Description:
	My high loaded PPPoE servers (mpd-5.5) panic seldom in dummynet code.
	Last time they were updated to 8.2-STABLE/amd64 18 October 2011
	and today I've got another panic. These panic generate crashdumps
	but kgdb just cannot read them for unknown reason:

# kgdb kernel.debug /path/to/vmcore.1
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Cannot access memory at address 0x746c75616620
(kgdb) bt
#0  0x0000000000000000 in ?? ()
Cannot access memory at address 0x0

	I've remote console also and at the moment of panic it shows:

ipfw: pullup failed
ipfw: pullup failed
ipfw: pullup failed
ipfw: pullup failed
ipfw: pullup failed
ipfw: pullup failed


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x308
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff802e7657
stack pointer           = 0x28:0xffffff81229e1970
frame pointer           = 0x28:0xffffff81229e1990
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (dummynet)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff801ab1ca = db_trace_self_wrapper+0x2a
kdb_backtrace() at 0xffffffff803294a7 = kdb_backtrace+0x37
panic() at 0xffffffff802f69ee = panic+0x27e
trap_fatal() at 0xffffffff804ddc00 = trap_fatal+0x290
trap_pfault() at 0xffffffff804ddfdf = trap_pfault+0x28f
trap() at 0xffffffff804de4bf = trap+0x3df
calltrap() at 0xffffffff804c5b74 = calltrap+0x8
--- trap 0xc, rip = 0xffffffff802e7657, rsp = 0xffffff81229e1970, rbp = 0xffffff81229e1990 ---
_mtx_lock_sleep() at 0xffffffff802e7657 = _mtx_lock_sleep+0x67
igmp_input() at 0xffffffff803dbc2d = igmp_input+0xf8d
ip_input() at 0xffffffff803fca3d = ip_input+0xad
netisr_dispatch_src() at 0xffffffff803bab7e = netisr_dispatch_src+0x7e
dummynet_send() at 0xffffffff803ee18e = dummynet_send+0x14e
dummynet_task() at 0xffffffff803ee386 = dummynet_task+0x1c6
taskqueue_run_locked() at 0xffffffff80335885 = taskqueue_run_locked+0x85
taskqueue_thread_loop() at 0xffffffff80335a1e = taskqueue_thread_loop+0x4e
fork_exit() at 0xffffffff802cab0f = fork_exit+0x11f
fork_trampoline() at 0xffffffff804c60be = fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff81229e1d00, rbp = 0 ---
Uptime: 27d16h2m57s
Dumping 871 out of 4079 MB:..2%..12%..21%..32%..41%..52%..61%..72%..81%..92%
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...

>How-To-Repeat:
	Panics occur seldom, about 6 weeks between them.
	This panic occured in a hour of least load.

	I use dummynet actively to rate-limits users:
	every user get its own pair of dynamic dummynet pipes for local traffic
	and another pair for external traffic:

ipfw add 1010 pipe tablarg ip from $mynets to 'table(13)' in recv $uplink
ipfw add 1020 pipe tablarg ip from 'table(14)' to $mynets in recv 'ng*'
ipfw add 1022 pipe 1 ip from $mynets to any in recv $uplink
ipfw add 1024 pipe 3 ip from any to $mynets in recv 'ng*'
ipfw add 5000 pipe tablearg ip from any to 'table(11)' in
ipfw add 5010 pipe tablearg ip from 'table(12)' to any in

	Here is my /etc/sysctl.conf tuning for dummynet:

net.inet.ip.dummynet.pipe_slot_limit=1000
net.inet.ip.dummynet.io_fast=1
net.inet.ip.fw.one_pass: 1

	All my pipes have queue length 1000 and 64 buckets and
	distinct bandwidth and burst values.

>Fix:

	Unknown.
>Release-Note:
>Audit-Trail:
>Unformatted: