Repeating kernel panic within dummynet

[Vlad Galu]

On Jul 11, 2011, at 1:42 PM, Eugene Grosbein wrote:

> Hi!
> 
> My FreeBSD 8.2/amd64 routers use dummynet heavily
> and keep panic with the *same* KDB backtrace:
> 
> dummynet: bad switch -256!
> 
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address   = 0x0
> fault code              = supervisor read instruction, page not present
> instruction pointer     = 0x20:0x0
> stack pointer           = 0x28:0xffffff81229d9a10
> frame pointer           = 0x28:0xffffff81229d9a40
> code segment            = base 0x0, limit 0xfffff, type 0x1b
>                        = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 0 (dummynet)
> trap number             = 12
> panic: page fault
> cpuid = 0
> KDB: stack backtrace:
> db_trace_self_wrapper() at 0xffffffff801aaaca = db_trace_self_wrapper+0x2a
> kdb_backtrace() at 0xffffffff80329667 = kdb_backtrace+0x37
> panic() at 0xffffffff802f6cb7 = panic+0x187
> trap_fatal() at 0xffffffff804d8b50 = trap_fatal+0x290
> trap_pfault() at 0xffffffff804d8f2f = trap_pfault+0x28f
> trap() at 0xffffffff804d940f = trap+0x3df
> calltrap() at 0xffffffff804c0b44 = calltrap+0x8
> --- trap 0xc, rip = 0, rsp = 0xffffff81229d9a10, rbp = 0xffffff81229d9a40 ---
> uart_z8530_class() at 0
> mb_dtor_pack() at 0xffffffff802e4787 = mb_dtor_pack+0x37
> uma_zfree_arg() at 0xffffffff8049ba5a = uma_zfree_arg+0x3a
> m_freem() at 0xffffffff803556a7 = m_freem+0x37
> dummynet_send() at 0xffffffff803e909d = dummynet_send+0x2d
> dummynet_task() at 0xffffffff803e93c6 = dummynet_task+0x1c6
> taskqueue_run_locked() at 0xffffffff80335a65 = taskqueue_run_locked+0x85
> taskqueue_thread_loop() at 0xffffffff80335bfe = taskqueue_thread_loop+0x4e
> fork_exit() at 0xffffffff802ca4bf = fork_exit+0x11f
> fork_trampoline() at 0xffffffff804c108e = fork_trampoline+0xe
> --- trap 0, rip = 0, rsp = 0xffffff81229d9d00, rbp = 0 ---
> Uptime: 2d5h17m39s
> Dumping 4087 MB (4 chunks)
>  chunk 0: 1MB (150 pages) ... ok
>  chunk 1: 3575MB (915072 pages) 3559 3543 3527 3511 3495 3479
> 
> 
> It does not finish writing dump and hangs until IPMI watchdog reboots the box.
> I've tried to use debug.minidump=1 but it still hangs while crashdumps is generating
> and stops responding to Ctrl-Alt-ESC meantime.
> 
> Sadly, I cannot add options INVARIANTS to the kernel because it makes my mpd-based
> routers to panic very often (every 2-3 hours) due to famous 'dangling pointer'
> problem - PPPoE user disconnects, its ngXXX interface got removed, then its traffic
> goes out various system queues (netisr, dummynet etc.) and another kind of panic
> occurs due to INVARIANTS' references to non-existent ifp.

Hi Eugene,
If your ISR threads aren't already bound to CPUs, you can bind them and try using INVARIANTS.