kern/154679: [Fatal trap 12] "em1 taskq" only at startup (8.1-RELEASE)

Fri Feb 11 12:10:10 UTC 2011

>Number:         154679
>Category:       kern
>Synopsis:       [Fatal trap 12] "em1 taskq" only at startup (8.1-RELEASE)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Feb 11 12:10:09 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Alexandr Shish
>Release:        8.1-RELEASE
>Organization:
>Environment:
FreeBSD leonet 8.1-RELEASE FreeBSD 8.1-RELEASE #2: Fri Jan 21 19:06:12 EET 2011     root at leonet:/usr/obj/usr/src/sys/M1  amd64

>Description:
At startup i see panic, but if unplug network cable at few min, then system well booting and after plug cable stability work under have load.
It is router for ~500users, bgp, ng_nat, few vlans, ng_car+ng_tee, ng_bpf, as firewall use ipfw.
Additional info about system there (if somebody it need): 
http://mystat.com.ua/a/
core.txt - some log generated by savecore
sysct.conf - /etc/sysctl.conf
kernel.config  kernel config
vmcore (1,2G)  kernel dump file

At system work ng_nat node with source ip at loopback interface lo0 (I think it is important, tomorrow i try to boot system without it).

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x18
fault code      = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff8032e051
stack pointer           = 0x28:0xffffff8000100900
frame pointer           = 0x28:0xffffff8000100940
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process     = 0 (em1 taskq)
trap number     = 12
panic: page fault
cpuid = 1
Uptime: 22s
..and dump kernel

(kgdb) backtrace
#0  doadump () at pcpu.h:223
#1  0xffffffff802d7399 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
#2  0xffffffff802d77cc in panic (fmt=0xffffffff805166a4 "%s") at /usr/src/sys/kern/kern_shutdown.c:590
#3  0xffffffff804c7048 in trap_fatal (frame=0xffffff0001464ba0, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:777
#4  0xffffffff804c7414 in trap_pfault (frame=0xffffff8000100850, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:693
#5  0xffffffff804c7c5a in trap (frame=0xffffff8000100850) at /usr/src/sys/amd64/amd64/trap.c:451
#6  0xffffffff804ad5b3 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:223
#7  0xffffffff8032e051 in m_copydata (m=0x0, off=0, len=107, cp=0xffffff0001eec495 "") at /usr/src/sys/kern/uipc_mbuf.c:817
#8  0xffffffff803b520d in ip_forward (m=0xffffff0001d3bc00, srcrt=Variable "srcrt" is not available.
) at /usr/src/sys/netinet/ip_input.c:1471
#9  0xffffffff803b6a7c in ip_input (m=0xffffff0001d3bc00) at /usr/src/sys/netinet/ip_input.c:735
#10 0xffffffff80386e6e in netisr_dispatch_src (proto=1, source=Variable "source" is not available.
) at /usr/src/sys/net/netisr.c:917
#11 0xffffffff80381fec in ether_demux (ifp=0xffffff000168f000, m=0xffffff0001d3bc00) at /usr/src/sys/net/if_ethersubr.c:901
#12 0xffffffff80382397 in ether_input (ifp=0xffffff000168f000, m=0xffffff0001d3bc00) at /usr/src/sys/net/if_ethersubr.c:760
#13 0xffffffff80381f0f in ether_demux (ifp=0xffffff00012da800, m=0xffffff0001d3bc00) at /usr/src/sys/net/if_ethersubr.c:810
#14 0xffffffff80382397 in ether_input (ifp=0xffffff00012da800, m=0xffffff0001d3bc00) at /usr/src/sys/net/if_ethersubr.c:760
#15 0xffffffff801d6e56 in em_rxeof (rxr=0xffffff0001409400, count=100) at /usr/src/sys/dev/e1000/if_em.c:4200
#16 0xffffffff801d7ec0 in em_handle_que (context=Variable "context" is not available.
) at /usr/src/sys/dev/e1000/if_em.c:1453
#17 0xffffffff80312193 in taskqueue_run (queue=0xffffff000146b180) at /usr/src/sys/kern/subr_taskqueue.c:239
#18 0xffffffff80312416 in taskqueue_thread_loop (arg=Variable "arg" is not available.
) at /usr/src/sys/kern/subr_taskqueue.c:360
#19 0xffffffff802af2e8 in fork_exit (callout=0xffffffff803123d0 <taskqueue_thread_loop>, arg=0xffffff80002d96e0, frame=0xffffff8000100c80)
    at /usr/src/sys/kern/kern_fork.c:844
#20 0xffffffff804ada8e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:562
....
#44 0x0000000000000000 in ?? ()
#45 0x000000000087c000 in ?? ()
#46 0x0000000000000000 in ?? ()
#47 0xffffff0001464ba0 in ?? ()
#48 0xffffffff806e7000 in affinity ()
#49 0xffffff00012f13e0 in ?? ()
#50 0xffffff80001004c0 in ?? ()
#51 0xffffff8000100478 in ?? ()
#52 0xffffff0001464ba0 in ?? ()
#53 0xffffffff802f9d0a in sched_switch (td=0xffffff80002d96e0, newtd=0xffffffff803123d0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1844
Previous frame inner to this frame (corrupt stack?)
(kgdb)

Sometime at startup i can se at startup some like this
kernel: panic: sbdrop
kernel: cpuid = 0 
after there system trying to dump kernel and stoped at firsts second
(I guess this trouble is interrelated witch "em1 taskq" panic ) 
I has patch kernel witch
http://www.freebsd.org/cgi/query-pr.cgi?prp=148807-1-diff&n=/patch-1.diff
posted at
http://www.freebsd.org/cgi/query-pr.cgi?pr=148807&cat=
but i have some other trouble witch sbdrop and this patch not required

>How-To-Repeat:
At this time i try to simulate this situation at wmware at linux. So far unsuccessfully. Maybe wmware Network adapter not fully simulate real ethercard?
But at 7.x upgrade to 8.0 at some configurated althoud ng_nat use one ip on internal interface, and now use few ip at loopback, this bug do not pop up.
>Fix:
As workaround unplug network cable at startup time.

>Release-Note:
>Audit-Trail:
>Unformatted: