kern/149917: freebsd 8.1 crash with ECMP

[Ingo Flaschberger]

>Number:         149917
>Category:       kern
>Synopsis:       freebsd 8.1 crash with ECMP
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 23 20:10:01 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Ingo Flaschberger
>Release:        8.1 stable
>Organization:
crossip communications gmbh
>Environment:
FreeBSD  8.1-STABLE FreeBSD 8.1-STABLE #16: Mon Aug 23 19:51:34 UTC 2010     root at jail.xip.at:/usr/obj/usr/src/sys/ROUTER  amd64
>Description:
system crash when adding 2nd route with other weight

Dump:
#0  doadump () at pcpu.h:224
224             __asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:224
#1  0xffffffff802b0e25 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
#2  0xffffffff802b126d in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:590
#3  0xffffffff80464b65 in trap_fatal (frame=0xffffff00017ed460, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:777
#4  0xffffffff80464f06 in trap_pfault (frame=0xffffff803e6394a0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:693
#5  0xffffffff8046553c in trap (frame=0xffffff803e6394a0) at /usr/src/sys/amd64/amd64/trap.c:451
#6  0xffffffff8044b984 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224
#7  0xffffffff80363cd3 in rtalloc_mpath_fib (ro=0xffffff803e639620, hash=192947925, fibnum=Variable "fibnum" is not available.
) at /usr/src/sys/net/radix_mpath.c:307
#8  0xffffffff8035016f in flowtable_lookup (ft=0xffffff80005a7000, ssa=0xffffff803e6396a0, dsa=0xffffff803e639720, fibnum=0, flags=2) at /usr/src/sys/net/flowtable.c:1229
#9  0xffffffff80350a67 in flowtable_lookup_mbuf (ft=0xffffff80005a7000, m=0xffffff00014e2100, af=Variable "af" is not available.
) at /usr/src/sys/net/flowtable.c:607
#10 0xffffffff80392f69 in ip_output (m=0xffffff00014e2100, opt=0x0, ro=0x0, flags=32, imo=0x0, inp=0xffffff00167c5a80) at /usr/src/sys/netinet/ip_output.c:164
#11 0xffffffff80393f07 in rip_output (m=0xffffff00014e2100, so=Variable "so" is not available.
) at /usr/src/sys/netinet/raw_ip.c:507
#12 0xffffffff80314b70 in sosend_generic (so=0xffffff00166e3550, addr=0xffffff00167b6160, uio=0xffffff803e639a40, top=0xffffff00014e2100, control=0x0, flags=0, td=0xffffff00164303e0)
    at /usr/src/sys/kern/uipc_socket.c:1260
#13 0xffffffff80316878 in kern_sendit (td=0xffffff00164303e0, s=3, mp=0xffffff803e639b10, flags=0, control=0x0, segflg=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:788
#14 0xffffffff80316aa8 in sendit (td=0xffffff00164303e0, s=3, mp=0xffffff803e639b10, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:724
#15 0xffffffff80316b90 in sendto (td=Variable "td" is not available.
) at /usr/src/sys/kern/uipc_syscalls.c:840
#16 0xffffffff804650f5 in syscall (frame=0xffffff803e639c80) at /usr/src/sys/amd64/amd64/trap.c:945
#17 0xffffffff8044bc62 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:377
#18 0x0000000800940e5c in ?? ()
>How-To-Repeat:
ifconfig em0 10.11.11.175/24
ifconfig em0 alias 10.20.20.1/24
route add -net 10.20.20.0/24 10.11.11.1 -weight 2

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x98
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80363cd3
stack pointer           = 0x28:0xffffff803e5da550
frame pointer           = 0x28:0xffffff803e5da580
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1767 (ping)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 3m46s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
>Fix:


Patch attached with submission follows:

--- /usr/src/sys/net/radix_mpath.c_org	2010-08-23 16:36:57.000000000 +0000
+++ /usr/src/sys/net/radix_mpath.c	2010-08-23 16:38:18.000000000 +0000
@@ -294,6 +294,9 @@
 	     weight >= rt->rt_rmx.rmx_weight && rn; 
 	     weight -= rt->rt_rmx.rmx_weight) {
 		
+		/* check if next key exists */
+		if (rn->rn_dupedkey == NULL)
+			break;
 		/* stay within the multipath routes */
 		if (rn->rn_dupedkey && rn->rn_mask != rn->rn_dupedkey->rn_mask)
 			break;


>Release-Note:
>Audit-Trail:
>Unformatted: