amd64/119516: panic: _mtx_lock_sleep: recursed on non-recursive mutex rtentry @ /usr/src/sys/net/route.c:1287

[Niels Bakker]

>Number:         119516
>Category:       amd64
>Synopsis:       panic: _mtx_lock_sleep: recursed on non-recursive mutex rtentry @ /usr/src/sys/net/route.c:1287
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jan 10 04:00:01 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Niels Bakker
>Release:        6.2-RELEASE-p8
>Organization:
>Environment:
FreeBSD burnout.tpb.net 6.2-RELEASE-p8 FreeBSD 6.2-RELEASE-p8 #44: Mon Oct 22 00:58:05 CEST 2007     niels at burnout.tpb.net:/usr/obj/usr/src/sys/BURNOUT  amd64

>Description:
# kgdb /boot/kernel/kernel.debug /var/crash/vmcore.14
kgdb: kvm_nlist(_stopped_cpus): 
kgdb: kvm_nlist(_stoppcbs): 
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:
panic: _mtx_lock_sleep: recursed on non-recursive mutex rtentry @ /usr/src/sys/net/route.c:1287

Uptime: 25d7h27m28s
Dumping 1023 MB (2 chunks)
  chunk 0: 1MB (159 pages) ... ok
  chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15

#0  doadump () at pcpu.h:172
172     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:172
#1  0xffffffff802854fb in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xffffffff80284fca in panic (
    fmt=0xffffffff80474d70 "_mtx_lock_sleep: recursed on non-recursive mutex %s @ %s:%d\n") at /usr/src/sys/kern/kern_shutdown.c:565
#3  0xffffffff8027bdc8 in _mtx_lock_sleep (m=0xffffff0018c9c3d8, 
    tid=18446742974534246400, opts=0, file=0x0, line=0)
    at /usr/src/sys/kern/kern_mutex.c:514
#4  0xffffffff8027bea5 in _mtx_lock_flags (m=0xffffff0018c9c3d8, opts=0, 
    file=0xffffffff804831e6 "/usr/src/sys/net/route.c", line=1287)
    at /usr/src/sys/kern/kern_mutex.c:288
#5  0xffffffff8030d813 in rt_check (lrt=0xffffffffa9a697b8, lrt0=0xffffffffa9a697c0, 
    dst=0xffffff0000c93490) at /usr/src/sys/net/route.c:1287
#6  0xffffffff8031111c in arpresolve (ifp=0xffffff0000b50000, rt0=0xffffff0018c9c318, 
    m=0xffffff000dfc8b00, dst=0xffffff0000c93490, desten=0xffffffffa9a69810 "")
    at /usr/src/sys/netinet/if_ether.c:404
#7  0xffffffff803069ea in ether_output (ifp=0xffffff0000b50000, m=0xffffff000dfc8b00, 
    dst=0xffffff0000c93490, rt0=0x0) at /usr/src/sys/net/if_ethersubr.c:174
#8  0xffffffff8032e677 in ip_output (m=0xffffff000dfc8b00, opt=0x54, 
    ro=0xffffffffa9a698d0, flags=33, imo=0x0, inp=0xffffff002925f390)
    at /usr/src/sys/netinet/ip_output.c:777
#9  0xffffffff8032eda3 in rip_output (m=0xffffff000dfc8b00, so=0x0, dst=64)
    at /usr/src/sys/netinet/raw_ip.c:335
#10 0xffffffff8032fb55 in rip_send (so=0xffffff0018453c08, flags=0, 
    m=0xffffff000dfc8b00, nam=0xffffff00096521b0, control=0x0, td=0x0)
    at /usr/src/sys/netinet/raw_ip.c:800
#11 0xffffffff802cb524 in sosend (so=0xffffff0018453c08, addr=0xffffff00096521b0, 
    uio=0xffffffffa9a69a80, top=0xffffff000dfc8b00, control=0x0, flags=0, 
    td=0xffffff00140be000) at /usr/src/sys/kern/uipc_socket.c:836
#12 0xffffffff802d25b4 in kern_sendit (td=0xffffff00140be000, s=3, 
    mp=0xffffffffa9a69b50, flags=0, control=0x0, segflg=64)
    at /usr/src/sys/kern/uipc_syscalls.c:772
#13 0xffffffff802d37e6 in sendit (td=0xffffff00140be000, s=3, mp=0xffffffffa9a69b50, 
    flags=0) at /usr/src/sys/kern/uipc_syscalls.c:712
#14 0xffffffff802d3ba4 in sendto (td=0x0, uap=0x0)
    at /usr/src/sys/kern/uipc_syscalls.c:830
#15 0xffffffff803fe892 in syscall (frame=
      {tf_rdi = 3, tf_rsi = 5273780, tf_rdx = 64, tf_rcx = 0, tf_r8 = 5339504, tf_r9 = 16, tf_rax = 133, tf_rbx = 56, tf_rbp = 64, tf_r10 = 1, tf_r11 = 582, tf_r12 = 5273780, tf_r13 = 0, tf_r14 = 0, tf_r15 = 0, tf_trapno = 12, tf_addr = 34368951312, tf_flags = 0, tf_err = 2, tf_rip = 34369247260, tf_cs = 43, tf_rflags = 514, tf_rsp = 140737488283704, tf_ss = 35}) at /usr/src/sys/amd64/amd64/trap.c:792
#16 0xffffffff803ebeb8 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:270
#17 0x000000080091181c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

>How-To-Repeat:
Uptime was about 25 days, this happened in the middle of the night with little to no other activity happening.  Perhaps dhcpd got a fresh lease (kern/84225).

There was an active ssh session over IPv6 at the time of the reboot.  This session sometimes gets reset by the server, where sshd logs a "Network unreachable" message trying to send a packet down a gif1 tunnel interface.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: