panic: mtx_lock() of destroyed mutex @ ../../../net/route.c:1306

[Gleb Smirnoff]

  Kris, Mehul,

  I think this patch (that is still a WIP) can cure your problems,
though you observe different problems in this rt_check().

I'd appreciate review and testing.

On Wed, May 02, 2007 at 02:24:54PM -0400, Kris Kennaway wrote:
K> One of my 7.0 systems has a flaky gateway, and when it goes down the
K> node often goes down with this panic:
K> 
K> panic: mtx_lock() of destroyed mutex @ ../../../net/route.c:1306
K> cpuid = 0
K> KDB: enter: panic
K> [thread pid 28619 tid 100074 ]
K> Stopped at      kdb_enter+0x68: ta              %xcc, 1
K> db> wh
K> Tracing pid 28619 tid 100074 td 0xfffff800140e87e0
K> panic() at panic+0x248
K> _mtx_lock_flags() at _mtx_lock_flags+0x8c
K> rt_check() at rt_check+0x128
K> arpresolve() at arpresolve+0x98
K> ether_output() at ether_output+0x94
K> ip_output() at ip_output+0xc64
K> udp_output() at udp_output+0x680
K> udp_send() at udp_send+0x38
K> sosend_dgram() at sosend_dgram+0x3e0
K> sosend() at sosend+0x74
K> kern_sendit() at kern_sendit+0x14c
K> sendit() at sendit+0x1d4
K> sendto() at sendto+0x48
K> syscall() at syscall+0x2f8
K> -- syscall (133, FreeBSD ELF64, sendto) %o7=0x40aa68ac --
K> 
K> I suspect locking is broken in an error case.  net/route.c:1306 is in
K> the senderr() macro in rt_check():
K> 
K>         /* XXX BSD/OS checks dst->sa_family != AF_NS */
K>         if (rt->rt_flags & RTF_GATEWAY) {
K>                 if (rt->rt_gwroute == NULL)
K>                         goto lookup;
K>                 rt = rt->rt_gwroute;
K> bewm -->        RT_LOCK(rt);            /* NB: gwroute */
K>                 if ((rt->rt_flags & RTF_UP) == 0) {
K>                         rtfree(rt);     /* unlock gwroute */
K>                         rt = rt0;
K> Kris



On Mon, May 07, 2007 at 07:52:32AM -0700, Mehul Vora wrote:
M> Hi,
M>    
M>   Current implementation (Version 6.2) of rt_check() routine defined in route.c is not completely  MPSAFE. I found an issue when i started routing with "directisr" enabled. For the first rcvd packet this function initializes rt_gateway of the passed rt_entry. This is done by calling "rtalloc1" routine. But "rt_check" function doesnt hold any lock while calling this function. So incase if we have multiple instances of "ip_input - netisr" running than more than one thread can call this routine which may lead to some corruption, in my case it leads to a dead lock. Problem doesn't happen if before sending heavy traffic a single packet of same kind is sent. But if initially itself heavy traffic is sent than this happens immediately. I have fixed this and it works well after it. Workaround patch for this issue is attached here with. Probably we need to define a macro in route.h for the hardcoded values in the patch. Can any one confirm this ? 
M>    
M>   Thanks,
M>   Mehul.
M> 
M>  
M> ---------------------------------
M> Sucker-punch spam with award-winning protection.
M>  Try the free Yahoo! Mail Beta.

Content-Description: 206142780-rt_check.patch.txt
M> 1260a1261
M> > try_again:
M> 1280a1282,1289
M> >
M> >               if(rt0->rt_flags & 0x80000000U){
M> >                       /*This rt is under process...*/
M> >                       RT_UNLOCK(rt);
M> >                       RT_UNLOCK(rt0);
M> >                       goto try_again;
M> >               }
M> >
M> 1281a1291
M> >                       rt0->rt_flags |= 0x80000000U;
M> 1288a1299
M> >                       rt0->rt_flags &= (~0x80000000U);

M> _______________________________________________
M> freebsd-net at freebsd.org mailing list
M> http://lists.freebsd.org/mailman/listinfo/freebsd-net
M> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"



-- 
Totus tuus, Glebius.
GLEBIUS-RIPN GLEB-RIPE
-------------- next part --------------
A non-text attachment was scrubbed...
Name: route.c.diff
Type: text/x-diff
Size: 1679 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20070606/e14778d9/route.c.bin