LOR route vr0

[Fredrik Lindberg]

Don Lewis wrote:
> On 27 Aug, M. Warner Losh wrote:
> 
>>In message: <20050828025721.X43518 at fledge.watson.org>
>>            Robert Watson <rwatson at FreeBSD.org> writes:
>>: 
>>: On Sat, 27 Aug 2005, M. Warner Losh wrote:
>>: 
>>: > : You need to add an entry to subr_witness.c creating a graph edge between
>>: > : the softc lock and the routing lock.  An example of an entry in
>>: > : subr_witness.c:
>>: > :
>>: > :          /*
>>: > :           * TCP/IP
>>: > :           */
>>: > :          { "tcp", &lock_class_mtx_sleep },
>>: > :          { "tcpinp", &lock_class_mtx_sleep },
>>: > :          { "so_snd", &lock_class_mtx_sleep },
>>: > :          { NULL, NULL },
>>: > :
>>: > : Note that sets of ordered entries are terminated with a double-null.  This
>>: > : declares that locks of type "tcp" preceed "tcpinp" which preceed
>>: > : "so_snd".
>>: >
>>: > So you have to have locks of type tcp BEFORE you take out tcpinp type 
>>: > locks?
>>: 
>>: Correct.  'tcp' reflects the global TCP state tables (pcbinfo) locks, and 
>>: 'tcpinp' is for individual PCBs.  If you acquire first a tcpinp and then 
>>: tcp, the above settings should cause WITNESS to generate a lock order 
>>: warning.  Likewise, both tcp and tcpinp preceed so_snd, so if you acquire 
>>: a protocol lock after a socket lock, it will get unhappy.  WITNESS handles 
>>: transitive relationships, so it gets connected up to the rest of the lock 
>>: graph, explicit and implicit, so indirect violations of orders are fully 
>>: handled.
>>
>>OK.  I've been seeing similar LORs in ed, sn, iwi (ed is my locked
>>version of ed, not in tree GIANT locked ed).
> 
> 
> Just as a datapoint, I've got fxp interfaces on all my machines running
> -CURRENT and I'm not seeing the problem here.
> 

I'm seeing both the rentry and the tcpinp LORs on my fxp interface
on a machine running a few days old -current (Aug 25).

lock order reversal
1st 0xc1e30d38 inp (tcpinp) @ /usr/src/sys/netinet/tcp_input.c:742
2nd 0xc1b74018 fxp0 (network driver) @/usr/src/sys/dev/fxp/if_fxp.c:1172

lock order reversal
1st 0xc1e06bb8 rtentry (rtentry) @ /usr/src/sys/net/route.c:1269
2nd 0xc1b74018 fxp0 (network driver) @/usr/src/sys/dev/fxp/if_fxp.c:1172

As for their backtraces they are almost identical to the
once already posted.

> 
>>I've made the following changes, and the LORs go away (except for
>>one, which was unrelated).  I further don't get the first place where
>>they locks happen that caused the original LORs, so I'm mightly
>>confused.
> 
> 
> What is the other LOR that you are seeing?  Does it go away if you
> unwire the MTX_NETWORK_LOCK order?  If so, that LOR is where witness is
> breaking the loop in the lock ordering graph.
> 
> As jhb mentioned, the output of "show witness" would be interesting in
> the case where the lock orders are not wired.
> 
> 
>>==== //depot/user/imp/newcard/kern/subr_witness.c#62 - /dell/imp/p4/newcard/src/sys/kern/subr_witness.c ====
>>@@ -273,6 +273,13 @@
>> 	{ "allprison", &lock_class_mtx_sleep },
>> 	{ NULL, NULL },
>> 	/*
>>+	 * Network driver locking order
>>+	 */
>>+	{ "rawinp", &lock_class_mtx_sleep },
>>+	{ MTX_NETWORK_LOCK, &lock_class_mtx_sleep },
>>+	{ "if_addr_mtx", &lock_class_mtx_sleep },
>>+	{ NULL, NULL },
>>+	/*
>> 	 * Sockets
>> 	 */
>> 	{ "filedesc structure", &lock_class_mtx_sleep },
>>@@ -309,6 +316,7 @@
>> 	{ "udp", &lock_class_mtx_sleep },
>> 	{ "udpinp", &lock_class_mtx_sleep },
>> 	{ "so_snd", &lock_class_mtx_sleep },
>>+	{ MTX_NETWORK_LOCK, &lock_class_mtx_sleep },
>> 	{ NULL, NULL },
>> 	/*
>> 	 * TCP/IP
>>@@ -316,6 +324,7 @@
>> 	{ "tcp", &lock_class_mtx_sleep },
>> 	{ "tcpinp", &lock_class_mtx_sleep },
>> 	{ "so_snd", &lock_class_mtx_sleep },
>>+	{ MTX_NETWORK_LOCK, &lock_class_mtx_sleep },
>> 	{ NULL, NULL },
>> 	/*
>> 	 * SLIP
>>
>>I'm not sure if I need to add the if_addr_mtx after each thing or
>>not.
> 
> 
> Nope.  You also don't need to add MTX_NETWORK_LOCK after so_snd more
> than once.
> 
> If you are finding that you need to wire the order of if_addr_mtx, that
> is a potential clue.  The only lock I see taken after if_addr_mtx is
> "UMA zone".  If you are seeing other locks under if_addr_mtx, maybe one
> of those is looping back to rtentry.  I also see taskqueue, "if send
> queue", and various memory subsystem locks under "network driver".  Both
> taskqueue and "if send queue" appear to be leaf locks.
> 
> 
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"