LOR route vr0

Robert Watson rwatson at FreeBSD.org
Sat Aug 27 17:20:26 GMT 2005

On Sat, 27 Aug 2005, M. Warner Losh wrote:

> In message: <Pine.BSF.4.53.0508270912550.969 at e0-0.zab2.int.zabbadoz.net>
>            "Bjoern A. Zeeb" <bzeeb-lists at lists.zabbadoz.net> writes:
> : > lock order reversal
> : >  1st 0xc17621ec rtentry (rtentry) @ /usr/src/sys/net/route.c:1269
> : >  2nd 0xc15ec938 vr0 (network driver) @ /usr/src/sys/pci/if_vr.c:1391
> :
> : added with ID 140: http://sources.zabbadoz.net/freebsd/lor.html#140
> I've noticed a *HUGE* number of LORs that look like this:
> ock order reversal
> 1st 0xc17490e4 rtentry (rtentry) @ sys/netinet/if_ether.c:445
> 2nd 0xc15c94b0 rl1 (network driver) @ sys/pci/if_rl.c:1451

Generally speaking, network interface device driver locks follow network 
stack locks in the lock order.  However, I've not really looked much at 
the route table locking so can't speak to whether that is the case 
specifically for routing locks.  If it is, the below traces reflect the 
correct order, and you might want to add a hard-coded entry to witness in 
order to catch the reverse order.  Lock order reversals between the 
network stack and device drivers tend to occur as a result of the device 
driver calling into the network stack while holding the device driver 
mutex.  Someone (tm) should work out if the right order is route locks -> 
device driver locks, as it's likely a common calss of bugs across many 

Robert N M Watson

> KDB: stack backtrace:
> kdb_backtrace(c07dcab1,c15c94b0,c160a1b0,c07c54d7,c07f401e) at kdb_backtrace+0x2e
> witness_checkorder(c15c94b0,9,c07f401e,5ab,c07e32bd) at witness_checkorder+0x6c3
> _mtx_lock_flags(c15c94b0,0,c07f401e,5ab,c152cc00) at _mtx_lock_flags+0x8a
> rl_start(c152cc00,1,c07e2eae,836) at rl_start+0x37
> if_start(c152cc00,0,c07e32bd,195,202) at if_start+0x99
> ether_output_frame(c152cc00,c169c100,6,c0562c28,c169c100)
> 	 at ether_output_frame+0x218
> ether_output(c152cc00,c169c100,cbfe79f0,0,2,c1740001,2302,c07e66e8,1bd,519)
> 	at ether_output+0x47e
> arprequest(c152cc00,c16cfcc8,cbfe7ae4,c15fa6ab,c05998a6) at arprequest+0x109
> arpresolve(c152cc00,c1749084,c169a400,cbfe7ae0,cbfe7a64) at arpresolve+0x32d
> ether_output(c152cc00,c169a400,cbfe7ae0,c1749084,0) at ether_output+0x7b
> ip_output(c169a400,0,cbfe7adc,0,0) at ip_output+0xb7a
> and am seeing the following in my newly locked ed driver:
> lock order reversal
> 1st 0xc1cb3588 rtentry (rtentry) @ net/route.c:1269
> 2nd 0xc1fd3420 ed1 (network driver) @ /dell/imp/p4/newcard/src/sys/modules/ed/../../dev/ed/if_ed.c:697
> KDB: stack backtrace:
> kdb_backtrace(0,ffffffff,c0680950,c067f5a0,c064bd44) at kdb_backtrace+0x29
> witness_checkorder(c1fd3420,9,c201ff8b,2b9) at witness_checkorder+0x52c
> _mtx_lock_flags(c1fd3420,0,c201ff8b,2b9,c1a86c00) at _mtx_lock_flags+0x5b
> ed_start(c1a86c00) at ed_start+0x1f
> if_start(c1a86c00) at if_start+0x7b
> ether_output_frame(c1a86c00,c1bbeb00,c04c0920,ffffffff,0) at ether_output_frame+0x1dc
> ether_output(c1a86c00,c1bbeb00,e5832a38,0,2) at ether_output+0x3e4
> arprequest(c1a86c00,c1d77ac8,e5832b08,c20236ab) at arprequest+0xd8
> arpresolve(c1a86c00,c1cb3528,c1bbed00,e5832b04,e5832aa8) at arpresolve+0x30b
> ether_output(c1a86c00,c1bbed00,e5832b04,c1cb3528,c1d77a00) at ether_output+0x6b
> ip_output(c1bbed00,0,e5832b00,0,0) at ip_output+0x78c
> udp_output(c1cb1168,c1bbed00,0,0,c1a8d600) at udp_output+0x4a7
> udp_send(c1d59c84,0,c1bbed00,0,0) at udp_send+0x1a
> sosend(c1d59c84,0,e5832c3c,c1bbed00,0) at sosend+0x5e3
> kern_sendit(c1a8d600,4,e5832cbc,0,0) at kern_sendit+0x104
> sendit(c1a8d600,4,e5832cbc,0,807a023) at sendit+0x163
> sendto(c1a8d600,e5832d04,6,0,206) at sendto+0x4d
> syscall(3b,3b,3b,805a000,28219fa4) at syscall+0x22f
> and was wondering if there's a common cause.
> Warner
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"

More information about the freebsd-current mailing list