LORs in ipfilter

[Giorgos Keramidas]

The locking changes of ipfilter have introduced a few LORs, which became
visible right  after the build fixes  committed by Scott.   Here are the
ones I've seen so far.

: lock order reversal
: 1st 0xc072d0a0 ifnet (ifnet) @ /usr/src/sys/contrib/ipfilter/netinet/fil.c:2146
: 2nd 0xc06f9380 ipf IP NAT rwlock (ipf IP NAT rwlock) @ /usr/src/sys/contrib/ipfilter/netinet/ip_nat.c:2836
: KDB: stack backtrace:
: kdb_backtrace(0,ffffffff,c0708df8,c07083f8,c06d9aac) at kdb_backtrace+0x29
: witness_checkorder(c06f9380,9,c0676e6c,b14) at witness_checkorder+0x54c
: _sx_xlock(c06f9380,c0676e6c,b14,3,c1e9a000) at _sx_xlock+0x50
: ip_natsync(c1e9a000,0,d95f9c84,c0448dd9,0) at ip_natsync+0x20
: frsync(0,c04f7994,c1d55fac,0,c068949f) at frsync+0x2e
: iplioctl(c1e98b00,80047249,c1fa09e0,3,c1fba450) at iplioctl+0x551
: devfs_ioctl_f(c1ff1d48,80047249,c1fa09e0,c1d67d80,c1fba450) at devfs_ioctl_f+0x87
: ioctl(c1fba450,d95f9d14,3,1,246) at ioctl+0x370
: syscall(2f,2f,2f,280556c0,bfbfeed4) at syscall+0x213
: Xint0x80_syscall() at Xint0x80_syscall+0x1f
: --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x280c67e7, esp = 0xbfbfea7c, ebp = 0xbfbfea98 ---

: lock order reversal
: 1st 0xc2103c84 inp (udpinp) @ /usr/src/sys/netinet/udp_usrreq.c:772
: 2nd 0xc06f92c0 ipf filter rwlock (ipf filter rwlock) @ /usr/src/sys/contrib/ipfilter/netinet/fil.c:1116
: KDB: stack backtrace:
: kdb_backtrace(0,ffffffff,c0709c30,c0708470,c06d9aac) at kdb_backtrace+0x29
: witness_checkorder(c06f92c0,1,c0676ca7,45c) at witness_checkorder+0x54c
: _sx_slock(c06f92c0,c0676ca7,45c,0,0) at _sx_slock+0x50
: fr_check(c1f9de84,14,c1e9a000,1,d9623ad4) at fr_check+0x330
: fr_check_wrapper(0,d9623ad4,c1e9a000,2,c2103bf4) at fr_check_wrapper+0x2a
: pfil_run_hooks(c072f5e0,d9623b48,c1e9a000,2,c2103bf4) at pfil_run_hooks+0xbd
: ip_output(c1f9de00,0,d9623b14,0,0) at ip_output+0x57e
: udp_output(c2103bf4,c1f9de00,c1e46830,0,c1fbca10) at udp_output+0x47d
: udp_send(c21013cc,0,c1f9de00,c1e46830,0) at udp_send+0x1a
: sosend(c21013cc,c1e46830,d9623c4c,c1f9de00,0) at sosend+0x70f
: kern_sendit(c1fbca10,16,d9623cc4,0,0) at kern_sendit+0x104
: sendit(c1fbca10,16,d9623cc4,0,c1e4e780) at sendit+0x159
: sendmsg(c1fbca10,d9623d14,3,0,286) at sendmsg+0x5a
: syscall(2f,2f,2f,1,82c34ec) at syscall+0x213
: Xint0x80_syscall() at Xint0x80_syscall+0x1f
: --- syscall (28, FreeBSD ELF32, sendmsg), eip = 0x28322a27, esp = 0xbfaed8fc, ebp = 0xbfaeda78 ---

Converting the sx locks used by ipfilter to mutexes fixed these LORs but
introduced a new one, which I'm not sure how to fix:

: 1st 0xc06f8ba0 ipf IP state rwlock (ipf IP state rwlock) @ /usr/src/sys/contrib/ipfilter/netinet/ip_state.c:793
: 2nd 0xc072c8e0 ifnet (ifnet) @ /usr/src/sys/net/if.c:1068
: KDB: stack backtrace:
: kdb_backtrace(0,ffffffff,c0707c38,c0708638,c06d7e5c) at kdb_backtrace+0x29
: witness_checkorder(c072c8e0,9,c0695132,42c) at witness_checkorder+0x54c
: _mtx_lock_flags(c072c8e0,0,c0695132,42c,2) at _mtx_lock_flags+0x5b
: ifunit(c2138cdc,da7bca2c,c2138c00,5006,da7bc9f8) at ifunit+0x1e
: fr_stinsert(c2138c00,1,1,0,0) at fr_stinsert+0x67
: fr_addstate(c1f9a8ac,da7bca2c,0,0) at fr_addstate+0x5f7
: fr_check(c1f9a8ac,14,c1e93414,1,da7bcad4) at fr_check+0x7cc
: fr_check_wrapper(0,da7bcad4,c1e93414,2,c2101bf4) at fr_check_wrapper+0x2a
: pfil_run_hooks(c072ee20,da7bcb48,c1e93414,2,c2101bf4) at pfil_run_hooks+0xbd
: ip_output(c1f9a800,0,da7bcb14,0,0) at ip_output+0x57e
: udp_output(c2101bf4,c1f9a800,c1e903d0,0,c20132e0) at udp_output+0x47d
: udp_send(c20ff3cc,0,c1f9a800,c1e903d0,0) at udp_send+0x1a
: sosend(c20ff3cc,c1e903d0,da7bcc4c,c1f9a800,0) at sosend+0x70f
: kern_sendit(c20132e0,16,da7bccc4,0,0) at kern_sendit+0x104
: sendit(c20132e0,16,da7bccc4,0,c1e90410) at sendit+0x159
: sendmsg(c20132e0,da7bcd14,3,0,282) at sendmsg+0x5a
: syscall(2f,2f,2f,1,82cb1ac) at syscall+0x213
: Xint0x80_syscall() at Xint0x80_syscall+0x1f
: --- syscall (28, FreeBSD ELF32, sendmsg), eip = 0x28322a27, esp = 0xbfaed73c, ebp = 0xbfaed8b8 ---

Any ideas about why this is a lock order reversal and how I can fix it? :-)