LORs with PF

[Robert Watson]

On Thu, 22 Jul 2004, Bjoern A. Zeeb wrote:

> > I don't think, it was reported yet, but here it goes:
> >
> > lock order reversal
> >  1st 0xc0632c80 pf task mtx (pf task mtx) @ /usr/src/sys/contrib/pf/net/pf.c:5822
> >  2nd 0xc066638c tcp (tcp) @ /usr/src/sys/contrib/pf/net/pf.c:2420
> 
> for reference: added it to "LOR page" with LOR # 017: 
> http://sources.zabbadoz.net/freebsd/lor.html#017

I have no immediate fix, but here's an explanation:

This likely occurs because calls are made into pf in the outgoing packet
path, which may hold protocol layer locks (inpcb and pcbinfo locks).  The
pf mutex will therefore typically follow the protocol locks in the lock
order.  However, when pf calls up to the pcb/socket code to find
potentially matching pcbs or sockets for a packet, it can't/shouldn't hold
the pf lock across that.  A similar bug exists in ipfw.

FWIW, this is actually symptomatic of a race: technically, that pcb lookup
could have a different result between when the firewall looks it up and
when the TCP or UDP code looks it up.  It's a property of what is
generally considered a layering violation, albeit a useful one.

The solution I'm looking at for ipfw, but haven't had a chance to
prototype yet, is to avoid holding the mutex for the duration of the call
to ipfw, but instead use some rule set reference count such that ipfw
consumers hold a reference on the rules to prevent them from changing out
from under them.  Note that this has interesting semantic consideration
such as starvation, which are usually remedied using copy-on-write, but
that may be expensive for large rule sets.  Just something to think
about...

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Principal Research Scientist, McAfee Research