Panic on boot with April 16 src (lengthy info attached)

Robert Watson rwatson at
Wed Apr 18 07:54:02 UTC 2007

On Tue, 17 Apr 2007, Robert Watson wrote:

>> I originally put it in there to work around a LOR that I was experiencing 
>> (based on you mentioning it in an email to current@ Sun 18 Mar 2007 15:50). 
>> doesn't show any changes 
>> to that particular LOR, do you happen to know if there's any ongoing work 
>> on this? I'm very willing to act as a test system.
> I chatted with Andre about the panic earlier this afternoon, and it sounds 
> like the fix is straight forward.  I would anticipate seeing it committed in 
> the near future.
> I'll send out an e-mail explaining the above lock order reversal tomorrow 
> morning.  I understand that several people have been looking at this, so 
> perhaps one of those people will reply talking about it before then. :-)

The essential problem of this lock order reversal has to do with the fact that 
higher network stack layers hold locks over lower network stack layers.  For 
example, the lock for a TCP connection is held over the operation to enqueue 
the TCP packet for transmission at a lower layer.  This is necessary in order 
to maintain TCP transmission order into the transmission queue between 
multiple threads operating on the same TCP connection, as if the "transmit and 
enqueue" operation were non-atomic with respect to the same TCP connection in 
another thread, quite damaging reordering could take place.  We directly 
dispatch the entire outbound network stack from that enqueue point, meaning 
that the per-TCP connection lock is held over that processing path, including 
the firewall.  As a result, PCB locks (TCP connection locks) preceed the 
firewall in the lock order.

Firewall locks are about protecting the rule state of the firewall from 
corruption when firewall rules are updated, allowing readers to interpret the 
rules using a static snapshot, and writers to avoid mangling the rules via 
simultaneous non-atomic update.  As such, when the firewall code is entered, 
the firewall lock is acquired, and held until the packet has been completely 
processed.  Things get sticky deep in the firewall code because our firewalls 
include credential-aware rules, which essentially "peek up the stack" in order 
to decide what user is associated with a packet before delivery to the 
connection is done.  The firewall rule lock is held over this lookup and 
inspection of TCP-layer state.  In the out-bound path, we pass down the TCP 
state reference (PCB pointer) and guarantee the lock is already held. 
However, in the in-bound direction, the firewall has to do the full lookup and 
lock acquisition.  Which reverses the lock order, and can lead to deadlocks.

debug.mpsafenet=0 places the Giant lock in front of all network stack lock 
acquisition, which effectively serializes all of the above.  It doesn't remove 
the lock order reversal, but it does eliminate simultaneous lock acquisition, 
removing one of the necessary causes of deadlock.  This trick of a serializing 
"global" lock in order to prevent lock order between "leaf" locks is not an 
uncommon technique, but in this case has a significant overhead (requiring 
non-parallelism in network processing), and needs to be fixed.

The key is to guarantee that the acquisition of the firewall reference will 
never be blocked waiting on a PCB lock -- i.e., that the firewall "lock" isn't 
a lock so much as a reference count that will never have to wait, removing the 
waiting requirement from the deadlock equation.  I know that Julian Elischer 
has been looking at doing this, and others may have also.  The model is 
essentially that you either starve writers to the firewall data, or you create 
a read-only snapshot to be used by readers in the event a writer arrives, 
allowing readers to pick up the new rules if available, or the old rules if 
not, and never wait indefinitely either way.

Robert N M Watson
Computer Laboratory
University of Cambridge

More information about the freebsd-current mailing list