NICs locking up, "*tcp_sc_h"

[Robert Watson]

On Sat, 14 Mar 2009, Nick Withers wrote:

> Right, here we go!
...

Turns out that the problem is a lock cycle triggered by the syncache calling, 
indirectly, the firewall during output, and the firewall trying to look up the 
connection for the packet.  Thread one:

> Tracing PID 31 tid 100030 td 0xffffff00012016e0
> sched_switch() at sched_switch+0xdf
> mi_switch() at mi_switch+0x18b
> turnstile_wait() at turnstile_wait+0x1c4
> _mtx_lock_sleep() at _mtx_lock_sleep+0x76
> _mtx_lock_flags() at _mtx_lock_flags+0x95
> syncache_lookup() at syncache_lookup+0xee
> syncache_expand() at syncache_expand+0x38
> tcp_input() at tcp_input+0x99b
> ip_input() at ip_input+0xaf
> ether_demux() at ether_demux+0x1b9
> ether_input() at ether_input+0x1bb
> fxp_intr() at fxp_intr+0x224
> ithread_loop() at ithread_loop+0xe9
> fork_exit() at fork_exit+0x112
> fork_trampoline() at fork_trampoline+0xe
> --- trap 0, rip = 0, rsp = 0xfffffffe80174d30, rbp = 0 ---

This thread holds TCP locks and is trying to acquire the syncache lock. 
Thread two:

> sched_switch() at sched_switch+0xdf
> mi_switch() at mi_switch+0x18b
> turnstile_wait() at turnstile_wait+0x1c4
> _rw_rlock() at _rw_rlock+0x9c
> ipfw_chk() at ipfw_chk+0x3ac1
> ipfw_check_out() at ipfw_check_out+0xb1
> pfil_run_hooks() at pfil_run_hooks+0xac
> ip_output() at ip_output+0x357
> syncache_respond() at syncache_respond+0x2fd
> syncache_timer() at syncache_timer+0x15a
> softclock() at softclock+0x270
> ithread_loop() at ithread_loop+0xe9
> fork_exit() at fork_exit+0x112
> fork_trampoline() at fork_trampoline+0xe

This is the syncache timer holding syncache locks, calling IP output, and IPFW 
trying to acquire TCP locks.

Am I right in thinking that you are using uid/gid/jail firewall rules?  They 
suffer from a fundamental architectural problem in that they require reaching 
"up" to a higher level of the stack at times when it's not always a good idea 
to do so.  In general we solve the problem by passing "down" the inpcb for a 
connection in the output path so that TCP doesn't have to look it up -- 
however, in the case of the syncache we actually don't have the inpcb easily 
in hand (or at least, we have it, but we can't just lock it because syncache 
locks are after TCP locks in the lock order...).  It transpires that what the 
firewall really wants is not the inpcb, but the credential, but those are 
interfaces we can't change right now.

I'll need to think a bit about a proper fix for this, but you'll find the 
problem likely goes away if you eliminate all uid/gid/jail rules from your 
firewall.  You could also tweak the syncache logic not to use a retransmit 
timer, which might slightly extend the time it takes for systems to connect to 
your host in the presence of packet loss, but would eliminate this 
transmission path entirely.  We'll need a real and more general fix, however, 
to commit, and I'll look and see what I can come up with.

Robert N M Watson
Computer Laboratory
University of Cambridge