Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Robert Watson]

On Mon, 7 Jul 2008, Andre Oppermann wrote:

> Robert Watson wrote:
>> Experience suggests that forwarding workloads see significant lock 
>> contention in the routing and transmit queue code.  The former needs some 
>> kernel hacking to address in order to improve parallelism for routing 
>> lookups.  The latter is harder to address given the hardware you're using: 
>> modern 10gbps cards frequently offer multiple transmit queues that can be 
>> used independently (which our cxgb driver supports), but 1gbps cards 
>> generally don't.
>
> Actually the routing code is not contended.  The workload in router is 
> mostly serialized without much opportunity for contention.  With many 
> interfaces and any-to-any traffic patterns it may get some contention.  The 
> locking overhead per packet is always there and has some impact though.

Yes, I don't see any real sources of contention until we reach the output 
code, which will run in the input if_em taskqueue threads, as the input path 
generates little or no contention of the packets are not destined for local 
delivery.  I was a little concerned about mention of degrading performance as 
firewall complexity grows -- I suspect there's a nice project for someone to 
do looking at why this is the case.  I was under the impression that, in 7.x 
and later, we use rwlocks to protect firewall state, and that unless stateful 
firewall rules are used, these are locked read-only rather than writable...

Robert N M Watson
Computer Laboratory
University of Cambridge