Network performance in a dual CPU system

[Andre Oppermann]

Luigi Rizzo wrote:
> 
> On Wed, Feb 15, 2006 at 09:20:05PM +0100, Andre Oppermann wrote:
> ...
> > >From my profiling with the Agilent tester there seem to be two areas where
> > the packet filters (ipfw in my test case) burn a lot of CPU per packet.
> > That is a) setup of lots of packet variables unconditionally at the entry
> > of ip_fw_chk() no matter whether they get looked at later or not, and b)
> > the switch() going through all the packet inspection options is for some
> > reason not optimized by the compiler and burns even more CPU.  Some sort
> > of JIT (as in the new bpf code) which replaces the case testing and jumps
> > directly to the proper place in the switch statement would go a long way
> > of making it way more performant.
> 
> i was expecting some overhead in the initial setting of
> variables but the cost of the switch() surprises me a bit.
> did you look at the assembly code produced, or otherwise
> could you explain a bit more how you think the switch
> affects performance ?
> Maybe one could make it cheaper through an indirect function call ?
> (in the end, instructions are already indexes for a jump table).

I didn't look at the assembler code as I can't do assembler.

In my testing (on UP) the peak forwarding rate on this particular hardware
with fastforwarding enabled dropped from 580kpps to 476kpps (ipfw allow all)
to 357kpps (30 non-matching rules on IP address).

The number of CPU instructions and branches per packet is as follows:

			maxkpps	instr.	branch	mispred	dcache	icfetch	icmiss
fastfwd			580	2238	300	 3.8	1429	 812	0.06
fastfwd+ipfw		476	2573	329	17.2	1721	1005	4.31
fastfwd+ipfw30		357	3493	508	15.2	2129	1500	3.35

The setup of the packet variables only happens once per packet. The overhead
thus must come from the micro-op evaluation.

-- 
Andre