freebsd router project. Problems with polling?

[Thomas Vogt]

Hello

Overview:

ATM, I've a 4.10 router (xeon 2.4ghz UP, dual on board em gigE
interfaces) in a productive enviroment. The routing software is quagga.
It's doing quite well.
It has an average of 200k pakets per seconds with the full routing
table. The System has polling enabled and interrupt is almost idle and
cpu too. 

New one: 
Now I'm building a FreeBSD 5.3 or -Stable based router system on CF
cards too. Intel whitepaper shows a max pps for the em cards at 1.4mpps.
I doubt that I can reach more than ~600k pps with the fully routing
table loaded! But this would be ok great. 

My system: Xeon 2.4Ghz UP, 2x onboard em GigE interface, 5.3-RELEASE-p4
with if_em.c,v 1.44.2.4. 

my configs:

sysctl.conf: 
vm.swap_enabled=0
kern.ipc.somaxconn=1024
kern.polling.enable=1
kern.random.sys.harvest.ethernet=0
kern.random.sys.harvest.interrupt=0
net.inet.udp.recvspace=65536
net.inet.tcp.sendspace=65536
net.inet.tcp.delayed_ack=0
net.inet.tcp.msl=5000
net.inet.ip.maxfragpackets=40
net.inet.ip.maxfragsperpacket=4
net.inet.ip.fw.one_pass=0
net.inet.ip.fw.dyn_max=8192
net.inet.ip.fw.dyn_udp_lifetime=15
net.inet.ip.fastforwarding=1

loader.conf:
kern.ipc.nmbclusters=32768

Kernel config:
http://www.bsdunix.ch/public/ROUTER5

Setup:

I've 3 similar machines.
I use my own udpsend.c and udprec.c to send packages and to count the
pakages. It can send up to 300-330k pps (udp size 64 bytes)

ATM, the full routing table is not loaded. It's a very basic setup. The
goal is to find the maximum pps throughput for the router with small
pakets. 
But atm I've problems with device polling.


Graphic:
 ------------------   
|10.0.1.2 udp send | 
 ------------------
        |
	| 
-------em0------
|freebsd router |
-------em1------
        |
	|
-----------------------
| 192.168.1.2 udp recv |
------------------------



netstat -w 1 (polling disabled) 
            input        (Total)           output
   packets  errs      bytes    packets  errs      bytes colls
    300531     0   23441438     300585     0   23444792     0
    300939     0   25872898     300895     0   25870184     0
    300738     0   23457584     300768     0   23460626     0
    300304    10   25826466     300304     0   25826876     0
    ...

Interrupt load is about 70% with net.inet.ip.fastforwarding enabled. If
I disable this the system becomes unusable. 
The system hasn't reach the limit yet. But the interrupt is much to
high. It's not worthy to add a second udp sender machine, at the moment.


netstat -w 1 (polling enabled)
            input        (Total)           output
   packets  errs      bytes    packets  errs      bytes colls
    150151 47647   12910330     150150     0   12911806     0
    150151     0   11711798     150152     0   11711876     0
    150151 47665   12910986     150151     0   12911810     0
    150151     0   11711798     150151     0   11711798     0
    ...

Interrupt load is about 10%. CPU is about 60% and with
kern.polling.idle_poll enabled it goes to 100% (as expected). 

As you see the speed is droping down to 50% with polling enable and and
I got a lot of errors. 
kern.polling.lost_polls: 188748 and kern.polling.suspect: 186919 are
also very high. I don't know why polling is so bad on this machine. All
SMP option are disabled in the kernel and bios. 
I tried to do as much as in
http://lists.freebsd.org/pipermail/freebsd-questions/2004-November/064427.html described.

I will prepare others tests with -STABLE and -CURRENT in the next few
days. At the mean time, are they some other magic things config option I
can try? Perhaps increase the HZ to 2000 in the kernel or remove polling
and try smp machine? I doubt that I can run the machine without polling.
If you see 70% interrupt load with 300k pps without polling.

regards
Thomas Vogt