Tuning for router performance

[Paul Civati]

I'm trying to tune 5.x for maximum router performance, in terms of
packets per second.

I'm starting to hit errors at 150Kpps.

            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
         1     0         66          1     0        178     0
     75751  3391    5124070          1     0        298     0
    150150     0    9609602          1     0        178     0
    150150 19837   10809670          1     0        186     0
    150150     0    9609602          1     0        178     0
    150150 19840   10809466          1     0        186     0
    150150     0    9609602          1     0        178     0
    150150 19842   10809466          1     0        186     0
    146029     0    9345858          1     0        178     0
    141518 32596   10206142          1     0        186     0


Set-up:

RELENG_5 as of Apr 03 21:33

* P4 3.0GHz / 2GB DDR400 RAM
* Supermicro P4SCi with 2 on-board Intel gige
* Additional 2-port Intel gige on 64-bit PCI-X

With this many interfaces I have disabled unnecessary devices in the
BIOS such as USB to reduce IRQ sharing.

APIC also seems to do quite some renumbering of IRQs, no idea if this
is a problem or not.

Kernel config additions over GENERIC:

options         DEVICE_POLLING  #network device mixed interrupt-polling handling
options         HZ=1000         #for polling
options         TCP_SIGNATURE   #TCP MD5 signatures
options         FAST_IPSEC      #IPSec needed for TCP MD5
options         IPFILTER        #ipfilter support
options         IPFILTER_LOG    #ipfilter logging

device          crypto          # crypto needed for TCP MD5
device          vlan            # 802.1q ethernet VLAN tagging
device          carp            # common address resolution protocol

/etc/sysctl.conf:
kern.polling.enable=1
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1

Empty ipfilter ruleset.

[ Full dmesg and kernel config etc at http://www.rackred.com/temp/ ]

My test set-up is to use a Linux based click-router with udpgen on
one side, and a Linux based click-router with udpcount on the other
side.

[ SRC host ] ---- [em0  router under test  em3] ---- [ DST host ]

So traffic test is passing data through the router.

Information gleaned from the archives suggests tweaking sysctl's
such as net.isr.enable and disabling harvest options but these
haven't really made any noticeable difference for me.

Does anyone have any suggestions?

I'm sure I have read of other people getting better performance than
this.

-Paul-

-- 
Paul Civati <paul(at)racksense.com>
Rack Sense Ltd - Managed/Business hosting - www.racksense.com   
RackRed - Value SSL certificates and servers - www.rackred.com