FreeBSD router question

["." at babolo.ru]

> Hello (just signed up to this list),
> 
>   I am wondering if anyone on the list has any experience using FreeBSD 5.3 as a
> router in a high traffic environment?  I am building a development cluster here
> and have decided to try using FreeBSD as my main network router instead of
> something like the Cisco 7200's, Force10, etc.  
> 
> I have 10 or 12 Xeon machines in my cluster so far, but may have as many as 50
> to 100 in the future (once our site goes live).  Right now I have a 2.40 GHz
> Xeon with 2GB of RAM running as the router using FreeBSD 5.3, ipf and ipnat
> (this may be upgraded to an AMD64 bit dual core shortly).  So far everything
> seems to work fine, but it has not been under heavy load yet.  The router has
> been up for 26 days with no problems and works great.
> 
> I've made the following tweaks (see end of message) to sysctl.conf in an effort
> to get things going the right direction.  I've also stripped down the kernel
> file and recompiled.  I read recently that FreeBSD was able to route 1Mpps,
> which sounded pretty good, but I don't know if there are any specific tweaks I
> need to make in order to obtain this sort of speed, or how fast it works "out of
> the box" with just a few modifications?  My main concern is that the router
> works okay now, but when traffic ramps up, it hits a wall without some large
> amount of exotic changes.  I'd like to feel comfortable that the machine will
> handle at least 50 to 100 megabits of traffic on a fairly sustained basis
> without facing any major problems.  Is that realistic or are there specific
> changes I should make to the OS?
> 
> If anyone on the list has any first hand information/experience that might steer
> me the right direction, that would be great.  Any feed back would be great,
> Thanks very much! :-)
We are using a lot of FreeBSD 4 routers.
They route up to 35..40 Tbytes/router,
4..70 vlans per router, natd and argus
runs for most of vlans, 1 natd and 1 argus
per vlan.
ipfw config is about 30..100 Kbyte, pipes
for about half of traffic.
Athlon XP on 760MPX mobo, 1Gbyte of memory.
2000 GHz (real) Athlon XP is 2+ faster router
compare to 2.6 GHz Pentium 4.
Configurators (route, arp, ipfw utilities)
are something buggy under high load
(we have up to 500 reconfigures/day),
and second CPU is not useful if Athlon MP is used.
I have bad impression on my FreeBSD 5 test
on our routers and good on DragonFlyBSD
test, but have no DragonFlyBSD router
under full load yet.

...
> net.inet.ip.fastforwarding=0            # not sure about this, but might want to
It is hard to build complex ipfw rules with
fastforwarding=1, dont know about ipf.

> net.inet.tcp.recvspace=65535            # increase TCP window size for better
> net.inet.tcp.sendspace=65535
Not used for routing.

> kern.ipc.somaxconn=1024                 # increase listen queue (defense against
> SYN attacks, better performance) [128]
Just close router fully, do not accept
any connect but from one control interface
from fully seperated internal net.