FreeBSD router question

[ray at redshift.com]

Hello (just signed up to this list),

  I am wondering if anyone on the list has any experience using FreeBSD 5.3 as a
router in a high traffic environment?  I am building a development cluster here
and have decided to try using FreeBSD as my main network router instead of
something like the Cisco 7200's, Force10, etc.  

I have 10 or 12 Xeon machines in my cluster so far, but may have as many as 50
to 100 in the future (once our site goes live).  Right now I have a 2.40 GHz
Xeon with 2GB of RAM running as the router using FreeBSD 5.3, ipf and ipnat
(this may be upgraded to an AMD64 bit dual core shortly).  So far everything
seems to work fine, but it has not been under heavy load yet.  The router has
been up for 26 days with no problems and works great.

I've made the following tweaks (see end of message) to sysctl.conf in an effort
to get things going the right direction.  I've also stripped down the kernel
file and recompiled.  I read recently that FreeBSD was able to route 1Mpps,
which sounded pretty good, but I don't know if there are any specific tweaks I
need to make in order to obtain this sort of speed, or how fast it works "out of
the box" with just a few modifications?  My main concern is that the router
works okay now, but when traffic ramps up, it hits a wall without some large
amount of exotic changes.  I'd like to feel comfortable that the machine will
handle at least 50 to 100 megabits of traffic on a fairly sustained basis
without facing any major problems.  Is that realistic or are there specific
changes I should make to the OS?

If anyone on the list has any first hand information/experience that might steer
me the right direction, that would be great.  Any feed back would be great,
Thanks very much! :-)

Ray


/etc/sysctl.conf

net.inet.ip.forwarding=1                # enable packet forwarding
net.inet.ip.fastforwarding=0            # not sure about this, but might want to
change to 1

net.inet.ip.check_interface=1           # verify incoming packets arrives on an
interface w/ address matching the packet
's destination address

net.link.ether.inet.log_arp_wrong_iface=0       # turn off ARP error messages -
see http://www.freebsdhowtos.com/102.htm
l

net.inet.tcp.blackhole=2                # drop SYN packets destine to
non-listening tcp/udp ports. This will
net.inet.udp.blackhole=1                # create a blackhole and protect against
stealth port scans

net.inet.tcp.recvspace=65535            # increase TCP window size for better
network performance
net.inet.tcp.sendspace=65535

kern.ipc.somaxconn=1024                 # increase listen queue (defense against
SYN attacks, better performance) [128]

net.inet.icmp.drop_redirect=1           # disable redirects [0]
net.inet.icmp.log_redirect=1            # [0]
net.inet.ip.redirect=0                  # [1]
# net.inet6.ip6.redirect=0              # not using IPv6

net.inet.ip.sourceroute=0               # disable source routing [0]
net.inet.ip.accept_sourceroute=0        # [0]

net.inet.icmp.bmcastecho=0              # disable broadcast ECHO response [0]
net.inet.icmp.maskrepl=0                # disable other broadcast probes [0]

net.link.ether.inet.max_age=1200        # ARP clean up time (prevent flooding
ARP requests) [1200]