Performance/lockup issues with FreeBSD as a router

[Peter Kieser]

--
Peter Kieser
pfak at telus.net


Quoting Benjamin Krueger <benjamin at seattlefenix.net>:

> * Peter Kieser (pfak at telus.net) [050526 10:49]:
> > Hello guys,
> > 
> > I'm not quite sure if this is the right list to address this to, as it's
> partly
> > a performance problem and partly otherwise.
> > 
> > I have a FreeBSD machine acting as a router (doing approx. 15-25Mbit/s of
> > traffic (lot's of small packets, about 45,000 pps)), however I'm currently
> > running into issues where one, or both of the NICs will stop transmitting
> > traffic. When I go onto the machine, and try to ping something I get "No
> buffer
> > space available"
> > 
> > The nics are if_dc, this is a stock FreeBSD 5.4-RELEASE installation (no
> > firewall or anything):
> > 
> > Here's my /etc/sysctl.conf:
> > 
> > net.inet.ip.rtexpire=1800
> > net.inet.ip.rtminexpire=1800
> > kern.maxfiles=32768
> > kern.maxfilesperproc=32768
> > kern.ipc.somaxconn=32767
> > net.inet.tcp.sendspace=256000
> > net.inet.tcp.recvspace=256000
> > kern.ipc.maxsockbuf=2097152
> > net.inet.ip.fastforwarding=1
> > net.inet.tcp.rfc1323=1
> > net.link.ether.inet.max_age=600
> > net.inet.tcp.msl=7500
> > net.inet.ip.fw.dyn_udp_lifetime=10
> > net.inet.ip.fw.dyn_buckets=1024
> > 
> > And here's my /boot/loader.conf:
> > 
> > kern.ipc.maxsockets="163840"
> > kern.maxusers="2048"
> > 
> > Is there anythign I'm overlooking that would be causing the machine to
> lockup
> > like this?
> > 
> > --Peter
> 
> What does 'netstat -m' say? Is there any reason you explicitely set maxusers
> instead of
> letting the kernel auto-tune the setting? I would watch your mbuf usage and
> when you
> find your average and peak usage, configure your mbuf allocation at boot and
> see if that
> helps. I'd also consider letting FreeBSD auto-tune your MAXUSERS.
> 
> -- 
> Benjamin Krueger
> SysAdmin, CarDomain Network
> 92 Toyota Turbo MR2
> 78 Datsun B-210
> 91 freakin Geo Prizm
> 


I forgot to mention that mbuf's aren't exhausted, here's the reporting on mbuf's
when the machine's NICs fail to work:

$ netstat -m
450 mbufs in use
448/132096 mbuf clusters in use (current/max)
0/2/33280 sfbufs in use (current/peak/max)
1008 KBytes allocated to network
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines