Gateway slowed down to barely usable

[Andrea Venturoli]

Bruce M. Simpson wrote:

>> Now the question is: in case this happens again, how do I find out 
>> what's wrong?
>> CPU usage was under 2% and so was swap usage... what else could I check?
>> What tools should I use?
> Points for further investigation:
> How long was the machine up for?

A couple of days.



> Exactly which network components in FreeBSD are you using?

Here's my ifconfig's output:

rl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
         options=8<VLAN_MTU>
         inet 192.168.99.1 netmask 0xffffff00 broadcast 192.168.99.255
         ether 00:00:e8:63:d1:25
         media: Ethernet autoselect (100baseTX)
         status: active
rl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
         options=8<VLAN_MTU>
         inet xxx.xxx.xxx.xxx netmask 0xfffffff8 broadcast 212.31.247.183
         ether 00:00:e8:63:d1:10
         media: Ethernet autoselect (100baseTX <full-duplex>)
         status: active
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
         options=8<VLAN_MTU>
         inet 192.168.100.201 netmask 0xffffff00 broadcast 192.168.100.255
         ether 00:a0:c9:d5:33:5d
         media: Ethernet autoselect (100baseTX <full-duplex>)
         status: active
xl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
         options=9<RXCSUM,VLAN_MTU>
         ether 00:50:04:0c:60:03
         media: Ethernet autoselect (100baseTX <full-duplex>)
         status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
         inet 127.0.0.1 netmask 0xff000000

rl1 is connected to an HDSL router with a public IP; xl0 is bridged to 
rl1; fxp0 and rl0 are private nets which require NAT.

It's working as a firewall with ipfw/natd/inetd and snort.
This box is also running squid, named and DHCP server for the internal nets.



> Do you have any figures on what kind of network load the machine was 
> dealing with?

Yes, I use cacti to graph that, so what I can say is:
_ most traffic goes from fxp0 through squid or natd to internet;
_ the HDSL is 1Mb/s and it is frequently saturated;
_ the other day, we had virtually constant 1Mb/s for about 6 hours, then 
the sudden performance drop.



> Can you rule out problems with an intermediate switch?

I don't understand; what do you mean?



> Based on what you've said I can only speculate that the possible causes 
> are either mbuf memory fragmentation or a driver problem; both are a 
> total stab in the dark.

WRT driver, xl0 was added at the beginning of this week, so it could be 
that. Is that driver known to be problematic? I've always used it on 
several other boxes without any problem.
As for mbuf, how do I check this?
(BTW I only have a rough idea of what mbufs are. Any good doc pointer?)

  bye
	av.