Help with high LA

[Ian Smith]

On Thu, 12 Feb 2009 10:04:41 -0200 (BRST) scuba at centroin.com.br wrote:

 > I need help for some strange problem with one of my servers, that can cost 
 > my job.
 > 
 > It's a FreeBSD 7.0-RELEASE-p5/amd64 running on a Dell PowerEdge III as a 
 > Virtual machine of VMware ESXi. There are only two VM in this box, and one 
 > of them (basicly a mail server) is running fine.
 > 
 > The problem is with high loads on the other one, that runs (besides other 
 > services) http and pop3.
 > 
 > TOP show LA from 40 to 90 most of the time.
 >
 > I thought, at first, that was a disk botleneck due to some big mailboxes, 
 > or something related to some Apache (2.2.9) fine tuning, but it's 
 > something else.
 > 
 > 
 > If I stop pop3 and apache services (the most active of the box), the LA 
 > drops to 1~2.
 > Starting only one of them (any one) the LA rise to 20~40. Sugesting that 
 > it's not tied to a specific service.
 > 
 > I did a test running just pop3 (Qpopper), pointing the mail spool to a 
 > empty directory, to make shure that it's not a disk problem. And the LA 
 > also goes to sky (~30). The same happens with only apache running pointing 
 > to a simple http page.
 > 
 > The console shows messages like:
 > 
 > ipfw: install_state: Too many dynamic rules

     net.inet.ip.fw.dyn_max: 4096	# (here)
             Maximum number of dynamic rules.  When you hit this limit, no
             more dynamic rules can be installed until old ones expire.

To see which traffic is creating 'too many' dynamic rules, check:

# ipfw -ted show | less -S ++G

(-td for just active rules, but the expired ones tell useful stories)

 > I know I must review my rules and limit the number of keep-state entries, 
 > but a tryed to rise the number of dynamic buckets via sysctl:
 > 
 > sysctl -w net.inet.ip.fw.dyn_buckets=2048
 > 
 > But it seems it's not working, since the number of current buckets doesn't 
 > pass 256:
 > 
 > net.inet.ip.fw.curr_dyn_buckets: 256

But did you remember to flush? :)  See ipfw(8) under 'SYSCTL VARIABLES'.

You might also want to monitor and/or play with some of the other 
net.inet.ip.fw.dyn_* sysctls to see what's happening and how many 
dynamic rules you need with comfortable headroom for your workload/s. 
For TCP, keepalive and *lifetime timeouts may be relevant.

I tend to use stateful rules for outbound UDP, and stateless setup and 
established rules for TCP services here, but your needs may differ.

 > I tryed to make some OS tuning, from the handbook, like increase the 
 > maxcon:
 > 
 > kern.ipc.somaxconn: 2048
 > 
 > but nothing seems to work.
 > 
 > Other entries in the logs:
 > Feb 12 09:06:20 host1 inetd[1248]: accept (for ftp): Software caused 
 > connection abort
 > Feb 12 09:06:20 host1 inetd[1248]: accept (for pop3): Software caused 
 > connection abort
 > 
 > I need some clues to undestand what is happening.
 > 
 > Thank you,
 > 
 >   - Marcelo

Yes, 'Too many dynamic rules'; further connections will surely fail.

cheers, Ian