zonelimit issues...

[gnn at freebsd.org]

At Tue, 22 Apr 2008 06:35:38 -0700,
Chris Pratt wrote:
> 
> 
> On Apr 21, 2008, at 12:43 AM, gnn at freebsd.org wrote:
> 
> > ...snip
> >
> > Well there are plenty of us motivated to get at these issues.  Can you
> > do me a favor and characterize your traffic a bit?  Is it mostly TCP,
> 
> The traffic that seems to take us out is TCP port 80. I'll make a
> generalized guess but it does seem to follow. We freeze on one of
> two dramatically heavy use days for our industry (Sunday and Monday
> evening). The hang will actually occur on Monday or Tuesday
> following these days if sufficient traffic hits us. It has not
> always followed this pattern but most frequently. There is always a
> high presence of high frequency attacks of various sorts. For
> example referer spam posts which hit us hard on our busy
> evenings. So it is TCP and I would presume we usually have the
> establishment of many useless sessions that could cause us to bump
> up against limits and cause exhaustion coupled with our real traffic
> peaks.
> 

Interesting, but with TCP it should be easier to tune this, in
particular because TCP has backoff once a packet drops.  I gather you
are using facilities, like accept filters, that make it easy to drop
less useful traffic?

> This thread has given me several things to try and I'm adjusting (e.g.,
> nmbclusters) upward to see what happens.

Sounds good.  Using netstat -m and netstat -an are a good way to watch
this issue.  -m is the number of mbufs/clusters in use and -an will
show you all sockets, but what you want to check on s the number of
bytes in the recv and send socket buffers, which are the 2nd and 3rd
columns.

> I should also mention that this system has the natural limitations
> on it's traffic ceiling of two T1s on two NICs and a 3rd LAN NIC
> fielding continuous round-robin mysql replication and rsync style
> mirroring.  It uses two bge interfaces and one server type em
> interface.  It's always troubled me that the zonelimit issues have
> always been associated with higher volume circuits (in what I've
> read). But since our issue is very directly related to traffic
> levels and seem to occur at times where my monitors show us way over
> committed on the two outward facing T1s, I'm still going to proceed
> with the adjustments and see if it increases our survivability.

Since zonelimit is a state reached when your system is out of
resources it makes sense that the higher the traffic the sooner you'll
reach it.  

> Thanks for your time on this.
> 

No problem, it's what I like to do :-)

Best,
George