Large number of http connections immediately dropped

[Ivan Voras]

Alexander Strange wrote:
> We're running a rather high-load webserver using FreeBSD 
> 7-RELEASE/amd64/nginx on an Intel em gigabit connection.
> Performance is good for our current bandwidth use (about 20Mbit and 
> ~2000 connections/sec at the moment), but a large number of HTTP 
> requests are being immediately dropped before getting to nginx. I see 
> complaints about this with earlier versions of FreeBSD - 
> http://forum.lighttpd.net/topic/171 - but no solutions. Does anyone know 
> what could be the problem, or anything we could do about it?
> 
> There are several other servers running earlier FreeBSDs on i386 which 
> don't seem to have this problem, but I still haven't ruled out upstream 
> hardware problems or Sandvine yet.
> 
> On the server:
> -nginx's error log is full of "accept() failed (53: Software caused 
> connection abort)", sometimes printing three or four at the same time.
> 
> -messages is full of:
> Limiting open port RST response from 441 to 200 packets/sec
> Limiting open port RST response from 488 to 200 packets/sec
> Limiting open port RST response from 399 to 200 packets/sec
> Limiting open port RST response from 434 to 200 packets/sec
> Limiting open port RST response from 308 to 200 packets/sec
> I'm not sure if that's related or not.

It's almost certainly related - in addition to other suggested tuning by 
Istvan, set net.inet.icmp.icmplim sysctl to something high - for example 
2000 in your case.

Actually, in your sysctl.conf it's set to 400 - you do know you have to 
run "/etc/rc.d/sysctl restart" to reaload sysctl.conf?