Large number of http connections immediately dropped

[Istvan Szukacs]

Hi!

Something to read:

http://people.freebsd.org/~hmp/utilities/satbl/sysctl-net.html

I have these in the sysctl.conf

kern.ipc.somaxconn=4096
net.inet.tcp.recvspace=78840
net.inet.tcp.sendspace=78840
kern.ipc.shmmax=67108864
kern.ipc.shmmni=200
kern.ipc.shmseg=128
kern.ipc.semmni=70

net.local.stream.sendspace=82320
net.local.stream.recvspace=82320
net.inet.tcp.local_slowstart_flightsize=10
net.inet.tcp.nolocaltimewait=1
net.inet.tcp.hostcache.expire=3900

and the loader.conf

kern.maxusers=512

kern.ipc.nmbclusters=32768
kern.ipc.maxsockets=81920
kern.ipc.maxsockbuf=1048576

net.inet.tcp.tcbhashsize=4096
net.inet.tcp.hostcache.hashsize=1024

Regards,
Istvan



Alexander Strange wrote:
> We're running a rather high-load webserver using FreeBSD 
> 7-RELEASE/amd64/nginx on an Intel em gigabit connection.
> Performance is good for our current bandwidth use (about 20Mbit and 
> ~2000 connections/sec at the moment), but a large number of HTTP 
> requests are being immediately dropped before getting to nginx. I see 
> complaints about this with earlier versions of FreeBSD - 
> http://forum.lighttpd.net/topic/171 - but no solutions. Does anyone 
> know what could be the problem, or anything we could do about it?
>
> There are several other servers running earlier FreeBSDs on i386 which 
> don't seem to have this problem, but I still haven't ruled out 
> upstream hardware problems or Sandvine yet.
>
> On the server:
> -nginx's error log is full of "accept() failed (53: Software caused 
> connection abort)", sometimes printing three or four at the same time.
>
> -messages is full of:
> Limiting open port RST response from 441 to 200 packets/sec
> Limiting open port RST response from 488 to 200 packets/sec
> Limiting open port RST response from 399 to 200 packets/sec
> Limiting open port RST response from 434 to 200 packets/sec
> Limiting open port RST response from 308 to 200 packets/sec
> I'm not sure if that's related or not.
>
> -sysctl.conf:
>
> net.inet.tcp.tso=1
> kern.ipc.somaxconn=10240
> kern.ipc.nmbclusters=65536
> net.inet.tcp.sendspace=65536
> net.inet.tcp.recvspace=65536
> net.inet.tcp.rfc1323=1
> kern.ipc.maxsockbuf=262144
> net.inet.tcp.blackhole=2
> net.inet.udp.blackhole=1
> net.inet.tcp.msl=7500
> net.inet.icmp.icmplim=400
> net.inet.tcp.drop_synfin=1
> net.inet.tcp.icmp_may_rst=0
> net.inet.tcp.fast_finwait2_recycle=1
>
> -netstat -m:
> 4677/6603/11280 mbufs in use (current/cache/total)
> 1017/2643/3660/65536 mbuf clusters in use (current/cache/total/max)
> 1017/1961 mbuf+clusters out of packet secondary zone in use 
> (current/cache)
> 9/514/523/12800 4k (page size) jumbo clusters in use 
> (current/cache/total/max)
> 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
> 3239K/8992K/12232K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0/0/0 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 9204 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
>
> nginx is not running any accept filters.
>
> Locally, after sending an HTTP request, I get a normal connection 
> close, then one RST with sequence 1, then another (possibly more than 
> one) RST with sequence 2. I can post a tcpdump sequence if necessary, 
> after I sanitize some cookies away.
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to 
> "freebsd-performance-unsubscribe at freebsd.org"