Random TCP 3 second delay

[Maxim Dounin]

Hello!

On Thu, May 12, 2011 at 01:01:23PM +0300, Michael wrote:

> I'm  running  nginx  in  load balancing mode for java application with
> around  400  requests/second.  For  every  request,  nginx  is doing a
> HTTP/1.0  connection  without  keep alive to jetty. Several
> times  per  minute this connection takes 3 seconds complete. I don't see
> 2 or 1 seconds  response from upstream, only 3 seconds. Listen backlog is not
> overflowed.

3 seconds delay usually indicate packet loss somewhere along the 
path.  You have to examine your network between nginx and your app 
to find out where (and why) packets are lost.

Some basic steps on FreeBSD include looking though "netstat -id", 
"netstat -s" output, "vmstat -z" output and your ethernet 
controller stats via "sysctl dev.<driver>" (if any).

You may also want to check if you by accident have some statefull 
firewall between nginx and backend.  E.g. pf(4) known to drop 
some packets when it aproaches limit on state table size.  Check 
"pfctl -s info" and friends if you are using pf.

Maxim Dounin