Sockets stuck in FIN_WAIT_1
rblayzor.bulk at inoc.net
Thu May 29 22:12:03 UTC 2008
On May 29, 2008, at 5:32 PM, Matthew Dillon wrote:
> Now, the connection is also in a half-closed state, which means
> one direction is closed. I can't tell which direction that is
> but my
> guess is that 220.127.116.11 (the apache server) closed the 18.104.22.168-
> direction and the 22.214.171.124 box has a broken TCP implementation and
> deal with it.
This is exactly what we're seeing, it's VERY strange. I did kill off
Apache, and all the FIN_WAIT_1's stuck around, so the kernel is in
fact sending these probe packets, every 60 seconds, which the client
responds to... (most of the time).
> I can suggest two things. First, the TCP connection is good but
> still may be able to tell Apache, in the apache configuration
> file, to
> timeout after a certain period of time and clear the connection.
I don't think this helps since Apache sees the connection as long
gone. As far as Apache is concerned (as far as I can tell), this
connection doesn't exist. This may be proved by killing off Apache,
the connection still lives and while Apache is running, I have the max
clients connected most of the time... so I don't think the linger
around and jam up sockets to Apache. If they did, I think Apache
would spiral down quite quickly.
> Secondly, it may be beneficial to identify exactly what the
> client and
> server were talking about which caused the client to hang with a
> tcp connection. The only way to do that is to tcpdump EVERYTHING
> on related to the apache srever, save it to a big-ass disk
> (like 500G), and then when you see a stuck connection go back
> the tcpdump log file and locate it, grep it out, and review what
> it was talking about. You'd have to tcpdump with options to tell
> it to
> dump the TCP data payloads.
Unfortunately it's not possible for me, not nearly enough space. This
is a VERY busy server, a spikey 20Mbps+ (8-12Mbps on average) of web
traffic almost constantly. The traffic is VERY static, just small
data files and occasional large ones (12Mb+), but the majority are
2-5k files. (it's a clamav mirror server)
> It seems likely that the client is running an applet or
> receives a stream over the connection, and that applet or
> program has locked up, causing the data sent from the server to
> build up
> and for the client's buffer space to run out, and start
> advertising the
> 0 window.
98% of the clients are clamav (freshclam) clients on various
platforms. Using p0f most of them are various flavors of Linux, but I
can't say what OS the clients are connecting to for sure since I'd
have to look at the OS finger print of the SYN packets...
Don't get me wrong, the server keeps up well, low CPU, lots of RAM
free, lots of network available, and 99% of all HTTP connections are
completed just fine. I just see these FIN_WAIT_1 connections build up
over time until the server runs out of socket space and then things
just stop working. Only way to correct it seems to reboot the
server... even under RELENG_7_0.... so the upgrade from 4_11 did not
fix the problem.
Robert Blayzor, BOFH
rblayzor at inoc.net
More information about the freebsd-stable