Sockets stuck in FIN_WAIT_1

[Robert Blayzor]

On May 29, 2008, at 5:32 PM, Matthew Dillon wrote:
>    Now, the connection is also in a half-closed state, which means  
> that
>    one direction is closed.  I can't tell which direction that is  
> but my
>    guess is that 1.1.1.1 (the apache server) closed the 1.1.1.1- 
> >2.2.2.2
>    direction and the 2.2.2.2 box has a broken TCP implementation and  
> can't
>    deal with it.


This is exactly what we're seeing, it's VERY strange.  I did kill off  
Apache, and all the FIN_WAIT_1's stuck around, so the kernel is in  
fact sending these probe packets, every 60 seconds, which the client  
responds to... (most of the time).

>    I can suggest two things.  First, the TCP connection is good but  
> you
>    still may be able to tell Apache, in the apache configuration  
> file, to
>    timeout after a certain period of time and clear the connection.

I don't think this helps since Apache sees the connection as long  
gone.  As far as Apache is concerned (as far as I can tell), this  
connection doesn't exist.  This may be proved by killing off Apache,  
the connection still lives and while Apache is running, I have the max  
clients connected most of the time... so I don't think the linger  
around and jam up sockets to Apache.  If they did, I think Apache  
would spiral down quite quickly.

>    Secondly, it may be beneficial to identify exactly what the  
> client and
>    server were talking about which caused the client to hang with a  
> live
>    tcp connection.  The only way to do that is to tcpdump EVERYTHING  
> going
>    on related to the apache srever, save it to a big-ass disk  
> partition
>    (like 500G), and then when you see a stuck connection go back  
> through
>    the tcpdump log file and locate it, grep it out, and review what  
> exactly
>    it was talking about.  You'd have to tcpdump with options to tell  
> it to
>    dump the TCP data payloads.


Unfortunately it's not possible for me, not nearly enough space.  This  
is a VERY busy server, a spikey 20Mbps+ (8-12Mbps on average) of web  
traffic almost constantly.  The traffic is VERY static, just small  
data files and occasional large ones (12Mb+), but the majority are  
2-5k files.  (it's a clamav mirror server)

>    It seems likely that the client is running an applet or  
> javascript that
>    receives a stream over the connection, and that applet or  
> javascript
>    program has locked up, causing the data sent from the server to  
> build up
>    and for the client's buffer space to run out, and start  
> advertising the
>    0 window.

98% of the clients are clamav (freshclam) clients on various  
platforms.  Using p0f most of them are various flavors of Linux, but I  
can't say what OS the clients are connecting to for sure since I'd  
have to look at the OS finger print of the SYN packets...

Don't get me wrong, the server keeps up well, low CPU, lots of RAM  
free, lots of network available, and 99% of all HTTP connections are  
completed just fine.  I just see these FIN_WAIT_1 connections build up  
over time until the server runs out of socket space and then things  
just stop working.  Only way to correct it seems to reboot the  
server...  even under RELENG_7_0.... so the upgrade from 4_11 did not  
fix the problem.

-- 
Robert Blayzor, BOFH
INOC, LLC
rblayzor at inoc.net
http://www.inoc.net/~rblayzor/