Sockets stuck in FIN_WAIT_1

[Matthew Dillon]

    I guess nobody mentioned the obvious thing to check:  Make sure
    TCP keepalive is turned on.

    sysctl net.inet.tcp.always_keepalive=1

    If you don't do this then dead TCP connections can build up, particularly
    on busy servers, due to the other end simply disappearing.

    Without this option the TCP protocol can get stuck, because it does not
    usually send packets to the other end of an idle connection unless 
    (1) its window has closed completely or (2) it has unackncowledged data
    or state pending.  The keepalive forces a probe to occur every so often
    on an idle connection (like once every 30min-2hrs, I forget what the
    default is), to check that the connection still exists.

    It is possible to get stuck during normal data operation and while in
    a half-closed state.  The 2MSL timeout does not activate until you
    go into a fully closed state (FIN2/TIME_WAIT).

    Pretty much if you are running any sort of service on the internet,
    and even if you aren't, you need to make sure keepalive is turned on
    for the long term health of your system.

						-Matt