Stale TIME_WAIT tcp connections

[Rumen Telbizov]

Hello everyone,

We have a server running 9.3-RELEASE which is exhibiting a high number of
TIME_WAIT tcp connections which are NOT being recycled. That is, netstat
reports them over and over again, no matter how long we wait for them to be
flushed out. Currently this server has been out of rotation for a couple of
hours and I still see the same tcp sockets there. Overall we have:

# netstat -na | grep TIME_WAIT | wc -l
   *30066*

Tracking one particular TCP socket in TIME_WAIT proves that it stays there
all the time.

Another observation is that pfctl shows a very large number of state
entries, even after pfctl -F all, or disable/enable sequence.

# pfctl -si
State Table                          Total             Rate
  current entries                    *59280*

At the same time though:

# pfctl -ss | wc -l
      18

After the problem was discovered we tried tweaking the following settings
without any luck:

net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.finwait2_timeout=5000
net.inet.tcp.maxtcptw=50000
net.inet.tcp.msl=100

​So it seems like this system is "stuck" and ​doesn't recycle those TCP
sockets. Again, the machine is out of rotation and not actively accepting
any traffic. I will keep it like that in case further investigation is
required. Please do let me know if there's anything else you'd like to know
from the state of the machine or something I could try.

​Regards,
-- 
Rumen Telbizov
Unix Systems Administrator <http://telbizov.com>