Stale TIME_WAIT tcp connections

[Michael Ross]

On Wed, 04 Mar 2015 01:36:18 +0100, Rumen Telbizov <telbizov at gmail.com>  
wrote:

> Hello everyone,
>
> We have a server running 9.3-RELEASE which is exhibiting a high number of
> TIME_WAIT tcp connections which are NOT being recycled. That is, netstat
> reports them over and over again, no matter how long we wait for them to  
> be
> flushed out. Currently this server has been out of rotation for a couple  
> of
> hours and I still see the same tcp sockets there. Overall we have:
>
> # netstat -na | grep TIME_WAIT | wc -l
>    *30066*
>
> Tracking one particular TCP socket in TIME_WAIT proves that it stays  
> there
> all the time.
>
> Another observation is that pfctl shows a very large number of state
> entries, even after pfctl -F all, or disable/enable sequence.
>
> # pfctl -si
> State Table                          Total             Rate
>   current entries                    *59280*
>
> At the same time though:
>
> # pfctl -ss | wc -l
>       18
>
> After the problem was discovered we tried tweaking the following settings
> without any luck:
>
> net.inet.tcp.fast_finwait2_recycle=1
> net.inet.tcp.finwait2_timeout=5000
> net.inet.tcp.maxtcptw=50000
> net.inet.tcp.msl=100
>
> ​So it seems like this system is "stuck" and ​doesn't recycle those TCP
> sockets. Again, the machine is out of rotation and not actively accepting
> any traffic. I will keep it like that in case further investigation is
> required. Please do let me know if there's anything else you'd like to  
> know
> from the state of the machine or something I could try.
>
> ​Regards,

Are you using any IPSEC?
I observed something similar a while back, haven't checked again since i  
reported this.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194690
Affected 9.2, too.

Michael