FreeBSD 7.0: sockets stuck in CLOSED state...

[Vlad GALU]

On 6/25/08, Ali Niknam <freebsd-net at transip.nl> wrote:
> Dear All,
>
>  Recently i've been upgrading some of my machines from FreeBSD 6.x amd64 to
> FreeBSD 7.0 amd64.
>
>  After upgrading I noticed a weird error/bug. It seems that after several
> thousand TCP connections some seem to hang in 'CLOSED' state.
>
>  netstat -n gives:
>  ...
>  tcp4      0       0  1.2.3.4.*          4.5.6.7.42149       CLOSED
>  tcp4      39      0  1.2.3.4.*          4.5.6.7.54103       CLOSED
>  tcp4      35      0  1.2.3.4.*          4.5.6.7.41718       CLOSED
>  tcp4      38      0  1.2.3.4.*          4.5.6.7.55618       CLOSED
>  tcp4      41      0  1.2.3.4.*          4.5.6.7.44230       CLOSED
>  tcp4      39      0  1.2.3.4.*          4.5.6.7.49439       CLOSED
>  ...
>
>  These never go away; they gradually increase and increase until the
> application starts giving errors (probably because some socket or
> filedescriptor limit is reached). When the application is killed these
> entries disappear.
>
>  The application in question is a self written DNS server, multithreaded,
> and running fine for years without any troubles on both BSD 5.x as well as
> 6.x. Also 32bits as well as 64bits on 6.x.
>
>  Ofcourse that doesn't mean that the application is error free, however,
> after doing extensive testing I really can not find anything wrong with the
> application itself, so I'm thinking maybe there's a change somewhere that
> causes this? I know that tcp/network has been completely redone...
>
>  What basically happens in the application is this:
>   - one main tcp thread runs an infinite while loop waiting for new
> connections to arrive
>   - as soon as one arrives a new thread is spawned that handles the newly
> created stream
>   - it reads some bytes, writes some bytes, then closes it
>   - thread exits
>
>  What appears to happen is this: after the new thread is spawned it tries to
> read 2 bytes (DNS tcp length information). It gets back 0 bytes (EOF) and
> therefore closes the sockets and calls pthread_exit. However in netstat that
> same stream oftenly appears to have bytes 'stuck' in the in queue...
>
>  I really can't see how this can cause hanging sockets in 'CLOSED' state.
> Even if the incoming queue isnt read entirely a call to close should close
> it. Also I really can't find any documentation in netstat, or elsewhere,
> about the 'CLOSED' state...
>
>
>  Any help would greatly be appreciated!
>
>
>  Kind Regards,
>
>
>  Ali Niknam
>  _______________________________________________


   This looks like an issue we used to have at work, where a streaming
application suddenly started getting kevents for sockets that had been
already closed. While that was happening, a netstat output looked just
like yours. We never tracked it down, as we moved to other projects :(


-- 
~/.signature: no such file or directory