bin/141683: nfsd(8) appears to hold onto closed connections indefinitely

[Steve Polyack]

>Number:         141683
>Category:       bin
>Synopsis:       nfsd(8) appears to hold onto closed connections indefinitely
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Dec 16 14:50:00 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     Steve Polyack
>Release:        FreeBSD 7.1-RELEASE amd64
>Organization:
Collaborative Fusion, Inc.
>Environment:
FreeBSD ***** 7.1-RELEASE-p5 FreeBSD 7.1-RELEASE-p5 #3: Fri May 22 07:56:55 EDT 2009     root@*****:/usr/obj/export/src-RELENG_7_1/src/sys/POWEREDGE  amd64
>Description:
A server running FreeBSD 7.1-RELEASE-p5 which acts as an nfs server does not seem to ever let go of closed connections.  The connections in question are from a service monitoring daemon (Nagios check_tcp).  Telnet connections which are then broken will also leave connections lying around in the CLOSE_WAIT state.

$ netstat -tan | grep CLOSE_WAIT |grep 2049 |wc -l
     454

We've witnessed the server build up to over 7000 connections kept in the CLOSE_WAIT state after a few weeks.  Restarting nfsd(8) will cause all of these held connections to disappear.
>How-To-Repeat:
Start nfsd(8) with the -t switch to serve TCP NFS.  Open several telnet connections to the NFS server on port 2049 and close them immediately after connection (or use Nagios to perform simple TCP port monitoring):

nfs-server$ netstat -tan | grep CLOSE_WAIT |grep 2049 |wc -l
     455

$ telnet nfs-server 2049
Connected to nfs-server
Escape character is '^]'.
^]
telnet> q
Connection closed.
$ telnet nfs-server 2049
Connected to nfs-server
Escape character is '^]'.
^]
telnet> q
Connection closed.
$ telnet nfs-server 2049
Connected to nfs-server
Escape character is '^]'.
^]
telnet> q
Connection closed.

$ netstat -tan | grep CLOSE_WAIT |grep 2049 |wc -l
     458

Wait minutes, hours, days, or weeks and see that the connections are still listed in CLOSE_WAIT by netstat.  Eventually restart nfsd(8) and witness the number of connections drop significantly:

$ sudo /etc/rc.d/nfsd restart
Stopping nfsd.
Starting nfsd.
$ netstat -tan | grep CLOSE_WAIT |grep 2049 |wc -l
       0

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: