File descriptor leaks under java 1.6

[Kurt Miller]

On Sunday 23 March 2008 3:13:33 pm Dominic Bishop wrote:
> I have an application that makes use of 2 way socket communcations, so uses
> Socket/ServerSocket, this is all standard IO, no java.nio is involved.
> 
> When running this under diablo 1.5 everything works fine, watching allocated
> file descriptors with lsof shows nothing abnormal whatsoever.
> 
> If I run this under a port build JDK 1.6 then it starts leaking file
> descriptors originating from TCP, until eventually the available descriptors
> are exhausted and the application crashes with a 'too many open files'
> error.
> 
> I've tried both patchset 3 and 4 of the port, under amd64 and i386
> architecture FreeBSD 6.3 and the same problem occurs under all
> circumstances.
> 
> The problem file descriptors in the lsof output appear as:
> 
> java    47065  djb   46u  IPv4 0xffffff01354a4000      0t0       TCP *:*
> (CLOSED)
> java    47065  djb   47u  IPv4 0xffffff006b2875f0      0t0       TCP *:*
> (CLOSED)
> java    47065  djb   48u  IPv4 0xffffff0001bdcbe0      0t0       TCP *:*
> (CLOSED)
> java    47065  djb   49u  IPv4 0xffffff015924c000      0t0       TCP *:*
> (CLOSED)
> java    47065  djb   50u  IPv4 0xffffff01375078e8      0t0       TCP *:*
> (CLOSED)
> 
> I ran the application with -Xrunhprof enabled under jdk 1.6 and then used
> jhat to look at the resulting dump.
> 
> I managed to determine from this that the FD number (ie 46u) corresponds to
> the 'fd' field in a java.io.FileDescriptor instance.
> 
> Using the Object Query Language in jhat I went looking for these file
> descriptor instances using:
> 
> 'select f from java.io.FileDescriptor f where f.fd == ##'
> 
> For numbers which were genuine file descriptors (ie connected sockets etc)
> this would always find the file descriptor instance and walking through the
> referenced objects/fields it was clear it had located the correct file
> descriptor. By that I mean the connected address, or for actual files, the
> pointed to filename agreed with the lsof output.
> 
> If however I tried this query looking for file descriptor numbers which lsof
> was showing in the CLOSED state then it would fail to find them at all, so
> no FileDescriptor instance in the java heap referenced these file descriptor
> numbers.
> 
> To make sure I did a:
> 'select f.fd from java.io.FileDescriptor f' 
> 
> This listed the 'fd' field value for all instances of java.io.FileDescriptor
> in the heap, I had numbers for the ones expected (ie actual bound sockets,
> open files/pipes etc that lsof showed) as well as quite a few set to -1 and
> a lot set to 0. Presumably these last 2 are either newly created instances
> or ones awaiting garbage collection.
> 
> I've basically completely run out of ideas as to where these file
> descriptors are leaking, and given the working behaviour under diablo 1.5
> and the lack of references in the heap dump I'm no longer sure that this is
> actually a problem in my code.
> 
> Anyone more familiar with the JVM internals got any suggestions as to where
> to go next?

Hi Dominic,

My next step would be to create a minimal test case and post it here to see if
others can reproduce it. A minimal test case is also useful to narrow the scope
of what JVM code needs to be reviewed to find the cause. 

-Kurt