java.net.SocketException: Too many open files

[Nick Gieczewski]

Hello folks,

My FreeBSD 4.8-STABLE server is running a Java server application that has recently had an increase in concurrent users. The JDK is jdk-1.4.2p5. Since the user surge, the main thread has been dying repeatedly during peak hours at ServerSocket.accept() with the following message:

Exception in thread "main" java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)

Now, here comes the weird stuff: My system is _not_ running out of file descriptors when this happens (kern.maxfiles = 16384, kern.openfiles = ~630), _nor_ is the per-process file descriptor limit being reached (it's set to unlimited). The heap is not full, either; its maximum size is set to 128 MB and it's under 60 MB when the thread dies. And there's plenty of both physical and virtual memory left.

I have written a small program that attempts to open 2,048 concurrent connections to the server in oder to reproduce this. On my FreeBSD box, the thread dies with the above exception at 500-600 open connections. When running the server application on Win32, on the other hand, I'm able to open all the 2,048 connections without a hitch.

In a nutshell, I'm getting a "Too many open files" error message but the message appears to be lying. The system-wide file descriptor limit is far from being reached, the per-process limit can never be reached, the heap is far from being full, and there's a reasonable amount of free physical and virtual memory. And the problem does not occur when the server application is run on the Win32 version of the JDK.

Has anyone run into this problem before, or does anyone know of a fix or whether it's been fixed in jdk-1.4.2p6? I could just go ahead and compile p6, but it would take many hours and I can't stress the old server that much while running this service.

Any input will be greatly appreciated.

Thanks,

Nick