sendfile returning ENOTCONN under heavy load

[Mike Silbersack]

On Fri, 26 Mar 2004, Kevin Day wrote:

> I'm using thttpd on a server that pushes 300-400mbps of static content,
> using sendfile(2).
>
> Once the load reaches a certain point (around 800-1000 clients
> downloading, anywhere from 150-250mbps), sendfile() will start randomly
> returning ENOTCONN, and the client is disconnected. I've raised
> kern.ipc.nsfbufs pretty high and that hasn't made any difference. Is
> there any easy way to tell exactly why the sockets are being closed? I
> can't seem to find any obvious signs of memory exhaustion or anything.
>
> Thanks,
>
> Kevin

The only place where I see sendfile returning ENOTCONN is:

        if ((so->so_state & SS_ISCONNECTED) == 0) {
                error = ENOTCONN;
                goto done;
        }

So presumably the connection was terminated at the TCP layer, not due to a
shortage in sfbufs.  Maybe you should set up your testbed that creates
this load, then set up one additional test computer.  Have that computer
tcpdump all of its traffic, hope that one of the dropped connections
happens to it, and see if you can find it in the dump.

Mike "Silby" Silbersack