Throughput problems with NFS between Linux and FreeBSD

[Richard Sharpe]

Hi,

We recently encountered a problem with NFS throughput between a FreeBSD 
server (we are using 4.6.2, but the same code seems to be in 5.1 as well).

When using Linux 2.4.19 or 2.4.21 as a client, although this might extend 
to other clients, and copying a large file, you will see the behavior 
shown in 
http://www.richardsharpe.com/ethereal-stuff.html#Time%20Sequence%20Graphs

This happens because Linux hangs onto the ack for the last segment of a 
32kB+header send for a while. The FreeBSD NFS server will not put anymore 
data in the socket because of an soreserve with a size of 32kB+header, so 
it waits for about 39mS until Linux finally sends the ack for the last 
segment. (Unless there is data, like another command, going the other way, 
that is).

Throughput is about 3MB/s on GigE.

The problem seems to be the following code

        if (so->so_type == SOCK_STREAM)
                siz = NFS_MAXPACKET + sizeof (u_long);
        else
                siz = NFS_MAXPACKET;
        error = soreserve(so, siz, siz);

in src/sys/nfs/nfs_syscalls.c.

We added a sysctl to allow finer control over what is passed to soreserve.

With the fix in, it goes up to around wire speed when lots of data is in 
the cache.

This was found by Chandu Gadhiraju with help from others.

Regards
-----
Richard Sharpe, rsharpe[at]ns.aus.com, rsharpe[at]samba.org, 
sharpe[at]ethereal.com, http://www.richardsharpe.com