Throughput problems with NFS between Linux and FreeBSD
rsharpe at richardsharpe.com
Fri Sep 19 10:33:21 PDT 2003
We recently encountered a problem with NFS throughput between a FreeBSD
server (we are using 4.6.2, but the same code seems to be in 5.1 as well).
When using Linux 2.4.19 or 2.4.21 as a client, although this might extend
to other clients, and copying a large file, you will see the behavior
This happens because Linux hangs onto the ack for the last segment of a
32kB+header send for a while. The FreeBSD NFS server will not put anymore
data in the socket because of an soreserve with a size of 32kB+header, so
it waits for about 39mS until Linux finally sends the ack for the last
segment. (Unless there is data, like another command, going the other way,
Throughput is about 3MB/s on GigE.
The problem seems to be the following code
if (so->so_type == SOCK_STREAM)
siz = NFS_MAXPACKET + sizeof (u_long);
siz = NFS_MAXPACKET;
error = soreserve(so, siz, siz);
We added a sysctl to allow finer control over what is passed to soreserve.
With the fix in, it goes up to around wire speed when lots of data is in
This was found by Chandu Gadhiraju with help from others.
Richard Sharpe, rsharpe[at]ns.aus.com, rsharpe[at]samba.org,
More information about the freebsd-hackers