bin/87792: very bad performance of cp(1) via NFS, possibly mmap() problem

[Andre Albsmeier]

>Number:         87792
>Category:       bin
>Synopsis:       very bad performance of cp(1) via NFS, possibly mmap() problem
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Oct 21 16:40:15 GMT 2005
>Closed-Date:
>Last-Modified:
>Originator:     Andre Albsmeier
>Release:        FreeBSD 5.4-STABLE i386
>Organization:
>Environment:

Two machines, a local one which copies files to a remote
machine via NFS using cp. Both machines use FreeBSD 5.4-STABLE.

>Description:

cp uses mmap() to copy files < 8MB. When files are copied
from the local host to a remote box via NFS data is transferred
very slowly. For bigger files, cp uses read() and write()
alternately which performs very well.

>How-To-Repeat:

I have two files, one is 6MB in size and the other one is 12MB.
I copy them from the local box to an NFS mounted directory named
/scratch:

andre at bali:/tmp>time cp 12mb /scratch
cp 12mb /scratch  0.00s user 0.12s system 10% cpu 1.142 total

andre at bali:/tmp>time cp 6mb /scratch
cp 6mb /scratch  0.00s user 0.13s system 0% cpu 14.274 total

The 12MB filecopy performs quite well as I would expect it
from a 100MBit network. The 6mb filecopy I won't comment on :-).
(This is an extreme case, other client-server combinations
are faster. The "fastest" one I found was 4 seconds which is
still bad for a 6MB file).

In the mmap() case, the whole source file is mmap'ed and
one write() call is used which tries to write the whole file
all at once. I have modified copy_file() in utils.c in a way
that instead of one "big" write(), several small ones are
being made (this is no fix, just for demonstration):

--- utils.c.ORI	Thu Aug 19 07:04:27 2004
+++ utils.c	Fri Oct 21 18:19:59 2005
@@ -140,7 +140,11 @@
 			wtotal = 0;
 			for (bufp = p, wresid = fs->st_size; ;
 			    bufp += wcount, wresid -= (size_t)wcount) {
+#if 0
 				wcount = write(to_fd, bufp, wresid);
+#else
+				wcount = write(to_fd, bufp, wresid > 65536 ? 65536 : wresid );
+#endif
 				wtotal += wcount;
 				if (info) {
 					info = 0;


65536 is just a value for playing; almost everything below
1M and not ridiculous small will do it:

andre at bali:/tmp>time cp 6mb /scratch 
cp 6mb /scratch  0.00s user 0.05s system 8% cpu 0.618 total

>Fix:

I have no idea. FreeBSD-4 uses the same mmap() thingy in utils.c
and here performance is ok so I assume the problem is somewhere
else than in the cp sources.
>Release-Note:
>Audit-Trail:
>Unformatted: