scp more perfectly fills the pipe than NFS/TCP

[Dan Nelson]

In the last episode (Dec 21), Zaphod Beeblebrox said:
> On Sun, Dec 20, 2009 at 12:27 AM, Dan Nelson <dnelson at allantgroup.com> wrote:
> > In the last episode (Dec 19), Zaphod Beeblebrox said:
> >> Here's an interesting conundrum.  I don't know what's different between
> >> the TCP that scp uses from the TCP that NFS uses, but given the same
> >> two FreeBSD machines, SCP fills the pipe with packets better.
> >>
> >> Examine the following graphic: http://www.eicat.ca/~dgilbert/example-mrtg.png
> >>
> >> The system doing the scp and the NFS server is FreeBSD-7.2-p1.  The
> >> system receiving the scp and the NFS client is FreeBSD-8.0-p1
> >>
> >> The scp transfer is the left hand side of the graph and the NFS
> >> transfer is on the right.
> >>
> >> The NFS is mounted with "-3 -T -b -l -i" and no other options.  Files
> >> are being moved over NFS with the system "mv" command.   The files in
> >> each case are large (50 to 500 meg files).
> >
> > If you increase the NFS blocksize (-r 32768 for example) you will get
> > slightly better performance, but you will likely never match the scp
> > results.   They're doing two different things under the hood: scp is
> > streaming the entire file in one operation, while NFS is performing many
> > "read 8k at offset 0", "read 8k at offset 8k", etc requests one after
> > another, so a high-latency connection will take a performance hit due to
> > the latency in issuing each command.   According to the mount_nfs
> > manpage, it looks like there is some prefetching that can be enabled
> > with the "-a ##" option.   It doesn't say what the default is, though.
> 
> While the link is slow, it is really directly connected with a latency
> of 10ms or so.  Isn't mv mmap()'ing large enough regions to cause
> there to be a reasonable queue to transfer?

I've never been impressed with FreeBSD's ability to detect sequential read
patterns and prefetch blocks ahead of time, even on local ufs filesystems.

-- 
	Dan Nelson
	dnelson at allantgroup.com