scp more perfectly fills the pipe than NFS/TCP

[Zaphod Beeblebrox]

On Sun, Dec 20, 2009 at 12:27 AM, Dan Nelson <dnelson at allantgroup.com> wrote:
> In the last episode (Dec 19), Zaphod Beeblebrox said:
>> Here's an interesting conundrum.  I don't know what's different between
>> the TCP that scp uses from the TCP that NFS uses, but given the same two
>> FreeBSD machines, SCP fills the pipe with packets better.
>>
>> Examine the following graphic: http://www.eicat.ca/~dgilbert/example-mrtg.png
>>
>> The system doing the scp and the NFS server is FreeBSD-7.2-p1.  The system
>> receiving the scp and the NFS client is FreeBSD-8.0-p1
>>
>> The scp transfer is the left hand side of the graph and the NFS transfer
>> is on the right.
>>
>> The NFS is mounted with "-3 -T -b -l -i" and no other options.  Files are
>> being moved over NFS with the system "mv" command.  The files in each case
>> are large (50 to 500 meg files).
>
> If you increase the NFS blocksize (-r 32768 for example) you will get
> slightly better performance, but you will likely never match the scp
> results.  They're doing two different things under the hood: scp is
> streaming the entire file in one operation, while NFS is performing many
> "read 8k at offset 0", "read 8k at offset 8k", etc requests one after
> another, so a high-latency connection will take a performance hit due to the
> latency in issuing each command.  According to the mount_nfs manpage, it
> looks like there is some prefetching that can be enabled with the "-a ##"
> option.  It doesn't say what the default is, though.

While the link is slow, it is really directly connected with a latency
of 10ms or so.  Isn't mv mmap()'ing large enough regions to cause
there to be a reasonable queue to transfer?