NFS reads vs. writes

[Mikhail T.]

On 03.01.2016 20:37, Rick Macklem wrote:
> This issue isn't new. It showed up when Sun introduced NFS in 1985.
> NFSv3 did change things a little, by allowing UNSTABLE writes.
Thank you very much, Rick, for the detailed explanation.
> If you use "sync=disabled"
>       (I'm not a ZFS guy, but I think that is what the ZFS option looks likes) you
>       *break* the NFS protocol (ie. violate the RFC) and put your data at some risk,
>       but you will typically get better (often much better) write performance.
Yes, indeed. Disabling sync got the writing throughput all the way up to
about 86Mb/s... I still don't fully understand, why local writes are
able to achieve this speed without async and without being considered
dangerous.
> Also, the NFS server was recently tweaked so that it could handle 128K rsize/wsize,
> but the FreeBSD client is limited to MAXBSIZE and this has not been increased
> beyond 64K.
I just tried lowering ZFS' recordsize to 64k to match MAXBSIZE, but that
didn't help NFS-writing (unless sync is disabled, that is).
> If this SSD is dedicated to the ZIL and is one known to have good write performance,
> it should help, but in your case the SSD seems to be the bottleneck.
It is a chunk of an older SSD, that also houses the OS. But it is
usually idle, because executables and libraries are cached in the
abundant RAM. I've seen it do 90+Mb/s (sequential)...

I just tried removing ZIL from the receiving pool -- to force direct
writes -- but it didn't help the case, where the writes go over NFS.
However, the local writes -- with reads from NFS -- went from the 56Mb/s
I was seeing earlier to 90Mb/s!..

There is got to be a better way to do this -- preferably, some
self-tuning smarts... Thanks again. Yours,

    -mi