NFS reads vs. writes

[Tom Curry]

On Mon, Jan 4, 2016 at 12:34 AM, Mikhail T. <mi+thun at aldan.algebra.com>
wrote:

> On 03.01.2016 20:37, Rick Macklem wrote:
> > This issue isn't new. It showed up when Sun introduced NFS in 1985.
> > NFSv3 did change things a little, by allowing UNSTABLE writes.
> Thank you very much, Rick, for the detailed explanation.
> > If you use "sync=disabled"
> >       (I'm not a ZFS guy, but I think that is what the ZFS option looks
> likes) you
> >       *break* the NFS protocol (ie. violate the RFC) and put your data
> at some risk,
> >       but you will typically get better (often much better) write
> performance.
> Yes, indeed. Disabling sync got the writing throughput all the way up to
> about 86Mb/s... I still don't fully understand, why local writes are
> able to achieve this speed without async and without being considered
> dangerous.
> > Also, the NFS server was recently tweaked so that it could handle 128K
> rsize/wsize,
> > but the FreeBSD client is limited to MAXBSIZE and this has not been
> increased
> > beyond 64K.
> I just tried lowering ZFS' recordsize to 64k to match MAXBSIZE, but that
> didn't help NFS-writing (unless sync is disabled, that is).
> > If this SSD is dedicated to the ZIL and is one known to have good write
> performance,
> > it should help, but in your case the SSD seems to be the bottleneck.
> It is a chunk of an older SSD, that also houses the OS. But it is
> usually idle, because executables and libraries are cached in the
> abundant RAM. I've seen it do 90+Mb/s (sequential)...
>
> I just tried removing ZIL from the receiving pool -- to force direct
> writes -- but it didn't help the case, where the writes go over NFS.
>

I assume you mean you removed the SLOG from the pool, in which case you
most definitely still have a ZIL, its now located on the pool itself.
Assuming you still have sync=standard I would venture a guess that writes
over NFS would now be measured in KB/s.


> However, the local writes -- with reads from NFS -- went from the 56Mb/s
> I was seeing earlier to 90Mb/s!..
>
> There is got to be a better way to do this -- preferably, some
> self-tuning smarts... Thanks again. Yours,
>
>
There is no getting around the performance impact of a synchronous
operation, whether its NFS or a database log. If you don't believe me hop
on your favorite Windows box, bring up the device manager and disable the
write cache on its drive then run some benchmark supporting sync writes.
One way to lessen the performance impact is to decrease the latency of
writes, which is why SSD SLOGs help so much. Which brings me to my next
point..

SSDs are so fast for three main reasons: low latency, large dram buffers,
and parallel workloads. Only one of these is of any benefit (latency) as a
SLOG. Unfortunately that particular metric is not usually advertised in
consumer SSDs where the benchmarks they use to tout 90,000 random write
iops consist of massively concurrent, highly compressible, short lived
bursts of data. Add that drive as a SLOG and the onboard dram may as well
not even exist, and queue depths count for nothing. It will be lucky to
pull 2,000 IOPS. Once you start adding in ZFS features like checksums and
compression, or network latency in the case of NFS that 2,000 number starts
to drop even more.

If you fancy a deeper understanding, this excellent article will take you
as far down the rabbit hole as you wish to go. It's very informative.

http://dtrace.org/blogs/brendan/2009/06/26/slog-screenshots/




>     -mi
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>