Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect

Rick Macklem rmacklem at uoguelph.ca
Wed Sep 7 22:19:37 UTC 2011


David Brodbeck wrote:
> On Mon, Sep 5, 2011 at 2:16 AM, Pawel Jakub Dawidek <pjd at freebsd.org>
> wrote:
> 
> > It would be good if you could try your test on FreeBSD directly.
> > This
> > way we could see if ZFS is to blame or NFS.
> >
> 
> Okay, it appears it is in fact an NFSv4 problem, and I've been barking
> up
> the wrong tree. I shared out an NFSv4 mountpoint with a UFS ramdisk as
> the
> backing store, and I got the same slow results. I was fooled by the
> fact
> that the numbers were similar to what I got using OpenSolaris with the
> ZIL
> enabled. (This is what I get for making assumptions.)
> 
> Interestingly enough, if I use NFSv3 instead of NFSv4, performance
> increases
> dramatically.
> 
Yea, I just tried a quick test using Fedora15 mounting the FreeBSD NFS
server and then extracted a small tarball. Here's what the RPC counts
were for NFSv3 and NFSv4 mounts (Open replaces Create for file creation
in NFSv4):

Client      Getattr Setattr Read Write Create Open
Fedora-NFSv3  12      249     35   75    60     0
Fedora-NFSv4 744      249     35   75     0    121
FreeBSDNFSv3  64      189     35   75    60     0
FreeBSDNFSv4 943      189     35   75     0     61

Two obvious things:
1 - Fedora15 does 2 Opens for each create, for NFSv4.
2 - NFSv4 does a massive # of Getattrs, however most of these will
    be in the compound RPCs with other Ops, so it doesn't mean
    that these are creating a larger load on the server.
    Put another way, the Getattrs done as a part of other RPCs
    don't get counted for NFSv3, but they are separate Ops in the
    NFSv4 compound and, therefore, do get counted.
*** This was NOT a serious benchmark and I didn't bother to list all
    the RPC counts. Do not read this as a serious performance comparison,
    it was simply meant to get a hint as to why there was a difference
    between NFSv3 and NFSv4 for this case.

So, my best guess is that doing twice as many Opens (vs NFSv3 Create RPCs)
is why you see the difference. I'd also guess that disabling the ZIL on
ZFS made the Opens perform faster, which helped alleviate the problem.
(An NFSv4 Open operation will end up doing both VOP_GETATTR() and
 VOP_CREATE(). It will also do a VOP_SETATTR() for the exclusive open
 case.)

> It appears it's FreeBSD's NFSv4 server, or the way it's interacting
> with
> Linux's client, that's the culprit here. I'm not sure if there are
> knobs I
> should be tweaking to make it perform better; any suggestions on what
> to try
> next?
> 
Sorry, but I can't think of anything within the NFS server. It just does
the operations that the client hands to it by performing the VOP_xxx() calls.
I will take a look at a packet trace, but I doubt I'll learn anything useful
from it.

rick
> --
> David Brodbeck
> System Administrator, Linguistics
> University of Washington
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"


More information about the freebsd-fs mailing list