Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file
creation performance, "sync=disabled" has no effect
Rick Macklem
rmacklem at uoguelph.ca
Wed Sep 7 22:19:37 UTC 2011
David Brodbeck wrote:
> On Mon, Sep 5, 2011 at 2:16 AM, Pawel Jakub Dawidek <pjd at freebsd.org>
> wrote:
>
> > It would be good if you could try your test on FreeBSD directly.
> > This
> > way we could see if ZFS is to blame or NFS.
> >
>
> Okay, it appears it is in fact an NFSv4 problem, and I've been barking
> up
> the wrong tree. I shared out an NFSv4 mountpoint with a UFS ramdisk as
> the
> backing store, and I got the same slow results. I was fooled by the
> fact
> that the numbers were similar to what I got using OpenSolaris with the
> ZIL
> enabled. (This is what I get for making assumptions.)
>
> Interestingly enough, if I use NFSv3 instead of NFSv4, performance
> increases
> dramatically.
>
Yea, I just tried a quick test using Fedora15 mounting the FreeBSD NFS
server and then extracted a small tarball. Here's what the RPC counts
were for NFSv3 and NFSv4 mounts (Open replaces Create for file creation
in NFSv4):
Client Getattr Setattr Read Write Create Open
Fedora-NFSv3 12 249 35 75 60 0
Fedora-NFSv4 744 249 35 75 0 121
FreeBSDNFSv3 64 189 35 75 60 0
FreeBSDNFSv4 943 189 35 75 0 61
Two obvious things:
1 - Fedora15 does 2 Opens for each create, for NFSv4.
2 - NFSv4 does a massive # of Getattrs, however most of these will
be in the compound RPCs with other Ops, so it doesn't mean
that these are creating a larger load on the server.
Put another way, the Getattrs done as a part of other RPCs
don't get counted for NFSv3, but they are separate Ops in the
NFSv4 compound and, therefore, do get counted.
*** This was NOT a serious benchmark and I didn't bother to list all
the RPC counts. Do not read this as a serious performance comparison,
it was simply meant to get a hint as to why there was a difference
between NFSv3 and NFSv4 for this case.
So, my best guess is that doing twice as many Opens (vs NFSv3 Create RPCs)
is why you see the difference. I'd also guess that disabling the ZIL on
ZFS made the Opens perform faster, which helped alleviate the problem.
(An NFSv4 Open operation will end up doing both VOP_GETATTR() and
VOP_CREATE(). It will also do a VOP_SETATTR() for the exclusive open
case.)
> It appears it's FreeBSD's NFSv4 server, or the way it's interacting
> with
> Linux's client, that's the culprit here. I'm not sure if there are
> knobs I
> should be tweaking to make it perform better; any suggestions on what
> to try
> next?
>
Sorry, but I can't think of anything within the NFS server. It just does
the operations that the client hands to it by performing the VOP_xxx() calls.
I will take a look at a packet trace, but I doubt I'll learn anything useful
from it.
rick
> --
> David Brodbeck
> System Administrator, Linguistics
> University of Washington
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
More information about the freebsd-fs
mailing list