Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect

[David Brodbeck]

On Wed, Sep 7, 2011 at 3:19 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote:

> Yea, I just tried a quick test using Fedora15 mounting the FreeBSD NFS
> server and then extracted a small tarball. Here's what the RPC counts
> were for NFSv3 and NFSv4 mounts (Open replaces Create for file creation
> in NFSv4):
>
> Client      Getattr Setattr Read Write Create Open
> Fedora-NFSv3  12      249     35   75    60     0
> Fedora-NFSv4 744      249     35   75     0    121
> FreeBSDNFSv3  64      189     35   75    60     0
> FreeBSDNFSv4 943      189     35   75     0     61
>
> Two obvious things:
> 1 - Fedora15 does 2 Opens for each create, for NFSv4.
> 2 - NFSv4 does a massive # of Getattrs, however most of these will
>    be in the compound RPCs with other Ops, so it doesn't mean
>    that these are creating a larger load on the server.
>    Put another way, the Getattrs done as a part of other RPCs
>    don't get counted for NFSv3, but they are separate Ops in the
>    NFSv4 compound and, therefore, do get counted.
> *** This was NOT a serious benchmark and I didn't bother to list all
>    the RPC counts. Do not read this as a serious performance comparison,
>    it was simply meant to get a hint as to why there was a difference
>    between NFSv3 and NFSv4 for this case.
>

I've also found this isn't just a Linux client problem.

I tested a FreeBSD-9.0 client against a FreeBSD-9.0 server.  Bonnie++ won't
run on FreeBSD NFS mounts due to PR kern/26142, so I hacked up a quick
script that times how long it takes to create 10,000 files.  I averaged
times over four runs; the backing filesystem was a UFS ramdisk.

FreeBSD client -> FreeBSD server, NFSv4: 95.43s
FreeBSD client -> FreeBSD server, NFSv3: 24.77s

So basically NFSv4 against a FreeBSD server is nearly 4x slower at doing
file creation than NFSv3, even when there's no disk latency involved.

I did the same test with Linux and the results were similar, although the
performance gap wasn't as wide:

Linux client -> FreeBSD server, NFSv4: 71.25s
Linux client -> FreeBSD server, NFSv3: 34.08s

Sorry, but I can't think of anything within the NFS server. It just does
> the operations that the client hands to it by performing the VOP_xxx()
> calls.
> I will take a look at a packet trace, but I doubt I'll learn anything
> useful
> from it.
>

Hm, okay.  Unfortunately this has a big impact on us, because file creation
speed happens to be one of the more user-visible aspects of server
performance on our system.

-- 
David Brodbeck
System Administrator, Linguistics
University of Washington