Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect

[Rick Macklem]

David Brodbeck wrote:
> On Wed, Sep 7, 2011 at 3:19 PM, Rick Macklem < rmacklem at uoguelph.ca >
> wrote:
> 
> 
> 
> Yea, I just tried a quick test using Fedora15 mounting the FreeBSD NFS
> server and then extracted a small tarball. Here's what the RPC counts
> were for NFSv3 and NFSv4 mounts (Open replaces Create for file
> creation
> in NFSv4):
> 
> Client Getattr Setattr Read Write Create Open
> Fedora-NFSv3 12 249 35 75 60 0
> Fedora-NFSv4 744 249 35 75 0 121
> FreeBSDNFSv3 64 189 35 75 60 0
> FreeBSDNFSv4 943 189 35 75 0 61
> 
> Two obvious things:
> 1 - Fedora15 does 2 Opens for each create, for NFSv4.
> 2 - NFSv4 does a massive # of Getattrs, however most of these will
> be in the compound RPCs with other Ops, so it doesn't mean
> that these are creating a larger load on the server.
> Put another way, the Getattrs done as a part of other RPCs
> don't get counted for NFSv3, but they are separate Ops in the
> NFSv4 compound and, therefore, do get counted.
> *** This was NOT a serious benchmark and I didn't bother to list all
> the RPC counts. Do not read this as a serious performance comparison,
> it was simply meant to get a hint as to why there was a difference
> between NFSv3 and NFSv4 for this case.
> 
> 
> 
> I've also found this isn't just a Linux client problem.
> 
> 
> I tested a FreeBSD-9.0 client against a FreeBSD-9.0 server. Bonnie++
> won't run on FreeBSD NFS mounts due to PR kern/26142, so I hacked up a
> quick script that times how long it takes to create 10,000 files. I
> averaged times over four runs; the backing filesystem was a UFS
> ramdisk.
> 
> 
> FreeBSD client -> FreeBSD server, NFSv4: 95.43s
> FreeBSD client -> FreeBSD server, NFSv3: 24.77s
> 
> 
> So basically NFSv4 against a FreeBSD server is nearly 4x slower at
> doing file creation than NFSv3, even when there's no disk latency
> involved.
> 
> 
> I did the same test with Linux and the results were similar, although
> the performance gap wasn't as wide:
> 
> 
> Linux client -> FreeBSD server, NFSv4: 71.25s
> Linux client -> FreeBSD server, NFSv3: 34.08s
> 
Yep, doesn't surprise me that the FreeBSD client generates the same
result. Bottom line is that an NFSv4 Open (especially the exclusive
open variant) is much more complex than an NFSv3 Create RPC. I showed
how many Ops are in the Linux NFSv4 Open in the last post.

Basically, unless you need the better byte range locking or ACLs, you can just
use NFSv3. I don't think NFSv4 will ever replace NFSv3. It was designed
for a somewhat different problem space.
> 
> 
> Sorry, but I can't think of anything within the NFS server. It just
> does
> the operations that the client hands to it by performing the VOP_xxx()
> calls.
> I will take a look at a packet trace, but I doubt I'll learn anything
> useful
> from it.
> 
> 
> 
> Hm, okay. Unfortunately this has a big impact on us, because file
> creation speed happens to be one of the more user-visible aspects of
> server performance on our system.
> 
> --
> 
> David Brodbeck
> System Administrator, Linguistics
> University of Washington