Why is NFSv4 so slow?

[Rick C. Petty]

First off, many thanks to Rick Macklem for making NFSv4 possible in
FreeBSD!

I recently updated my NFS server and clients to v4, but have since noticed
significant performance penalties.  For instance, when I try "ls a b c" (if
a, b, and c are empty directories) on the client, it takes up to 1.87
seconds (wall time) whereas before it always finished in under 0.1 seconds.
If I repeat the test, it takes the same amount of time in v4 (in v3, wall
time was always under 0.01 seconds for subsequent requests, as if the
directory listing was cached).

If I try to play an h264 video file on the filesystem using mplayer, it
often jitters and skipping around in time introduces up to a second or so
pause.  With NFSv3 it behaved more like the file was on local disk (no
noticable pauses or jitters).

Has anyone seen this behavior upon switching to v4 or does anyone have any
suggestions for tuning?

Both client and server are running the same GENERIC kernel, 8.1-PRERELEASE
as of 2010-May-29.  They are connected via gigabit.  Both v3 and v4 tests
were performed on the exact same hardware and I/O, CPU, network loads.
All I did was toggle nfsv4_server_enable (and nfsuserd/nfscbd of course).

It seems like a server-side issue, because if I try an nfs3 client mount
to the nfs4 server and run the same tests, I see only a slight improvement
in performance.  In both cases, my mount options were
"rdirplus,bg,intr,soft" (and "nfsv4" added in the one case, obviously).

On the server, I have these tunables explicitly set:

kern.ipc.maxsockbuf=524288
vfs.newnfs.issue_delegations=1

On the client, I just have the maxsockbuf setting (this is twice the
default value).  I'm open to trying other tunables or patches.  TIA,

-- Rick C. Petty