Why is NFSv4 so slow?

Rick Macklem rmacklem at uoguelph.ca
Tue Jun 29 16:14:11 UTC 2010



On Mon, 28 Jun 2010, Rick C. Petty wrote:

>>>
>> It would be interesting to see if the performance problem exists for
>> NFSv3 mounts against the experimental (nfsv4) server.
>
> Hmm, I couldn't reproduce the problem.  Once I unmounted the nfsv4 client
> and tried v3, the jittering stopped.  Then I unmounted v3 and tried v4
> again, no jitters.  I played with a couple of combinations back and forth
> (toggling the presence of "nfsv4" in the options) and sometimes I saw
> jittering but only with v4, but nothing like what I was seeing before.
> Perhaps this is a result of Jeremy's TCP tuning tweaks.
>
> This is also a difficult thing to test, because the server and client have
> so much memory, they cache the date blocks.  So if I try my stutter test
> on the same video a second time, I only notice stutters if I skip to parts
> I haven't skipped to before.  I can comment that it seemed like more of a
> latency issue than a throughput issue to me.  But the disks aren't ever
> under a high load.  But it's hard to determine accurate load when the
> disks are seeking.  Oh, I'm using the AHCI controller mode/driver on those
> disks instead of ATA, if that matters.
>

I basically don't have a clue what might be the source of the problem. I
do agree that it sounds like an intermittent latency issue.

The only thing I can think of that you might try is simply increasing
the number of nfsd threads on the server. They don't add much overhead
and the default of '4'is pretty small. (It's just the "-n N" option on
nfsd, just in case you weren't aware of it.)

> One time when I mounted the v4 again, it broke subdirectories like I was
> talking about before.  Essentially it would give me a readout of all the
> top-level directories but wouldn't descend into subdirectories which
> reflect different mountpoints on the server.  An unmount and a remount
> (without changes to /etc/fstab) fixed the problem.  I'm wondering if there
> isn't some race condition that seems to affect crossing mountpoints on the
> server.  When the situation happens, it affects all mountpoints equally
> and persists for the duration of that mount.  And of course, I can't
> reproduce the problem when I try.
>

If it happened for a "hard mount" (no "soft,intr" mount options) then it
is a real bug. The server mount point crossings are detected via a change
in the value of the fsid attribute. I suspect that under some 
circumstances, the wrong value of fsid is getting cached in the client.
(I just remembered that you use "rdirplus" and it "might" not be caching
the server's notion of fsid in the right place.)

If you were really keen (if you ever look up "keen" in Webster's, it's
not what we tend to use it for at all. It was actually a "wail for the 
dead" and a keener was a professional wailer for the dead, hired for
funerals of important but maybe not that well liked individuals. But I
digress...), you could try a bunch of mounts./dismounts without "rdirplus"
and see if you can even get it to fail without the option.

> I saw the broken mountpoint crossing on another client (without any TCP
> tuning) but each time it happened I saw this in the logs:
>
> 	nfscl: consider increasing kern.ipc.maxsockbuf
>
> Once I doubled that value, the problem went away..  at least with this
> particular v4 server mountpoint.
>

If this had any effect, it was probably timing/latency related to a bug
w.r.t. caching of the server's notion of fsid. I'll poke around and see
if I can spot where this might be broken.

> At the moment, things are behaving as expected.  The v4 file system seems
> just as fast as v3 did, and I don't need a dozen mountpoints specified
> on each client thanks to v4.  Once again, I thank you, Rick, for all your
> hard work!
>
Btw, if the mountpoint crossing bug gets too irritating, you can do the
multiple mounts for NFSv4 just like NFSv3. (That's what you have to do
do for the Solaris10 NFSv4 client, because its completely broken w.r.t.
mountpoint crossings.)

rick




More information about the freebsd-stable mailing list