Strange NFS problem implicating nfsuserd?

[Graham Allan]

I spent a few days digging into a strange NFSv4 problem at our site, 
which I think I may have finally resolved but don't really understand why.

We have a bunch of large-ish NFS servers running FreeBSD 9.3 exporting 
ZFS filesystems to mostly "RHEL-clone" linux clients. Over the last few 
weeks I started getting reports that peoples' jobs would fail 
erratically with i/o errors, and it became apparent that they pointed in 
general to all our FreeBSD NFS servers rather than just one.

Ultimately I could trivially reproduce the problem running
   "find . -type f -exec cat {} > /dev/null \;"
on one of the NFS-mounted filesystems.

Linux clients would eventually error with "Input/output error"
FreeBSD clients would eventually error with "Permission denied" on files 
or directories which should be readable.

Reverting to earlier patch releases didn't make any difference, though 
it seemed like the problem started roughly when I updated p8->p13.

Finally I seem to have pinpointed it to one change made in rc.conf for 
nfsuserd, which I committed at around the right date:

nfsuserd_flags="-usermax 500 -usertimeout 600 16"

became:

nfsuserd_flags="-domain xxx.yyy.zzz -usermax 500 -usertimeout 600 16"

probably because I saw a user mapping failure somewhere previously, and 
decided to make the domain explicit.

Undoing this change appears to eliminate the problem - but this makes no 
sense to me. Starting nfsuserd with either set of options (adding 
-verbose) prints the same output:

Starting nfsuserd.
nfsuserd: domain=xxx.yyy.zzz usermax=500 usertimeout=36000

So the domain chosen by default is the same as the one explicitly 
specified (as I would expect).

I've reproduced this across 4-5 different servers and a similar number 
of different client systems. I'm wondering if any plausible explanation 
suggests itself?

Graham
--