FreeBSD 9.1 NFSv4 client attribute cache not caching ?

Sun Apr 14 11:08:28 UTC 2013

On 14 Apr 2013, at 5:00 , Rick Macklem <rmacklem at uoguelph.ca> wrote:

Thanks for taking the effort to send such an extensive reply.

> Paul van der Zwan wrote:
>> On 12 Apr 2013, at 16:28 , Paul van der Zwan <paulz at vanderzwan.org>
>> wrote:
>> 
>>> 
>>> I am running a few VirtualBox VMs with 9.1 on my OpenIndiana server
>>> and I noticed that make buildworld seem to take much longer
>>> when the clients mount /usr/src and /usr/obj over NFS V4 than when
>>> they use V3.
>>> Unfortunately I have to use V4 as a buildworld on V3 hangs the
>>> server completely...
>>> I noticed the number of PUTFH/GETATTR/GETFH calls in in the order of
>>> a few thousand per second
>>> and if I snoop the traffic I see the same filenames appear over and
>>> over again.
>>> It looks like the client is not caching anything at all and using a
>>> server request everytime.
>>> I use the default mount options:
>>> 192.168.178.24:/data/ports on /usr/ports (nfs, nfsv4acls)
>>> 192.168.178.24:/data/src on /usr/src (nfs, nfsv4acls)
>>> 192.168.178.24:/data/obj on /usr/obj (nfs, nfsv4acls)
>>> 
>>> 
>> 
>> I had a look with dtrace
>> $ sudo dtrace -n '::getattr:start { @[stack()]=count();}'
>> and it seems the vast majority of the calls to getattr are from open()
>> and close() system calls.:
>> kernel`newnfs_request+0x631
>> kernel`nfscl_request+0x75
>> kernel`nfsrpc_getattr+0xbe
>> kernel`nfs_getattr+0x280
>> kernel`VOP_GETATTR_APV+0x74
>> kernel`nfs_lookup+0x3cc
>> kernel`VOP_LOOKUP_APV+0x74
>> kernel`lookup+0x69e
>> kernel`namei+0x6df
>> kernel`kern_execve+0x47a
>> kernel`sys_execve+0x43
>> kernel`amd64_syscall+0x3bf
>> kernel`0xffffffff80784947
>> 26
>> 
>> kernel`newnfs_request+0x631
>> kernel`nfscl_request+0x75
>> kernel`nfsrpc_getattr+0xbe
>> kernel`nfs_close+0x3e9
>> kernel`VOP_CLOSE_APV+0x74
>> kernel`kern_execve+0x15c5
>> kernel`sys_execve+0x43
>> kernel`amd64_syscall+0x3bf
>> kernel`0xffffffff80784947
>> 26
>> 
>> kernel`newnfs_request+0x631
>> kernel`nfscl_request+0x75
>> kernel`nfsrpc_getattr+0xbe
>> kernel`nfs_getattr+0x280
>> kernel`VOP_GETATTR_APV+0x74
>> kernel`nfs_lookup+0x3cc
>> kernel`VOP_LOOKUP_APV+0x74
>> kernel`lookup+0x69e
>> kernel`namei+0x6df
>> kernel`vn_open_cred+0x330
>> kernel`vn_open+0x1c
>> kernel`kern_openat+0x207
>> kernel`kern_open+0x19
>> kernel`sys_open+0x18
>> kernel`amd64_syscall+0x3bf
>> kernel`0xffffffff80784947
>> 2512
>> 
>> kernel`newnfs_request+0x631
>> kernel`nfscl_request+0x75
>> kernel`nfsrpc_getattr+0xbe
>> kernel`nfs_close+0x3e9
>> kernel`VOP_CLOSE_APV+0x74
>> kernel`vn_close+0xee
>> kernel`vn_closefile+0xff
>> kernel`_fdrop+0x3a
>> kernel`closef+0x332
>> kernel`kern_close+0x183
>> kernel`sys_close+0xb
>> kernel`amd64_syscall+0x3bf
>> kernel`0xffffffff80784947
>> 2530
>> 
>> I had a look at the source of nfs_close and could not find a call to
>> nfsrpc_getattr, and I am wondering why close would be calling getattr
>> anyway.
>> If the file is closed what do we care about it's attributes....
>> 
> Here are some random statements w.r.t. NFSv3 vs NFSv4 that might help
> with an understanding of what is going on. I do address the specific
> case of nfs_close() towards the end. (It is kinda long winded, but I
> threw out eveything I could think of..)
> 
> NFSv3 doesn't have any open/close RPC, but NFSv4 does have Open and
> Close operations.
> 
> In NFSv3, each RPC is defined and usually includes attributes for files
> before and after the operation (implicit getattrs not counted in the RPC
> counts reported by nfsstat).
> 
> For NFSv4, every RPC is a compound built up of a list of Operations like
> Getattr. Since the NFSv4 server doesn't know what the compound is doing,
> nfsstat reports the counts of Operations for the NFSv4 server, so the counts
> will be much higher than with NFSv3, but do not reflect the number of RPCs being done.
> To get NFSv4 nfsstat output that can be compared to NFSv3, you need to
> do the command on the client(s) and it still is only roughly the same.
> (I just realized this should be documented in man nfsstat.)
> 
I ran nfsstat -s -v 4 on the server and saw the number of requests being done.
They were in the order of a few thousand per second for a single FreeBSD 9.1 client 
doing a make build world.

> For the FreeBSD NFSv4 client, the compounds include Getattr operations
> similar to what NFSv3 does. It doesn't do a Getattr on the directory
> for Lookup, because that would have made the compound much more complex.
> I don't think this will have a significant performance impact, but will
> result in some additional Getattr RPCs.
> 
I ran snoop on port 2049 on the server and I saw a large number of lookups.
A lot of them seem to be for directories which are part of the filenames of
the compiler and include files which on the nfs mounted /usr/obj.
The same names keep reappering so it looks like there is no caching being done on 
the client.

> I suspect the slowness is caused by the extra overhead of doing the
> Open/Close operations against the server. The only way to avoid doing
> these against the server for NFSv4 is to enable delegations in both
> client and server. How to do this is documented in "man nfsv4". Basically
> starting up the nfscbd in the client and setting:
> vfs.nfsd.issue_delegations=1
> in the server.
> 
> Specifically for nfs_close(), the attributes (modify time)
> is used for what is called "close to open consistency". This can be
> disabled by the "nocto" mount option, if you don't need it for your
> build environment. (You only need it if one client is writing a file
> and then another client is reading the same file.)
> 
I tried the nocto option in /etc/fstab but it does not show when mount shows
the mounted filesystems so I am not sure if it is being used.
On the server netstat shows an active connection to port 7745 on the client
but snoop shows no data flowing on that session.

> Both the attribute caching and close to open consistency algorithms
> in the client are essentially the same for NFSv3 vs NFSv4.
> 
> The NFSv4 Close operation(s) are actually done when the v_usecount for
> the vnode goes to 0, since mmap'd files can do I/O on pages after
> the close syscall. As such, they are only loosely related to the close
> syscall. They are actually closing Windows style Openlock(s).
> 
I had a look at the code of the NFS v4 client of Illumos ( which is basically what
my server is running ) and as far as I  understand it they only do the gettatr only when
the close was for a file that  was opened for write and when there was actually something 
written to the file.
The FreeBSD code seems to do the getattr for all close() calls.
For files that were never written, like executables or source files that seems
to cause quite a lot of overhead.

> You mention that you see the same file over and over in a packet trace.
> You don't give specifics, but I'd suggest that you look at both NFSv3
> and NFSv4 for this (and file names are in lookups, not getattrs).
> 
> I'd suggest you try enabling delegations in both client and server, plus
> trying the "nocto" mount option and see if that helps.
> 
Tried it but it does not seem to make any noticable difference.

I tried a make buildworld buildkernel with /usr/obj a local FS in the Vbox VM
that completed in about 2 hours. With /usr/obj on an NFS v4 filesystem it takes
about a day. A twelve fold increase is elapsed time makes using NFSv4 unusable 
for this use case.
Too bad the server hangs when I use nfsv3 mount for /usr/obj.
Having a shared /usr/obj makes it possible to run a make buildworld on a single VM
and just run make installworld on the others.

	Paul

> rick
> 
>> 
>> Paul
>> 
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> 