NFS writes calling FSYNC and ASYNC not consistent

[Brent Jones]

Hello FreeBSD users,
I am running into some performance problems with NFSv3/v4 mounts.
I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS shares
The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram
and a FreeBSD 7.1 32bit with same hardware.

The issue I am seeing, is that for certain file types, the FreeBSD NFS
client will either issue an ASYNC write, or an FSYNC.
However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP
versions of the protocol, so that should be the default.
Issuing FSYNC's for every compete block transmitted adds substantial
overhead and slows everything down.

The two test files I have that can reproduce this data are a file
created by 'dump' which is just binary data:

$ file testbinery
testbinery: data

ASCII text file from a Maildir format:

$ file ascittest
ascittest: ASCII mail text

My NFS mount command lines I have tried to get all data to ASYNC write:

$ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/
$ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/
$ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/

Here is an excerpt from a snoop from the binary data file:

$ snoop rpc nfs

obsmtp02.local -> pdxfilu01    NFS C ACCESS3 FH=57D3
(read,lookup,modify,extend,delete,execute)
   pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend)
obsmtp02.local -> pdxfilu01    NFS C LOOKUP3 FH=BB85 testbinery
   pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3
obsmtp02.local -> pdxfilu01    NFS C ACCESS3 FH=57D3
(read,lookup,modify,extend,delete,execute)
   pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend)
obsmtp02.local -> pdxfilu01    NFS C SETATTR3 FH=57D3
   pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK
obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=57D3 at 0 for 32768 (ASYNC)
   pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=57D3 at 582647808 for
32768 (ASYNC)
   pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=57D3 at 592871424 for
32768 (ASYNC)
   pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=57D3 at 605421568 for
32768 (ASYNC)
   pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)


And on and on.. it will acheive near full wire-speed, about 110MB/sec
during the copy


Here is the same snoop, only copying the ASCII mail file:

$ snoop rpc nfs

   obsmtp02.local -> pdxfilu01    NFS C LOOKUP3 FH=BB85 ascittest
   pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory
obsmtp02.local -> pdxfilu01    NFS C LOOKUP3 FH=BB85 ascittest
   pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory
obsmtp02.local -> pdxfilu01    NFS C CREATE3 FH=BB85 (UNCHECKED) ascittest
   pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3
obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=69D3 at 0 for 32768 (FSYNC)
   pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC)
obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=69D3 at 32768 for 32768 (FSYNC)
   pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC)
obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=69D3 at 65536 for 32768 (FSYNC)
   pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC)


And so on. I've reproduced this with several files, and the only
difference between tests is the file type.
Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the
file type/contents?
If so, is there a tuneable setting to make all write ASYNC?
Otherwise, FSYNC'ing for every block written over NFS will cause so
many IOPS on the NFS server, that performance will degrade severely.

Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for
any file type, if mounted with NFSv3 of NFSv4 (TCP).

Any ideas?

Thanks in advance!

-- 
Brent Jones
brent at servuhome.net