NFS writes calling FSYNC and ASYNC not consistent

[Alfred Perlstein]

Apologies for being terse, in a hurry here.

1) -o async doesn't work with NFS, don't use that.
2) how big are the text versus binary files?
3) how are you copying them over nfs?

I suspect, (could be wrong of course) that the ascii files
are a lot smaller than the binary files, so what's happening
is that for binary files, the client is issuing write-behind
async, however for ascii files its issuing the writes at
close time which will force the sync flag.

-Alfred

* Brent Jones <brent at servuhome.net> [090128 19:38] wrote:
> Hello FreeBSD users,
> I am running into some performance problems with NFSv3/v4 mounts.
> I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS shares
> The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram
> and a FreeBSD 7.1 32bit with same hardware.
> 
> The issue I am seeing, is that for certain file types, the FreeBSD NFS
> client will either issue an ASYNC write, or an FSYNC.
> However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP
> versions of the protocol, so that should be the default.
> Issuing FSYNC's for every compete block transmitted adds substantial
> overhead and slows everything down.
> 
> The two test files I have that can reproduce this data are a file
> created by 'dump' which is just binary data:
> 
> $ file testbinery
> testbinery: data
> 
> ASCII text file from a Maildir format:
> 
> $ file ascittest
> ascittest: ASCII mail text
> 
> My NFS mount command lines I have tried to get all data to ASYNC write:
> 
> $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/
> $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/
> $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/
> 
> Here is an excerpt from a snoop from the binary data file:
> 
> $ snoop rpc nfs
> 
> obsmtp02.local -> pdxfilu01    NFS C ACCESS3 FH=57D3
> (read,lookup,modify,extend,delete,execute)
>    pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend)
> obsmtp02.local -> pdxfilu01    NFS C LOOKUP3 FH=BB85 testbinery
>    pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3
> obsmtp02.local -> pdxfilu01    NFS C ACCESS3 FH=57D3
> (read,lookup,modify,extend,delete,execute)
>    pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend)
> obsmtp02.local -> pdxfilu01    NFS C SETATTR3 FH=57D3
>    pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK
> obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=57D3 at 0 for 32768 (ASYNC)
>    pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
> obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=57D3 at 582647808 for
> 32768 (ASYNC)
>    pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
> obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=57D3 at 592871424 for
> 32768 (ASYNC)
>    pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
> obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=57D3 at 605421568 for
> 32768 (ASYNC)
>    pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
> 
> 
> And on and on.. it will acheive near full wire-speed, about 110MB/sec
> during the copy
> 
> 
> Here is the same snoop, only copying the ASCII mail file:
> 
> $ snoop rpc nfs
> 
>    obsmtp02.local -> pdxfilu01    NFS C LOOKUP3 FH=BB85 ascittest
>    pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory
> obsmtp02.local -> pdxfilu01    NFS C LOOKUP3 FH=BB85 ascittest
>    pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory
> obsmtp02.local -> pdxfilu01    NFS C CREATE3 FH=BB85 (UNCHECKED) ascittest
>    pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3
> obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=69D3 at 0 for 32768 (FSYNC)
>    pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC)
> obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=69D3 at 32768 for 32768 (FSYNC)
>    pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC)
> obsmtp02.local -> pdxfilu01    NFS C WRITE3 FH=69D3 at 65536 for 32768 (FSYNC)
>    pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC)
> 
> 
> And so on. I've reproduced this with several files, and the only
> difference between tests is the file type.
> Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the
> file type/contents?
> If so, is there a tuneable setting to make all write ASYNC?
> Otherwise, FSYNC'ing for every block written over NFS will cause so
> many IOPS on the NFS server, that performance will degrade severely.
> 
> Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for
> any file type, if mounted with NFSv3 of NFSv4 (TCP).
> 
> Any ideas?
> 
> Thanks in advance!
> 
> -- 
> Brent Jones
> brent at servuhome.net
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"

-- 
- Alfred Perlstein