ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect

[Rick Macklem]

David Brodbeck wrote:
> I originally posted this on FreeBSD-questions, but it was suggested
> that I
> bring it here.
> I'm testing FreeBSD 9.0-BETA with an eye toward eventually using
> FreeBSD 9.0 to replace some existing OpenSolaris 2008.11
> installations. I've found NFS file creation performance (as measured
> by Bonnie++) is equally slow for both with default settings. However,
> on OpenSolaris I disable the ZIL to improve file creation performance.

I know nothing about ZFS, so all I can do is pass along what others
have said in previous posts. (I'd suggest you look through the freebsd-fs@
archives.)

One post explained how disabling the ZIL can result in up to 5seconds worth
of changes being lost if/when the server crashes. (A lot can change on a file
system in 5sec. The NFS protocol assumes all fs changes related to a file
creation are done before the server replies to the RPC. As such, disabling the
ZIL does violate the protocol specs and means you are living dangerously.)

> This tuning parameter was removed from FreeBSD 9.0; its replacement
> is supposed to be the per-filesystem flag "sync", but setting this
> flag seems to have no effect.
> 
> I did recompile the FreeBSD kernel without debugging features before
> doing the tests, so I don't think this is a case of debugging code
> slowing things down.
> 
> Here's the relevant data; these are all from bonnie++'s "sequential
> create" benchmark. The NFS client was RedHat Enterprise Linux 5.6.
> 
> OpenSolaris 2008.11, default settings: 58/second
> OpenSolaris 2008.11, with "zil_disable=1": 1258/second
> 
> FreeBSD 9.0-BETA, default settings: 107/second
> FreeBSD 9.0-BETA, with "sync=disabled": 106/second
> 
> 
> So it appears the "sync" ZFS parameter has no effect in FreeBSD. Has
> anyone else seen this? Is there a way to improve NFS file creation
> performance now that zil_disable has been removed?
> 
Some have reported good results from putting the ZIL on a dedicated
device. Some use an SSD, but there are write bandwidth issues related
to this. If I understood them correctly, you need to make sure that the
SSD is designed to provide good write performance and should be configured
to a much larger size than what the ZIL would use. Again, there were
good threads discussing this on freebsd-fs at . I have never used either
ZFS nor an SSD, so the above is just paraphrasing my understanding from
these threads and could be way off base. (I do know that NFS clients
expect the changes related to file creation to be stored on non-volatile
storage before replying to the creation RPC.)

> --
> David Brodbeck
> System Administrator, Linguistics
> University of Washington
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"