Low nfs write throughput

[Jeremy Chadwick]

On Tue, Nov 29, 2011 at 10:36:44AM -0500, John Baldwin wrote:
> On Monday, November 28, 2011 7:12:39 pm Daryl Sayers wrote:
> > >>>>> "Bengt" == Bengt Ahlgren <bengta at sics.se> writes:
> > 
> > > Daryl Sayers <daryl at ci.com.au> writes:
> > >> Can anyone suggest why I am getting poor write performance from my nfs setup.
> > >> I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards,
> > >> 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with
> > >> onboard Gb network cards connected to an idle network. The results below show
> > >> that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It
> > >> improves if I use async but a smbfs mount still beats it. I am using the same
> > >> file, source and destinations for all tests. I have tried alternate Network
> > >> cards with no resulting benefit.
> > 
> > > [...]
> > 
> > >> Looking at a systat -v on the destination I see that the nfs test does not
> > >> exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t.
> > >> For the record I get reads of 22Mb/s without and 77Mb/s with async turned on
> > >> for the nfs mount.
> > 
> > > On an UFS filesystem you get NFS writes with the same size as the
> > > filesystem blocksize.  So an easy way to improve performance is to
> > > create a filesystem with larger blocks.  I accidentally found this out
> > > when I had two NFS exported filesystems from the same box with 16K and
> > > 64K blocksizes respectively.
> > 
> > > (Larger blocksize also tremendously improves the performance of UFS
> > > snapshots!)
> > 
> > Thanks to all that answered. I did try the 'sysctl -w vfs.nfsrv.async=1' with
> > no reportable change in performance. We are using a UFS2 filesystem so the
> > zfs command was not required. I did not try the patch as we would like to stay
> > as standard as possible but will upgrade if the patch is released in new
> > kernel.
> 
> If you can test the patch then it is something I will likely put into the
> next release.  I have already tested it as far as robustness locally, what
> I don't have are good performance tests.  It would really be helpful if you
> were able to test it.

John,

We'd like to test this patch[1], but need to know if it needs to be
applied to just the system acting as the NFS server, or the NFS clients
as well.

[1]: http://www.freebsd.org/~jhb/patches/nfs_server_cluster.patch

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |