Low nfs write throughput

[John Baldwin]

On Thursday, November 17, 2011 10:10:27 pm Daryl Sayers wrote:
> 
> Can anyone suggest why I am getting poor write performance from my nfs 
setup.
> I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother 
boards,
> 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with
> onboard Gb network cards connected to an idle network. The results below 
show
> that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It
> improves if I use async but a smbfs mount still beats it. I am using the 
same
> file, source and destinations for all tests. I have tried alternate Network
> cards with no resulting benefit.
> 
> oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2
> 1950511+1 records in
> 1950511+1 records out
> 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec)
> 1950477+74 records in
> 1950511+1 records out
> 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec)     
(98Mb/s)
> 
> 
> oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt
> oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
> 7619+1 records in
> 7619+1 records out
> 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec)     
(15Mb/s)
> 
> 
> oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs 
/mnt
> oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
> 7619+1 records in
> 7619+1 records out
> 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec)     
(19Mb/s)
> 
> 
> oguido# mount -t smbfs //gemini/ufs /mnt
> oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
> 7619+1 records in
> 7619+1 records out
> 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec)     
(33Mb/s)
> 
> Looking at a systat -v on the destination I see that the nfs test does not
> exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t.
> For the record I get reads of 22Mb/s without and 77Mb/s with async turned on
> for the nfs mount.

I don't know if it will help with your performance, but I have some patches
to allow the NFS server to cluster writes.  You can try 
www.freebsd.org/~jhb/patches/nfs_server_cluster.patch.  I've tested it on 8, 
but it should probably apply fine to 9.

-- 
John Baldwin