Poor throughput using new NFS client (9.0) vs. old (8.2/9.0)

[Thomas Johnson]

We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0. Almost
immediately, we received reports from users of poor performance. The
upgraded hosts are PXE-booted, with an NFS-mounted root. Additionally, they
mount a number of other NFS shares, which is where our users work from.
After a week of tweaking rsize/wsize/readahead parameters (per guidance),
it finally occurred to me that 9.0 defaults to the new NFS client and
server. I remounted the user shares using the oldnfs file type, and users
reported that performance returned to its expected level.

This is obviously a workaround, rather than a solution. We would prefer to
get our hosts using the newnfs client, since presumably oldnfs will be
deprecated at some point in the future. Is there some change that we should
have made to our NFS configuration with the upgrade to 9.0, or is it
possible that our workload is exposing some deficiency with newnfs? We tend
to deal with a huge number of tiny files (several KB in size). The NFS
server has been running 9.0 for some time (prior to the client upgrade)
without any issue. NFS is served from a zpool, backed by a Dell MD3000,
populated with 15k SAS disks. Clients and server are connected with Gig-E
links. The general hardware configuration has not changed in nearly 3 years.

As an example of the performance difference, here is some of the testing I
did while troubleshooting. Given a directory containing 5671 zip files,
with an average size of 15KB. I append all files to an existing zip file.
Using the newnfs mount, I found that this operation generally takes ~30
seconds (wall time). Switching the mount to oldnfs resulted in the same
operation taking ~10 seconds.

tom at test-1:/test-> ls 53*zip | wc -l
    5671
tom at test-1:/test-> ll -h BIG*
-rw-rw-r--  1 tom      claimlynx   8.9M Oct 17 14:06 BIGGER_PILE_1.zip
tom at test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip
0.646u 0.826s 0:51.01 2.8%    199+2227k 0+2769io 0pf+0w
...reset and repeat...
0.501u 0.629s 0:30.49 3.6%    208+2319k 0+2772io 0pf+0w
...reset and repeat...
0.601u 0.522s 0:32.37 3.4%    220+2406k 0+2771io 0pf+0w

tom at test-1:/-> cd /
tom at test-1:/-> sudo umount /test
tom at test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test
tom at test-1:/-> mount | grep test
server:/array/test on /test (oldnfs)
tom at test-1:/-> cd /test
...reset and repeat...
0.470u 0.903s 0:13.09 10.4%    203+2229k 0+5107io 0pf+0w
...reset and repeat...
0.547u 0.640s 0:08.65 13.6%    231+2493k 0+5086io 0pf+0w
tom at test-1:/test-> ll -h BIG*
-rw-rw-r--  1 tom      claimlynx    92M Oct 17 14:14 BIGGER_PILE_1.zip

Thanks!

-- 
Thomas Johnson
ClaimLynx, Inc.