Poor throughput using new NFS client (9.0) vs. old (8.2/9.0)

[Rick Macklem]

Ronald Klop wrote:
> On Thu, 18 Oct 2012 18:16:16 +0200, Thomas Johnson <tom at claimlynx.com>
> wrote:
> 
> > We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0.
> > Almost
> > immediately, we received reports from users of poor performance. The
> > upgraded hosts are PXE-booted, with an NFS-mounted root.
> > Additionally,
> > they
> > mount a number of other NFS shares, which is where our users work
> > from.
> > After a week of tweaking rsize/wsize/readahead parameters (per
> > guidance),
> > it finally occurred to me that 9.0 defaults to the new NFS client
> > and
> > server. I remounted the user shares using the oldnfs file type, and
> > users
> > reported that performance returned to its expected level.
> >
> > This is obviously a workaround, rather than a solution. We would
> > prefer
> > to
> > get our hosts using the newnfs client, since presumably oldnfs will
> > be
> > deprecated at some point in the future. Is there some change that we
> > should
> > have made to our NFS configuration with the upgrade to 9.0, or is it
> > possible that our workload is exposing some deficiency with newnfs?
> > We
> > tend
> > to deal with a huge number of tiny files (several KB in size). The
> > NFS
> > server has been running 9.0 for some time (prior to the client
> > upgrade)
> > without any issue. NFS is served from a zpool, backed by a Dell
> > MD3000,
> > populated with 15k SAS disks. Clients and server are connected with
> > Gig-E
> > links. The general hardware configuration has not changed in nearly
> > 3
> > years.
> >
> > As an example of the performance difference, here is some of the
> > testing
> > I
> > did while troubleshooting. Given a directory containing 5671 zip
> > files,
> > with an average size of 15KB. I append all files to an existing zip
> > file.
> > Using the newnfs mount, I found that this operation generally takes
> > ~30
> > seconds (wall time). Switching the mount to oldnfs resulted in the
> > same
> > operation taking ~10 seconds.
> >
> > tom at test-1:/test-> ls 53*zip | wc -l
> >     5671
> > tom at test-1:/test-> ll -h BIG*
> > -rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip
> > tom at test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip
> > 0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w
> > ...reset and repeat...
> > 0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w
> > ...reset and repeat...
> > 0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w
> >
> > tom at test-1:/-> cd /
> > tom at test-1:/-> sudo umount /test
> > tom at test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test
> > tom at test-1:/-> mount | grep test
> > server:/array/test on /test (oldnfs)
> > tom at test-1:/-> cd /test
> > ...reset and repeat...
> > 0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w
> > ...reset and repeat...
> > 0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w
> > tom at test-1:/test-> ll -h BIG*
> > -rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip
> >
> > Thanks!
> >
> 
> 
> You might find this thread from today interesting.
> http://lists.freebsd.org/pipermail/freebsd-fs/2012-October/015441.html
> 
There are a couple of patches you guys want to have, if you are using
an NFS server on FreeBSD9.0. However, I'm not convinced either of them would cause
the performance issues reported, although the second one would result
in a constipated server after a while.

For the server:
- http://people.freebsd.org/~rmacklem/nfsd-enoent.patch
  Without this patch, there can be
  spurious ENOENT replies to lookups, because ni_topdir wasn't initialized.
- http://people.freebsd.org/~rmacklem/namei-leak.patch
  This patch is ZFS specific and without it, each Remove RPC results in
  leakage of 1 lookup buffer. (This would eventually result in a constipated
  server.)

I didn't realize these hadn't made it into 9.0 until I looked, rick

> Ronald.
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"