Poor throughput using new NFS client (9.0) vs. old (8.2/9.0)

[Rick Macklem]

Ronald Klop wrote:
> On Thu, 18 Oct 2012 18:16:16 +0200, Thomas Johnson <tom at claimlynx.com>
> wrote:
> 
> > We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0.
> > Almost
> > immediately, we received reports from users of poor performance. The
> > upgraded hosts are PXE-booted, with an NFS-mounted root.
> > Additionally,
> > they
> > mount a number of other NFS shares, which is where our users work
> > from.
> > After a week of tweaking rsize/wsize/readahead parameters (per
> > guidance),
> > it finally occurred to me that 9.0 defaults to the new NFS client
> > and
> > server. I remounted the user shares using the oldnfs file type, and
> > users
> > reported that performance returned to its expected level.
> >
> > This is obviously a workaround, rather than a solution. We would
> > prefer
> > to
> > get our hosts using the newnfs client, since presumably oldnfs will
> > be
> > deprecated at some point in the future. Is there some change that we
> > should
> > have made to our NFS configuration with the upgrade to 9.0, or is it
> > possible that our workload is exposing some deficiency with newnfs?
> > We
> > tend
> > to deal with a huge number of tiny files (several KB in size). The
> > NFS
> > server has been running 9.0 for some time (prior to the client
> > upgrade)
> > without any issue. NFS is served from a zpool, backed by a Dell
> > MD3000,
> > populated with 15k SAS disks. Clients and server are connected with
> > Gig-E
> > links. The general hardware configuration has not changed in nearly
> > 3
> > years.
> >
> > As an example of the performance difference, here is some of the
> > testing
> > I
> > did while troubleshooting. Given a directory containing 5671 zip
> > files,
> > with an average size of 15KB. I append all files to an existing zip
> > file.
> > Using the newnfs mount, I found that this operation generally takes
> > ~30
> > seconds (wall time). Switching the mount to oldnfs resulted in the
> > same
> > operation taking ~10 seconds.
> >
> > tom at test-1:/test-> ls 53*zip | wc -l
> >     5671
> > tom at test-1:/test-> ll -h BIG*
> > -rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip
> > tom at test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip
> > 0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w
> > ...reset and repeat...
> > 0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w
> > ...reset and repeat...
> > 0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w
> >
> > tom at test-1:/-> cd /
> > tom at test-1:/-> sudo umount /test
> > tom at test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test
> > tom at test-1:/-> mount | grep test
> > server:/array/test on /test (oldnfs)
> > tom at test-1:/-> cd /test
> > ...reset and repeat...
> > 0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w
> > ...reset and repeat...
> > 0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w
> > tom at test-1:/test-> ll -h BIG*
> > -rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip
> >
> > Thanks!
> >
> 
> 
> You might find this thread from today interesting.
> http://lists.freebsd.org/pipermail/freebsd-fs/2012-October/015441.html
> 
Yes, although I can't explain why Alexey's problem went away
when he went from 9.0->9.1 for his NFS server, it would be
interesting if Thomas could try the same thing?

About the only thing different between the old and new NFS
clients is the default rsize/wsize. However, if Thomas tried
rsize=32768,wsize=32768 for the default (new) NFS client, then
that would be ruled out. To be honest, the new client uses code
cloned from the old one for all the caching etc (which is where
the clients are "smart"). They use different RPC parsing code,
since the new one does NFSv4 as well, but that code is pretty
straightforward, so I can't think why it would result in a
factor of 3 in performance.

If Thomas were to capture a packet trace of the above test
for two clients and emailed them to me, I could take a look
and see if I can see what is going on. (For Alexey's case,
it was a whole bunch of Read RPCs without replies, but that
was a Linux client, of course. It also had a significant # of
TCP layer retransmits and out of order TCP segments in it.)

It would be nice to figure this out, since I was thinking
that the old client might go away for 10.0 (can't if these
issues still exist).

rick

> Ronald.
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"