misc/145189: nfsd performs abysmally under load

Bruce Evans brde at optusnet.com.au
Tue Mar 30 20:20:09 UTC 2010

The following reply was made to PR misc/145189; it has been noted by GNATS.

From: Bruce Evans <brde at optusnet.com.au>
To: Rich <rercola at acm.jhu.edu>
Cc: Bruce Evans <brde at optusnet.com.au>, freebsd-gnats-submit at freebsd.org,
        freebsd-bugs at freebsd.org
Subject: Re: misc/145189: nfsd performs abysmally under load
Date: Wed, 31 Mar 2010 07:11:32 +1100 (EST)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed
 Content-Transfer-Encoding: QUOTED-PRINTABLE
 On Tue, 30 Mar 2010, Rich wrote:
 > On Tue, Mar 30, 2010 at 11:50 AM, Bruce Evans <brde at optusnet.com.au> wrot=
 >>> For instance, copying a 4GB file over NFSv3 from a ZFS filesystem with =
 >>> following flags
 >>> [rw,nosuid,hard,intr,nofsc,tcp,vers=3D3,rsize=3D8192,wsize=3D8192,slopp=
 >>> client, the above is the server), I achieve 2 MB/s, fluctuating between=
 >>> and 3. (pv reports 2.23 MB/s avg)
 I also tried various nfs r/w sizes and tcp/udp.  The best sizes are
 probably the fs block size or twice that (normally 16K for ffs).  Old
 versions of FreeBSD had even more bugs in this area and gave surprising
 performance differences depending on the nfs r/w sizes or application
 i/o sizes.  In some cases smaller sizes worked best, apparently because
 they avoided the stalls.
 >>> ...
 >> Enabling polling is a good way to destroy latency. =A0A ping latency of
 >> ...
 > Actually, we noticed that throughput appeared to get marginally better wh=
 > causing occasional bursts of crushing latency, but yes, we have it on in =
 > kernel without using it in any actual NICs at present. :)
 > But yes, I'm getting 40-90+ MB/s, occasionally slowing to 20-30 MB/s,
 > average after copying a 6.5 GB file of 52.7 MB/s, on localhost IPv4,
 > with no additional mount flags. {r,w}size=3D8192 on localhost goes up to
 > 80-100 MB/s, with occasional sinks to 60 (average after copying
 > another, separate 6.5 GB file: 77.3 MB/s).
 I thought you said you often got 1-3MB/S.
 > Also:
 > 64 bytes from icmp_seq=3D0 ttl=3D64 time=3D0.015 ms
 > 64 bytes from icmp_seq=3D1 ttl=3D64 time=3D0.049 ms
 > 64 bytes from icmp_seq=3D2 ttl=3D64 time=3D0.012 ms
 Fairly normal slowness for -current.
 > 64 bytes from [actual IP]: icmp_seq=3D0 ttl=3D64 time=3D0.019 ms
 > 64 bytes from [actual IP]: icmp_seq=3D1 ttl=3D64 time=3D0.015 ms
 Are these with external hardware NICs?  Then 15 uS is excellent.  Better
 than I've ever seen.  Very good hardware might be able to do this, but
 I suspect it is for the local machine.  BTW, I don't like the times
 been reported in ms and sub-uS times not being supported.  I sometimes
 run Linux or cygwin ping and don't like it not supporting sub-mS times,
 so that it always reports 0 for my average latency of 100 uS.
 >> After various tuning and bug fixing (now partly committed by others) I g=
 >> improvements like the following on low-end systems with ffs (I don't use
 >> zfs):
 >> - very low end with 100Mbps ethernet: little change; bulk transfers alwa=
 >> =A0went at near wire speed (about 10 MB/S)
 >> - low end with 1Gbps/S: bulk transfers up from 20MB/S to 45MB/S (local f=
 >> =A050MB/S). =A0buildworld over nfs of 5.2 world down from 1200 seconds t=
 o 800
 >> =A0seconds (this one is very latency-sensitive. =A0Takes about 750 secon=
 ds on
 >> =A0local ffs).
 > Is this on 9.0-CURRENT, or RELENG_8, or something else?
 Mostly with 7-CURRENT or 8-CURRENT a couple of years ago.  Sometimes with
 a ~5.2-SERVER.  nfs didn't vary much with the server, except there were
 surprising differences due to latency that I never tracked down.
 I forgot to mention another thing you can try easily:
 - negative name caching.  Improves latency.  I used this to reduce makeworl=
    times significantly, and it is now standard in -current but not
    enabled by default.

More information about the freebsd-bugs mailing list