misc/145189: nfsd performs abysmally under load

Rich Ercolani rercola at acm.jhu.edu
Tue Mar 30 05:10:03 UTC 2010

>Number:         145189
>Category:       misc
>Synopsis:       nfsd performs abysmally under load
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Mar 30 05:10:02 UTC 2010
>Originator:     Rich Ercolani
>Release:        8.0
FreeBSD manticore.acm.jhu.edu 8.0-RELEASE FreeBSD 8.0-RELEASE #1: Mon Mar 22 20:13:43 EDT 2010     root at manticore.acm.jhu.edu:/usr/obj/usr/local/ncvs/src/sys/DTRACE_POLL  amd64

nfsd performs abysmally on this machine under conditions in which Solaris's NFS implementation is reasonably fast, and while local IO to the same filesystems is still zippy.

For instance, copying a 4GB file over NFSv3 from a ZFS filesystem with the following flags [rw,nosuid,hard,intr,nofsc,tcp,vers=3,rsize=8192,wsize=8192,sloppy,addr=X.X.X.X](Linux client, the above is the server), I achieve 2 MB/s, fluctuating between 1 and 3. (pv reports 2.23 MB/s avg)

Locally, on the server, I achieve 110-140 MB/s (at the end of pv, it reports 123 MB/s avg).

I'd assume network latency, but nc with no flags other than port achieves 30-50 MB/s between server and client.

Latency is also abysmal - ls on a randomly chosen homedir full of files, according to time, takes:
real    0m15.634s
user    0m0.012s
sys     0m0.097s
while on the local machine:
real	0m0.266s
user	0m0.007s
sys	0m0.000s

The server in question is a 3GHz Core 2 Duo, running FreeBSD RELENG_8. The kernel conf, DTRACE_POLL, is just the stock AMD64 kernel with all of the DTRACE-related options turned on, as well as the option to enable polling in the NIC drivers, since we were wondering if that would improve our performance.

We tested this with a UFS directory as well, because we were curious if this was an NFS/ZFS interaction - we still got 1-2 MB/s read speed and horrible latency while achieving fast throughput and latency local to the server, so we're reasonably certain it's not "just" ZFS, if there is indeed any interaction there.

Read speed of a randomly generated 6500 MB file on UFS over NFSv3 with the same flags as above: 1-3 MB/s, averaging 2.11 MB/s
Read speed of the same file, local to the server: consistently between 40-60 MB/s, averaging 61.8 MB/s [it got faster over time - presumably UFS was aggressively caching the file, or something?]
Read speed of the same file over NFS again, after the local test:
Amusingly, worse (768 KB/s-2.2 MB/s, with random stalls - average reported 270 KB/s(!)).
1) Mount multiple NFS filesystems from the server
2) Watch as your operations latency and throughput rapidly sink to near-zero


More information about the freebsd-bugs mailing list