kern/97686: massive NFS server performance break under "options QUOTA"

[Ralf S. Engelschall]

>Number:         97686
>Category:       kern
>Synopsis:       massive NFS server performance break under "options QUOTA"
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue May 23 12:00:39 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator:     Ralf S. Engelschall
>Release:        FreeBSD 6.1-STABLE i386
>Organization:
FreeBSD
>Environment:
System: FreeBSD sv0 6.1-STABLE FreeBSD 6.1-STABLE #0: Tue May 23 12:05:59 CEST 2006 root at sv0:/usr/obj/usr/src/sys/SV0  i386

>Description:
   After an upgrade we experienced massive serious NFS server problems
   on a NFS server under FreeBSD 6.1-STABLE (as of 2006-05-23). It
   still worked fine (exactly same kernel configuration) under FreeBSD
   5.4-STABLE (as of about December 2005).
   
   The effect is that the machine becomes totally unusable (nearly a
   system freeze) after about 10 minutes of just moderate NFS traffic
   of a single NFS client. We figured out that once the "options QUOTA"
   was removed from the kernel configuration the machine performance
   was 100% ok and we do not experience any performance breaks again.
   Even disabling SMP or deconfiguring all Netgraph configurations (for
   interface bundling and Netflow statistics) caused no effect. Only the
   "options QUOTA" caused the problem and its removal finally fixed the
   problem.

   There seems to be a strange interaction between NFS and UFS Quotas...

>How-To-Repeat:
   - build a FreeBSD 6.1 kernel with "options QUOTA" and "options NFSSERVER".
   - configure nfs_server_enable="YES" and nfs_server_flags="-u -n 8" in rc.conf.
   - cause some usual NFS traffic by unpacking a large distribution
     tarball (lots of small files), removing lots of files or just doing a
     "cvs update" in a large source tree.
   - in parallel on the NFS server watch a steadily increasing CPU
     utilitzation from nfsd(8) and about 45% of "CPU time in system
     mode" (= "sy") in "iostat 1" outputs (numbers on a 2 CPU HP DL380
     box). Also the system loads increases steadily up to 15-20.
   - after maximally 10 minutes the box becomes more and more totally
     unusable. Even typing in the SSH session becomes a problem. Once
     the NFS traffic is stopped, the machines becomes usable within
     30-60 seconds again.

>Fix:
   workaround by removing "options QUOTA" from the kernel configuration
   the machine works fine and no performance problems occur at all. With
   the above "How-To-Repeat" procedure the NFS performance is fully ok
   and the NFS server itself also never reaches any high loads. Also
   "iostat 1" shows never a "CPU time in system mode" of more than 5%.

>Release-Note:
>Audit-Trail:
>Unformatted: