kern/131360: [nfs] poor scaling behavior of the NFS server under load

[Martin Birgmeier]

>Number:         131360
>Category:       kern
>Synopsis:       [nfs] poor scaling behavior of the NFS server under load
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 04 12:40:01 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     Martin Birgmeier
>Release:        FreeBSD 7.1-RELEASE i386
>Organization:
MBi at home
>Environment:
System: FreeBSD gandalf.xyzzy 7.1-RELEASE FreeBSD 7.1-RELEASE #0: Sun Jan 4 10:53:24 CET 2009 root at atpcdvvc.xyzzy:/usr/VOL/OBJ/FreeBSD/RELENG_7_1_0_RELEASE/src/sys/XYZZY i386

>Description:
	[Please note that the email address given is not valid due
	to security reasons. Just reply to the gnats entry, and
	I'll follow it via the web interface.]

	Between FreeBSD 6.3 and 7.1, the behavior of the NFS server
	changed for the worse. Under 6.3, the load generated by the
	nfsd's would never exceed their number (I am using
	nfs_server_flags="-u -t -n 8"). With 7.1, when the client
	generates a lot of requests, it seems that the load on the
	NFS server can grow nearly without bounds, rendering the
	server all but unusable.

	I have been testing this with two scenarios:

	- running a FreeBSD buildworld -j4 with src on NFS and obj
	  local, so there are only NFS reads, and

	- compiling openoffice.org-3-devel with WRKDIRPREFIX set
	  to an NFS-mounted directory, and MAXMODULES=3, MAXPROCESSES=3,
	  resulting in both reads and writes.

	In both cases, the NFS server reaches absurdly high load
	values: with 'buildword -j4' on the client I currently read
	from 'systat -vm 1' on the server:

	- a load just passing the 15 mark

	- nearly 100% 'sys'

	- there seems to be nearly no disk i/o activity

	- very poor general network throughput: an xterm from the
	  server to the client is nearly unresponsive (in fact, I
	  again have to suspend the build on the client right now
	  in order to be able to write this bug report in an xterm
	  running on the server).

	- resuming the build immediately sends the load skyrocketing
	  again.

	I am quite sure that this is a server issue, as for a while
	I had the client running 7.0 and server 6.3, and in this
	scenario this did not happen.

	BTW, the NFS mounts on the client are done via amd - this
	probably influences which mount type is used.

>How-To-Repeat:
	Generate a high NFS load on a client, and watch the server
	load skyrocket.

>Fix:
	Unknown.

>Release-Note:
>Audit-Trail:
>Unformatted: