[patch, try 1] Re: Trouble with NFSd under 6.1-Stable, any ideas?

[Konstantin Belousov]

On Thu, May 25, 2006 at 01:19:26AM -0400, Kris Kennaway wrote:
> On Wed, May 24, 2006 at 11:48:53PM -0400, Howard Leadmon wrote:
> 
> > So what's changed at that delta, under the one that works vfs_lookup.c is:
> > 
> >  Edit src/sys/kern/vfs_lookup.c
> >   Add delta 1.80.2.6 2006.03.31.07.39.24 kris
> > 
> > 
> > Under the one that fails the vfs_lookup.c is:
> > 
> >  Edit src/sys/kern/vfs_lookup.c
> >   Add delta 1.80.2.7 2006.04.30.03.57.46 kris
> > 
> > 
> > 
> >  So I stand corrected on my last post, the issue is in fact in this module, as
> > just taking that module back to 1.80.2.6 fixes the problem with my server.   I
> > even took multiple NFS clients and gave them a heavy workload, and CPU still
> > remained reasonable, and very responsive.  As soon as I rev to the new
> > version, NFS breaks badly and even a single client doing something like a du
> > of a directory structure results in sluggishness and extreme CPU usage.
> 
> Yep, unfortunately this commit was necessary to fix other bugs.  Jeff
> said he should have time to look at it next week.
> 
> Kris

I tried to debug the problem. First, I have to admit that I cannot
reproduce the problem on GENERIC kernel. Only after QUOTAS where added,
and, correspondingly, UFS started to require Giant,
I get described behaviour. Below are the changes to GENERIC config file
I made to reproduce problem.

Index: amd64/conf/GENERIC
===================================================================
RCS file: /usr/local/arch/ncvs/src/sys/amd64/conf/GENERIC,v
retrieving revision 1.439.2.11
diff -u -r1.439.2.11 GENERIC
--- amd64/conf/GENERIC	30 Apr 2006 17:39:43 -0000	1.439.2.11
+++ amd64/conf/GENERIC	25 May 2006 14:44:14 -0000
@@ -26,6 +26,19 @@
 #hints		"GENERIC.hints"		# Default places to look for devices.
 
 makeoptions	DEBUG=-g		# Build kernel with gdb(1) debug symbols
+options 	KDB
+options 	KDB_TRACE
+#options 	KDB_UNATTENDED
+options 	DDB
+options 	DDB_NUMSYM
+options 	BREAK_TO_DEBUGGER
+options         INVARIANTS
+options         INVARIANT_SUPPORT
+options         WITNESS
+options         DEBUG_LOCKS
+options         DEBUG_VFS_LOCKS
+options         DIAGNOSTIC
+options		MUTEX_PROFILING
 
 #options 	SCHED_ULE		# ULE scheduler
 options 	SCHED_4BSD		# 4BSD scheduler
@@ -34,6 +47,7 @@
 options 	INET6			# IPv6 communications protocols
 options 	FFS			# Berkeley Fast Filesystem
 options 	SOFTUPDATES		# Enable FFS soft updates support
+options		QUOTA
 options 	UFS_ACL			# Support for access control lists
 options 	UFS_DIRHASH		# Improve performance on big directories
 options 	MD_ROOT			# MD is a potential root device

After that, server machine easily panics on 

	KASSERT(!(debug_mpsafenet == 1 && mtx_owned(&Giant)),
		    ("nfssvc_nfsd(): debug.mpsafenet=1 && Giant"));

from nfsserver/nfs_syscalls.c, line 570.

As I understand the problem, kern/vfs_lookup.c:lookup() could
aquire additional locks on Giant, indicating this by GIANTHELD
flag in nd. All processing in nfsserver already goes with Giant held,
so, I just dropped that excessive locks after return from lookup.
System with patch applied survived smoke test (client did
du on mounted dir, patch was generated from exported fs, etc.).
nfsd eats no more than 25% of CPU (with INVARIANTS).

Please, users who reported the problem and willing to help,
try the patch (generated against STABLE) and give the feedback.

Index: nfsserver/nfs_serv.c
===================================================================
RCS file: /usr/local/arch/ncvs/src/sys/nfsserver/nfs_serv.c,v
retrieving revision 1.156.2.2
diff -u -r1.156.2.2 nfs_serv.c
--- nfsserver/nfs_serv.c	13 Mar 2006 03:06:49 -0000	1.156.2.2
+++ nfsserver/nfs_serv.c	25 May 2006 14:44:25 -0000
@@ -569,6 +569,10 @@
 
 			error = lookup(&ind);
 			ind.ni_dvp = NULL;
+			if (ind.ni_cnd.cn_flags & GIANTHELD) {
+				mtx_unlock(&Giant);
+				ind.ni_cnd.cn_flags &= ~GIANTHELD;
+			}
 
 			if (error == 0) {
 				/*
@@ -1915,6 +1919,10 @@
 
 			error = lookup(&nd);
 			nd.ni_dvp = NULL;
+			if (nd.ni_cnd.cn_flags & GIANTHELD) {
+				mtx_unlock(&Giant);
+				nd.ni_cnd.cn_flags &= ~GIANTHELD;
+			}
 			if (error)
 				goto ereply;
 
@@ -2141,6 +2149,10 @@
 
 		error = lookup(&nd);
 		nd.ni_dvp = NULL;
+		if (nd.ni_cnd.cn_flags & GIANTHELD) {
+			mtx_unlock(&Giant);
+			nd.ni_cnd.cn_flags &= ~GIANTHELD;
+		}
 
 		if (error)
 			goto out;
@@ -2878,6 +2890,10 @@
 
 		error = lookup(&nd);
 		nd.ni_dvp = NULL;
+		if (nd.ni_cnd.cn_flags & GIANTHELD) {
+			mtx_unlock(&Giant);
+			nd.ni_cnd.cn_flags &= ~GIANTHELD;
+		}
 
 		if (error == 0) {
 			bzero((caddr_t)fhp, sizeof(nfh));
Index: nfsserver/nfs_srvsubs.c
===================================================================
RCS file: /usr/local/arch/ncvs/src/sys/nfsserver/nfs_srvsubs.c,v
retrieving revision 1.136.2.2
diff -u -r1.136.2.2 nfs_srvsubs.c
--- nfsserver/nfs_srvsubs.c	4 Apr 2006 15:29:51 -0000	1.136.2.2
+++ nfsserver/nfs_srvsubs.c	25 May 2006 14:44:25 -0000
@@ -875,6 +875,10 @@
 	}
 	if (!lockleaf)
 		cnp->cn_flags &= ~LOCKLEAF;
+	if (cnp->cn_flags & GIANTHELD) {
+		mtx_unlock(&Giant);
+		cnp->cn_flags &= ~GIANTHELD;
+	}
 
 	/*
 	 * nfs_namei() guarentees that fields will not contain garbage
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20060525/8bf6ee49/attachment.pgp