Trouble with NFSd under 6.1-Stable, any ideas?

[Howard Leadmon]

 I need to follow up to the below, as I am not sure why the below test with
the vfs_lookup.c didn't pan out the first time, but with my new found
knowledge on cvs I was determined to regress the system till I found the
smoking gun so to speak, which I have done.

 First let me say that instead of running RELENG_6_1 like Rong-en is, I am
running the RELENG_6 tree that I know updates more often, but seems to work
well for me. 

 OK, so as I said above I started to regress the system a couple days at a
time, till suddenly NFS stared working again, so I knew at that point it was a
change that was made.  So then I started to narrow the time range, till I got
to the point that it broke.   Sure enough under the RELENG_6 branch, this time
was as follows:

*default tag=RELENG_6 date=2006.04.30.03.57.00   (Works OK)

*default tag=RELENG_6 date=2006.04.30.03.58.00   (Broken)


So what's changed at that delta, under the one that works vfs_lookup.c is:

 Edit src/sys/kern/vfs_lookup.c
  Add delta 1.80.2.6 2006.03.31.07.39.24 kris


Under the one that fails the vfs_lookup.c is:

 Edit src/sys/kern/vfs_lookup.c
  Add delta 1.80.2.7 2006.04.30.03.57.46 kris



 So I stand corrected on my last post, the issue is in fact in this module, as
just taking that module back to 1.80.2.6 fixes the problem with my server.   I
even took multiple NFS clients and gave them a heavy workload, and CPU still
remained reasonable, and very responsive.  As soon as I rev to the new
version, NFS breaks badly and even a single client doing something like a du
of a directory structure results in sluggishness and extreme CPU usage.

 I am not a coder, so not sure why this module was changed, but unless there
is some good reason why the changes were needed I would suspect it needs to be
rolled back, or something fixed.   So Rong-en Fan, I think you were dead on
with your analysis that the issue is in fact inside the vfs_lookup.c module.


I hope this helps...



---
Howard Leadmon - howard at leadmon.net
http://www.leadmon.net

 

> -----Original Message-----
> From: owner-freebsd-stable at freebsd.org 
> [mailto:owner-freebsd-stable at freebsd.org] On Behalf Of Howard Leadmon
> Sent: Wednesday, May 24, 2006 1:23 PM
> To: 'Rong-en Fan'
> Cc: 'Konstantin Belousov'; freebsd-stable at freebsd.org
> Subject: RE: Trouble with NFSd under 6.1-Stable, any ideas?
> 
> 
>    Hello Rong-en,
> 
> 
> As an update, I did the below, and I still had the issue with 
> either version
> of vfs_lookup.c compiled in and running.   
> 
> On the bright side, I didn't realize you could step through 
> the cvs by date, guess I just never paid attention.  So I 
> just stepped back to 'tag=RELENG_6 date=2006.04.20.00.00.00' 
> on my server, rebuilt and violla nfs is now running
> perfect.   
> 
>  So backing out something has fixed my problem, now to figure 
> out just what it
> was.   As I don't know what has caused this, I have done 
> complete buildworlds
> to make sure everything updates which takes a few hours.    I 
> am going to
> start moving the cvs date forward till I get the problem 
> back, once I nail this down a bit more, I'll let you know 
> what I come up with.
> 
> 
> 
> ---
> Howard Leadmon
> http://www.leadmon.net
> 
>  
> 
> > -----Original Message-----
> > From: Rong-en Fan [mailto:grafan at gmail.com]
> > Sent: Tuesday, May 23, 2006 3:09 PM
> > To: Howard Leadmon
> > Cc: freebsd-stable at freebsd.org
> > Subject: Re: Trouble with NFSd under 6.1-Stable, any ideas?
> > 
> > On 5/23/06, Howard Leadmon <howard at leadmon.net> wrote:
> > >
> > >    Hello Rong-en,
> > >
> > >  Thanks for the info on getting the debugger configured,
> > and on the serial
> > > console.   I will have to try and play with the serial 
> > console thing more, I
> > > just tried putting in the flags and the damn thing hung, I
> > had to boot
> > > from CD and take the stuff back out.
> > >
> > >  One thing you mention below that concerns me is that you
> > have version 1.90 of
> > > the vfs_lookup.c file.   I just did a less on 
> > /usr/src/sys/kern/vfs_lookup.c
> > > and I see the following:
> > >
> > > FreeBSD: src/sys/kern/vfs_lookup.c,v 1.80.2.7 2006/04/30
> > 03:57:46 kris
> > > Exp
> > >
> > >
> > >  I even did a cvsup (I use cvsup2.FreeBSD.org) to make sure
> > I had the
> > > current stuff before rebuilding the kernel just now, and
> > still I see the same thing.
> > > Is something fishy going on here, or did you by chance 
> make a typo??
> > 
> > Sorry for the confusion. rev 1.90 is the number for -HEAD. 
> To back out 
> > this MFC'ed change for RELENG_6_1, please cvsup to
> > RELENG_6_1 date=2006.04.30.03.57.00. Then you should see it is
> > 
> > 1.80.2.6 2006/03/31 07:39:24 kris
> > 
> > To verify the effect of this revision. Please run RELENG_6_1 with 
> > 2006.04.30.03.57.00 and 2006.04.30.04.00.00.
> > 
> > Regards,
> > Rong-En Fan
> > 
> 
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to 
> "freebsd-stable-unsubscribe at freebsd.org"
>