Trouble with NFSd under 6.1-Stable, any ideas?
howard at leadmon.net
Wed May 24 20:49:16 PDT 2006
I need to follow up to the below, as I am not sure why the below test with
the vfs_lookup.c didn't pan out the first time, but with my new found
knowledge on cvs I was determined to regress the system till I found the
smoking gun so to speak, which I have done.
First let me say that instead of running RELENG_6_1 like Rong-en is, I am
running the RELENG_6 tree that I know updates more often, but seems to work
well for me.
OK, so as I said above I started to regress the system a couple days at a
time, till suddenly NFS stared working again, so I knew at that point it was a
change that was made. So then I started to narrow the time range, till I got
to the point that it broke. Sure enough under the RELENG_6 branch, this time
was as follows:
*default tag=RELENG_6 date=2006.04.30.03.57.00 (Works OK)
*default tag=RELENG_6 date=2006.04.30.03.58.00 (Broken)
So what's changed at that delta, under the one that works vfs_lookup.c is:
Add delta 22.214.171.124 2006.03.31.07.39.24 kris
Under the one that fails the vfs_lookup.c is:
Add delta 126.96.36.199 2006.04.30.03.57.46 kris
So I stand corrected on my last post, the issue is in fact in this module, as
just taking that module back to 188.8.131.52 fixes the problem with my server. I
even took multiple NFS clients and gave them a heavy workload, and CPU still
remained reasonable, and very responsive. As soon as I rev to the new
version, NFS breaks badly and even a single client doing something like a du
of a directory structure results in sluggishness and extreme CPU usage.
I am not a coder, so not sure why this module was changed, but unless there
is some good reason why the changes were needed I would suspect it needs to be
rolled back, or something fixed. So Rong-en Fan, I think you were dead on
with your analysis that the issue is in fact inside the vfs_lookup.c module.
I hope this helps...
Howard Leadmon - howard at leadmon.net
> -----Original Message-----
> From: owner-freebsd-stable at freebsd.org
> [mailto:owner-freebsd-stable at freebsd.org] On Behalf Of Howard Leadmon
> Sent: Wednesday, May 24, 2006 1:23 PM
> To: 'Rong-en Fan'
> Cc: 'Konstantin Belousov'; freebsd-stable at freebsd.org
> Subject: RE: Trouble with NFSd under 6.1-Stable, any ideas?
> Hello Rong-en,
> As an update, I did the below, and I still had the issue with
> either version
> of vfs_lookup.c compiled in and running.
> On the bright side, I didn't realize you could step through
> the cvs by date, guess I just never paid attention. So I
> just stepped back to 'tag=RELENG_6 date=2006.04.20.00.00.00'
> on my server, rebuilt and violla nfs is now running
> So backing out something has fixed my problem, now to figure
> out just what it
> was. As I don't know what has caused this, I have done
> complete buildworlds
> to make sure everything updates which takes a few hours. I
> am going to
> start moving the cvs date forward till I get the problem
> back, once I nail this down a bit more, I'll let you know
> what I come up with.
> Howard Leadmon
> > -----Original Message-----
> > From: Rong-en Fan [mailto:grafan at gmail.com]
> > Sent: Tuesday, May 23, 2006 3:09 PM
> > To: Howard Leadmon
> > Cc: freebsd-stable at freebsd.org
> > Subject: Re: Trouble with NFSd under 6.1-Stable, any ideas?
> > On 5/23/06, Howard Leadmon <howard at leadmon.net> wrote:
> > >
> > > Hello Rong-en,
> > >
> > > Thanks for the info on getting the debugger configured,
> > and on the serial
> > > console. I will have to try and play with the serial
> > console thing more, I
> > > just tried putting in the flags and the damn thing hung, I
> > had to boot
> > > from CD and take the stuff back out.
> > >
> > > One thing you mention below that concerns me is that you
> > have version 1.90 of
> > > the vfs_lookup.c file. I just did a less on
> > /usr/src/sys/kern/vfs_lookup.c
> > > and I see the following:
> > >
> > > FreeBSD: src/sys/kern/vfs_lookup.c,v 184.108.40.206 2006/04/30
> > 03:57:46 kris
> > > Exp
> > >
> > >
> > > I even did a cvsup (I use cvsup2.FreeBSD.org) to make sure
> > I had the
> > > current stuff before rebuilding the kernel just now, and
> > still I see the same thing.
> > > Is something fishy going on here, or did you by chance
> make a typo??
> > Sorry for the confusion. rev 1.90 is the number for -HEAD.
> To back out
> > this MFC'ed change for RELENG_6_1, please cvsup to
> > RELENG_6_1 date=2006.04.30.03.57.00. Then you should see it is
> > 220.127.116.11 2006/03/31 07:39:24 kris
> > To verify the effect of this revision. Please run RELENG_6_1 with
> > 2006.04.30.03.57.00 and 2006.04.30.04.00.00.
> > Regards,
> > Rong-En Fan
> freebsd-stable at freebsd.org mailing list
> To unsubscribe, send any mail to
> "freebsd-stable-unsubscribe at freebsd.org"
More information about the freebsd-stable