NFS/Sendmail issue since upgrading to 4.9-STABLE

[Adam Maloney]

Sorry for the noise on the list about this.  We think we've identified the
issue.

We made the change to make the inbound queues local, and the problem occured
again almost immediately.  That ruled out NFS, and left us with only one
other realistic option.

This custom application reads a directory list and for each qf file,
attempts a LOCK_EX.  For the past 2 years this has appeared to work fine
with Sendmail - if Sendmail is still working on the qf file, we aren't able
to get an exclusive lock, so we go on to the next file, etc.  (Maybe Claus
can confirm some of this behavior?)

It appears that Sendmail changed from 8.12.9 (FreeBSD 4.8R) to 8.12.10
(FreeBSD 4.9-STA).  What we believe is happening is that the qf file is
being created, and our application is getting the exclusive lock before
Sendmail.  This makes Sendmail spit out the "resource temporarily
unavailable" error, since it can't lock the file, and it leaves our
application with an empty qf file that we are trying to process.

It will take several more hours of testing before we're relatively confident
that this is the issue.  Hopefully I'll be posting one last time to confirm
that everything is fixed.

----- Original Message ----- 
From: "Adam Maloney" <adam at whee.org>
To: <freebsd-stable at freebsd.org>
Sent: Wednesday, November 26, 2003 1:40 PM
Subject: Re: NFS/Sendmail issue since upgrading to 4.9-STABLE


> Just an update for anyone paying attention to this issue.
>
> We are going to temporarily move the inbound Sendmail queue to be local
> rather than NFS.  We'll run like this through the holiday weekend and see
if
> the problems we're having continue or not.  If they stop happening, then
we
> can be pretty confident that some change in the NFS client code is causing
> this (the problems started when we upgraded the 1st client, before making
> any changes to the server).
>
> We've also noticed another problem that we believe to be NFS related as
well
> with these same machines.  It started happening at exactly the same time
> (though less frequently).  This 2nd issue is in a custom application that
we
> are running.  At this time I don't have a lot of detail, but we are fairly
> confident that it is also NFS related.  Making the inbound Sendmail queues
> local will also isolate this to be NFS or not.
>
> I will report back on Monday with our results.
>
> I appreciate everyone's input.
>
> ----- Original Message ----- 
> From: "Adam Maloney" <adam at whee.org>
> To: <freebsd-stable at freebsd.org>
> Sent: Wednesday, November 26, 2003 8:16 AM
> Subject: NFS/Sendmail issue since upgrading to 4.9-STABLE
>
>
> > Hi all,
> >
> > I have 3 servers accepting mail, with sendmail in Queue-only mode
dropping
> > mail into an NFS queue (mounted from a 4th machine.)  All 4 machines
> > involved are running FreeBSD.
> >
> > I upgraded the 3 NFS clients from 4.8 to 4.9-STABLE on Monday, and
started
> > seeing this sendmail message:
> >
> > Nov 26 06:51:29 unix20 sm-mta[48069]: (real-message-id): SYSERR(root):
> > queueup: cannot create queue file ./qf(real-message-id), euid=0:
Resource
> > temporarily unavailable
> >
> > The df(message-id) file is created and appears to be complete, however
the
> > qf file is 0 bytes.
> >
> > This happens sporadically, a few times a day on each machine (together
> doing
> > about 150,000 messages/day).
> >
> > I upgraded the NFS server from 4.8 to 4.9 yesterday, thinking that might
> > help, but the problem is still there.
> >
> > Any ideas how to troubleshoot this one?
> >
> > _______________________________________________
> > freebsd-stable at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to
"freebsd-stable-unsubscribe at freebsd.org"
> >
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>