Phantom /var full messages

[Bill Moran]

Paul Schmehl <pauls at utdallas.edu> wrote:
> I'm running snort 2.1.3 and mysql 3.23.58 on FreeBSD 4.9 RELEASE.  All 
> applications are built from ports.
> 
> Periodically I get /var full messages and everything comes to a grinding 
> halt.  The problem is, /var isn't full.
> 
> df -h will show /var at 104%, but du -h /var shows /var at 40% (for 
> example).

This typically happens because a file has been deleted, but some program
still has it open.  The filesystem can't actually free up the data blocks
until no other programs are using them.  But du doesn't see the usage
because there's no longer a file there to attribute it to.

> If I shut down snort and mysql, wait for a minute and then start 
> them back up, df agrees with du again.

You've already done the first diagnostic step.  You know that either snort
or MySQL is keeping a file handle after a file is deleted.

> The system works fine because only /var is full (although things can get 
> squirrelly if I let it go long enough because the system can't write to the 
> logs or the mail spool), so I can still ssh in and run utilities.

Yup.

> I suspect this is some sort of filehandle not being released issue, but I'm 
> not sure how to track it down.  I've got lsof installed, but I'm not an 
> expert on it yet.
> 
> Any hints would be welcomed.  What's the best way to troubleshoot this 
> problem?

First, if you could isolate it to just snort or just MySQL.

Typically, folks have this problem because they try to rotate log files
without restarting the program that's logging to them.  The rotate program
compresses the current log file into a new file, then deletes the original
file ... but the program is still logging to it.  Thus the space fills up,
but there is no file to see the space in.  Restarting the program doing
the logging causes the old file to disappear, and a new log file to be
created.

On a guess, Snort would be the first thing I'd look at.  However, MySQL
can create a TON of data if logging is enabled, so you may want to look
closely at it as well.

HTH

-- 
Bill Moran
Potential Technologies
http://www.potentialtech.com