Log analysis server suggestions?

[Nathan Vidican]

Ashley Moran wrote:
> Until recently I had a server running syslog-ng set to archive all logs into 
> server/year/month/day/ directories.  Now the server is running in amd64, 
> we've lost our hi-res scrolling display so I want to look at a better log 
> watching system.
> 
> I've read about logging to a database.  I quite like the idea of storing our 
> logs in PostgreSQL (I don't like MySQL and don't want to get involved in 
> administering a second database).  I know I can log to a PG database quite 
> easily, but I don't know how I can get the data back out without writing 
> manual queries.
> 
> Here is what I need:
> 
> - Logs stored for the last 6 months or so, and easily searchable
> - Live log watching
> - Log analysis
> 
> I might try swatch for the live log watching as this is not affected by the 
> choice of log storage and seems the best tool for the job.
> 
> As for searching / analysis, I've seen php-syslog-ng 
> ( http://www.vermeer.org/projects/php-syslog-ng ), which looks very basic, 
> and phpLogCon ( http://www.phplogcon.com/ ), which does not support PG 
> anyway.  Is there anything better GUI-wise?
> 
> Maybe I am best keeping the logs in text files for now, and spending more time 
> on swatch.
> 
> Any thoughts?
> 
> Cheers
> Ashley
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"
> 
> 
In my experience, logfiles are best NOT in SQL, flat-files are easy to deal 
with, and through a few simple Perl scripts you could accomplish all you need 
to. You can run a tail -f and dump output to stdout, or even pipe it to a socket 
and monitor remotely. Also, various programs have great open-source analysers, 
for specific logs, (ie Apache, sendmail, etc).

I would advise against trying to log everything into SQL records, aside from the 
performance hit on translating log/write outputs to SQL inserts/queries then 
having the SQL server write to disk anyway, it just complicates things 
uneccessarily.

My advice would be to take a step back and look at what's important to you. 
Decide which logs need to be monitored in real-time, are there certain criteria 
that require immediate attention? What about alhpa-numeric pager systems, or 
emailed warnings? Are customers going to require reports/information (ie web 
server stats, sendmail relay stats or spam logs, bandwidth usage, etc). It of 
course depends on your overall system, and your users more than anything... but 
in the end I find it's best to work with a mixture of things and hack your own 
scripts to fill in the gaps.

Just my two cents, hope it helps.

-- 
Nathan Vidican
nvidican at wmptl.com
Windsor Match Plate & Tool Ltd.
http://www.wmptl.com/