Aggregating logs from numerous FreeBSD machines

[Mark Johnston]

Hi folks,

My stack of trusty FreeBSD servers always seems to be growing, and it's 
getting to the point where the daily and security output mail is too much to 
make good use of.  I'm looking for suggestions for log monitoring and 
aggregation tools, especially from a monitoring-for-security perspective.

If I had to imagine an ideal system, it would be a central server that 
securely collects syslog messages from all my servers, indexes them by server 
and severity, and gives a reasonable management interface.  Given expressions 
based on facility, severity, log message, and the like, it could throw away 
useless messages, or page me for critical ones.  This would tie into 
AIDE/Samhain/Tripwire (haven't picked one yet) and maybe even different 
flavors of IDS.  It could even warn me when processes run away with the CPU 
or RAM, or disks get too full.

I've found a variety of things that almost do this.  Nagios is good at paging 
for service failures, disk full warnings, and that sort of thing, but it 
doesn't seem well-suited for aggregating log messages.  The Prelude IDS seems 
to have some kind of console, as does Samhain, but I want to try to avoid 
having different interfaces for each service type.

I realize this is something that could be had using IPSec-protected remote 
logging with some greps and interface stuff bolted on, but if there's a 
ready-made tool, it'd save me a fair bit of implementation time.  What kind 
of things are other security-minded admins using to stay on top of all the 
logs?

Thanks,
Mark