duplicate message removal

Mel fbsd.questions at rachie.is-a-geek.net
Sat Feb 23 12:24:04 UTC 2008

On Saturday 23 February 2008 11:32:41 Robin Becker wrote:

> Trouble is many of the emails get huge because of repeated messages
> typically stuff like
> xxx.yyy.com login failures:
> Feb 22 20:07:54 app3 sshd[56886]: reverse mapping checking getaddrinfo
> for 216-194-26-66.ny.ny.metconnect.net failed - POSSIBLE BREAKIN ATTEMPT!
> etc etc
> All these servers are running denyhosts, but we still see lots of these
> messages.
> I was wondering if there's any simple compression script which notices
> the repetitions (apart from timestamp) and can remove the many
> duplicates etc etc.

Well, there's two things you can do:
Attack the problem:
Install sysutils/grok on the machines and start it with following config:

file "/var/log/auth.log" {
        type "ssh-illegal-user" {
                match = "Invalid user %USERNAME% from %IP%";
                threshold = 5;   # 5 hits ...
                key = "%IP%";    # from a single ip ...
                interval = 60;   # in 1 minutes
                reaction = "/sbin/pfctl -t sshscan -Tadd %IP%";

        type "ssh-scan-possible" {
                match = "Did not receive identification string from %IP%";
                threshold = 3;
                interval = 60;
                reaction = "/sbin/pfctl -t sshscan -Tadd %IP%";

If you replace:
my $TAIL = "tail -0f";
my $TAIL = "/usr/bin/tail -0F"; # capital F

in grok, you will never have to look at it again. Else you will have to 
restart it whenever newsyslog(8) rotates the auth.log.
Note that this uses pf, and assumes the table <sshscan> is blocking table.
I'm sure it can be done with other firewalls as well.

Attack the report:
daily_status_security_loginfail_enable="NO" >>/etc/periodic.conf

Then write your own and put it in /usr/local/etc/periodic/security/. I've 
written something similar with php for mail rejects, that groups sender or 
hostname and error message together, but any scripting language that you're 
comfy with should work.


More information about the freebsd-questions mailing list