duplicate message removal

[Robin Becker]

We have a bunch of FreeBSD 6.x servers which we administer remotely. As 
part of that we get the normal root job mails emailed to a mailing list 
which the admins(mostly me) can inspect at leisure and also use for 
historical purposes.

Trouble is many of the emails get huge because of repeated messages 
typically stuff like

xxx.yyy.com login failures:
Feb 22 20:07:54 app3 sshd[56886]: reverse mapping checking getaddrinfo 
for 216-194-26-66.ny.ny.metconnect.net failed - POSSIBLE BREAKIN ATTEMPT!

etc etc

All these servers are running denyhosts, but we still see lots of these 
messages.

I was wondering if there's any simple compression script which notices 
the repetitions (apart from timestamp) and can remove the many 
duplicates etc etc.
-- 
Robin Becker