duplicate message removal

[Mel]

On Saturday 23 February 2008 11:32:41 Robin Becker wrote:

> Trouble is many of the emails get huge because of repeated messages
> typically stuff like
>
> xxx.yyy.com login failures:
> Feb 22 20:07:54 app3 sshd[56886]: reverse mapping checking getaddrinfo
> for 216-194-26-66.ny.ny.metconnect.net failed - POSSIBLE BREAKIN ATTEMPT!
>
> etc etc
>
> All these servers are running denyhosts, but we still see lots of these
> messages.
>
> I was wondering if there's any simple compression script which notices
> the repetitions (apart from timestamp) and can remove the many
> duplicates etc etc.

Well, there's two things you can do:
Attack the problem:
Install sysutils/grok on the machines and start it with following config:

file "/var/log/auth.log" {
        type "ssh-illegal-user" {
                match = "Invalid user %USERNAME% from %IP%";
                threshold = 5;   # 5 hits ...
                key = "%IP%";    # from a single ip ...
                interval = 60;   # in 1 minutes
                reaction = "/sbin/pfctl -t sshscan -Tadd %IP%";
        };

        type "ssh-scan-possible" {
                match = "Did not receive identification string from %IP%";
                threshold = 3;
                interval = 60;
                reaction = "/sbin/pfctl -t sshscan -Tadd %IP%";
        };
};

If you replace:
my $TAIL = "tail -0f";
with
my $TAIL = "/usr/bin/tail -0F"; # capital F

in grok, you will never have to look at it again. Else you will have to 
restart it whenever newsyslog(8) rotates the auth.log.
Note that this uses pf, and assumes the table <sshscan> is blocking table.
I'm sure it can be done with other firewalls as well.

Attack the report:
daily_status_security_loginfail_enable="NO" >>/etc/periodic.conf

Then write your own and put it in /usr/local/etc/periodic/security/. I've 
written something similar with php for mail rejects, that groups sender or 
hostname and error message together, but any scripting language that you're 
comfy with should work.


-- 
Mel