machine hangs on occasion - correlated with ssh break-in attempts

[Jeremy Chadwick]

On Thu, Aug 21, 2008 at 01:38:38PM -0400, Mikhail Teterin wrote:
> Hello!
>
> A machine I manage remotely for a friend comes under a distributed ssh  
> break-in attack every once in a while. Annoyed (and alarmed) by the  
> messages like:
>
> Aug 12 10:21:17 symbion sshd[4333]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:18 symbion sshd[4335]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:20 symbion sshd[4337]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:21 symbion sshd[4339]: Invalid user mythtv from 85.234.158.180
>
> I wrote an awk-script, which adds a block of the attacking IP-address to  
> the ipfw-rules after three such "invalid user" attempts with:
>
>    ipfw add 550 deny ip from ip
>
> The script is fed by syslogd directly -- through a syslog.conf rule  
> ("|/opt/sbin/auth-log-watch").
>
> Once in a while I manually flush these rules... I this a good (safe)  
> reaction?
> I'm asking, because the machine (currently running 7.0 as of July 7)  
> hangs solid once every few weeks... My only guess is that a spike in  
> attacks causes "too many" ipfw-entries created, which paralyzes the  
> kernel due to some bug -- the machine is running natd and is the gateway  
> for the rest of the network...
> The hangs could, of course, be caused by something else entirely, but my  
> self-defense mechanism is my first suspect...
>
> Any comments? Thanks!

Yes, I have quite a few comments on this matter:

The above looks like sshguard.  I've personally never trusted something
that *automatically* adjusts firewall rules based on data read from text
logs or packets coming in off the Internet.  The risks involved are
insanely high.

Stop for a moment and think what would happen to your box if a
distributed brute-force attack (e.g. 300,000 different IPs) was launched
against it; someone executing 20-30 SSH login attempts per IP.  I'm
willing to bet adding 300,000 individual ipfw entries would cause some
serious havok on your machine (speculative: exhausted kernel memory, or
at a bare minimum, exhaust the number of remaining ipfw rule entries)

And yes, the liklihood of someone doing this is quite high.

Try re-thinking your firewall logic.  Instead of "allow any, deny
specific IPs dynamically", how about "allow specific IPs, deny all
others"?

Surely you don't have that many users who SSH into the NAT router from
random public IPs all over the world, rather than via the LAN?  Surely
if you yourself often SSH into your NAT router from a Blackberry device,
that you wouldn't have much of a problem adding a /19 to the allow list.
That's a hell of a lot better than allowing 0/0 and denying individual
/32s.

A different approach: consider putting sshd on a different port, rather
than the default of 22.  A lot of people I know do this, solely to
decrease the number of brute-force attempts you see above; I've never
seen any of those brute-force attacking programs portscan, then attack
against a port which returns a OpenSSH string.

Finally, consider moving to pf instead, if you really feel ipfw is
what's causing your machine to crash.  You might be pleasantly surprised
by the syntax, and overall administrative usability (it is significantly
superior to ipfw, IMHO).

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |