SSH login takes very long time...sometimes

[Atanas]

David Malone said the following on 02/16/06 13:24:
>> Just a thought, wouldn't this open a new possibility for denial of 
>> service attacks?
> 
> I doubt it. I'm guessing you're thinking of an attack where someone
> makes many connections to sshd in a short time and runs you out of
> processes? I think you can protect against this with the MaxStartups
> directive in sshd_config. The amount of time that an attacker has
> to open many connections is probably not that important, as you can
> open a lot of TCP connections in 1 second even with a small link.
> 
These were different types of attacks, primarily originating from single 
IP addresses:

1. Dictionary attacks taking as much concurrent unauthenticated 
connections as possible and with speeds as fast as the server can 
respond. These were happening like a few up to several times a day, 
sometimes lasting hours.

2. Time based attacks taking again all of the available MaxStartups, but 
then doing nothing until the LoginGraceTime expires, then again, etc. 
These were not so frequent, but had the worst impact on the ssh 
availability.

3. Network scans on machines hosting some hundreds (or in some cases 
thousands) of IP addresses, causing outages lasting just a few minutes 
or so.

>> Last year I already had to decrease the LoginGraceTime from 120 to 30 
>> seconds on my production boxes, but it didn't help much, so on top of 
>> that I got to implement (reinvent the wheel again) a script tailing the 
>> auth.log and firewalling bad gyus in order to secure sshd and let my 
>> legitimate users in.
> 
> Are you trying to prevent the ssh scanners that just try well-known
> combinations of usernames and passwords? It is not clear that you
> gain much by firewalling these off, other than having fewer log
> messages.
> 
All of the above three. It wasn't just a matter of too much log 
messages. The type 1. for instance, besides the ssh unavailability, and 
depending on the MaxStartups setting, can bring a server to its knees by 
dedicating all of its available resources for bouncing unauthenticated 
ssh requests.

I tried setting a 'limit' ipfw firewall rule, something like:

   ipfw add allow tcp from any to any 22 in setup limit src-addr 5

I already had success with such a rule for a first level SMTP DoS 
protection before sendmail got its per-client and rate connection limits 
built in (since 8.13.0), and still keep that on, just in case.

But unlike sendmail, ssh instances when hammered with bogus requests are 
way more CPU intensive. I couldn't afford limiting the ssh connectivity 
to just one single session per client IP (someone might need multiple 
ssh sessions while working on something, right?), and in case of 
multiple sessions enabled, machines would be still vulnerable to CPU 
overload (i.e. bouncing tons of useless ssh authentication attempts).

So the best option for me was to implement a log analyzer script placing 
temporary blocks on the firewall when necessary. Like after 5 "Illegal 
user" or "Failed password for" or "Did not receive identification 
string" events, the script simply denies that IP right away on the 
firewall for one hour. So far this works well (for about 6 months 
already) and I no longer see unusual load spikes or ssh connectivity 
outages like before.

>> I really miss the inetd features. A setting like "nowait/100/20/5" 
>> (/max-child[/max-connections-per-ip-per-minute[/max-child-per-ip]]) 
>> would effectively bounce the bad guys, but AFAIK (correct me if I'm 
>> wrong), ssh is no longer supposed to work via inetd and still has no 
>> such capabilities.
> 
> You can still run sshd through inetd (or, at least, the -i option
> is still documented in the sshd man page). If does suggest that you
> may need to reduce the key size to make this practical (increasing
> LoginGraceTime here may help too ;-)
> 
I knew that, but actually never tried it thinking it would be too slow. 
Now I just ran a ssh-keygen and found that it takes only a few seconds 
for a 1024-bit and several for 2048-bit key. So it's not that much bad 
and running it with 512 or 1024-bit key through inetd seems feasible enough.

The default ssh key length in FreeBSD-6 however just got doubled from 
1024 to 2048-bit. I believe there's a reason for that and don't like the 
idea of going down.

Regards,
Atanas