Coping With 1 Million SSH Authentication Failures?

An anonymous reader writes "I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses. Our production servers, which host about 50 sites and generate ~20K hits/week, are managed by a 3rd party that I'm sure many on Slashdot would recognize. Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year. I contacted the ISP, who had promised me that server security would be actively managed, and their recommendation was, 'change the SSH port!' Of course this makes sense and may help to an extent, but it still doesn't solve the problem I'm facing: how do you manage server security on a tight budget with literally no system admin (except for me and I know I'm a n00b)? User passwords are randomly generated, we use a non-standard SSH port, and do not use any unencrypted services such as FTP. Is there a server monitoring program you would recommend? Is there an ISP or Web-based service that specializes in this?"

Comments

fail2ban

[Anonymous Coward]

fail2ban, key-auth

Re:fail2ban

[jimpop]

> fail2ban, key-auth

+1

Change port. Use iptables to only allow access from known subnets/hosts.

Re:fail2ban

[Sandman1971]

Not only use something like DenyHost or Fail2Ban, but firewall off all IP classes from Asia, South America and any other region that you know you'll never SSH from. I can easily get the /8s and such that are in use by each region (just look up Apnic, RIPE, etc...).

Re:fail2ban

[gmack]

fail2ban firewalls off the port for a time you specify

DenyHosts blocks the ip in /etc/hosts.deny

I find fail2ban to be much more effective since I can use it for more than just SSH (on my system: ftp, imap, pop3, ssh, smtp). Some of the newer botnets will attempt to crack the password using another service and then try the resulting password on ssh so it's important to have more complete coverage.

Hardware firewall or use bfd

[Golbez81]

bfd (and apf if you like it) are probably the best software solutions your gonna find, but if you're facing 1 million+ auth failures, I would seriously consider a hardware firewall and VPN of some sort.

Tar Pitting

[spydum]

I'm a big fan of tarpitting SSH connections myself. It dramatically cuts down on the repeated ssh attempts, and keeps my logs much cleaner.

Basically, an iptables rule:
-A INPUTCHAIN -m state --state NEW -m recent -p tcp --dport 22 --update --seconds 30 --hitcount 4 --rttl --name SSH -j DROP

Re:Tar Pitting

[jonesy2k]

That's not tarpitting. Tarpitting involves keeping the connection open in order to tie up the resources of the attacking host.

Re:Tar Pitting

[schon]

iptables -N autoban
iptables -I INPUT -p TCP --dport 22 -j autoban
iptables -A autoban -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
iptables -A autoban -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP

Any site that connects more than 4 times in 60 seconds gets all packets subsequently dropped.

You could change DROP to TARPIT (if your kernel supports it) to fuck with them a little more.

Passwords?

[fm6]

Here's my excuse to ride my usual hobbyhorse about passwords being obsolete. SSH supports certificate-based authentication, which is not only more secure, it's less of a hassle for the user. Don't know if it would be practical for your application (you might tell us more about that) but it's worth a look.

fwknop

[c0d3g33k]

They can't fail authentication if they don't know it's there. http://cipherdyne.org/fwknop/

You are being brute-forced

[Anonymous Coward]

Welcome to the internet -- this happens to every site with a public IP.

First off, do not change your SSH port. It won't do a whole lot for you, and it will be more hassle than it works.

There are a whole lot of programs available to deal with SSH brute forcing. sshguard is one of them, and it's not too bad (you can apt-get install it). It's a bit of a hack -- it just watches your logs and takes appropriate action -- but it does work.

The other thing you can do immediately is simply turn off password authentication in ssh. Anyone getting in will need a key. This is what sourceforge and github both do. This isn't always practical for every site, but it will damn sure keep your passwords from being brute-forced.

Fail2ban or denyhosts

[mystik]

Fail2ban or denyhosts activly target ssh. fail2ban includes rules for other services, but denyhosts includes a mechanism for sharing lists of your denied hosts w/ other admins, as well as using their ban lists to protect your ssh logins.

SSH public key authentication

[whamett]

This kind of brute-forcing is common. The simplest way to deal with it is to set up SSH public key authentication, disable SSH password authentication, and forget about it.

Move to a higher order port and use denyhosts

[deadmongrel]

I use one or more of these on my public facing servers.
1. Move the default ssh port to a higher order port (5000+)
2. Use Denyhosts http://denyhosts.sourceforge.net/ to block repeated attempts
3. use key exchange instead of username/password
4. use network based IPS.
Just moving the ssh port reduced the ssh brute force attack for me. Either stop being a noob or hire a sys admin.

Throttle Inbound Connections

[Padrino121]

I have a similar situation and cannot limit to very specific IP ranges. I have done the following with good success. I pulled some examples from my configuration that can be tweaked for yours if you like.

1. Limit incoming SSH attempts to a low number. In my case I limit to 2 connections in 60 seconds. I can tighten it even more but this did a lot to kill brute force attempts.
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -j DROP
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state NEW -m limit --limit 2/min -j ACCEPT
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state RELATED,ESTABLISHED -j ACCEPT

2. Automatic blacklist via DenyHosts. This helps cut down attempts from known ranges without even giving them the chance even at a slow rate. http://denyhosts.sourceforge.net/

Re:Spend more than 10 minutes?

[Gearoid_Murphy]

add the ip address and/or hostname of all the hosts you use to access your servers into /etc/hosts.allow. If denyhosts picks up 3 failed logins from a single ip address, that address is added to /etc/hosts.deny, if this happens to be your machine (and you're having a bad day entering your password), you could get locked out of your system.

hashlimit

[suso]

You can also use the hashlimit module for iptables. I find it works pretty well for allowing people in who need to access and block things like the ssh worm that was causing ssh to run out of resources.

You might check out the end of this presentation that I made 4 years ago.

http://www.bloomingtonlinux.org/wiki/15th_meeting

I've been using hashlimit successfully for years in a web hosting environment.

I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!

proactive blocking

[bugi]

In addition to using reactive tools like fail2ban and denyhost, also block most of the world proactively:

git clone git://github.com/bugi/iptables-by-country.git