Slashdot Mirror
Coping With 1 Million SSH Authentication Failures?
An anonymous reader writes "I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses. Our production servers, which host about 50 sites and generate ~20K hits/week, are managed by a 3rd party that I'm sure many on Slashdot would recognize. Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year. I contacted the ISP, who had promised me that server security would be actively managed, and their recommendation was, 'change the SSH port!' Of course this makes sense and may help to an extent, but it still doesn't solve the problem I'm facing: how do you manage server security on a tight budget with literally no system admin (except for me and I know I'm a n00b)? User passwords are randomly generated, we use a non-standard SSH port, and do not use any unencrypted services such as FTP. Is there a server monitoring program you would recommend? Is there an ISP or Web-based service that specializes in this?"
fail2ban, key-auth
If you have a public site, it'll get scanned night and day by APNIC. If they see a banner for SSH, they'll jump on that. As long as you are using patched software and have good user security, what's the real emergency?
bfd (and apf if you like it) are probably the best software solutions your gonna find, but if you're facing 1 million+ auth failures, I would seriously consider a hardware firewall and VPN of some sort.
I'm a big fan of tarpitting SSH connections myself. It dramatically cuts down on the repeated ssh attempts, and keeps my logs much cleaner.
Basically, an iptables rule:
-A INPUTCHAIN -m state --state NEW -m recent -p tcp --dport 22 --update --seconds 30 --hitcount 4 --rttl --name SSH -j DROP
Here's my excuse to ride my usual hobbyhorse about passwords being obsolete. SSH supports certificate-based authentication, which is not only more secure, it's less of a hassle for the user. Don't know if it would be practical for your application (you might tell us more about that) but it's worth a look.
They can't fail authentication if they don't know it's there. http://cipherdyne.org/fwknop/
Welcome to the internet -- this happens to every site with a public IP.
First off, do not change your SSH port. It won't do a whole lot for you, and it will be more hassle than it works.
There are a whole lot of programs available to deal with SSH brute forcing. sshguard is one of them, and it's not too bad (you can apt-get install it). It's a bit of a hack -- it just watches your logs and takes appropriate action -- but it does work.
The other thing you can do immediately is simply turn off password authentication in ssh. Anyone getting in will need a key. This is what sourceforge and github both do. This isn't always practical for every site, but it will damn sure keep your passwords from being brute-forced.
Fail2ban or denyhosts activly target ssh. fail2ban includes rules for other services, but denyhosts includes a mechanism for sharing lists of your denied hosts w/ other admins, as well as using their ban lists to protect your ssh logins.
Why aren't you encrypting your e-mail?
This kind of brute-forcing is common. The simplest way to deal with it is to set up SSH public key authentication, disable SSH password authentication, and forget about it.
Seriously. I'm in the same shoes and it's easy. I came across it pretty quick and all these SSH log in problems went away.
Check out...waaaaait...
What's most troubling is this:
I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses
You operate a business and you haven't figure out this chief problem yet...but you want us to help you out?
Well...Here's the solution: denyhosts
Someone else'll chime in with it....Just hope you read up and configure it properly....or you'll find yourself locked out of your own servers.
import system.cool.Sig;
I use one or more of these on my public facing servers.
1. Move the default ssh port to a higher order port (5000+)
2. Use Denyhosts http://denyhosts.sourceforge.net/ to block repeated attempts
3. use key exchange instead of username/password
4. use network based IPS.
Just moving the ssh port reduced the ssh brute force attack for me. Either stop being a noob or hire a sys admin.
We started using BlockHosts to feed iptables rules, and our failure logs went from 30-50k per day to 100. Basically, with more than 'x' failed logins within 'y' time frame, the source IP is blocked for 'z' time period. Since it uses iptables, you could block it from just the ssh port, or the entire system (we do the latter).
All three variables are configurable, and we also have whitelisted a few select standby IPs for contingency use. (As another poster said, you **will** lock yourself out eventually.)
I have a similar situation and cannot limit to very specific IP ranges. I have done the following with good success. I pulled some examples from my configuration that can be tweaked for yours if you like.
1. Limit incoming SSH attempts to a low number. In my case I limit to 2 connections in 60 seconds. I can tighten it even more but this did a lot to kill brute force attempts.
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -j DROP
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state NEW -m limit --limit 2/min -j ACCEPT
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state RELATED,ESTABLISHED -j ACCEPT
2. Automatic blacklist via DenyHosts. This helps cut down attempts from known ranges without even giving them the chance even at a slow rate. http://denyhosts.sourceforge.net/
You can also use the hashlimit module for iptables. I find it works pretty well for allowing people in who need to access and block things like the ssh worm that was causing ssh to run out of resources.
You might check out the end of this presentation that I made 4 years ago.
http://www.bloomingtonlinux.org/wiki/15th_meeting
I've been using hashlimit successfully for years in a web hosting environment.
I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!
In addition to using reactive tools like fail2ban and denyhost, also block most of the world proactively:
git clone git://github.com/bugi/iptables-by-country.git
Instead of generating passwords, have your users supply their SSH key and turn off passwords. Much better solution.
The above is not worth reading.
- you configure several wrong password attempts and/or shorter blocking times - then the attacks do not get slowed quite as much, but it hinders your users less.
There are some things you might consider, though -
3 fails / 10 min != 10 fails/ 1 sec