Slashdot Mirror
Coping With 1 Million SSH Authentication Failures?
An anonymous reader writes "I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses. Our production servers, which host about 50 sites and generate ~20K hits/week, are managed by a 3rd party that I'm sure many on Slashdot would recognize. Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year. I contacted the ISP, who had promised me that server security would be actively managed, and their recommendation was, 'change the SSH port!' Of course this makes sense and may help to an extent, but it still doesn't solve the problem I'm facing: how do you manage server security on a tight budget with literally no system admin (except for me and I know I'm a n00b)? User passwords are randomly generated, we use a non-standard SSH port, and do not use any unencrypted services such as FTP. Is there a server monitoring program you would recommend? Is there an ISP or Web-based service that specializes in this?"
fail2ban, key-auth
bfd (and apf if you like it) are probably the best software solutions your gonna find, but if you're facing 1 million+ auth failures, I would seriously consider a hardware firewall and VPN of some sort.
I'm a big fan of tarpitting SSH connections myself. It dramatically cuts down on the repeated ssh attempts, and keeps my logs much cleaner.
Basically, an iptables rule:
-A INPUTCHAIN -m state --state NEW -m recent -p tcp --dport 22 --update --seconds 30 --hitcount 4 --rttl --name SSH -j DROP
sshblack works very well for me. It monitors failed ssh connections and inserts entries into iptables to block the IPs.
Here's my excuse to ride my usual hobbyhorse about passwords being obsolete. SSH supports certificate-based authentication, which is not only more secure, it's less of a hassle for the user. Don't know if it would be practical for your application (you might tell us more about that) but it's worth a look.
They can't fail authentication if they don't know it's there. http://cipherdyne.org/fwknop/
Welcome to the internet -- this happens to every site with a public IP.
First off, do not change your SSH port. It won't do a whole lot for you, and it will be more hassle than it works.
There are a whole lot of programs available to deal with SSH brute forcing. sshguard is one of them, and it's not too bad (you can apt-get install it). It's a bit of a hack -- it just watches your logs and takes appropriate action -- but it does work.
The other thing you can do immediately is simply turn off password authentication in ssh. Anyone getting in will need a key. This is what sourceforge and github both do. This isn't always practical for every site, but it will damn sure keep your passwords from being brute-forced.
Fail2ban or denyhosts activly target ssh. fail2ban includes rules for other services, but denyhosts includes a mechanism for sharing lists of your denied hosts w/ other admins, as well as using their ban lists to protect your ssh logins.
Why aren't you encrypting your e-mail?
This kind of brute-forcing is common. The simplest way to deal with it is to set up SSH public key authentication, disable SSH password authentication, and forget about it.
Seriously. I'm in the same shoes and it's easy. I came across it pretty quick and all these SSH log in problems went away.
Check out...waaaaait...
What's most troubling is this:
I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses
You operate a business and you haven't figure out this chief problem yet...but you want us to help you out?
Well...Here's the solution: denyhosts
Someone else'll chime in with it....Just hope you read up and configure it properly....or you'll find yourself locked out of your own servers.
import system.cool.Sig;
If you really have secure passwords, the random guessers aren't going to get them. Log it and move on.
I get thousands of Chinese hackers attempting to break into the battery monitor in my tool shed, big deal. I don't know why my battery voltage and solar current is so important to them, but they can knock themselves out all day.
If the logs bother you then install fail2ban and configure it to lock people out after a few bad guesses. (Then be ready to unlock yourself from an alternate IP when you inevitably lock yourself out.)
I use one or more of these on my public facing servers.
1. Move the default ssh port to a higher order port (5000+)
2. Use Denyhosts http://denyhosts.sourceforge.net/ to block repeated attempts
3. use key exchange instead of username/password
4. use network based IPS.
Just moving the ssh port reduced the ssh brute force attack for me. Either stop being a noob or hire a sys admin.
We started using BlockHosts to feed iptables rules, and our failure logs went from 30-50k per day to 100. Basically, with more than 'x' failed logins within 'y' time frame, the source IP is blocked for 'z' time period. Since it uses iptables, you could block it from just the ssh port, or the entire system (we do the latter).
All three variables are configurable, and we also have whitelisted a few select standby IPs for contingency use. (As another poster said, you **will** lock yourself out eventually.)
He could get trolled on slashdot by the very people he's coming to ask for help to become *less* of a noob.
I'll bet you teach your kid gun safety by shooting him in the neck.
I have a similar situation and cannot limit to very specific IP ranges. I have done the following with good success. I pulled some examples from my configuration that can be tweaked for yours if you like.
1. Limit incoming SSH attempts to a low number. In my case I limit to 2 connections in 60 seconds. I can tighten it even more but this did a lot to kill brute force attempts.
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -j DROP
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state NEW -m limit --limit 2/min -j ACCEPT
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state RELATED,ESTABLISHED -j ACCEPT
2. Automatic blacklist via DenyHosts. This helps cut down attempts from known ranges without even giving them the chance even at a slow rate. http://denyhosts.sourceforge.net/
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name SSH --rsource -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j LOG --log-prefix "SSH_brute_force "
-A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j DROP
Stops 'em *somewhat* dead. If you want to whitelist hosts so they are not impacted by this rule, just add;
-A INPUT -s your.ip.address -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
Before the throttling ruleset.
-- The unsig...
you know what I am saying?
You can't handle the truth.
http://lmgtfy.com/?q=pam_abl
Share and enjoy!
If somebody tries to perform surgery without a proper medical education, do you offer pointers, or do you tell them to drop the knife and find a real surgeon?
Companies running public-facing servers without good technical expertise are a menace.
So there have been a million failed attempts to log in over SSH in a year. That's about one failed attempt every 30 seconds.
What's wrong with that? It's not like they're getting in, right?
Pretty good is actually pretty bad.
That's easy, just move the front door to where one of your upstairs windows is and install tiny robots that will draw the curtains if the traffic noise gets too loud.
That's a beautiful and witty analogy but to reverse it:
If he asked "How can I stop my house being burgled?" and you said, "Without a full alarm, CCTV and patrol system, don't even bother. Leave it to the professionals."
Some others might see some practical value in suggesting, "Maybe lock your door at night."
My opinion in this situation is first to take what advice you can and do something yourself as soon as possible (since any extra security is better than nothing). Next, as you suggested and when he has the resources to employ someone to do it properly, seek professional help. Simply doing nothing until he can do it properly sounds a bit dangerous.
Since it seems to have escaped your attention, I'd like to point out the fact that unqualified surgeons can (will ?) kill people; unqualified admins won't.
Which seems to be a somewhat relevant matter.
The Cloud - because you don't care if your apps and data are up in the air.
1 million per year over 50 sites == 20,000 per year per site, or 54 per day per site. Change the passwords every few months (and use different ones for each site).
Further - 1,200 different remote hosts means what, 17 attempts per remote host per site per year. Probably randomly p0wned PCs that hit addresses at random, make a few attempts using default or ocmmon passwords, then move on.
But a million cars haven't driven past. Only ~1200 --- in the past year. This is no big deal. Hell, I start thinking something is wrong when I DON'T see many failed SSH attempts in the daily log reports.
You can also use the hashlimit module for iptables. I find it works pretty well for allowing people in who need to access and block things like the ssh worm that was causing ssh to run out of resources.
You might check out the end of this presentation that I made 4 years ago.
http://www.bloomingtonlinux.org/wiki/15th_meeting
I've been using hashlimit successfully for years in a web hosting environment.
I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!
In fairness, surgery is WAY MORE FUCKING IMPORTANT than server management.
People without perspective are a menace.
Practically speaking, it really isn't different.
The surgeon who kills someone might, if the victim's family is very lucky, be assessed damages of a couple million. An idiot with a cracked server farm can easily cause that much damage on the net.
WTF? In one case, a web server has been vandalized. In the other, a person is dead.
Society places economic value on everything, and the amount of damage a cracked server farm can potentially do far outstrips the economic value of a human life.
You forgot to start that sentence with, "you might be a psychopath if..."
I'm sorry you're too psychologically immature to recognize economic reality, but that's really not my problem, and hardly makes me a psychopath.
It's not a sign of being "psychologically mature" to hold the notion that human life is a fungible commodity.
A poor person in the slums of Bangalore is economically worth more dead (for his organs) than alive. You state that society places an economic value on everything, including people. Yet for some reason, society has outlawed murder, even if the result is a net monetary profit for society.
Your complete failure to comprehend that a human life is worth more than its economic value to society is why I called you a psychopath.
I actually go one step further: after IPv6-enabling my site, I only allow v6 ssh inbound. Since Teredo makes it possible to get v6 from nearly anywhere, it doesn't cause any inconvenience, and ssh attacks have basically vanished. 'course, it won't last forever, but it works great right now.
In addition to using reactive tools like fail2ban and denyhost, also block most of the world proactively:
git clone git://github.com/bugi/iptables-by-country.git
It's fucking computer. He can learn how to secure it. There's no reason to hire someone else to do something that he can do himself, especially when it's something that he needs to know how to do.
Don't take life so seriously. No one makes it out alive.
Instead of generating passwords, have your users supply their SSH key and turn off passwords. Much better solution.
The above is not worth reading.
I can't believe the person who set the server up left this port open to the world. If you want the server secure then you will close all the ports that are not actually needed - in your case this likely means you keep 80 and 443 open.
For remote management make get a few IPs for the desktops that are meant to connect to it and make sure the ISP opens SSH port only to the IPs.
You can tunnel X over SSH and do file management, whatever you need so it should be enough.
Don't bother changing it to some random port, security through obscurity is total bullshit in this age of port scanners.
All those moments will be lost in time, like tears in rain.
I am using fail2ban, but I do not think it's a particularly great tool when you see how many different IP addresses the attacks come from, so you're somewhere stuck in the middle trying to optimise how to make fail2ban more effective, while not being a problem for your own users:
- you configure blocking from 1 login attempt and a long block time cuts down on most of the outside attacks, but if some legitimate user mistypes his password, he'll be locked out for that same long block time...
- you configure several wrong password attempts and/or shorter blocking times - then the attacks do not get slowed quite as much, but it hinders your users less.
There are some things you might consider, though -
- if you can, and your users always access the system from their own computers/laptops, force the use of public-key authentication and disable use of password logins; then outside access attempts via password cannot succeed.
- if you can't do this, at least force root to use public key (and/or su/sudo from another login. (sshd_config either
'PermitRootLogin without-password' (to allow root access through authorized_keys, but not through password, or
'PermitRootLogin no' to block root access via ssh-login altogether.
- if there are only specific accounts you might want to allow access to, look into pam_require, then limit to those users in pam's ssh
config: 'account required pam_require.so @ssh-users myaccount' (allows user 'myaccount' and all users in group 'ssh-users'
to log in via ssh - everyone else will be kicked out, even with a valid password).
- keep an eye on the log to see whether it's actual users that get hit, or whether you see access attempts to non-existent users
(many of the attacks are basically dictionary based). If you find one of the valid accounts gets hit too much, consider renaming that
user account.
- you configure several wrong password attempts and/or shorter blocking times - then the attacks do not get slowed quite as much, but it hinders your users less.
There are some things you might consider, though -
3 fails / 10 min != 10 fails/ 1 sec