Coping With 1 Million SSH Authentication Failures?

An anonymous reader writes "I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses. Our production servers, which host about 50 sites and generate ~20K hits/week, are managed by a 3rd party that I'm sure many on Slashdot would recognize. Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year. I contacted the ISP, who had promised me that server security would be actively managed, and their recommendation was, 'change the SSH port!' Of course this makes sense and may help to an extent, but it still doesn't solve the problem I'm facing: how do you manage server security on a tight budget with literally no system admin (except for me and I know I'm a n00b)? User passwords are randomly generated, we use a non-standard SSH port, and do not use any unencrypted services such as FTP. Is there a server monitoring program you would recommend? Is there an ISP or Web-based service that specializes in this?"

Comments

fail2ban

[Anonymous Coward]

fail2ban, key-auth

Re:fail2ban

[jimpop]

> fail2ban, key-auth

+1

Change port. Use iptables to only allow access from known subnets/hosts.

Re:fail2ban

[Kugrian]

Someone fucked up big time when they taught you html:
[url]http://www.fail2ban.org/[/url]

Re:fail2ban

[mellon]

This is really cool until you find yourself trying to log in from the same access point where somebody with a virus was attached earlier in the day. Better to just use crypto (key-based authentication only) and rate-limiting.

Re:fail2ban

[daveime]

Because it your production server is *nix, you are probably *already* running a shitload of random scripts written by unknown cunts ?

Re:fail2ban

[Sandman1971]

Not only use something like DenyHost or Fail2Ban, but firewall off all IP classes from Asia, South America and any other region that you know you'll never SSH from. I can easily get the /8s and such that are in use by each region (just look up Apnic, RIPE, etc...).

Re:fail2ban

[codeguy007]

Nobody's going to guess a 2048-bit RSA key.

No but they can steal if from your users' computers. It's one thing to have staff use keys when they are on your secure network but having users who are out on the web using keys when you can't control the security on their machines is only as secure as your dumbest client. Of course if they can hack and steal keys, they can probably steal passwords as well. For remote access one of the securest ways is using a security token. Every time user logs in they have to enter a different number.

Re:fail2ban

[gmack]

fail2ban firewalls off the port for a time you specify

DenyHosts blocks the ip in /etc/hosts.deny

I find fail2ban to be much more effective since I can use it for more than just SSH (on my system: ftp, imap, pop3, ssh, smtp). Some of the newer botnets will attempt to crack the password using another service and then try the resulting password on ssh so it's important to have more complete coverage.

Hardware firewall or use bfd

[Golbez81]

bfd (and apf if you like it) are probably the best software solutions your gonna find, but if you're facing 1 million+ auth failures, I would seriously consider a hardware firewall and VPN of some sort.

Tar Pitting

[spydum]

I'm a big fan of tarpitting SSH connections myself. It dramatically cuts down on the repeated ssh attempts, and keeps my logs much cleaner.

Basically, an iptables rule:
-A INPUTCHAIN -m state --state NEW -m recent -p tcp --dport 22 --update --seconds 30 --hitcount 4 --rttl --name SSH -j DROP

Re:Tar Pitting

[jonesy2k]

That's not tarpitting. Tarpitting involves keeping the connection open in order to tie up the resources of the attacking host.

Re:Tar Pitting

[Anonymous Coward]

I use a variation on tarpitting that has worked very well for me.
It cut the attempts down from 60,000/day to 20 or 30 per day.

I just add a small delay between the initial connection attempt
and when I send the username/password prompt. The delay (in
seconds) is the number of attempts in the last 30 minutes,
squared. This makes all but the most determined attacker
give up and go away very quickly.

I have been using this with both FTP and SSH for the last
year, and it works great.

Re:Tar Pitting

[Minwee]

Fail2ban can do stuff that automatically!

Let me see if I have this right. If iptables can keep count of incoming connections within the kernel and drop incoming connections as they happen without any intervention from a userspace program, that's _not_ automatic.

But running a gigantic shell script to scrape text messages out of your system logs and then call randomly chosen commands which may or may not have any effect on the observed problem some time after it occurs, then that's "doing that stuff automatically"?

Maybe those words don't mean what one of us thinks they do.

Re:Tar Pitting

[cptdondo]

That, and create only a single user who can log in, that takes you to the real log in prompt. That way an attacker has to guess the one user+password, and have a legitimate userid+password to gain access.

It's not foolproof, but it foils the vast majority of script attackers.

And DISABLE ROOT LOGIN!

Re:Tar Pitting

[IQgryn]

How would one go about implementing this delay?

Re:Tar Pitting

[schon]

iptables -N autoban
iptables -I INPUT -p TCP --dport 22 -j autoban
iptables -A autoban -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
iptables -A autoban -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP

Any site that connects more than 4 times in 60 seconds gets all packets subsequently dropped.

You could change DROP to TARPIT (if your kernel supports it) to fuck with them a little more.

Re:Tar Pitting

[JackieBrown]

Incidentally, I would quite like to know how this AC went about implementing this.

He'll be back. He has to wait an hour before being able to log in again.

Re:Tar Pitting

[StripedCow]

But what happens if you call scp from a number of scripts in parallel? You will get banned?

The problem is that you want to act only on the failed login attempts.

Passwords?

[fm6]

Here's my excuse to ride my usual hobbyhorse about passwords being obsolete. SSH supports certificate-based authentication, which is not only more secure, it's less of a hassle for the user. Don't know if it would be practical for your application (you might tell us more about that) but it's worth a look.

Re:Passwords?

[markdavis]

I agree that PKI would be more secure, but it is a LOT more hassle for most users. The simple fact is, a tarpit is *extremely* effective. Even a relatively weak password is going to be nearly impossible to break if the attackers are limited to only a few tries per hour or day or whatever.

fwknop

[c0d3g33k]

They can't fail authentication if they don't know it's there. http://cipherdyne.org/fwknop/

You are being brute-forced

[Anonymous Coward]

Welcome to the internet -- this happens to every site with a public IP.

First off, do not change your SSH port. It won't do a whole lot for you, and it will be more hassle than it works.

There are a whole lot of programs available to deal with SSH brute forcing. sshguard is one of them, and it's not too bad (you can apt-get install it). It's a bit of a hack -- it just watches your logs and takes appropriate action -- but it does work.

The other thing you can do immediately is simply turn off password authentication in ssh. Anyone getting in will need a key. This is what sourceforge and github both do. This isn't always practical for every site, but it will damn sure keep your passwords from being brute-forced.

Fail2ban or denyhosts

[mystik]

Fail2ban or denyhosts activly target ssh. fail2ban includes rules for other services, but denyhosts includes a mechanism for sharing lists of your denied hosts w/ other admins, as well as using their ban lists to protect your ssh logins.

SSH public key authentication

[whamett]

This kind of brute-forcing is common. The simplest way to deal with it is to set up SSH public key authentication, disable SSH password authentication, and forget about it.

Re:SSH public key authentication

[ColaMan]

The simplest way is to drop SSH and just use telnet. They won't be expecting that!

So?

[victim]

If you really have secure passwords, the random guessers aren't going to get them. Log it and move on.

I get thousands of Chinese hackers attempting to break into the battery monitor in my tool shed, big deal. I don't know why my battery voltage and solar current is so important to them, but they can knock themselves out all day.

If the logs bother you then install fail2ban and configure it to lock people out after a few bad guesses. (Then be ready to unlock yourself from an alternate IP when you inevitably lock yourself out.)

Move to a higher order port and use denyhosts

[deadmongrel]

I use one or more of these on my public facing servers.
1. Move the default ssh port to a higher order port (5000+)
2. Use Denyhosts http://denyhosts.sourceforge.net/ to block repeated attempts
3. use key exchange instead of username/password
4. use network based IPS.
Just moving the ssh port reduced the ssh brute force attack for me. Either stop being a noob or hire a sys admin.

Re:Move to a higher order port and use denyhosts

[sootman]

1. Move the default ssh port to a higher order port (5000+)

Agreed. The higher the better. For the ultimate in security, I recommend 65536.

Re:whatcouldposiblygowrong

[jo_ham]

He could get trolled on slashdot by the very people he's coming to ask for help to become *less* of a noob.

I'll bet you teach your kid gun safety by shooting him in the neck.

Throttle Inbound Connections

[Padrino121]

I have a similar situation and cannot limit to very specific IP ranges. I have done the following with good success. I pulled some examples from my configuration that can be tweaked for yours if you like.

1. Limit incoming SSH attempts to a low number. In my case I limit to 2 connections in 60 seconds. I can tighten it even more but this did a lot to kill brute force attempts.
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -j DROP
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state NEW -m limit --limit 2/min -j ACCEPT
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state RELATED,ESTABLISHED -j ACCEPT

2. Automatic blacklist via DenyHosts. This helps cut down attempts from known ranges without even giving them the chance even at a slow rate. http://denyhosts.sourceforge.net/

An iptables recipie

[Simon Carr]

-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name SSH --rsource -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j LOG --log-prefix "SSH_brute_force "
-A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j DROP

Stops 'em *somewhat* dead. If you want to whitelist hosts so they are not impacted by this rule, just add;

-A INPUT -s your.ip.address -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT

Before the throttling ruleset.

OpenBSD PF

[roman_mir]

#SSH
pass in inet proto tcp from any to $ext_if port ssh
pass out inet proto tcp from any to $ext_if port ssh
pass quick proto tcp from any to $ext_if port ssh \
        flags S/SA keep state \
        (max-src-conn 15, max-src-conn-rate 5/3, \
          overload <bruteforce> flush global)

you know what I am saying?

Re:Spend more than 10 minutes?

[Gearoid_Murphy]

add the ip address and/or hostname of all the hosts you use to access your servers into /etc/hosts.allow. If denyhosts picks up 3 failed logins from a single ip address, that address is added to /etc/hosts.deny, if this happens to be your machine (and you're having a bad day entering your password), you could get locked out of your system.

Re:Exactly

[Minwee]

"How can I stop people from driving past my house and seeing if the lights are on?"

That's easy, just move the front door to where one of your upstairs windows is and install tiny robots that will draw the curtains if the traffic noise gets too loud.

Re:whatcouldposiblygowrong

[dintech]

That's a beautiful and witty analogy but to reverse it:

If he asked "How can I stop my house being burgled?" and you said, "Without a full alarm, CCTV and patrol system, don't even bother. Leave it to the professionals."

Some others might see some practical value in suggesting, "Maybe lock your door at night."

My opinion in this situation is first to take what advice you can and do something yourself as soon as possible (since any extra security is better than nothing). Next, as you suggested and when he has the resources to employ someone to do it properly, seek professional help. Simply doing nothing until he can do it properly sounds a bit dangerous.

Re:Ignore it?

[tomhudson]

There's no "brute forcing" going on. Look at the numbers.

1 million per year over 50 sites == 20,000 per year per site, or 54 per day per site. Change the passwords every few months (and use different ones for each site).

Further - 1,200 different remote hosts means what, 17 attempts per remote host per site per year. Probably randomly p0wned PCs that hit addresses at random, make a few attempts using default or ocmmon passwords, then move on.

hashlimit

[suso]

You can also use the hashlimit module for iptables. I find it works pretty well for allowing people in who need to access and block things like the ssh worm that was causing ssh to run out of resources.

You might check out the end of this presentation that I made 4 years ago.

http://www.bloomingtonlinux.org/wiki/15th_meeting

I've been using hashlimit successfully for years in a web hosting environment.

I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!

Re:whatcouldposiblygowrong

[node 3]

Society places economic value on everything, and the amount of damage a cracked server farm can potentially do far outstrips the economic value of a human life.

You forgot to start that sentence with, "you might be a psychopath if..."

proactive blocking

[bugi]

In addition to using reactive tools like fail2ban and denyhost, also block most of the world proactively:

git clone git://github.com/bugi/iptables-by-country.git