Slashdot Mirror
Coping With 1 Million SSH Authentication Failures?
An anonymous reader writes "I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses. Our production servers, which host about 50 sites and generate ~20K hits/week, are managed by a 3rd party that I'm sure many on Slashdot would recognize. Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year. I contacted the ISP, who had promised me that server security would be actively managed, and their recommendation was, 'change the SSH port!' Of course this makes sense and may help to an extent, but it still doesn't solve the problem I'm facing: how do you manage server security on a tight budget with literally no system admin (except for me and I know I'm a n00b)? User passwords are randomly generated, we use a non-standard SSH port, and do not use any unencrypted services such as FTP. Is there a server monitoring program you would recommend? Is there an ISP or Web-based service that specializes in this?"
fail2ban, key-auth
If you have a public site, it'll get scanned night and day by APNIC. If they see a banner for SSH, they'll jump on that. As long as you are using patched software and have good user security, what's the real emergency?
bfd (and apf if you like it) are probably the best software solutions your gonna find, but if you're facing 1 million+ auth failures, I would seriously consider a hardware firewall and VPN of some sort.
I'm a big fan of tarpitting SSH connections myself. It dramatically cuts down on the repeated ssh attempts, and keeps my logs much cleaner.
Basically, an iptables rule:
-A INPUTCHAIN -m state --state NEW -m recent -p tcp --dport 22 --update --seconds 30 --hitcount 4 --rttl --name SSH -j DROP
sshblack works very well for me. It monitors failed ssh connections and inserts entries into iptables to block the IPs.
if you can manage the set of users of your server, you can use OpenVPN and then SSH. OpenVPN has a "feature" that if each packet of the VPN is not digitally signed by a previously arranged (and distribuyed) key, then the packet is "ignored". After the VPN session is established/authenticated, your users can log in using ssh. There are even some virtual appliances and special distros (untangle.com) that have a "openvpn appliance" built in for this purpose. The how-to for openvpn is also easy to follow.
People are just paranoid. "How can I stop people from driving past my house and seeing if the lights are on?" You can't, cheaply.
what could go wrong?
worst case scenario? Slashdot trolls.
Here's my excuse to ride my usual hobbyhorse about passwords being obsolete. SSH supports certificate-based authentication, which is not only more secure, it's less of a hassle for the user. Don't know if it would be practical for your application (you might tell us more about that) but it's worth a look.
They can't fail authentication if they don't know it's there. http://cipherdyne.org/fwknop/
Welcome to the internet -- this happens to every site with a public IP.
First off, do not change your SSH port. It won't do a whole lot for you, and it will be more hassle than it works.
There are a whole lot of programs available to deal with SSH brute forcing. sshguard is one of them, and it's not too bad (you can apt-get install it). It's a bit of a hack -- it just watches your logs and takes appropriate action -- but it does work.
The other thing you can do immediately is simply turn off password authentication in ssh. Anyone getting in will need a key. This is what sourceforge and github both do. This isn't always practical for every site, but it will damn sure keep your passwords from being brute-forced.
Use public key authentication. It's super easy and much more secure, unless your random passwords happen to be over 300 characters long (ssh-keygen makes 2048-bit keys by default; assume 6 bits per character in a random password if each character is in the base64 set). Also, sshguard is your friend. After a small number of invalid logins - 4 by default - that IP gets firewalled for a short period of time.
Dewey, what part of this looks like authorities should be involved?
Fail2ban or denyhosts activly target ssh. fail2ban includes rules for other services, but denyhosts includes a mechanism for sharing lists of your denied hosts w/ other admins, as well as using their ban lists to protect your ssh logins.
Why aren't you encrypting your e-mail?
This kind of brute-forcing is common. The simplest way to deal with it is to set up SSH public key authentication, disable SSH password authentication, and forget about it.
If your not willing to invest money into intrusion detection, vulnerability scanning, or moving to a more responsive provider, accept that your server is going to constantly be scanned by the droves of script kiddies out there throwing everything but the kitchen sink at any server that responds to a ping and keep monitoring those logs, which hopefully are stored on a separate server in case you are ever actually compromised. In the mean time, try installing Fail2Ban on your server. It will block an IP address after X number of failed authentication attempts, which will alleviate the noise of the brute force password guessing attempts.
Everybody wants free porn. You're just going to have to live with that. You sleep with dogs, you're bound to get flees (unless you have mandatory government filters).
Seriously. I'm in the same shoes and it's easy. I came across it pretty quick and all these SSH log in problems went away.
Check out...waaaaait...
What's most troubling is this:
I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses
You operate a business and you haven't figure out this chief problem yet...but you want us to help you out?
Well...Here's the solution: denyhosts
Someone else'll chime in with it....Just hope you read up and configure it properly....or you'll find yourself locked out of your own servers.
import system.cool.Sig;
Great tool that pools the resources of multiple servers to proactively block IPs that are trying to brute-force the SSH port on any server in the pool:
http://denyhosts.sourceforge.net/
I use it, and it has Just Worked for years.
Matthew Walker
http://www.tweeterdiet.com/ - My Diet Tracking Tool
If you really have secure passwords, the random guessers aren't going to get them. Log it and move on.
I get thousands of Chinese hackers attempting to break into the battery monitor in my tool shed, big deal. I don't know why my battery voltage and solar current is so important to them, but they can knock themselves out all day.
If the logs bother you then install fail2ban and configure it to lock people out after a few bad guesses. (Then be ready to unlock yourself from an alternate IP when you inevitably lock yourself out.)
I use one or more of these on my public facing servers.
1. Move the default ssh port to a higher order port (5000+)
2. Use Denyhosts http://denyhosts.sourceforge.net/ to block repeated attempts
3. use key exchange instead of username/password
4. use network based IPS.
Just moving the ssh port reduced the ssh brute force attack for me. Either stop being a noob or hire a sys admin.
Key-based authentication is really what you need. If your situation doesn't allow for key-based auth, Fail2Ban can help reduce brute-force attempts for password-based auth, and allowing connections from specific IPs, if possible, can heighten security quite nicely.
We started using BlockHosts to feed iptables rules, and our failure logs went from 30-50k per day to 100. Basically, with more than 'x' failed logins within 'y' time frame, the source IP is blocked for 'z' time period. Since it uses iptables, you could block it from just the ssh port, or the entire system (we do the latter).
All three variables are configurable, and we also have whitelisted a few select standby IPs for contingency use. (As another poster said, you **will** lock yourself out eventually.)
He could get trolled on slashdot by the very people he's coming to ask for help to become *less* of a noob.
I'll bet you teach your kid gun safety by shooting him in the neck.
I have a similar situation and cannot limit to very specific IP ranges. I have done the following with good success. I pulled some examples from my configuration that can be tweaked for yours if you like.
1. Limit incoming SSH attempts to a low number. In my case I limit to 2 connections in 60 seconds. I can tighten it even more but this did a lot to kill brute force attempts.
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -j DROP
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state NEW -m limit --limit 2/min -j ACCEPT
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state RELATED,ESTABLISHED -j ACCEPT
2. Automatic blacklist via DenyHosts. This helps cut down attempts from known ranges without even giving them the chance even at a slow rate. http://denyhosts.sourceforge.net/
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name SSH --rsource -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j LOG --log-prefix "SSH_brute_force "
-A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j DROP
Stops 'em *somewhat* dead. If you want to whitelist hosts so they are not impacted by this rule, just add;
-A INPUT -s your.ip.address -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
Before the throttling ruleset.
-- The unsig...
http://www.fail2ban.org/wiki/index.php/Main_Page
That should cover automatic banning of repeat offenders. It works by reading logs, so it can work with many services. You could for example make it block all those attempting to exploit your CMSes.
you know what I am saying?
You can't handle the truth.
http://lmgtfy.com/?q=pam_abl
Share and enjoy!
If somebody tries to perform surgery without a proper medical education, do you offer pointers, or do you tell them to drop the knife and find a real surgeon?
Companies running public-facing servers without good technical expertise are a menace.
So there have been a million failed attempts to log in over SSH in a year. That's about one failed attempt every 30 seconds.
What's wrong with that? It's not like they're getting in, right?
Pretty good is actually pretty bad.
Peter N. M. Hansteen has written a nice article about a similar atack. http://bsdly.blogspot.com/2009/11/rickrolled-get-ready-for-hail-mary.html The first thing I would do (at install time) is to disable root login over ssh.
We use denyhosts on our production servers. Works like a charm. Just make sure you have an alternate ip to log on from in case you lock yourself out.
OSSEC is what I've been using for years and it works well. IT's much more comprehensive a security package than fail2ban but uses log monitoring as it's basis.
It's worth a look.
Change the port and use private certificates for authentication.
However, changing the port to something will get the bots off your back. They still don't try other ports.
---- aut viam inveniam aut faciam
Change the default port, disable SSH access for root, disable password access entirely (login with public keys), install fcheck to monitor changed files and hence intrusions. If you have the luxury, remove SSH access entirely from your web server and block everything but ports 80 and 443, and enter via another server behind the same firewall. As a nOOb, I gained quite a lot from following this Hardening Linux Web Servers guide: http://www.freesoftwaremagazine.com/articles/hardening_linux
Although the solutions like fail2ban and denyhosts are undeniably useful, you will cut down well over 90% of the drive-by ssh attempts by simply changing the port.
None of the scripts, that I'm aware of, actually portscan your machine looking for the SSH login headers, they all hit port 22 and try to log in. It takes too long to scan a machine so if nothing is listening on port 22, they will move on.
You will still get the odd attempt from someone who has portscanned you - these will be relatively rare. In these cases simply disable password logins altogether and use the public key authentication that is a LOT more secure (and convenient once it's all set up) anyway.
Specialist Mac support for creative pros, Melbourne
RSA keys are notoriously time intensive to brute-force. As far as I know, no one is crazy enough to automate an attack. If you disable password authentication altogether, you eliminate your risk of an automated dictionary attack. It might confuse the hell out of some users, which is a con.
--dport 22 -i DROP
Use a high SSH port and allow logins from two static IPs -j DROP everything else.
Hire me to fix the rest ;)
This is pretty common stuff trying random SSH logins on public facing servers, at least for me - just change the SSH port and have strong SSH passwords - that's all I do and stops it dead.
If you are really worried, firewall/iptables to fixed IP addresses.
First..... 1 million password *failures* is not a security problem by itself. They're supposed to fail if they don't have the right password.
So - moving this to a more obscure port will definitely cut down on the automated scanning.. but it's not really the heart of the issue.
The heart of the issue is that it took you so long to realize someone was even doing this. That probably boils down to your running a business requiring sysadmin work without a sysadmin.
There are lots and lots of ways to approach this, some of which people have touched on already.
1) You need awareness of what's going on - daily log analysis. OSSEC is neat, give it a shot.
2) Moving SSH to a weird port is something some people do - I don't prefer that - it buys some obscurity and cuts down on some scanning, but doesn't address any real security issues.
3) OSSEC in active mode with a whitelist so you don't block out legitimate users - works pretty well.
4) Hire someone to deal with security/deployment/backups/recovery/service providers (sysadmin?)
Obviously the port change idea is unpopular because it's just obscurity, which != security. However, most, of not all, of those attempts are probably from scripts rather than actual human attackers actively attempting to compromise your servers. The script kiddies rely on automation to rapidly find vulnerable hosts using easily-scripted vulnerabilities or brute force. The simple act of changing your ssh port will render those kinds of attacks useless against you. I had a similar scenario where my logs were full of failed ssh attempts and by changing the ssh port, they all stopped. ALL OF THEM. I haven't had an unknown IP attempt to log in via SSH in around five years. fail2ban, strong passwords, or whatever additional security you want to apply is great, and recommended, but a port change will solve more of the problem than you might think.
That's a hell of a lot better than some "l33t h$ck3r" who does not know his ass from a whole in the ground. I will take the "noob" any day. They at least show an open mind and admit their weaknesses.
I've used ISPs that provided separate VPN access for protected ports such as SSH, so you only had to open HTTP to the public. But seriously, there options such as port knockers and the like. Of course, if they are actively managing, they could try to have a firewall that automatically filtered obvious break in attempts.
And then just ignore the attempts. fail2ban can sometimes be ok, but be aware that it creates a denial-of-service vulnerability that is exploitable by attackers who can can spoof source addresses.
http://outcampaign.org/
Then the answer to the question is the link to someone who can provide that.
I doubt the responses would have been quite so hostile if the topic had been car repair - poorly maintained vehicles are a menace on the roads!
The answer does not necessarily have to be "here is what you do" - it can be "find someone who knows what they are doing". Trolling him is not likely to create any positive effect, except among the already insular server admin clique. Save the "oh my god, so this noob totally connected a 54-50 to a 35-27 with no logging!" drinking stories for the bar.
"Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year."
What's to cope with? This isn't a problem - this is the internet. Unless servers were actually breached, or it's actually costing you money - why do you care? If your password practices are solid as you say - you should be fine.
People attempting to log in is not a security issue - people logging in is.
How do you manage server security with no system admin? Either pay the extra money for managed hosting where they do this for you, or hire someone to do it for you on staff. If those aren't viable - change your business plan (I mean, your business plan involves administering servers - somewhere along the way you made a decision that administering them yourself was the way to go for your business..... so why would you not hire a sysadmin?)
That's a beautiful and witty analogy but to reverse it:
If he asked "How can I stop my house being burgled?" and you said, "Without a full alarm, CCTV and patrol system, don't even bother. Leave it to the professionals."
Some others might see some practical value in suggesting, "Maybe lock your door at night."
My opinion in this situation is first to take what advice you can and do something yourself as soon as possible (since any extra security is better than nothing). Next, as you suggested and when he has the resources to employ someone to do it properly, seek professional help. Simply doing nothing until he can do it properly sounds a bit dangerous.
I hate to sound unsympathetic here, but the Internet is a hostile place. If you have port 22 open to the world, you should expect to get pummelled with password cracking attempts more or less constantly.
Either learn to live with it, or at least take some steps to slow them down. I find that throttling back the number of connections any one IP address can make in a short time pretty much slows it down to a reasonable level. Alternatively, you can also put some "trap" logins on your system. Usernames like "admin" or "oracle" with the password set to the same as the username are good. When the login succeeds, you immediately drop the connection and lock the source IP address out using iptables.
Unless it's really causing a lot of pain, though, please don't harass the ISP from which the connection is originating. You're just going to make some overworked sysadmin's day a little more unpleasant. Chances are, they're already doing everything they can to keep rogues off their network.
Tired of FB/Google censorship? Visit UNCENSORED!
what could go wrong?
Lots.
worst case scenario?
Depends how realistic we are being with our range of possible scenarios.
Worst not-ridiculously-unlikely case: ill configured service, service minus important patches, or poorly chosen authentication/privilege setup, or something similar, allows an automated hack in and therefore the server becomes yet another bot in the net or possibly even a C&C box for the botnet.
A zombied server is usually more trouble for the rest of the 'net then a zombied home machine because they generally have access to better bandwidth resources and are more likely to be openly connectible from the outside world so can more easily be used to host services such as phishing websites or a botnet C&C relay.
There is also the potential loss/damage to the information in the server itself to consider. If a "noob" admin is running the server then it is not unlikely that the services and data held on the server are not adequately backed up.
Since it seems to have escaped your attention, I'd like to point out the fact that unqualified surgeons can (will ?) kill people; unqualified admins won't.
Which seems to be a somewhat relevant matter.
The Cloud - because you don't care if your apps and data are up in the air.
http://en.wikipedia.org/wiki/Port_knocking
http://www.portknocking.org/
I'd tell them to wait a minute, then run and grab a video camera. But that might not work with the analogy...
You are making them work for their attempt. Soon, you're servers are just too expensive time wise to make it a worthwhile investment.
If you are randomly generating your passwords and they are of a decent length then you don't really need to do anything. If your passwords contain lower-case letters only (not recommended), but are eight characters long then your million authentication attempts would represent only a 0.0005% chance of success. If you passwords contain numbers and upper-case characters too, then the likelihood is 1000 times less.
He could get trolled on slashdot by the very people he's coming to ask for help to become *less* of a noob.
I'll bet you teach your kid gun safety by shooting him in the neck.
LMAO! Where are my mod points?
There's no place like
I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla
Right off the bat, you're far too focused on the security of least concern, ssh. Unless you set an idiotic password, it's not going to be guessed. It sounds like you have that covered.
But.. the part you've missed is your FAR more vulnerable applications. You say your ISP is "actively managing" your security, but how certain are you they're doing a decent job of it? Do they have experts in each of these different CMS systems?
If you're serious about security, find experts to manage the security settings and versions of each of your chosen applications. I know for a fact that Joomla has historically had some issues with it. The others you mentioned wouldn't be terribly surprising to find problems as well. We're not talking a lot of money here. A good price would be in the neighborhood of $200 per application/site for an assessment of your configuration. Look for people that are reputable, not some fly-by-night "I know all applications" joker who just runs a security scanner and charges you $200.
And no, scanners aren't going to replace expert human beings that know what to look for.
AccountKiller
Just ignore it.
A good method would be to configure your iptables to drop all packets that come in from a new connection on all ports by default, and only allow through the specific ports you want the outside world to have access to. It is also an extremely good idea to move the ssh port higher than 5k. With the right iptable rules you can configure your system to ignore repeat attacks at the kernel level. And for the love of all that is good do not use passwords use key auth only, and disable root ssh login.
Some others might see some practical value in suggesting, "Maybe lock your door at night."
The problem is there is no simple laymans advice like that that would solve his problem. He needs someone to fit new locks to his doors, etc not just turn the key in the door.
Somebody might want to check my math here...my stats are a bit rusty
Ok let's assume a few things:
1) your passwords are 8 chars long and use upper and lowercase alpha and numbers only
2) they are essentially random passwords that are NOT in the dictionary.
This would mean that there are 2^62 possible passwords in your password space. That's 4,611,686,018,427,387,904 possible passwords. The chances of guessing one password in that space within a million random tries is 1,000,000/4,611,686,018,427,387,904. That's about one chance in 4,611,686,018,427. I believe that one in 4.5 trillion is about as remote a possibility as you getting struck by a meteor.
I think it's much more likely that someone will exploit a buffer overflow or sql injection vulnerability on your servers than someone guessing a password. You should make sure that you've applied whatever patches that you can and that any sensitive data traveling to or from your servers is encrypted.
I also like the iptables suggestion -- this means your machine will ignore entire regions of the internet so your server's exposure is at least limited to friendly subnets.
Another useful technique is to prohibit root access to SSH on initial login. This means that everyone has to login with some other username before they can acquire root privileges. For hackers, this means guessing not just a password but also a username that is allowed to login via SSH on the machine.
when surgery was first performed, anyone could and did do it
to much suffering, yes
but you can't expect the regimented laws and well-established protocols of long experience with a brand new technology
and yes, the internet is STILL brand new technology
we are only beginning to see standards and laws and social changes from its appearance
as well as further technological change
you can't expect the standards you are asking for
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
That should keep the CPU usage down, and your system secure from that type of attack. You can’t do much more than that.
You could run a script to find the owner of all those IPs... maybe do a bit of statistics to see what you can find out... But in the end it will be someone that you can not sue or contact in any way anyway.
Any sufficiently advanced intelligence is indistinguishable from stupidity.
I use 'denyhosts' which actively monitors and blacklists based on failed login attempts. I usually set it to ban anyone that tries to login as root more than once since root logins are disabled anyway no one should be trying that. And to ban anyone trying to login as an unknown user. Valid users are allowed a lot more leeway, 10 retries I think.
use an ACL to drop all traffic from Africa, Russia and China... problem solved
You can also use the hashlimit module for iptables. I find it works pretty well for allowing people in who need to access and block things like the ssh worm that was causing ssh to run out of resources.
You might check out the end of this presentation that I made 4 years ago.
http://www.bloomingtonlinux.org/wiki/15th_meeting
I've been using hashlimit successfully for years in a web hosting environment.
I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!
Serious, I shifted my port for ssh several years ago. I have yet to see a failed attempt that wasn't a legitimate user who screwed up since I changed. Sure, I know that someone could be port scanning and stuff, but really, I just don't see it happening.
Companies running public-facing servers without good technical expertise are a menace.
Umm... just out of curiosity: a menace to whom, exactly?
deadmongrel and PhunkySchtuff have really answered your question the best of all.
I can't add much to it except for these comments. denyhosts works great and we use a threshold of 1 failure on unknown logins and only 3 failures on known logins. It takes attention to detail as a remote user but it really works well. We set the thing to completely shut out someone who pulls that SSH crap on us, assuming that they are a bad buy. By extension, we use our firewall to log things that shouldn't be happening and tell us about the source IP. Again we assume they are bad guys (or gurls) and simply shut em out forever. Same thing goes for POP3 scans, etc. Hey, there are only a few billion possibilities (in IP4), knock em down and coordinate your data with others to save them time.
Don't let administrative ports/pages be open to the world on your production environment. If you think you have to (Need to log in to troubleshoot from a hotel etc) you don't.
Use a development host in an unprivileged network with unprivileged users as a jump host if absolutely needed. Don't store host keys on jump hosts. If you're just on a single host well.. too bad. See above messages about fail2ban alternative ports etc.
Patch regularly to help prevent local privilege escalation vulnerabilities.
Disallow remote root ssh logins. Use sudo with as needed command configuration. Use file groups/permissions wisely so users don't need privilege escalation to do their daily work. (777 is not wisely!)
Disallow shell access for people that only deal with content files. (sftponly for content. nologin for users that need neither file nor shell access)
And of course use strong passwords.
By defaut, I drop connections on port 22. /tmp, the file contains one IP. When the file is created, my script opens the 22/tcp port for the specific IP.
I made a small script which basically waits for a file to be created in
To create the file, I simply load a web page with a browser.
This is easy and works really great.
If the webpage were indexed, I just have to move it somewhere else. And even if the port gets opened, it is no big deal...
The script:
#!/bin/sh
FILE_IP=/tmp/unlock
IPTABLES_RULE=INPUT
while :
do
sleep 1
[ -f $FILE_IP ] || continue
IP=`cat $FILE_IP`
if iptables-save | grep -q $IP /dev/null 2>&1
then
sleep 1
else
cat
then
iptables -I $IPTABLES_RULE -s $IP -p tcp --dport 22 -j ACCEPT
fi
fi
rm $FILE_IP 2>/dev/null
done
------
The web page (this is bash, but easy to write as php too) :
#!/bin/sh
echo Content-type: text/plain /tmp/unlock
echo ""
cat
$REMOTE_ADDR
END
If there's any difference between "that stuff", and "stuff that!", and I think there is, then maybe he knew what he was saying all along?
1 - Disable direct root login - allow login only with a wheel user that is allowed to su-
2 - Use hosts_deny to deny ssh service to any ip other than clients'. you can allow clients' ip ranges in narrow range if they have dynamic ip.
3 - Change ssh port.
Read radical news here
or use portknocking
In fairness, surgery is WAY MORE FUCKING IMPORTANT than server management.
People without perspective are a menace.
1) Quickest and easiest: Change the port to something oddball.
2) Slightly harder: Set iptables to only permit SSH sessions from selected domains.
3) Getting geeky: Kill password authentication and force sshd to use certificates only.
4) Advanced: Set iptables to DROP IP addresses after three authentication failures within sixty seconds, for sixty seconds.
5) Paranoid: 2+3.
6) Very paranoid: 2+3+4.
7) Go-get-the-guys-in-the-white-suits paranoid: 1+2+3+4.
#4 is my personal favorite. It only takes about three lines in the iptables settings, and it drives script kiddies crazy when their "leet warez" hang.
Regards;
... many, many (dozens per second) SSH probes recently. Make sure your passwords are solid.
I've seen quite a few posts about using `iptables` (which I'm 100% for and there were some pretty stout ideas that I hadn't thought of myself) however, regardless of running SSH on it's standard service port or picking an alternative, I didn't see a lot of comments about just implementing straight ACL's other on the tcpwrappers level in /etc/hosts.{allow,deny}. Everyone talks about blacklists, but I think it's import to have whitelists as well. A new method I've started to incorporate is a strict ACL whitelist `iptables` custom table along with using a custom port knocking sequence prior to opening up access for a particular window of time.
I second every suggestion to get a hardware firewall, and use VPN if possible. But I know with many large hosting providers, particularly those who specialize in VM's, this isn't an option. I have a couple of VM's through one of those hosts (and I *love* said host), so you have to do the next best thing: some sort of software protection. If you are running Ubuntu, UFW does excellent. I use UFW and allow SSH connections only from two spots - my work IP, and my shell server. My shell server sits in a co-lo behind my PIX, and I VPN into it. I'm kinda weird like that, I'll admit. I have a PIX at home as well. Okay I'm a bit of a freakshow. Anyway, if you aren't running ubuntu, hosts.deny, PKI, and moving SSH to a very high numbered port will *help*. Other than that, contact the hosting provider and see if they will help you on a consulting basis. Remember many of them are running barebones to give you good prices - don't expect too much, and you never know - if you offer they might not even take you up on it. Finally - good luck.
you can always use iptables and hashlimit module to limit the number of connections to port 22 as well as using sshdfilter
First, hire someone who knows what they're doing. You just posted to a pool of roughly 1 million technical folks, and at least a couple hundred thousand who know what they're doing. A good percentage of them are unemployed or underemployed right now.
Changing the SSH port is an excellent idea. Folks scan huge swaths of the Internet looking for ports with services that they expect. Leave SSH on port 22, they'll assume you haven't done much for security and have a chance of exploiting a buggy version or finding an easy password. I knew someone who allowed root logins, with the password of "password" with SSH on port 22. The box was compromised within days.
These folks generally go for low hanging fruit. Let that be someone else.
Make your machines invisible. Well, almost invisible. Make them so they won't ping, and won't respond to anything but an authorized request. Tools like nmap default to ICMP pings first, and then will follow up with a detailed port scan. If you have high visibility sites, you'll still get curious people who look hard to find something. Make it impossible for them to get to anything sensitive.
Set up iptables (or the firewall applicable for your platform) is essential for security. Here's a little script that I used for a long time. Networks and some ports have been modified to protect the guilty^H^H^H^H^H^H innocent. This script is for Linux. It may require modification for your platform. The general idea will apply to any *nix platform. Modify as necessary. I had an ipchains version, but that was retired long ago. You may want to add to your enemies list (in the code below) automatically based on some honeypot services. Maybe code up a little something as an inetd process that simply appends their IP to enemies.list (and distributes to your peers) if someone tries to SSH in on port 22. Use that with caution though. It's a shame when you lock out yourself accidentally.
You could either start DROP rules on the fly, or work up the list automagically and have this script rerun at an interval (every 5 minutes, ever hour, every week, whatever suits your tastes).
#!/usr/bin/perl
# rc.firewall
# secure yer webserver
# (c) 2010 - JWSmythe
# Free to reproduce and modify with attribution.
# Publicly accessible ports. These are any ports you want the general public to hit.
@pub_ports = (
"80", # HTTP
"443", # HTTPS
"8080" # Streaming something
);
# These are private ports. Only our private_net users. Any other ports will be completely inaccessible.
@private_ports = (
"22222", # Our SSH port
"33066" # Our DB port
);
# Permitted Networks/Hosts.
# Use / subnet notation
# Use the smallest subnet possible.
@private_net = (
"192.168.1.0/24", # Where our servers are
"198.81.129.0/24", # Another companion network
"63.97.94.0/24", # Someone else that works with us.
"216.34.181.45", # home
);
# These are known enemies. You can collect these
$enemy_list = "/etc/firewall/enemies.list";
#
# Normally you won't modify beyond here.
#
Serious? Seriousness is well above my pay grade.
I think the point here is that the server is supposed to be "managed" already. He already understands that he's too n00b to properly handle the problem. What he IS requesting is any additional advice for simple activities or procedures that might help to improve the security and are within his abilities to implement on his own.
He's not asking how to perform surgery.. He's asking for advice on the proper way to apply a bandage.
Play with my webcams and lights here
Change port. Believe it or not, it does help enourmously - I didn't beleive it, but then I've done it for shits and giggles, I haven't had a single brute force attempt on my server.
Also, use sshdfilter (or fail2ban, etc).
Those 2 will take care of most SSH brute force attacks.
And if you don't want any SSH brute force attacks on your server, use public key authentication.
Whether you implement access controls in a stateless firewall appliance or server side with iptables, you will need to accept some measure of responsibility for the security of your infrastructure at some point. Additional access controls can be introduced through SSH daemon configuration, for example by restricting access to specific users or groups. Tcpwrappers can also provide an additional layer of protection.
it's simple, change your port. all you're looking for is to relieve some of the bot login attempts to your ssh server. it's not a security measure, it's a simply a annoyance measure. bots don't have the time to search for ssh service on other ports, and if one did, so what, let it try to brute force, your server is no less secure on another port but will have several million fewer log lines.
Set up knockd:
http://www.zeroflux.org/projects/knock
Port knocking took my web servers from millions of failed attempts per month to zero.
Computer security can not be achieved by responding to attacks. It's achieved by not being vulnerable to them.
Attacks may happen at any moment, and may target anything -- nothing that happened in the past can predict what might happen in the future considering that any host on the Internet can talk to your server. The amount of connection would matter if it is high enough to produce a DoS attack or number of attempt is so high, a brute force password guessing becomes viable. You are so far from both, pretty much any action you can perform would DECREASE the security because attackers might find a way to turn it against you. For example, if you block a host that tried to establish too many connections, they may try to spoof your host, so you will end up locking yourself out.
Contrary to the popular belief, there indeed is no God.
Practically speaking, it really isn't different.
The surgeon who kills someone might, if the victim's family is very lucky, be assessed damages of a couple million. An idiot with a cracked server farm can easily cause that much damage on the net.
To whom, exactly, is it more important?
Society places economic value on everything, and the amount of damage a cracked server farm can potentially do far outstrips the economic value of a human life.
http://en.wikipedia.org/wiki/Two-factor_authentication
This works great for reducing the effectiveness of dictionary attacks:
http://freshmeat.net/projects/sshutout
How do you manage security if you're a n00b? Only you can manage your security. Only you care enough to do the work.
Start reading. And FAST!
aka Matthew at SlashNOT/!
Practically speaking, it really isn't different.
The surgeon who kills someone might, if the victim's family is very lucky, be assessed damages of a couple million. An idiot with a cracked server farm can easily cause that much damage on the net.
WTF? In one case, a web server has been vandalized. In the other, a person is dead.
I know a lot of people have suggested fail2ban, and its a great solution that's easy to implement. The best part is it uses sendmail to mail logs to the root user about brute-force attempts. This is mainly for my curiosity.
The big thing is the ban timer on fail2ban doesn't need to be very long. The ban can be as short as 2-3 minutes and still get the message across. Once they realize they're disconnected, they'll go elsewhere. It'll slow them down enough that brute forcing ssh isn't practical anymore, usually at that point they'll move on to another host.
An easier solution is to just use public-key authentication though, there are plenty of articles out there that deal with this.
Really, just all the spam in my logs to be amusing. I had someone use a web scanner on my box, and after it was done and found nothing of interest, it made a request for
Society places economic value on everything, and the amount of damage a cracked server farm can potentially do far outstrips the economic value of a human life.
You forgot to start that sentence with, "you might be a psychopath if..."
Google for it. It is a pretty up to date list of all the undesirable IP ranges and your attempts will go way down.
Most recently they added 1.0.0.0 :)
Hey KID! Yeah you, get the fuck off my lawn!
Learn iptables. There are many good books available covering iptables and server security topics.
Sudheer Satyanarayana
www.techchorus.net
No, a web server has been cracked and turned to spamming, DDoS, and cracking of other servers, causing immense financial damage to other companies.
And we're not talking about the happy-go-lucky land of every-life-is-precious or the insufferable "if even one child is saved..." fantasies, we're in the real world, and in the real world, the economic damage to be wrought from a single company's negligence in security their servers can meet or exceed the economic damage of a surgeon's negligence.
I'm sorry you're too psychologically immature to recognize economic reality, but that's really not my problem, and hardly makes me a psychopath.
It's not really that bad. I manage several Unix systems. I have done so for all of my professional life. Some servers get in someone's list, some others don't.
I currently work for an international telco, with base in the US, and servers in the US, Brazil and Argentina. It's a small firm, and I manage the whole operation. Some servers get attack attempts all the time. Others don't.
The first thing you have to find out is: Are those attacks targeted? Many times, you'll get a new server up, assign a fresh IP, and immediately inherit shitloads of connection attempts. Those've been probably targeted at that IP for years. Most probably aren't even meant for your particular platform. I get shitloads of IIS+Windows attacks targeted at my Slackware + Apache2 servers. I see connections to my SSH daemons trying to exploit vulnerabilities that have been patched for at least 5 years.
But mostly, I get stupid dictionary attacks. With oh so crappy dictionaries that look mostly for usual names and system defaults.
This are the steps you should follow:
1st: Move SSH out of port 22. That'll filter out 75% of all attempts.
2nd: Use Portsentry. You'll eventually have to whitelist some guy unfairly banned, but most of the time, it works like a charm.
3rd: Implement way of the many scripts (or write your own) that bans IPs or ranges of IPs that have failed auth many times to your opensshd. This is your most important defense here.
4rd: Implement longer delays after failed passwords
5th: Seat back, and relax. Change passwords regularly. Keep them strong. Watch logs regularly, keep an eye for sofisticated, targeted attacks, and ignore the script kiddies.
Most of the time, those logs just mean that you are out there on a public network, and you've got an IP on way too many people's lists. If it's sucking up too much bandwidth, ask your ISP for a new IP.
If you ever became worried about some guys, feel no shame about banning whole ranges of IPs. If you truly expect no traffic from those places, dropping China, Brasil and Russia from ever reaching your ssh port is a great idea. Actually, if you only expects connections from certain places, whitelist those ranges and drop anything else.
Finally, don't focus so much on SSH. OpenSSH is stable and secure, and it'll rarely be the source of you getting rooted. Focus on more vulnerable services, like crappy php webapps, ftp servers, etc.
WTF am I doing replying to an AC at 5 A.M on a Friday night?
what ever happened to port knocking?
1. Worry about securing Drupal, Joomla and Wordpress first...
- or -
2. Use a VPN and hardware firewall
- or -
3. Use iptables 'recent' or 'limit' modules
- or -
4. SSH keys
- or -
5. Find a managed service provider to do 1-4 so you can worry about managing the sites (check out Secure-24.com maybe)
I'm sorry you're too psychologically immature to recognize economic reality, but that's really not my problem, and hardly makes me a psychopath.
It's not a sign of being "psychologically mature" to hold the notion that human life is a fungible commodity.
A poor person in the slums of Bangalore is economically worth more dead (for his organs) than alive. You state that society places an economic value on everything, including people. Yet for some reason, society has outlawed murder, even if the result is a net monetary profit for society.
Your complete failure to comprehend that a human life is worth more than its economic value to society is why I called you a psychopath.
works quite well for me
http://www.snowman.net/projects/ipt_recent/
its been around for years, and has kept my ssh service nice and available for almost as long.
basically, keeps a contrack record for tcp new attempts on the configured port(s), with threshholds for how many attempts before being temporarily blacklisted, then a timeout for how long before they can go again.
fail2ban and denyhosts fee way to high up in the stack for my liking
To name a few...
1) All the zombie pcs their servers are controlling.
2) All the computers the zombie pcs attack.
3) All the email addresses the zombies spam.
This is not the funny you're looking for.
Your complete failure to comprehend that a human life is worth more than its economic value to society is why I called you a psychopath.
Whether or not I recognize any such thing is entirely irrelevant to the discussion at hand. There are no moral imperatives involved in the question of whether self-described 'n00bs' should be running world-facing servers, it is purely an economic question, thus making any non-economic value of a human life entirely irrelevant.
Your complete inability to disconnect your base emotions from a purely economic discussion is why I called you psychologically immature.
There are lots of comments here and I doubt "Anonymous Coward" will make for a respectable post, but here goes anyway on my experiences.
- So, random passwords are a good start. My experience has been that most attacks on my systems start with specific organized password hacks.
- I don't agree with using off-the-wall port numbers or things like port-knocking. Here is why: When I connect to a standard port number for a specific service with an established site, I do have the impression that security is usually addressed and that malicious attacks will get nowhere. When I connect to a non-standard port or use port-knocking or whatever on an established site, I have the impression that the sys-admins don't know jack about real security, that they are scared, and that "security through obscurity" has become the main line of defense. I'd much rather put things out in the open and show a locked door, than give the impression of an unlocked door that is hidden.
- fail2ban, as others have mentioned, is a great tool. By default, it will only ban an IP for 10 minutes or so... which is more than enough time to cause the attacking machine to effectively lose interest. But, if you want to make sure other machines on your network are protected, a plan on action could be as follows: I run fail2ban on multiple machines and tally the results daily with logwatch, the corresponding bad IP addresses are taken (there are usually not many) and blocked using hosts.deny on each machine. I've found over and over again that this drastically limits the amount of malicious traffic that I get attacking multiple machines.
- An obvious point from above is to disable root via ssh. This doesn't disable your specific login's super user root access, btw.
- If you find that you are attacked more from specific domains (and you do have the ability to address entire domains and subnets with most of these options) and it causes system resource problems, then try to understand how your machines are communicating with the attacking machines... which are 100% of the time scripted. Using the TARPIT option in IPTABLES may be a good approach here. The main advantage that a script has in attacking your machines is that it can blow through multiple attempts in a short time... and so you basically drag it out so the script becomes useless or until the script thinks your machine is not responding.
~1,000,000 auth failures per year = ~2740 failures per day. from 1200 IPs, that's 833 failures PER IP ADDRESS. how in the bloody hell does that go amiss? i like to think most people could figure out their password within 10 guesses (per ip). for a problem this minimal, you don't even need to consider adding a delay between auth requests.
One million login hack attempts over a year is pretty normal on the standard port 22.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
and advertising that fact on /.
An SQL query goes to a bar, walks up to a table and asks, "Mind if I join you?"
I'll bet you teach your kid gun safety by shooting him in the neck.
I bet he is terrified of guns after, the system works.
An SQL query goes to a bar, walks up to a table and asks, "Mind if I join you?"
The problem is that all solutions need more work from the customer and that's sometimes something they just don't want to deal with. If I'm really really stuck, what I do is set fail2ban to block on the first failed attempt. That server also gets removed from the VPN network and thus do not get any backup or MySQL replication. No matter what, there is no root password login enabled.
In addition to using reactive tools like fail2ban and denyhost, also block most of the world proactively:
git clone git://github.com/bugi/iptables-by-country.git
If everyone gets told that, it's not long before there are no surgeons left.
Also, running a server is not a critical as surgery. No one is going to die if a server crashes or gets hacked.
Don't take life so seriously. No one makes it out alive.
It's fucking computer. He can learn how to secure it. There's no reason to hire someone else to do something that he can do himself, especially when it's something that he needs to know how to do.
Don't take life so seriously. No one makes it out alive.
Reasonable people understand that economic damages do not equate with lost lives. Drop the pedantry...you sound like a moron.
Don't take life so seriously. No one makes it out alive.
Surgeons get paid around $300k. System admins get paid around $70k. (Those are median wages pulled from salary.com). It seems that society places a much higher value and importance on surgeons.
Don't take life so seriously. No one makes it out alive.
If you are using FreeBSD, you can add this (and tune it) to /etc/inetd.conf to cope with login bots.
# throttle bot login attempts : limit sshd to 10 child processes, /usr/sbin/sshd sshd -i -4
# 3 connections per IP per minute, and 5 simultaneous invocations per
# IP address
ssh stream tcp nowait/10/3/5 root
you're not really successful. stop calling yourself developer and stop calling your bedroom studio. go back to serving burgers at mcd.
Instead of generating passwords, have your users supply their SSH key and turn off passwords. Much better solution.
The above is not worth reading.
SSH is attacked often because a lot of sites run it.
Disable SSHD on all machines, and run Kerberized telnet instead.
htaccess, fail2ban, denyhost, and iptables + snort
WTF? In one case, a web server has been vandalized. In the other, a person is dead.
But your whole argument crumbles if the web server in question is WOPR and that pesky hacker has just started World War Three.
Cool! Amazing Toys.
However, zombie servers get noticed. When you find that all your email goes to spam, and blacklists are yelling at you... you realize something is wrong. Abuse complaints come in, and your server gets nullrouted. Home users don't notice until it is EXTREMELY bad, and even then often blame it on the wrong things.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Happy-go-lucky? Fuck you.
Accidents and such are one thing. Malpractice and deliberate killing is another. These are fucking [b]lives[/b], not imaginary numbers that may or may not represent a quantity of gold.
Again... fuck you.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Reading your comments in this story... I only have one thing to say:
You need counseling.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
I can't believe the person who set the server up left this port open to the world. If you want the server secure then you will close all the ports that are not actually needed - in your case this likely means you keep 80 and 443 open.
For remote management make get a few IPs for the desktops that are meant to connect to it and make sure the ISP opens SSH port only to the IPs.
You can tunnel X over SSH and do file management, whatever you need so it should be enough.
Don't bother changing it to some random port, security through obscurity is total bullshit in this age of port scanners.
All those moments will be lost in time, like tears in rain.
Hi There,
I know its not the cheapest solution around but we use Fortinet Firewalls with their IPS feature. There is a signature for brute force SSH attacks. It works really well. We have seen a 96% drop in SSH connections since installing these devices in front of our servers. So far we have had no reported false positive issues. Coupling that with all the other signatures in the IPS database and we have a huge drop in attacks overall.
Supposing that a hacker knocks out a power grid...or jams up a medical database? Or...gets into a military system?
Causing damage or disruption in a server might not be directly dangerous, but it can make things indirectly quite hazardous indeed.
A hospital might lose power and patients in the middle of surgery could die. That's probably why hospitals have backup generators though.
Medical databases getting corrupted could easily get someone a fatal drug dose of the wrong type, or quantity.
And hacking into the army, possibly launching an attack. Well, duh.
Computers these days are quite capable of causing harm if they're malfunctioning, thank you.
You get a lot of recommendations here, but people tend to just blurt out the flavor of the hour they are using without considering the essential motto of any proffesional, "the right tool for the job".
Simplest, do you need SSH access? I once maintained servers with absolutely no remote access capability of any kind. Didn't need to, lived within 5 minutes of the hosting location and simply worked from there. Saves a LOT of security hassle when only physical access can be used on a server. Most likely not possible, but I hope it does show that you must THINK about what is needed/possible in your situation.
For instance, moving the SSH port. It is possible, security through obscurity at its finest, but how will it affect your customers? If you then got to deal with endless service calls to explain the non-standard port, or worse, client software can't handle alternate ports, then it is not the right tool.
An alternative to this is switching the IP. Nothings says that the IP used to access the webserver must be the same as the one used to access SSH. Again security through obscurity, but it might help.
Also, there is no law that says every machine should be remotely accesible. If you got a lot of servers, why not create an Admin box? It can connect through all the servers over the internal network, but the world facing servers got no outside admin connections. Yes, if the Admin server is borked, you are in trouble, but I found that people in the name of "Shit might happen" open themselves up to a lot more shit.
An other simple approach is to block parts of the world that you don't expect legit connections to come from. If you are never in China and neither are your customers, why would you allow China to connect to port 22? Just remember that you got such a block in place before you book your holiday in Hong Kong.
An even simpler approach is to block root from using SSH, and then use only somewhat unique account names. You need two parts to login to SSH. Brute forcing passwords is only half the job unless I have been missing something all these years.
And on the whole, 1 million failed login's is a good thing. It is the successful login that you should worry about.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
It's cheap, it's professional, it's fast and it's done correctly with lots of experience. What's not to like there?
Of course, it's not a answer to everyone, but in your case seems to be good match.
http://www.platinumservermanagement.com/
I've considered using them in the past to save my time and they seem to be good, and is recommended by many.
Pulsed Media Seedboxes
that's why you pay a good sysadmin (for a lot of money) if you need good security, and don't if you don't.
if your server is important economically or even supports some life saving tools/etc and it doesn't have a good sysadmin, that's your error.
If you get an unimportant server hacked, well, no big deal. Rather that than being disabled or die etc by like.. far.
on the other hand, you probably want 100% of the surgeries to be successful.
You do the best you can. Sheesh. Learn about the system and how to secure it. It's not slashdot's problem you're under-funded and under-resourced.
There's a triangular spectrum. Corner 1 is how much you're willing to pay. Corner 2 is how much you're willing to learn and do by yourself. Corner 3 is your probability of getting r00ted. If you don't want to spend any money on sysadmin and you don't want to spend any time to secure your system, then expect to get r00ted sometime.
http://www.aczoom.com/cms/blockhosts/
The advantage is that it does things live. So no log scanning (well, it does the first time, just to see what it can add) and therefore even if several hundred scans are done from the same IP at the same time, you do not need to wait a minute for some cron job to start and read all the adresses again.
As it works with hosts.allow (or deny) you can determine IP addresses that will never fail, so you have always access. You can also determine the amount of times users can try as well as how long you want to block the IP address.
Don't fight for your country, if your country does not fight for you.
I am using fail2ban, but I do not think it's a particularly great tool when you see how many different IP addresses the attacks come from, so you're somewhere stuck in the middle trying to optimise how to make fail2ban more effective, while not being a problem for your own users:
- you configure blocking from 1 login attempt and a long block time cuts down on most of the outside attacks, but if some legitimate user mistypes his password, he'll be locked out for that same long block time...
- you configure several wrong password attempts and/or shorter blocking times - then the attacks do not get slowed quite as much, but it hinders your users less.
There are some things you might consider, though -
- if you can, and your users always access the system from their own computers/laptops, force the use of public-key authentication and disable use of password logins; then outside access attempts via password cannot succeed.
- if you can't do this, at least force root to use public key (and/or su/sudo from another login. (sshd_config either
'PermitRootLogin without-password' (to allow root access through authorized_keys, but not through password, or
'PermitRootLogin no' to block root access via ssh-login altogether.
- if there are only specific accounts you might want to allow access to, look into pam_require, then limit to those users in pam's ssh
config: 'account required pam_require.so @ssh-users myaccount' (allows user 'myaccount' and all users in group 'ssh-users'
to log in via ssh - everyone else will be kicked out, even with a valid password).
- keep an eye on the log to see whether it's actual users that get hit, or whether you see access attempts to non-existent users
(many of the attacks are basically dictionary based). If you find one of the valid accounts gets hit too much, consider renaming that
user account.
Recognizing the reality for what it is without making a value judgement one way or another is now grounds for counseling? What next, not believing in God is grounds for re-education?
- you configure several wrong password attempts and/or shorter blocking times - then the attacks do not get slowed quite as much, but it hinders your users less.
There are some things you might consider, though -
3 fails / 10 min != 10 fails/ 1 sec
There *is* brute-forcing going on. I'm surprised no-one has mentioned the string of articles on the Net on this very topic (Then again, this is slashdot where everyone posts first, then thinks. So naaaaaaah, forget that).
The graph at this site http://stats.denyhosts.net/stats.html shows you what's been going on for the last few months. And this article http://www.immutablesecurity.com/index.php/2009/10/06/low-and-slow-ssh-brute-force-attacks/ has some info. about how this has been going on for a long time.
At our tiny company (four remote users) we moved ssh to a non-standard port, and don't allow password access, only private key auth. Requiring a key makes ssh bullet-proof against password guessing. Moving the port saved us a bit of bandwidth - early on in the brute-forcing the attackers would hammer away as fast as they could. Now, with lots of people using an IDS, they attack more slowly, and from a large number of different IPs.
So to answer the original question - I would move the ssh port and go for an auth key. It may seem like a bit of trouble, but a lot less trouble than re-building your server.
Do you have many customers who access SSH, or is it mostly you? That could help define your problem and your solution.
As others have pointed out, a million connections divided by 50 sites, divided by 365 days, may not be all that unreasonable.
On the other hand, if it is only you, or if you like to go through your log files regularly and find it too cumbersome with all the extra noise, then there may be better solutions.
I like to use pageknocking (not portknocking) to open the SSH port for access. This prevents access from worms and limits the number of log entries created, although others claim (incorrectly) that it is just security by obscurity... (It should be trivial to modify it for use with Linux) www.otterhole.ca/knock/
You forgot to start that sentence with, "you might be a psychopath if..."
Or an economist. An economist would realise that effort is fungible[1], and the effort that goes into cleaning up malware could save a lot of lives if it was applied to, say, medicine. So yes, enough compromised servers are worth more than a human life.
[1] On long enough timescales. Allow a decade or so for retraining and changed career courses.
You have bigger security problems than SSH.
I see others have already mentioned my articles such as http://bsdly.blogspot.com/2009/11/rickrolled-get-ready-for-hail-mary.html, and if you take a peek in the list of addresses I put there, I would not be at all surprised if there's a great deal of overlap with the hosts that keep sniffing your servers (my data for that round has a little over 4000 hosts). In fact, it would be interesting to know how large or small the overlap is. They will keep trying (in fact I'm just seeing the start of another alphabetic phase over the last few days here) but there are a few things you can do to make it less likely they will succeed.
The general advice is, as you have heard many times before, to enforce a policy of no passwords, usin only key authentication, of course disable root logins and if practical limit where you can log in from to 'known good' IP addresses or ranges. The first two won't rid you of the logged attempts, but sensible in any case and makes the probability of ssh-based compromise quite a bit less likely.
Rate limiting helps get rid of the classical rapid-fire variety password guesser, but will not help at all when you're faced with the coordinated 'hail mary cloud' where each individual host could be attempting to access your system or network only every few hours.
As for portknocking, I seriously think the port knockers would be equally well served by switching all passwords to unicode. That provides a practical alphabet of the same number of unique characters (16 bits, remember), and for anyone with a large enough fleet of password guessers, the mechanics of guessing the right one is not all that different. Oh well, I just spilled the beans for the main point of an upcoming column, that won't spoil the fun later, I hope.
-- That grumpy BSD guy - http://bsdly.blogspot.com/
Okay, so they're making a million attempts a year.
Let's say you plan to keep this server for, what, twenty years? After that you will replace it with an entirely new system and new passwords.
Assuming that the number of attempts they make increases geometrically, starting from one million this year, in twenty years they could generate how many attempts? A billion or so? Maybe ten billion? Let's say a hundred billion, because you don't want to take chances with your server security.
So just make sure all your passwords are sufficiently long and complex that the search space magnitude well and truly exceeds a hundred billion. Ideally you want to make the hundred-billion number look absolutely ridiculous. This is not particularly hard. Even if you don't look beyond lowercase letters (because you want to type the password quickly with no shifting keys), 26^N goes past a trillion after just nine characters.
Personally, I'd set the minimum length to at least twenty, and make the root password more like twice that long (if you allow remote root login). It's overkill, but when it comes to security a little bit of overkill is good. It just may save you if you've underestimated some important element of risk (like, say, how many attempts they can make in a given time period).
Cut that out, or I will ship you to Norilsk in a box.
We use DenyHosts, a program watching your secure-log for failed logins and after a few of those adds their IP to /etc/hosts.deny. It can also sync with a central repository.
That way an attacker has to guess the one user+password, and have a legitimate userid+password to gain access.
Use public key authentication.
Also. The root account should not have a password at all. Passwords are a weakness which should be phased out entirely. It's (long past) time brute forcing passwords was made futile.
Deleted
A million attempts over the course of 1 year can easily be defeated by changing the password every 3 months.
Well, that works as long as you never, ever have to log in to from a random terminal. Which I do.
And if your laptop gets stolen, then your key gets stolen with it. Ditto USB stick, etc.
So pubic key is less vulnerable to brute force attacks, but more vulnerable to physical attacks. Pick which one you want.
The short answer is to get someone to read all your logs daily and email you only if there's something non-routine in them. That will find anything that gets logged, so you will have good coverage and rapid notification.
Since no-one in their right mind is going to offer that service for a reasonable price, you can have cron run an 'artificial ignorance" script nightly. The tern coems from Marcus Ranum, and there's a complete discussion of filtering syslog at Sherlock Holmes on Log Files.
In practice, I get a few emails a month about new things, and add them to the list in the script if they're uninteresting.
--dave
davecb@spamcop.net
You need to read this: http://news.slashdot.org/comments.pl?sid=1573970&cid=31389232
From the article which discusses the economic value of a human life, from the standpoint of the wife of a man who cost over $600,000 to try and keep alive. Do you think she needs counseling because she compared that cost to what else the money could have bought?
It's trivial to put in rules to only accept ssh connections from specific IP addresses. Perhaps you might consider only accepting connections from your customer's IP's, or doing the reverse and blocking abusing IP subnets.
-- Programming with boost is like building a house with lego. It's a cool but I wouldn't want to live in it
If some random botnet on the internet can cause any significant effect to any of the systems you mention in your post, they are broken and need to be re-engineered.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Not entirely true. Economic damage can directly cause jobs to be lost. That can easily lead to lack of healthcare, or even shelter and food for those previously employed individuals. Which can lead directly to lost lives.
Run SSH server in a different port, use 22 for a honeypot:
http://code.google.com/p/kippo/
Just setup a VPN and make your iptables allow access to the IP block that the VPN will use. Also, allow access to the subnet of your office only(if you have one).
Instead of installing tools use the iptables rate-limit and automatically ban them.
Well, sometimes unqualified admins can kill living things.
Or have we forgotten the sage advice: "always mount a scratch monkey"?
there is a certain view that the proper valuation of a human life is best done by an accountant. I would give examples, but goodwin's law gets in the way.
He didn't say he agrees with it. But he's right. There are a lot of instances where the loss of a human life had far fewer financial implications than a theft of some customer database. Sad but true.
Except that we have probably 100:1 ratio of system admins to surgeons which means society pays faaaar more to employ sysadmins than surgeons. So when you look at the broad picture you are completely incorrect.
You could consider implementing port knocking. It will be somewhat of an inconvenience for your clients, but it will almost eliminate your ssh authentication failures.
Is there a piece of software similar to SSHGuard but instead of adding local IPTables rules can function in conjunction with a hardware firewall (eg - Cisco ASA) ? I'm thinking of a device inside the private network pulling logs off hosts in the DMZ and using the results to dynamically manage firewall denies.
I think everyone that has a SSH server must have the exact same issues. I have a CentOS server behind my firewall that I use for a little web development (personal projects, nothing major or really interesting) and just testing out different things but for some reason the chinese army seems hell bent on breaking in. I just set iptables to block every /8 IP range that I could find that was even loosely related to china. Personally, I don't understand why anyone would allow their network/machine to communicate with chinese IP ranges, there just seems to be no real need. Honestly, this cut invalid login attempts from thousands to a handful. As said before make use of your authorized_keys file and I'd also recomend that you double check that login by root is disabled.
I looked at my logs the other day on my FreeBSD router and noticed the same thing this guy had. Random IP addresses (different class A's very few repeat IP addresses) and invalid logins. Each time the username would alphabetically increment. So all these different IP's each taking the enxt name on the list to try to login. It was pretty scary.
I let it get through it's list up to about 'j' before I blocked port 22.
I highly recommend OSSEC (ossec.net). It will parse your system logs (SSH, Apache, ModSecurity, Snort, and others) and block attacker's IP addresses automatically. There's even a Wordpress plugin that will output to syslog and therefore OSSEC. For more information, please see: http://securityonion.blogspot.com/2010/02/defense-in-depth-using-ossec-and-other.html
You're finding the 1% of cases where a server crashing might cause lost lives.
If a surgeon messes up, the chances are near 100% that someone will die or be maimed for the rest of their life.
Don't take life so seriously. No one makes it out alive.
You're talking about pay. I'm talking about value.
Don't take life so seriously. No one makes it out alive.
a single server farm isn't going to do $millions of damage
and an unlicensed surgeon who kills someone is facing jail time
Some advice:
Hack the ssh program, let them in, rate limit them after they give you data, then dump them. Start all over again.
I'll let you know how this works as I have a research project to do this.
I am looking for ssh friends :)
knockd if only *you* or other technically savvy people connect to them, or option B (DenyHost) if customers actually use it too.
DenyHost can be set where it will blacklist IPs that fail ssh more than $x times and you can even whitelist IPs.
I already see it as a 5. Oh well... "I get thousands of Chinese hackers attempting to break into the battery monitor..." I'm still laughing.
Still hoping for Gentle Treatment...
thus making any non-economic value of a human life entirely irrelevant.
No, the non-economic value of a human life is never irrelevant.
Ever.
You were equating the need for trained doctors with the need for trained sysadmins. The reason we need highly trained doctors is not due to the economic impact of poorly trained doctors, but because of the human impact of poorly trained doctors.
Your complete inability to disconnect your base emotions from a purely economic discussion is why I called you psychologically immature.
The value of a human life is not a "base" emotion. I can easily talk about the value of a human life in terms of economic output, the value of the elements in the body, the value as a customer or an employee, etc. But one must never forget that these are also humans.
To dissociate the value of a human as a human from a human, is the key ingredient of a psychopath.
1. Dont have open ssh ports on the net...
2. Use vpn tunnles to get to your network
3. ONly allow port 22 connections from your vpn network.
4. Stop getting hacked
Who the hell leaves an ssh port open to the net on a production server ?
Also while fail2ban is very effective as i have used it in the past, its not a perfect solution.
again while in gods name do you have ssh ports open to the net, realy you should not have any admin access avaible strait from the net, if you do your just asking to get burned
http://www.hexten.net/wiki/index.php/Pam_abl
Sig out of date
I think the realization is that economics are psychopathic. The comparison of a surgeon to sys. admin. above may be a poor analogy because of the societal (i.e. human) difference between the two jobs. The suggestion that a human life counts less than a server farm's integrity may be an economic perspective, it as also a psychopathic perspective. The point may have been phrased better.
Letting psycopaths (i.e. large corporations) have a large amount of political control is probably an extremely bad idea for a human society
Invest in a hardened VPN box, block all ports on your webserver from outside world except 80 and 443. Only allow SSH access from the VPN. Log into your environment only after you first have a VPN connection. Outside world would only see the ports they are allowed access -- you would have more control over who has secure access.
Best,
Justin
If you had everything you wanted, you'd just want more.
OWASP Live CD http://www.appseclive.org/content/downloads
You can't discuss economy in isolation, specially disconnected from human emotion and moral judgements.
That should be obvious to even the most derided, but educated, person.
IANAL but write like a drunk one.
I did a reverse lookup on the ip addresses that were trying to randomly guess user/pass on my machine and found that the majority were from: China, Korea, india, and Japan. I then filled the hosts.deny file with the subnets from those countries and cut back on 97% of the break-in attempts.
I also told my boss noone from those countries was ever going to be able to see our website let alone login w/o bouncing it through a different computer first. I also block root login and users that can login have to be in the sshd group and don't have simple login names like "fred",
I believe something fail2ban-like to be a better solution in the long run, but this was faster for me to setup and understand exactly what was going on.
This is why i use ClearOS. It comes with Intrusion Prevention via Snort right out of the box. It proactively blocks failed SSH attempts and keeps my boxes off the lists happy and smiley!
If your passwords are randomly generated and long, it doesn't matter how many attempts to guess them are tried. The likelyhood of a random guess getting through are lower than your chances of winning the lottery. Let people waste their time on futile attempts.
To further decrease your chances, use public keys authentication instead of passwords, or two factor authentication, or limit connections by IP address.
Changing the post does fool most SSH scanners as well.
I don't like fail2ban because it can lead to DoS vectors.
Bottom line is that logged attacks that have no hope of getting through shouldn't cause a panic.
non-standard port & TARPIT ( IPTables ), and you reduce the discovery of your actual port ( by making scans take nearly-forever )
+1 for sshguard. Compiled (not interpreted by perl, python or ruby), fast, stable, works great with any firewall -- pf, iptables, etc.
TossableDigits.com: Temporary Phone Numb
In fairness, I have noticed that in terms of self-importance, the two professions are about equal.
I am wondering why no one did not mention OTP solutions as well?
You can have Mobile OTP. Also Port Knocking or SPA with SSH certificates.
If you're going to have services open to the Internet without IP restricions, expect millions of random connections. Just use ssh-key authentication, disable password over ssh and forget about it.
POKE 36879,8