Slashdot Mirror
The Hail Mary Cloud and the Lessons Learned
badger.foo writes "Against ridiculous odds and even after gaining some media focus, the botnet dubbed The Hail Mary Cloud apparently succeeded in staying under the radar and kept compromising Linux machines for several years. This article sums up the known facts about the botnet and suggests some practical measures to keep your servers safe."
The solution to low-frequency brute force attempts is Denyhosts. It just blocks any host with repeated failed login attempts. I've been using it for longer than I can remember, probably longer than this "Hail Mary" botnet has been in existence. I'm not sure why this author seems to have never heard of it.
1^2=1; (-1)^2=1; 1^2=(-1)^2; 1=-1; 1=0.
This is about the low-intensity password brute-forcing via ssh that's been going on for years -- the only difference between this and any other password brute-forcing via ssh is that fail2ban and such scripts are ineffective, because the attempts are so low-frequency that it's practically impossible to distinguish them from users fumbling their passwords.
The simple solution is to disable password authentication for all users, and make them use keys -- this renders you 100% safe from this botnet. If that's infeasible, be damn sure you've disabled password authentication for root (i.e. "PermitRootLogin no" or "PermitRootLogin without-password" if you still want key-based root logins). If you do allow password logins for any or all users, enforce strong password requirements.
"I'm not really in a mind to offer help or advice to the people running those scripts, but it might be possible to scan the internet from 255.255.255.255 downwards next time."
Yes, start with all the multicast addresses. That'll work for them! ;)
It looks like you didn't read through TFA. The author did address higher port numbers. While they may not be attacked as much, they will still be attacked eventually. I have seen higher port numbers on a couple of my servers used for targeting SSH myself. Bad fix. Just puts off the inevitable.
As far as root logins being enabled... you would be amazed at how many "IT admin wannabes" never address killing off root access on SSH. I routinely turn this setting off on new client sites once to twice a month.
Actually, not permitting root logins via SSH was one of the points he repeated several times. He also said, explicitly, "There's no safety in high port numbers anymore.". Perhaps he's wrong, but he did consider your point.
Personally, I don't feel very exposed, so I'm not willing to do much more than that, plus picking a secure password. I may have set things up so that no SSH logins are possible in any way, but I'd need to check again to find out. (I'm not much of an administrator, and my system isn't internet facing, but as I never use SSH except in a browser, there's no reason to allow it.)
But did you notice that not all of the attacks were attempts to log into root? (I don't know why anyone would replace root with admin, but apparently enough people do that that was a secondary target. If I changed the root account's name it would be to something like "doofus" or "notHere". (I.e., easy to remember and short, but not easily predictable. So you'd need a "dictionary attack" on the account name as well as the password.) I don't feel threatened enough, however, to bother with that.
I think we've pushed this "anyone can grow up to be president" thing too far.
Yeah, that's a great idea if you're a) a hosting provider or b) a security researcher. Ignore the problem and prevent your users from easily reaching their hosts.
You cracked the fucking case, Murder She Wrote's Angela Lansbury.
We really need to record all bytes txmitted and then review the dumps when we get core files. Forward to Kaspersky for analysis. Stick it to them.
I think this article points to the fact the author is retarded.
Considering the retarded author in question is someone who is a respected author on OpenBSD ''pf'', firewalls and security in general, I think your answer prove you are the retarded one.
The right to offend is far more important than the right not to be offended. (Rowan Atkinson)
One of the reasons that "admin" may be highly targeted is due to its use on some customized distros. There was an asterisk distro, trixbox, that had an admin user account installed by default. While the password was changable, on installation I came across had a rather simple p$ssw0rd1 set by the original integrator. Access to a sip trunk and the security settings to boot would be a high commodity item.
Sometimes you have to access from other IPs, or don't have a known, fixed IP from where you could have to connect, or you could have to fix something when in a trip. I'd suggest a mix of several of those (no root access, non standard port, explicitely enabling in fw just the IPs you know that must enter by that service) but adding portknocking for the rest of the world (only if you could need to access from elsewhere), specially using Single Packet Authorization to prevent the chance of someone (specially 3-letters agencies) capturing/replaying how you entered there. And not just for ssh, every service that is not meant for the whole internet shouldn't be even visible for the rest of the world.
not permitting direct login as root at all is a good idea - it what su was invented for (amongst other things)
On what OS is root login via ssh even enabled by default?
"I've managed to get my name on slashdot a lot"
"Use key auth instead of passwords"
"My references are my own blog posts"
There's nothing interesting to see here. Don't allow password logins to your system, because you can't trust people to use good passwords. It's 2013, there's no cake for pointing this out.
I am writing this on a BSD machine, but this guy is a disingenuous SHILL. If you have a computer connected to the interwombs, just USE A FUCKING STRONG PASSWORD: NOT: Anna123, PeterPan, Fuckme, dumbass, microsoft, sleaze, shill, whore, foss USE: 767.211.856.543. Too much work for you ? Then don't have an ssh port open on the interwombs. Please don't reproduce either, because you are simply too retarded to get the obvious fix (strong password).
Doesn't matter what his reputation is; the article is banal, and the idea that he's basically written ten just like it and had them posted on slashdot is pathetic.
Not to mention several routers use admin and support ssh connections.
Router software virtually never gets updated.
Sig Battery depleted. Reverting to safe mode.
not permitting direct login as root at all is a good idea - it what su was invented for (amongst other things)
Acually, on a new Freebsd install, out of the box, ssh to root was not permitted nor was su to root.
So on new installs, you have to pick your poison before you walk away from the console.
The rationale for not allowing su to root is because an attacker who (by what ever means) logs into some random account has a dramatically more advantageous position than one banging on the doors of ssh. That being said, unless you have convenient access to the console you are going to
have to choose ssh or su or something similar.
By far the easiest, and the one that most admins default to using is ssh with big keys, and don't allow password logins (or su to root).
(I make this claim without a shred of evidence to support it, other than knowing what I see among the Admins I work with. When an admin quits or moves on, its easy to remove his key from Authorized_keys but hard to remove a password from his head. I'm sure someone will school me on why this is wrong headed).
Sig Battery depleted. Reverting to safe mode.
If you're still allowing password-only auth AND you're not enforcing strong passwords, you deserve to get owned.
Anyone who's ever seen the logs after a dictionary attack already know how utterly dumb the average SSH brute force dictionary is. You literally have to as dumb as a bag of rocks to use a username and password that's likely to appear in any brute force dictionary.
At least some versions of RHEL.
Fully agreed, it's banal.
(I'm a different anonymous coward thatn the previous one)
This is one reason why people recommend sudo instead of su. The admin logs in as himself and gains root privilege using his personal password. There is no shared root password, so you only have to disable the old admin's account and sudo access. The root password can be set to something very unique/random and perhaps entered into a physical logbook that is kept under lock and key by the management, or just obliterated if you know you can recover systems through local physical access regardless of password.
We like the luks encrypted filesystems too, for a similar reason. We can enter multiple passphrases so that different attending admins can unlock an encrypted filesystem at boot using a personal passphrase, and these can be revoked individually, where we need encryption at rest for private data stores.
Of course there is a level of trust with admins no matter what. Did they install any backdoor software etc.? If you have to salt the earth every time staff changes, it is hard to get anything useful done...
He actually mentions port knocking, but I think he mis-understands it. He makes a mathematical argument, without addressing the fact that the sequence of ports can be as long as you want. He seems to think port knocking is used to grant access, when most of the users I know who use it do so only to start sshd so that they can then log in via secure means.
But having to log in from a multitude of places is pretty common. Less common these days than in the past is having to log in from some random machine. Everyone has a cell phone or tablet or laptop. With those, you can store your private keys, and then totally disallow password logins via ssh.
You can then decide whether to allow root login via ssh or not as a separate issue.
Sig Battery depleted. Reverting to safe mode.
Towards the end he actually mentions the solution: keys. Setting up sshd to not allow passwords solves the brute force issue pretty quickly. But he kind of glosses over it so that he can maintain his narrative.
You can never know everything, and part of what you do know will always be wrong. Perhaps even the most important part.
STOP USING PASSWORDS FOR AUTHENTICATION! why not only allow connections in authorized_keys??? if you feel so inclined, add a password on top of the authorized key.
there is only so much programmers can do to make it user friendly and secure.
Anons need not reply. Questions end with a question mark.
or you could have to fix something when in a trip.
I suggest you come down from your trip before attempting to fix the problem. God knows what kind of things you'll fuck up, otherwise.
Use key authentication. Secure your private key. Job done.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
This is one reason why people recommend sudo instead of su. The admin logs in as himself and gains root privilege using his personal password. There is no shared root password, so you only have to disable the old admin's account and sudo access.
The problem with sudo is that often the complexity of setting it up is time consuming when you have a fleet of admins for a large network, or even a small network with more than a couple admins. Not only do you have to kill the departing admin's account, you have to review the sudoers to make sure no other accounts crept in there as a root equivalent. Some middle management account it unlikely to ever discover they have sudo capability, so someone could simply add that account to sudoers as a root equivalent, steal a password or insert their public key into said person's authorized keys, and ssh in as job-Payroll, and su to root.
I like having only one door to guard.
SSH with key only access, no passwords, no su to root, no sudo, just seems more secure. Each admin has his own public key appended to authorized_keys for root in one place. Nobody has to actually know root's real password (or it can be kept in a vault as you mention).
So one line gets added to authorized keys for each admin, and one line gets deleted when they leave.
Sig Battery depleted. Reverting to safe mode.
Or you could just set up key based authentication only. If you need to access devices that do not support key based auth, set up a server that DOES do key based auth as the only location they are accessible via SSH from.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Centos6.3 and 6.4, AIX 5.3.
Battlemaster--Game with friends in medival realms
Sorry i meant to say only allow su to root only from the physical console
I'd go so far as to say if you are permitting password auth from the internet, you deserve to get owned. secure passwords or not. key based auth is available. use it. if your device doesn't support keys, block access to it from the internet. If you need to access it remotely, set up a secure launchpad type SSH server that does key auth and permit password based access only form that host. All this fucking about with cookies, state tables, and trying to match remote connection attempts is just dumping a turd in glitter to make it look better. it's still a turd.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Sorry i meant to say only allow su to root only from the physical console
I didn't even know it was possible to limit su to a specific terminal. I suppose with SELinux it would be, or via pam.
Sig Battery depleted. Reverting to safe mode.
Pick a host from our pool, assign a user name and password (picked from a list, dictionary or pool)
Implies that each host will, from a predefined directive, try certain user names. I have seen ones better coordinated than that, where they are going through the list alphabetically across a large number of systems. To me this implies a tighter degree of central control on the attack.
...). Any sane admin has root disabled for ssh access, in particular, so the number of attempts they make on that is irrelevant; and the rest shouldn't be allowed much of anything, ever.
That said, the list of users that they try are almost always first names only, aside from the usual collection of system names (root, toor, admin, oracle, games
At any rate, my point remains: this is an old game they've been doing. Interesting that we now have assigned a name to the botnet but otherwise not really news. It would be interesting to know more about the systems that are part of it, but they mostly come from the usual collection of countries and hence your likelihood of getting useful information on them is pretty well zero.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Here is the guide we provide to the SSH users at our University: https://it.wiki.usu.edu/ssh_description
Some of the major points:
Ow please.. This is so old.. haven't allowed pasword logins in the last decade or so..
Why on earth would anyone have allowed password logins for the last 10 years? Or: Ever?
Someone that's savvy enough to get a shell account is savvy enough to use a key pair.
It's 2013. I mean, seriously, PASSWORDS? for SSH?? You must be joking.
-f
That won't help if tomorrow someone finds a vulnerability in the openssh server that enables to bypass that (maybe something like this one from 2011). And that someone instead of announcing it worldwide (i.e. the NSA) start to use it to deploy their own backdoors in your server. Not having access to the service in the first place will avoid potential future exploits on it. Of course, could be exploits for the portknocker daemon, but as is simpler than the sshd (or any other service you have published that is not meant for the world) should be easier to check/audit it (only 2 vulnerabilities were found so far that im aware of, and implies or already being logged in the system, or being successfully authenticated.)
And, btw, the Single Packet Authentication uses a certificate too to open the port for your IP. And then you can use your own ssh certificate or password to login.
I guess this guy never reads his own blog posts. Otherwise he'd see that the log files that he's included are cut off after about 59 characters making them useless for the reader.
You have to do this no matter what privilege escalation method you use, because a rogue administrator might have left a random setuid binary around somewhere. Or has put a logic bomb in the script. Or something like that. Having only one door to guard is no use when the inside of the building carries a bunch of materials for building extra doors.
(1)DOCOMEFROM!2~.2'~#1WHILE:1<-"'?.1$.2'~'"':1/.1$.2'~#0"$#65535'"$"'"'&.1$.2'~'#0$#65535'"$#0'~#32767$#1"
Yeah we always don't allow root login with a password, only with a key. In sshd config:
PermitRootLogin without-password
I wish the argument were named differently. I think some people have been scared that would allow roologin with no authentication . "require-secure-key" would more accurately describe what it does.
Not quite. It's retarded admins that use password authentication on public facing SSH services. I have had a public facing SSH server for over 5 years now and it ONLY permits key-based authentication. I have NEVER had an unauthorized login. But them I'm an unpublished no-name IT guy who only follows those "best practices" that the so-called experts keep railing on about but don't seem to follow themselves.
I am certain you can also have a secure Windows server that has a public facing connection on the internet ... I've just never had the patience to try it.
OpenBSD fanboy then. Pfffft. Whatever.
the reason for seeing so many admin attempts is all those god damn cheap routers that have admin/pw setups by default.
Mod me up/Mod me down: I wont frown as I've no crown
Yes. People who put trust in "password strength" limits don't seem to consider that you cannot prevent users from reusing passwords across multiple hosts and services to cut down the number of "strong" passwords they have to remember.
Password authentication opens you up to all kinds of attacks other than just brute force. If a password is divulged by any other trick on another service, the attacker can go around and try the same password on other servers including yours. When you have something like SSH known_hosts files for each user, it is trivial to discover a working set of target hosts to attack, after compromising a user's account somewhere. They might have used rainbow tables, keyloggers, or trojan horse attacks to get the initial password at a more vulnerable location.
Problem with that is you cannot trust users to secure their keys any more than you can trust them to choose good passwords. Once you give them a keyfile they can change the password to whatever they want, or leave it naked.
Really if you want to excercise hard security, it should be both password and key. No, not password protected key, password stored on the remote host and require a key. What's truly retarded is the failure of most people including crypto software developers to see the use case for this, and endless arguments between the relative merits of one or the other.
Someone had to do it.
Sorry i meant to say only allow su to root only from the physical console
I didn't even know it was possible to limit su to a specific terminal. I suppose with SELinux it would be, or via pam.
I'm sure with SELinux you could permit su access only from the console (yours or the remote NSA one).
In the free world the media isn't government run; the government is media run.
This is one reason why people recommend sudo instead of su. The admin logs in as himself and gains root privilege using his personal password. There is no shared root password, so you only have to disable the old admin's account and sudo access.
They recommend it as safer in theory.
In practice sudo is source of jokes like:
Q: H0w d0 I h4ck ubuntu?
A:
user
user
sudo
user
Sorry but what a (looong) and useless article. Thank you captain obvious.
You can do the same with iptables on Linux using the module "limit". See the manual page for "iptables-extensions" for the details. DenyHosts may have it's good points, but mostly it just complicates things. There is already a lot of functionality in the packet filter that you can use, whether on Linux or BSD.
However, what I see now, in contrast to years ago, are slower paced attacks. These come in steadily but at a rate that just passes under the threshold. One of these days I ought to look at what is blocked to see if it's just the slow ones getting through or if all the probes are now timed that way.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
You don't see the bots trying a million passwords with user "jeleay", they are trying "root", so not allowing root via password is significant.
It's not just banal, it's insulting and contradictory. He takes time out of his busy writing schedule to pan port-knocking, says he could tell you how to do it easily but won't, and falsely equates it to just adding bits of complexity, when it in fact adds a time domain as well.
My favorite part is when he says:
There's nothing magical about TCP/UDP ports. It's a 16 bit number, and in our context, it's just another alphabet.
Like iterating over 16 bits multiplied by the password database is trivially fast. This is some real ivory tower shit, but I guess that's to be expected when you are hyping the botnet equivalent of the Model T.
Debian 6.
Every trollism an AC posts is prefixed, in my mind, with "A. Coward whined, in a weak and cowardly voice:"
I don't advise that anyone else do as we have done, however. If everyone starts reading log files and defending themselves the attackers will stop being simple minded and try harder.
"Be a good sport and take the fall for the herd", eh? :-)
This is impossible. Everyone knows that Linux is so great because this can't happen. It is open source and there are no holes that allow thing like this because of the "many eyes" thing. This only happens on WinBlow$ boxes. ~
Who the fuck tries passwords against all the ports?
Are you retarded?
There was never any safety in using high port numbers for SSH.
That is security through obscurity at its worst.