Slashdot Mirror
First Botnet of Linux Web Servers Discovered
The Register writes up a Russian security researcher who has uncovered a Linux webserver botnet that is coordinating with a more conventional home-based botnet of Windows machines to distribute malware. "Each of the infected machines examined so far is a dedicated or virtual dedicated server running a legitimate website, Denis Sinegubko, an independent researcher based in Magnitogorsk, Russia, told The Register. But in addition to running an Apache webserver to dish up benign content, they've also been hacked to run a second webserver known as nginx, which serves malware [on port 8080]. 'What we see here is a long awaited botnet of zombie web servers! A group of interconnected infected web servers with [a] common control center involved in malware distribution,' Sinegubko wrote. 'To make things more complex, this botnet of web servers is connected with the botnet of infected home computer(s).'"
Just waiting for the flamefest here of Linux vs Windows botnets.
Live today, because you never know what tomorrow brings
Awkward...
Funny may not give karma, but +5 Informative never made anyone snort coffee out their nose.
It's ready for the botnet!
We can blame our hate pet OS for all of the internet evil out there, but we need to remember one important thing: people are almost always the week link in security. If someone knows what they are doing, it is very hard to penetrate a linux server... or a windows server. There will always be those that can break through the best security, but there is a lot of low hanging fruit and not just on the windows tree.
Does this mean Linux finally has reached a point of user friendliness equal to Windows?
What's so special about this one that we haven't seen in the last 5 years? Linux or BSD systems have been durned into rogue IRC servers (for C&C purposes) for zombies all the time.
Whether sweeps for vulnerable AWStats installations, badly configured PHP installations or archaic PHPBB installs, webservers are hammered with automated exploits all day. Maybe "DataCha0s 2.0" rings a bell for some.
"With about 100 nodes". The average windows botnet (at least the one that make into the news) have from hundreds of thousands to millons of nodes. Not sure how "automatic" was the creation of this botnet, or how much at risk are generic linux users. Considering how are installed some and how careful are some admins about "security", is not amazing that a few out there could be rooted.
In fact, if those servers already had apache, and some old vulnerable web application that enables somewhat transfer and execute binaries, in no recently patched kernels 2.4+ there are ways to escalate priviledges and get root to install what is needed. But probably normal users using modern distributions or admins caring a little about security are safe.
The only part of this article that is news is the part that is incorrect. Botnets of Windows machines often have compromised Linux servers working as a control channel or update channel. It is not at all unusual. What would be unusual would be for a worm or virus to actually compromise Linux machines in an automated fashion and make them bots. That does not seem to be what has happened here as the Linux systems seem to have been manually hacked in a normal, directed attack.
Basicaly, nothing new or newsworthy happened here, except someone mistakenly referred to the compromised Linux servers as bots.
The article speculated that, since the iframe code was injected to legitimate webpages using stolen FTP credentials, it may be that a few "root" credentials are obtained the same way. FTP credentials can be stolen by malware running on the client computer, for example a computer an admin uses to control the server, from well-known FTP client software.
I once had a signature.
If your customers put up vulnerable software on your shared, dedicated, or virtual hosting service and they don't update it or you don't detect it, someone's going to find it and exploit it.
Had something similar happen to my me. If you're monitoring server load, a webserver sending spam will definitely raise an alarm. As for services on odd ports, block everything except the real ports. Blocking outgoing traffic on IRC ports helps too in minimizing damage. The script kids are already making use of the recent Linux local root exploit (wunderbar_emporium) so make sure you do some yum updates!
nginx, so that's what the worm is called? I'd better check my company's webservers so they aren't running this evil hacker malware.
Oh my... all of them had been infected. No worries though, I managed to clean them all up. A good day's work well done.
It's unclear exactly how the servers have become infected. Sinegubko speculates they belong to careless administrators who allowed their root passwords to be sniffed.
If Sinegubko is right and the attack vector was sniffed passwords, then it is likely that those passwords got sniffed by an existing Windows Botnet.
We don't see the world as it is, we see it as we are.
-- Anais Nin
Rather than getting consumed in an OS holy-war, perhaps we should focus on how exactly these systems were compromised and how to detect whether your server has been compromised. Linux servers being compromised is not a new thing. If you run old-enough libraries and software on them or configure things improperly, they'll eventually be compromised.
Does anyone know if a particular vulnerability was used to gain access to systems?
Does anyone know how to detect whether your system is compromised in this manner (is doing "ps -aux nginx" simple enough to detect it)?
Spare everyone the OS holy-war and fanboism and let's figure out what the problem is, how to detect it, and what to do to fix it.
Faith is a willingness to accept something w/o complete proof and to act on it. Reason allows you to correct that faith.
It's nice to be able to apt-get yourself the latest stable copy of apache2 and php5 and mysql and postfix humming with just a command or two, also nice to be able to apt-get upgrade them after you apt-got updated. Those who maintain, clean and contribute to the large public repositories that apt and yum and rpm and pkg_add, good people and they generally do a bang up job for 99% of the Linux and UNIX and UNIX-like folks. However, when you maintain servers which are not completely hidden behind a nat with these programs for years and once in a blue moon compile something you downloaded in a gzipped tar, you put yourself on admin autopilot and that can bite you in the ass.
Give you one example: I installed RoundCube, the most badass webmail client there will ever be, ever, with apt (the first time). Ran it for a while without incident. Had my system on weekly cron apt updates so I figured I was safe. Eventually I discover someone made it onto my system and put a malware installing js line in my web pages. Looking through the guy's bash history I discovered they got in through a RoundCube vulnerability. I checked out RoundCube's site, something I should have done first thing but did not, and it turns out their stable version was much newer than what apt realized and that this vulnerability would not have been on my system about five months ago had I downloaded straight from their site and stayed on the ball with their support resources which are things that are less necessary when you just let apt-get rip.
Bottom line, apt-get update/upgrading would not patch a glaring vulnerability in software I found with apt originally with the default Debian sources.list and I doubt it would have on most other distros' package management systems. It wasn't RoundCube's fault, the patched release was their Stable build for a long time but I was left wide open to anyone who went on a rootkit site and googled for roundcube hosts and I got nailed. Learned my lesson and I don't fault the repository maintainers for being behind the ball a bit on less popular software in their enormous archives but if you ask me software should not be available on the default repositories for Linux variants that the maintainers are not confident that they can keep up to date or don't have some kind of way to be quickly and effectively notified by the authors/vendors in the event of a critical upgrade being available and to put it live right quick. Put it on the people who want to install such software themselves -- if they can make it past that hump I'd say their odds of running the software safely will be substantially higher than Joe Yum. And spreading awareness of cvs/svn would be nice too.
Can't believe I just admitted I got compromised.
Calling out bogus battery capacity claims.
Actually, the article says that FTP passwords were used. Meaning they were probably sniffed either on FTP Users personal computer, or over the wire somewhere between the user and the server on one of the hops, which could be dangerous.
Moral of the story, use SSH!
Can't believe I just admitted I got compromised.
Much better than the fanbois who have tried everything under the sun to defend their pet project against the evil meanies who don't have a problem admitting that every system has weaknesses.
More than once I heard "I just use Linux, so I'm gonna have a secure system anyway". Yes, Linux is more secure by design than windows, but this attitude makes ppl dumb and lazy.
On second thought, let's not go to Camelot. It is a silly place.
Back around 2001, I found a "botnet" comprising a perl script that ran on websites. Because it ran as a child of Apache, it showed up as "http" in ps. It would log into an IRC server, and wait for commands which appeared to be little more than arbitrary bash commands that were shelled out.
Bone-headedly simple. Ran well on any unix website host running perl scripts, installed via an insecure formmail.pl script. I penetrated the IRC network and watched for a few hours while the operator attacked a few hosts. There were some 50 hosts or so. Then I killed the script and updated all copies of formmail.pl hosted on the server...
Is this new news?
What's next? "Hammers can be used to smack things, even if they aren't nails." !?!?!
Truth is this: no operating system is 100% secure. But this "botnet" isn't necessarily even a compromise of the Operating System! Port 8080 is above 1024, so non-root controlled processes can open sockets there. This may be nothing more than something like the perl script I mentioned and having nothing to do with the Operating System in question. The server wasn't compromised, just a bad script was running that had to be deleted, then killed with an Apache restart.
Given the parameters I just mentioned, there isn't an Operating System around that would stop this from happening. It's just that the "Mom's basement" fanbois get all riled up because it's gospel that Linux is immune to $allBadThings.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
I can't imagine how you came to the conclusion that the fault was with *apt* of all things.. did you think it works by magic? Blame the Debian "It's not moldy, so it's not for us" maintainers instead, or even yourself for using a distribution known to ship ancient software no longer supported by upstream.
Firstly, it's my fault for running a webmail client I got from browsing through apt-cache, installed with apt-get and configured mostly with dpkg-reconfigure instead of grabbing the official current build and reading the readme and man pages and faq, and doing this on a somewhat important machine. Did the same thing with Gallery and PHPNuke several years ago. Even webmin in my reckless and stupid experimental days. That's painting a target on yourself to get malware on your sites and start running irc bots or worse. Have you looked at some of these rootkit sites? Disturbing how finding and proliferating vulnerabilities in Linux, not just MS, is a full-time hobby/living for so many people. Then you install something like snort from apt-get thinking Yeah I'm on top of my security now, but you have no idea that you're using a six month old release of software with a demo package of ancient rules when it needs heavy configuration that dpkg doesn't handle and fresh rules with a subscription and a key in the right place to be effective.
That said, yeah, Debian's reputation for waiting a ... conservative amount of time to make new releases of various software available on their repositories, whether it's gimp or gaim or kde or nmap, maybe I assumed that that behavior of deliberately (?) waiting a little while longer than the rest of the world to catch up to the developers' latest releases for the sake of not releasing anything that may contribute to snafus, that Debian's actually doing what's best for me. Maybe my roundcube adventure was anomalous. Regardless, I love Debian, I certainly love apt (so much I just tried Debian KFreeBSD to hang onto apt). By naming the package management systems of the other distros/OSs I was trying to suggest another point that Linux is becoming too easy. Lower learning curve, more people who may make my mistake and surrender their machines to China, Russia and 4chan by installing the wrong package.
It would be great if apt had svn/cvs behavior embedded into it to somehow investigate whether or not everything on your system is up to date by logging not just onto Debian's repositories but to servers maintained by developers. Can't expect apt to then install the next version but just to let me know what it found so I could deal with it myself. Maybe such a thing already exists -- guess I should apt-cache search it. :P
Calling out bogus battery capacity claims.
A beowolf cluster run by a beowolf klutz
That Debian runs older stable software, does not stop them from installed patched versions of software when it comes to security. You still get security updates in stable.. Not pointing fingers or anything.. but if I do a search for roundcube in debian stable I don't find anything.. testing, unstable, and experimental yes.. but stable no.. So perhaps the whole idea of running the creaky old software makes sense.
waiting for ad.doubleclick.net
Let this be a lesson to everyone who reads the article. Security is not something that happens by accident.
I've said for a long time that binary packaging is, fundamentally, a Hell-spawned abomination masquerading as a convenience; incidents like this only prove the point.
Compile yourself a minimalistic base system, a la Hardened Linux From Scratch.
Then get the absolute minimum number of packages you need for a working system, such that you've got some chance of keeping them updated. Firefox for web browsing, maybe. A single media player; VLC or Xine. Vim/Emacs as an editor. OpenOffice.org if you need that. Whatever servers you need, but keep that list small. A firewall, which is hopefully obvious.
Use a minimal window manager which doesn't have a dep list as long as your arm, as well. I use Ratpoison. Do not laugh until you've tried it. It is very, very fast, and resource consumption is virtually nil. It's basically an X version of GNU Screen.
Once you've got this small list of packages, take full, ruthless, practical advantage of the fact that your system is open source. Subscribe to the announce or bug related mailing lists for the apps you've got, and keep local virgin tarballs. This way, whenever there is a bug or potential exploit, and the patch gets posted within a few minutes or hours, you can get it the moment it goes to CVS, patch your own source tarball, and recompile. The same goes for the kernel itself.
You won't be vulnerable to exploits, because you'll get the solutions to them as they are implemented, and you're also far less likely to end up with a compromised machine as a result.
Brainless Windows refugees, who will sneer at me, and/or complain about how this isn't, "user friendly," don't even bother. This post isn't for you. We already know that you've committed yourselves to being servile, unthinking sheep, and you are therefore invited to accept the consequences of your (lack of) actions in that regard.