Cracked Linux Boxes Used to Wield Windows Botnets

m-stone writes "Online auction house eBay recently did a threat assessment to better understand the forces ranging against them. The company is keeping the fine details under wraps, but the biggest source of danger for the company is apparently botnets. You're never going to guess who was running them. '[Dave Cullinane, eBay's chief information and security officer] noticed an unusual trend when taking down phishing sites. 'The vast majority of the threats we saw were rootkitted Linux boxes, which was rather startling. We expected Microsoft boxes,' he said. Rootkit software covers the tracks of the attackers and can be extremely difficult to detect. According to Cullinane, none of the Linux operators whose machines had been compromised were even aware they'd been infected. Because Linux is highly reliable and a great platform for running server software, Linux machines are desired by phishers, who set up fake websites, hoping to lure victims into disclosing their passwords."

Comments

Confirmed

[mccalli]

I've noticed a large increase in attempts to crack my co-lo Linux servers recently, and it must be said that two got through (shared site, some customers running old content management apps and the kits hit). When we watched the behaviour of the cracked box, it was connecting back to...I think undernet.org or similar?...and sending controls via IRC. Plus doing a spot of spamming of its own bat.

Our set-up is that we have a host OS install doing nothing but running VMware Server and then any real stuff gets done in a VM, so this was easy for us to recover from quickly via VM snapshotting. But still, it's a trend that's noticeably on the increase.

Cheers,
Ian

Re:Confirmed

[mccalli]

How are they trying? My logs show lots of attempts at phpbb, etc. vulnerabilities.

phpbb, Drupal and PHPNuke attempts mostly. Plus old sshd vulnerabilities, though we're up to date there and nothing got through.

Cheers,
Ian
(oh yeah, and first post! Only took a mere eight years or so...)

Re:Confirmed

[Bert64]

I've not been dossed per se, but some of the ssh/ftp brute force attempts and scans for common website vulns are incredibly aggressive.
My biggest problems stem from foolish users, i host a lot of customers who have the ability to run PHP apps and choose their own passwords. Fortunately, each user has their own account, so i can easily check which user owns any malicious processes or files that appear on the system.

Re:Confirmed

[mccalli]

Fortunately, each user has their own account, so i can easily check which user owns any malicious processes or files that appear on the system.

May want to be careful about that assumption. A lot of these things go out under the apache user and the mails via the www-data@somehost.invalid account.

Look for tell-tale things like apache processes running when you're an apache2-only site (they're disguised processes that are really something else, obviously). Do an ls -al in all the home directories, look for directories whose name is just a space character, check /tmp isn't mounted executable...that kind of thing.

Cheers,
Ian

Re:Confirmed

[Bert64]

All of the Apache processes show up as /usr/sbin/apache2
A process just called "apache" or "./apache" would stick out like a sore thumb...
Apache doesn't run as a single user either, each site runs under it's own userid.
There is also trusted path execution enabled on the server, so the web users can only execute programs which are owned by root, and located inside a directory owned by root, so they can't upload and execute arbitrary binaries, all they could really leave running is a php script.

Re:Confirmed

[NeverVotedBush]

Hard for me to say. But if you want to clean up the simple things, turn off every service you aren't actually using. If using sshd, edit /etc/ssh/sshd_config to disallow root logins, only allow protocol 2, and set allow users to just whoever needs access. The post above about using a different port for ssh is also a good idea. Security through obscurity is still security for some attacks.

Use strong passwords. Make sure all other users use strong passwords. With some exploits just needing a user account, if someone can't get in as a user, it blocks that path.

There are blacklist websites that when you are browsing allow you to redirect any accesses to them to go to 127.0.0.1 instead. Makes pages load faster and reduces exposure to web-based ad exploits. But there are also blacklist website lists that you can use to deny all traffic from into your machine regardless. Speaking of, make sure you have your firewall up and running. Make sure your gateway to your ISP is running a firewall and since you have more than one box, you are probably running with NAT. If not, set it up.

That's the short list. You already say you keep up with patches. That's also very important. Good luck!

Re:Confirmed

[AndroSyn]

Well one could mount /tmp on its own filesystem or perhaps using tmpfs so that it ends up effectively using swap space for /tmp. Then you mount it with the noexec flag which in simple terms tells the operating system not to run executables from here. However this does not stop people from being able to run shell or perl scripts from here as they could simply do /bin/sh /tmp/somescript.sh or /usr/bin/perl /tmp/someperlscript.pl or so.

Re:Confirmed

[TheRaven64]

Having a different filesystem makes sense for /tmp, because you do not care particularly about integrity. You can mount it async and disable journalling, because you don't expect data there to persist beyond a reboot. Having it as a separate mount (which can be achieved via remount) also allows you to disable support for setuid binaries on it, or even just disable execution. That's why it's always been fairly standard practice for it to be a separate slice. The same applies to /var, which typically wants to be write-optimised, while /usr wants to be read-optimised.

Re:Confirmed

[betterunixthanunix]

I see lots of attempts on PERL and PHP scripts, many of which I cannot identify, in the logs for my personal web server. lbc.php, prx.php, awstats.pl, to name a few. Just another reason to keep SELinux in enforcing mode.

Re:Confirmed

[lazy_playboy]

A server should really have /tmp located on a seperate partition that is only mounted read/write and not executable.
At least, I think that's generally thought to be a good thing to do.

Re:Confirmed

[B2382F29]

Use fail2ban, and regarding banning all of China, I didn't see getting more than average attacks from there, a lot of attacks are also from US IPs, this seems to be a worldwide issue.

Re:Confirmed

[Qzukk]

all they could really leave running is a php script

Of course, PHP these days has all of the network goodies needed to make "just a php script" still a serious problem. A good idea is to make sure that anything the webserver can serve can't be written from the webserver's user, i.e. uploaded data goes outside the docroot. Of course then we'll get tears about how hard it is for everyone to install forum software and their blog since they can't just unzip/untar their code and mark it all writable.

That's the problem - "Its Linux, so its secure!"

[Anonymous Coward]

So many people have that mentality or were converted by hearing sayings like that.

They don't realize, like any other operating system, if you want it secure, you have to work to make it secure. Everything from using good passwords, to not running unecessary services, to getting behind a firewall or two.

And, as usually, the biggest security hole is between the keyboard and the chair.

Windows vs. Linux

[derian_cf]

I work for a fairly well known dedicated server provider. If I had to give a rough estimate, I'd say we're 40% Windows and 60% Linux environments. Not surprisingly, the number of boxes that get hacked (rooted entirely or not) is about equal between the two, however the purpose for which they're hacked is generally quite different. 80% of the hacked Linux boxes are used for UDP floods, things like that. Also IRC bots. Interestingly enough, in my 6 months working there, I don't believe I've ever seen a Windows box used for phishing. They're always used for FTP servers hosting movies/music/programs and/or IRC servers doing the same thing.

Brute Force Attacks

[superbrose]

I'm sure pretty much everybody who is running a Linux server (or any server as a matter of fact), especially with services like SSH enabled, is currently subject to brute force attacks.

When I looked at my auth log I noticed a huge amount of brute force attacks for all my servers, so I installed denyhosts, which seems to work fine.

I guess the problem is also that in many distributions SSH servers are configured to allow root logins, and if nobody looks at the log files these go totally unnoticed.

Re:Brute Force Attacks

[Russell Coker]

Run your sshd on a port other than 22. Most attackers only scan the well-known ports. Running your sshd on a different port removes a lot of the noise from your logs and allows you to concentrate on the real issues.

The "Host" sections in the /etc/ssh/ssh_config file allows you to specify which port to use for each host you connect to (so you don't need to type "-p 1234" every time you connect).

Re:Brute Force Attacks

[walruz]

Run your sshd on a port other than 22. Or instead of that, just disable password authentication and allow only RSA/DSA keys auth. OTOH, moving the service to another port may save bandwith from the constant login attempts, but in certain scenarios (behind a router or provider with strict policies, like only allowing port 22, 25 & 80 forwardings) this may not be an option, in which case, disabling passwords is the best option IMHO.

Re:Brute Force Attacks

[networkBoy]

While it is security through obscurity, that is not a bad thing. What people often don't understand is that obscurity is a valid security measure, just not one that you can rely on as an end-all.
Example:
SSH on 22 with no authentication -> bad
ssh on 2200 with no authentication -> just as bad
ssh on 22 with strong authentication -> good and not burdensome
ssh on 2200 with strong auth -> better and not burdensome.

As Bruce says it is all about layers of security and understanding the deficits of each. By running on port 22 you subject your strong auth to multiple automated attacks, thus increasing its exposure to being comprised. By taking the simple step of moving to a non standard port you have eliminated the bulk of the automated attacks, which as PP noted allows you to focus on fewer log entries, likely showing a wider breadth of attacks per unit of log entries.
-nB

Re:Good News & Bad News

[morgan_greywolf]

Yes. You should Here's what I do. (I guess you could say these are some security tips for those running Linux boxes at home and leaving them up on the Net):

• Run a hardware NAT firewall/router. Any ol' Linksys, Dlink or Netgear thang will do. Just remember it's not the be all and end all to security problems.
• Open as few ports as absolutely possible. I have nothing open on my router except port 22 and BitTorrent, and I don't leave BitTorrent running all the time
• Check your logs at least once a day. Look for any suspicious signs -- missing log entries, ssh connects you weren't expecting, services running that you don't normally have running, NICs going into promiscuous mode unexpectedly, excessive mail being pumped through any MTAs, etc.
• When running OpenSSH, I disallow password authentication. This prevents problems with users due to the use of stupid passwords. My sshd only accepts a valid RSA key exchange as acceptable authorization.
• Regularly update and run rootkit checkers. These are not be all end all, but they help spot obvious rootkits
• Make cron jobs that regularly scan your system for unusual permissions -- world writeable, binaries that are setuid, etc. and for suspicious files. There are programs and scripts that will do this for you. STFW or check with your distro.
• Perform MD5 checking on your files and executables, espcially.
• Regularly check your /etc/passwd and /etc/group files for new or unusual entries.
• Don't run NIS -- it's inherently insecure. You should be using OpenLDAP if you need directory authorization on your network.

Re:Good News & Bad News

[alexhs]

Many points you make are done automatically on a default FreeBSD install :

Suspicious network activities (bad logins, reverse DNS issues...) are printed on the console and e-mailed to you each day.

Every root logins also printed on the console.

System modifications (user accounts, system files permissions, disk usage, start scripts modifications) are alse mailed to you (some maybe only once a week)

I only check regularly the console, and once a month or so I check the e-mails. (It's my home server BTW, don't need much maintenance).

Comment removed

[account_deleted]

Comment removed based on user account deletion

Root not needed to host phishing scams

[rjamestaylor]

I am a supporting system administrator for Linux/UNIX servers at a large hosting
company. I have come across many Linux servers that are compromised and being
used to host phishing scams, spamware, IRC servers, etc. Rarely, however, do I
see a "root'ed" server -- that is, a server on which an unauthorized
person or program has gained root privileges illicitly. In fact, having root
access is not necessary to host web content, send mail or provide other
Internet-facing services.

All that is needed is the privilege to put content served by the web server in
place. That could be a script for server-side execution, page or fragment for
browser- (client-) side execution, etc. If you can upload to the web content
(DocumentRoot or include) directories and the web server automatically servers
that content, you, too, can host a phishing scam or illicit media for download.

If a directory in the DocumentRoot tree on a web server can be written to by the
web server (the apache or nobody system account) then it is easy to inject one's
illicit content on that server. OS is irrelevant at that point. In fact, if a
web server has world- or apache-writable directories in the web content area the
OS *must* allow any web client to upload whatever they desire to that server.
It is the responsibility of the owner of the server to restrict who gets to
upload what content to his/her server.

I try to explain to web designers that granting write access to the
apache/nobody user is BAD, but often I hear back: "Ya, but, I can't make
the script work without opening the permissions." Usually, this is done on
PHP Content Management System portal sites that allow content to be uploaded
directly from the web browser by arbitrary users. There is a little bit of
effort required to make doing this difficult -- and it can be tricky to get
right -- but forcing the script to work by removing world/apache write
privileges is EASY:

$ sudo chmod -R 777 /var/www/html

Ugh. Then, when that same customer is complaining that, "Hey! I've been
hacked!" I respond, "no, you haven't. You been compromised. You
allowed *anyone* to upload *anything* to your server and set apache to
automatically server that content. You were trusting *everyone* on the Internet
to behave. Your trust was broken and now your server is distributing phishing
scams/malware/kidde porn/spam."

If you ever think you need to "open up" permissions so your PHP script
will "run right" you either need a different PHP script or help making
the script run "safely." It's harder than chmod'ing 777 but it's
definitely worth doing.

One server I worked on had a lazy owner who allowed apache full write and
execute access to his web content directories. He would not upgrade his PHP
scripts to patched versions that plugged well-publicized holes. After repeated
warnings I received a frantic call from him that his server was
"hacked" and running a banking phishing scam. I checked the weblogs
and found that 20,000 people had clicked the phishing scam links from their
webmail inbox and retrieved the malware-ladden web pages with Internet Explorer
-- meaning many of these people were sending their data right to the
Russian/terrorist criminals for funding their illicit operations. The customer
asked that I call the FBI to "find out who is responsible" and I said
I didn't need to make that call to find out: he was responsible.

That customer is now fully-turned around and is complying with the necessary
steps to ensure that his server is not used for illicit purposes any longer.

Root was never required for these compromises. Just poor administration.

Re:I'm not surprised

[TheRaven64]

There are three things that make a server machine, Linux or otherwise, more attractive than a home machine:

• Upstream bandwidth. A server is likely to be on a 10Mbit or 100Mbit upstream pipe, while a home machine will rarely have more than 0.5Mbits. You get between 20 and 200 times the available bandwidth to launch new attacks with a cracked server.
• Uptime. Home machines are rarely on for more than 8 hours a day, servers are on 24/7. In terms of daily bandwidth, you get at least three times as much from a server as a desktop on the same connection. Combine this with the previous point and a server is worth 60 to 600 desktops.
• Static IP. If you crack a home machine, you need it to periodically ping you to let you know it has changed IP, making it easier to trace you.

Some comments on rootkits

[ajs318]

There's a particularly nasty rootkit out there which overwrites certain system programs (such as ls, ps, netstat, md5sum and a few others) with modified versions, then does a chattr to stop you overwriting them (though lsattr is left alone). And while attempting to clean up a machine so infected, I've seen Perl scripts changing the value of $0. This means even if you've got a "clean" ps around (like a copy of busybox in your own non-root home directory ..... you do have a non-root login, don't you?), it will report the "wrong" thing. Another clue that this rootkit is installed, is that (at least on Debian and Slackware) coloured directory listings don't work properly, and invoking ls generates a non-fatal error message. (The "special" ls must be based on an older version.)

The www-data (Debian / Ubuntu) or apache (Fedora) user should not be running any process other than apache2 or httpd. If you see something like "accepting connections", that's a sign that someone could be running something nasty.

In general, watch for world-writable directories (they list with a green background in Debian) because that's one of the first steps in cracking a box ..... install a script in a user's home directory, then persuade it to run. Beware of badly-written PHP scripts which don't chmod uploaded files to make them non-executable (turning off short open tags is also surprisingly effective). And what you think might be a DDoS (repeated attempts to retrieve mail on nonexistent accounts via POP3) might actually be a password-guesser. Block the /24 with an iptables rule at once. Note, if you aren't within walking distance of your co-lo, make your first firewall rule
iptables -I INPUT 1 -s 10.20.30.40/32 -j ACCEPT
(replace 10.20.30.40/32 by a subnet specifier which will always contain your own IP address -- get this from your broadband company -- and just to make you all jealous, my one ends in /32 because my IP is static) and never, ever use -I INPUT 1; use -I INPUT 2 or -A INPUT instead. It's too easy to block yourself out with an injudiciously-applied rule (and I do live within walking distance of my co-lo). If you see a process running that looks suspicious, leave it running long enough to examine its /proc entry before applying kill -9. Give users who don't need shell access a "shell" of /bin/true or /usr/games/fortune -o; but be sure to include whatever "shell" you gave them in /etc/shells -- otherwise they will not be able to use FTP. (If they don't have any web space on your server, just e-mail, then use /bin/false and don't put that in /etc/shells. That will make it harder to use an ftpd-based exploit.)

Note that the binaries in this rootkit are 32-bit ..... so running 64-bit Debian (which has *no* 32-bit libraries) will break them. Personally, I'd like to see a patch that will make Perl give a segmentation fault if any script tries to alter $0. In fact, I'd like to see a kernel patch that will break any binary that was not compiled locally.

Re:Security is not simple...

[deftcoder]

Try Denyhosts. http://denyhosts.sourceforge.net/

Most distros come with it available right in their package manager.

Useful UDP services

[canadiangoose]

I can think of a few right off the top of my head. There's SIP, RTP, OpenVPN and DNS, just to name a few. I'm sure there are more.

This isn't a Linux botnet. PHP is a pox.

[Tracy Reed]

A friend emailed me about this just this morning. Here is what he wrote and my reply:

> I'm going to chalk this up (tentatively) to the increasing popularity of
> Linux, which means that a subset of users will be those who don't actually
> know what they're doing, and how to protect a box-- something long the norm
> in the Windows world:
>
> http://computerworld.co.nz/news.nsf/scrt/CD0B9D97EE6FE411CC25736A000E4723
>
> While there, he noticed an unusual trend when taking down phishing sites.
>> "The vast majority of the threats we saw were rootkitted Linux boxes,
>> which was rather startling. We expected Microsoft boxes," he said.

I am not surprised in the least that this was their conclusion. I don't chalk it up to the increasing popularity of Linux at all. I have never (not once) run across a Linux box operating in a botnet. Nor can anyone name a botnet software that infects Linux boxes. In the last 5 years I have found only one Linux box that had a security issue and that was because of PHP (*spit*) which had an XML-RPC exploit a while back and allowed someone to make the box host a fishing website that looked like some bank website. It seems very rare that a Linux desktop (not a webserver) would fall victim to this. I have never seen a security incident such as a botnet on a Linux desktop. I have seen that phishing page on the Linux server that hosted the bogus PHP install. That's it.

And I suspect that they are using terminology incorrectly. A Linux box hosting a fishing site is not part of a botnet. I can understand how Linux boxes would be more popular for fishing websites. PHP is popular and is a pox on Linux as PHP released a bunch of absolute garbage which only happens to run on Linux. It can run on Windows also but that is the expensive and less reliable way to do it so few people do. If people make a conscious decision to install software on Linux that lets just about anyone use the box for whatever they want such as PHP often does I don't think counts against Linux security.

Glancing over the article I immediately spotted this:

"eBay recently did an in-depth analysis of its threat situation, and while the company is not releasing the results of this analysis, it did uncover a huge number of hacked, botnet computers, said Dave Cullinane, eBay's chief information and security officer, speaking at a Microsoft-sponsored security symposium at Santa Clara University."

I challenge anyone to find a single MS sponsored paper or symposium which DOESN'T come to a conclusion favorable to MS and unfavorable to Linux. Just one. And they won't release the raw data. How much is a large botnet? 10? 100? Among millions of infected MS machines. I would also like to know what this alleged Linux botnet software is called.

I am positive that Linux will not be nearly so adversely affected by users who do not know what they are doing. Linux is very different from Windows and is architected for performance, security, and utility instead of being architected to make someone a boatload of money and maintaining monopoly lock-in. (See the fine the EU just imposed on MS.)

Some technical features which help ensure that even if Linux becomes popular on the desktop it won't suffer the same fate as Windows:

* Linux users don't run as admin/root.

* Email programs do not automatically execute attachments.

* Does not depend on filename extensions for anything.

* Does not auto-run anything from inserted media (Worth a laugh: http://www.foxnews.com/story/0,2933,299155,00.html )

* System of mandatory access controls (SE Linux) which really locks things down (some people still turn that off but it is improving rapidly, I use it on my desktop).

* Linux also takes advantage of NX (non-executable memory) which is a recent feature of x86 cpu's

Re:Your sig [OT]

[808140]

In this case, it should be "If I was able to see further." Use of the subjunctive mood in English (If I were) indicates that the statement is contrary to hypothesis. For example, "If I were a dog, I would lick myself" implies that I am not, in fact, a dog, and am only speaking hypothetically. Whereas: "If I was a dog, it was only because I was selfish at heart" implies that you were a dog (in this case, the meaning is figurative, obviously).

Here, your sig does not introduce any information that is contrary to hypothesis. When you say "If I was able to see further, it is because..." you are actually giving an explanation for why you were able to see further. Saying "If I were able to see further" implies that you were not, in fact, able to see further, which is not what you meant.

Hope this helps.