Ongoing Linux/Solaris Compromise Epidemic

An anonymous reader writes to point out that Stanford's Information Technology Systems and Services "has written a summary of a series of compromises that have been happening at universities, research institutions, and high performance computing centers, for the last month or more. The attackers are using known vulnerabilities in Linux and Solaris, along with compromised user accounts, to gain access and control of systems, from standalone servers to HPC clusters ... (the attacks are still ongoing)."

Comments

Re:Nothing to worry about

[arazor]

What diffrence does it make what the attackers motives are. If he is doing it that means the blackhats can do it as well. This is something we should -all- be concerned about.

Windows is not the only vulnerable OS

[ObviousGuy]

It is important that when we wave our flags and cheer when Microsoft is laid low by the latest security flaw that we not close our eyes to the very real vulnerabilities in the Unix/Linux system. No OS can be fully secured, and it is absolutely mandatory that we remain vigilant to the possibility of a heretofore unknown security hole in our systems, regardless of the system OS.

Assuming that Unix/Linux is invulnerable to security holes is deadly. Though the OS may have more security features and "more eyes" on the code than closed source operating systems, we must not rest on our laurels watching Windows implode while our own house is burning.

Re:Windows is not the only vulnerable OS

[morelife]

You're joking.

All the vulns mentioned have patches/fixes/replacements for the faulty code.

The System Administrators are at fault FOR NOT MAINTAINING THEIR SYSTEMS PROPERLY.

Re:Windows is not the only vulnerable OS

[FrYGuY101]

How does that differ from the worms which get released for Microsoft almost a year after the patch was released? I hear people railing Microsoft all the time for not 'getting it right the first time' when THAT happens...

Re:Windows is not the only vulnerable OS

[Monkelectric]

Eh in theory yes, in practice it is a little more difficult. Closed source kernel modules really complicate the whole upgrade right now issue, and even alot of open source modules can break between kernel versions. VMWare is used in alot of operating system courses (and thus on alot of acedemic computers), at the *very* least its modules need to be recompiled, although its pretty good at not breaking between kernel versions ... same with alsa, the nvidia-kernel, bestcrypt, and a million other modules.

That being said, when the choice is compile modules or get o3ned, your path is clear :) The price of freedom is eternal vigilance.

Re:Windows is not the only vulnerable OS

[EvilTwinSkippy]

I am a religious patcher. Hell, I've almost gotten a fired a few times when patches went wrong. Bosses just don't understand that machines don't just "work". They require constant intervention. The computers, that is, not the bosses.

Now that said, you have an interesting slant on ethics. By that mindset, a burglar is perfectly entitled to break into your apartment because your door could be kicked in. A theif can swipe your radio because, hey, it was only glass between him and what he wanted.

Yes, there is a certain amount to be said for not painting a target on yourself. But regardless of how much you "had it coming" it's still a crime to break into your dwelling, steal your property, or damage your person or posessions. System intrusion is a crime, and a matter for law enforcement.

Re:Windows is not the only vulnerable OS

[bebing]

Wow this got modded up to +5 while there were only 55 replies to the article, that's fast. Either you're popular or there are a lot of pissed off MS fans out there.

It is important that when we wave our flags and cheer when Microsoft is laid low by the latest security flaw that we not close our eyes to the very real vulnerabilities in the Unix/Linux system.

Is there really flag waving and cheering going on? Perhaps joking and laugher. Also Linux vs. Microsoft(leaving Unix out for now) is not comparable to say Rocky vs. Apollo Creed, but David vs. Goliath. Microsoft does not need you to defend them, they have billions of dollars and a monopoly. We do have to stick up for Linux because we are Linux, and there is nothing close to a monopoly or billion dollar bank accounts. Now that I think about it maybe cheering is ok when the bully takes a blow to the chin, it happens in the movie theatres.

Re:Windows is not the only vulnerable OS

[SemperFiDownUnda]

Most companies don't get it right the first time. If they did there would never be patches would there!

People do like to slam MS about holes that have known fixes for them along with newly discovered holes

I agree that MS have tighten up about security because of market share but this doesn't change the fact that some people will look at a situation like this in the linux world and point fingers at the admin for not having things up to date but in the MS world they'll blame MS first not the admin that haven't kept up with patches and procedures.

Re:Windows is not the only vulnerable OS

[RT Alec]

There is a well founded fear many Windows admins have about MS patches. They tend to break things. Patch Win2k, and MS-SQL does not work upon reboot. Or that third party medical charting software suddenly does not work.

Windows is very complex (many would say "too complex"), and certainly suffers from the "integration" of its parts. Therefore, unintentional side effects of patches are envitable. With Unix(ish) systems, the descrete parts can be patched, well, descretely. You can patch Sendmail, or MySQL, or OpenSSL all by itself (although sometimes you must recompile applications that depend on shared libraries, such as OpenSSL).

Re:Windows is not the only vulnerable OS

[JDevers]

I read his comment more as the admins are more responsible for the problem than the programmers behind the OS. He never mentioned the people actually breaking in to the systems, obviously THEY are most at fault.

Extending your analogy to what he actually said, Masterlock isn't responsible when you don't actually LOCK the damned lock. Which, of course, they aren't nor should anyone blame them for losses suffered from the inability of the purchasers of their equipment to properly USE that equipment.

Re:Windows is not the only vulnerable OS

[Izago909]

When your business plan involves running a monopoly on the home desktop market you should be held to a different standard. You can't expect millions of computer novices to be knowledgeable or even aware of exploits or updates like the administer of a HPC pruning a UNIX based OS. The Internet is a community almost like any other. I don't need to list the advantages of having so many people wired, especially within such a short amount of time; but you can't expect the rapid influx of unsuspecting users to know every way to protect themselves.

It's much likehow the tobacco industry operates. Get as many people as you can to start using your product, then rake in the revenues from here on out. Except that the wave of suits over the last decade has shown that corporations can be held liable for their irresponsibility for exploiting the ignorant. (Note: The difference between ignorance and stupidity is that an ignorant person just hasn't been told yet.)

Re:Windows is not the only vulnerable OS

[nathanh]

How does that differ from the worms which get released for Microsoft almost a year after the patch was released?

It's no different.

I hear people railing Microsoft all the time for not 'getting it right the first time' when THAT happens...

I also hear other people calling those first people idiots. No software is perfect. Security is a process. Patching is forever.

Re:Windows is not the only vulnerable OS

[Anonymous Coward]

We all know no patch has ever caused any problems with any server.(heavy sarcasm)

I work with a large organization with hundreds of servers and no patch gets install until the patch is tested to make sure it does not break the business app. That means setting up a lab with as close to production setup as possible, install the patch and try to run some realistic tests to confirm that things work. If everything checks out then you can update that server. Repeat process for each application. Don't forget the months of negotiation to get the time to patch/reboot the server for the upgrade.

I have been waiting 4 months to do patches because the users refuse to let the server to be shutdown for even a few minutes a year. They want mainframe uptime on PC budgets. It is a case of the golden rule, and I don't have the gold.

Not ever unpatched system is the fault of bad administrators.

Re:Windows is not the only vulnerable OS

[Anonymous Coward]

The problem with patching is that it's not reasonable to take some slab of code that's been put on the 'Net by the software manufacturer and throw it on the computer.

Why not?

Well, what happens if that system just happens to be the payroll system, for example? What happens if the patch just manages to break the system so that the fortnightly payroll run doesn't happen? What happens when that money, which you expected to be in your bank account, doesn't appear? What happens when your mortgage provider goes to pull out your fortnightly mortgage repayment, and finds that there's no money in there to grab?

It isn't as simple as "Here's a patch, you're now secure as long as you apply it." We're talking real-world systems, with real-world conflicts and requirements. If you step outside the known and tested, you're liable to break things.

In other words: have a second system which you can throw patches onto and pound away on for a week or two, to make sure that those patches don't break anything important. Then throw the patches onto the live, production system. Doing it any other way could cause serious problems.

Sometimes, it's a case of having a choice: either you're secure, or your business is functioning. This is not a choice that I would want anybody to have to make, but you need to know that that choice is entirely possible, every time a new patch is released from your vendor, whether that vendor be Microsoft, Sun, IBM, HP, SGI, Apple, or Linus. Note that I'm not talking about deliberately (or through slacking off) avoiding application of patches; I'm talking about verifying that the patches still let you function as a business.

Or, in other words: IT exists to serve the business. The business does not operate to serve IT. Most of the time, there is no conflict between the two, but when there is, you need to make damn sure that the right one wins.

Re:Windows is not the only vulnerable OS

[ichimunki]

Good god, man! Microsoft has been crappy software since Day 1 and people have been complaining about the health effects of tobacco use since the U.S. was just a bunch of colonies. Anyone who gets "hooked" on either deserves every last drop of what they get.

Why is it that these discussions always have to descend into Windows vs Linux flame wars? Frankly, I don't give a rat's ass about Windows right now. What I want to see is a good discussion of what's going on at Stanford. What can the rest of us do to make sure we don't fall victim to the same problems, etc etc.

I mean, when I recommend Linux to my friends and family, how can I know that we're going to be safe from this stuff? Telling them that Windows is worse or that Microsoft should be held to a higher standard isn't a good enough answer. If it were, I would have easily convinced everyone I know to switch a long time ago.

Re:Windows is not the only vulnerable OS

[4of12]

has shown that corporations can be held liable for their irresponsibility for exploiting the ignorant.

I wish.

Our whole damn culture is a corporate strategy to create fools who will part with their money.

In other words

[Rosco P. Coltrane]

a variety of local exploits, including the do_brk() and mremap() exploits on Linux

In other words, Stanford doesn't keep its Linux boxes up to date. These exploits have been fixed. Linux too requires maintenance and patching, not just Windows.

Re:In other words

[winkydink]

Maintaining a large, heterogenous environment (where administrative control may be decided by political or monetary reasons) is not easy to do. This may explain why you see so many really bright sysadmins at .edu's, but even they have difficulty breaking the political & financial layers.

Re:In other words

[randyest]

Actually, it's users who are not following rules (assuming they have rules against using insecure telnet, which I'm sure they do):

The attacks start with the compromise of an unprivileged local user account. Usually this is because the attacker's captured the password from somewhere else: it's been sniffed off the network (through the use of insecure protocols like telnet), it's been collected when the user signs on to or from another compromised machine, it's been harvested from the password file on a compromised system.

So, we have user passwords as the source, which users freely give away by (1) using telnet instead of SSH, (2) just being very uninformed or gullible users, enough to plug in his/her unix password to a web form, and (3) once-removed version of (1) or (2) since these are just obtained from other compromised machines.

(1) and (2) are arguably the same problem, so that boils down to: users breaking rules -- surprise! But, that's easy to say, but hard to fix without more power . What to do? Seriously? Fine users for breaking rules?

Re:In other words

[KrispyKringle]

I don't think they mean clusters as in MOSIX, etc. The term seems to be used frequently in academia to refer to a group of machines, with load balancing between them, used for services like shell access, web and mail serving, etc. Additionally, individual servers are being attacked as well. Many schools have a very, shall we say, fragmented IT infrastructure; I'm at a medium-sized private university (about 10,000 undergrads, perhaps) with four different undergraduate schools and perhaps twice as many graduate schools. Each has its own IT department. The larger ones are well-run, but some of the smaller ones aren't even on the newsgroup of which all the IT departments are supposed to belong because they can't figure out how to use the news server (or so it's been said, at any rate). Point is, academia has some great admins, and some psych professors running servers out of their classrooms.

Academic computing is the epitome of *available* computing, in the sense that availability is the highest priority. Financial institutions may prioritise (or at least, should prioritise) security and a good administration over availability, but by its nature, academic computing involves disparate infrastructures, various levels of admins with various goals, and so forth. All students, faculty, and staff need access; frequently, granting loose, unsecure access is simply more efficient for the time being than making things secure. Such is life.

Re:In other words

[modecx]

At my university nearly everyone used telnet to check their mail, and FTP on the big computer (ran AIX, probably still does). It's really quite stupid, especially when Free software exists for pretty much all platforms under the sun to easiy mitigate that risk.

I once approaced one of the computer dorks at the lab about making PuTTY available to everyone on the lab computers, explaining packet sniffing (what's worse is that most of the individual labs were hubbed), and he turned me into the administration for hacking, and they froze my account. I wrote a letter to the network admins and CS staff, and got my account back explaining this--that I hadn't attempted sniffing passwords, and that I was just illustrating a point. But that's what you get for trying to do the right thing. No good deed goes unpunished, as they say.

So don't doubt that at many universities around the world there's passwords--and all sorts of other good stuff floating around in plaintext--ripe for sniffing.

Admins just need to turn off telnet and FTP where applicable, and force their users to use other methods. That's what it comes down to.

Re:In other words

[FireFury03]

In most cases, the attacker gets access to a machine by cracking or sniffing passwords. Local user accounts are escalated to root privileges by triggering a variety of local exploits

The machines should of course be patched up to date, but I think the real failing here is the sysadmins not enforcing secure protocols - it doesn't take much to disable the telnet and ftp servers and make people use ssh and scp, etc instead. As soon as users are allowed to send authentication details in the clear instead of encrypting them you open up all the local exploits to network attack, and security holes that can be accessed remotely by arbitrary users are far more of a security risk than holes that are only exploitable by users who have legitimate access to the system.

Sloppy work all around

[fastpage]

What gets me is that you can tell the white hats and black hats are both lazy.

If the sysadmins had actually patched their servers with the appropriate security patches the "hackers" would have never gotten in, in the first place. If you read the counter measure section this isn't anything new that they shouldn't be doing every day and enforcing.

If you look at the section entitled Evidence of compromise you can see that the people breaking into the systems are leaving a pretty big trail to follow. In my job, when customers start complaining that their servers are working quite right, when you take a look at whats going on you can see a root kits been installed. The whole idea of a root kit is to cover your tracks. If these guys did a better job you'd never know you were hacked. Its quite sad really. Laziness is the biggest security problem if you ask me.

they wanna know WHAT?

[ChipMonk]

If you believe your Unix computer has been affected by these intrusions, please contact the Information Security Services office (650-723-2911 or security@stanford.edu). Please include the name or IP address of the affected machine, as well as any compromised userIDs.

Never mind the compromised machines. Let's try social engineering instead. I know! We'll make a security alert, get it on Slashdot, and the poor trusting souls will beat a path to our POP3 account!

Seriously, you might as well just hand them your hard drive and credit card number.

HPC question

[abrotman]

could someone more familiar with HPC systems please explain to me why any cluster is attached to the internet? I'm assuming these are externally routable addresses. I just dont understand why you would do this.

Re:Hmm, doesn't seem very unusual.

[eclectro]

Someone is sniffing passwords off the network (telnet or http sessions probably) or cracking badly chosen ones

They could be using hardware keyloggers, in which case NO machine is invulnerable.

The problem with passwords

[xixax]

...Is that you cannot make sure your users are careful.

You pretty much have to assume that black-hats are going to be able to runs escalation exploits and work accordingly. That or severely limit how users are allowed to interact with the machine (if they only need to access email or upload files, WTF should they be able to run anything else?).

But yeah, good passwords limit the opportunities.

Xix.

Wait, isn't the same true for Microsoft

[melted]

Every single god damn worm would not work if users would patch their god damn systems. That's not news. Tell me something new to support that "Linux is secure" myth.

Bonjour, Monsieur Straw

[freeweed]

Straw, meet man.

It is important that when we wave our flags and cheer when Microsoft is laid low by the latest security flaw that we not close our eyes to the very real vulnerabilities in the Unix/Linux system.

No one is. Work is always being done to find and fix vulnerabilities in *nix variants.

No OS can be fully secured

No one with a brain ever claimed that was the case.

Assuming that Unix/Linux is invulnerable to security holes is deadly.

See last comment.

Though the OS may have more security features and "more eyes" on the code than closed source operating systems

Which is true...

we must not rest on our laurels watching Windows implode while our own house is burning.

Last time, NO ONE IS.

Geez. I know your nick is "Obvious Guy", and that's pretty much all you're saying. Well, except for the entire argument about "watching Windows implode while we rest on our laurels", which no one is doing, talking about doing, nor thinking about doing.

Straw, meet man. I'm still befuddled as to the upwards moderation you consistently get, however.

Re:Nothing to worry about

[Daniel Dvorkin]

No, no, just ignore this. When Windows is being compromised that's cause for gleeful giggles and jokes on slashdot. When Linux is being compromised that's for social misfits to blush about and shamefacedly ignore.

When Windows is being compromised, that's cause for Microsoft to ignore, deny, and lie about the problem, and if that fails, spend a few billion dollars on PR. When Linux is being compromised, that's for knowledgeable programmers to study, work on, and fix the vulnerability.

That's So Lame!!! Microsoft looks desperate!!!

[Anonymous Coward]

Every day we see the constant stream of Microsoft security failures.

And those aren't minor, obscure failures. They affect millions of Windows users. They fill up our our reject logs. And they don't require special conditions -- Windows exploits can hit you simply because you browsed a webpage, played an MP3, received an e-mail, or just by having your PC connected to the Internet.

In fact, not only was there a story about three new Windows vulnerabilities, just two stories before this one, but Windows vulnerabilities set an all time record in February for the number of new exploits in a month. According to The Washington Times, "Internet attacks in February caused an estimated $68 billion to $83 billion in damages worldwide."

And to counter the impression that Windows has bad security, we are presented with... wait for it... a single Linux site, whose faulty administration procedures have left their machines vulnerable to local exploits, requiring the cracker(s) to first sniff a password.

And then the parent poster suggests that the two are somehow equivalent???

How lame!!!

No

[spoco2]

No, it doesn't... many of the same types of reports about windows attacks are ALSO due to UNPATCHED machines.

It's the one-eyed, severely slanted nature of the Slashdot readership that:
* Microsoft is evil, stupid, moronic, evil, nasty, unsafe, did I mention evil?
* Linux is the shining non-denominational grail.

For god sake, there are security vulnerabilities in both people... and they aren't taken advantage of within the *nix world, because... hey, guess what? The majority of users are computer savvy, and know about passwords and firewalls and not leaving ports open etc.

Windows users on the whole have issues programming their VCRs.

As you start to get what you want, which is widespread Linux adoption, you'll start getting more of the VCR no-hopers using Linux, not patching it, not having secure passwords... and GUESS WHAT? Linux will start having major security issues in the same way as Windows does now... not as severe most likely due to better design, but they'll be widespread... there'll be a doozy, and it'll cause all sorts of problems and then people will be "Hey, I thought when we all moved to Linux the world would be a safer place for me and my little children, but now that a vulnerability has allowed my Linux box to be used as a Spam mail distribution point, I feel dirty and scared. I might install XP again."

Stop being so damn one sided.

Re:Libsafe protects against buffer overflow exploi

[Anonymous Coward]

Well theres 2 sides to that coin. Some say its really bad to rely on libsafe because the underlying source never gets fixed, therefore libsafe becomes and indispensible middlelayer you rely on more and more to protect legacy code which is inelegant. So in the long run much better to sort out the original source and do the job properly from the top. Just another 0.2c from a different school of thought.

More education, less laws

[green_crocadilian]

The p;roblem, among others, is that we don't have enough real punishment going on for hacking activities.

The problem is that the concentration of clue among sysadmins is just too low. If you are still running a do_brk vulnerable kernel 5 months after the vulnerability was discovered and patched and widely publicised (remember the Debian and Gentoo server compromises that were all over the news?), you deserve whatever you get. I mean, sure, if you were hacked on December 5, my sympathy goes out to you, but if you are running unpatched 2.4.22 right now, there is no excuse.

As for jail time for hackers: to justify that, you would need to show that a moderately skilled sysadmin, one that reads a security-related news source at least on a quarterly basis, physically cannot protect his/her system from a moderately skilled attacker. For example, suppose someone proved P=NP and made a polynomial-time ssh decryptor. Only then we would need laws against password sniffing, because once you let a government have a taste of regulating the Internet, it will not stop until it has, so to speak, filled its belly with electronic freedoms.

Re:Nothing to worry about

[jaavaaguru]

Do a fresh install of the original edition of Windows XP, and do a fresh install of Red Hat from the same time period. Which has more known security holes? They're probably quite similar.

Apply all known patches to each installation. Now which has more known holes? I think you'll find a list of things still currently broken in Windows, but Red Hat (and therefore other Linuxes too) will have their problems patched already.

The parent post looks initially like just another one of those Linux has no holes and Windows is full of them posts, but if you're looking at the situation five days after a security hole is announced, it's perectly true.

Re:Nothing to worry about

[terrox]

well get a refund on your Linux.. oh dear was it free?
when people pay $200 for something they expect it to damn well work - if it causes your $2000 appliance to become totally useless then anger can be justified right?
if more people used linux then more people would be making it easier for non-techs to use - so just BE PART OF THE SOLUTION and stop defending corporations, they wont be defending YOU when it comes to the crunch.

Re:Nothing to worry about

[Anonymous Coward]

"Do a fresh install of the original edition of Windows XP, and do a fresh install of Red Hat from the same time period. Which has more known security holes? They're probably quite similar."

Not at all, my friend!!!

Red Hat from that time, has exactly 0 (zero) remote explotaible bugs when properly installed. Specially since it is a Red Hat from *that* time period: all bugs are clearly stated from ages, and I *can and will* install with no service opened by default till I can upgrade to latest known stable versions, and that *only* for services I really need (all the others won't even be installed, since I tend not to worry too much about software it is not even installed).

Now, try to do the same with XP: probably you won't even end up the installation procedure and you will already be infected with some of the RPC hole bugs. And know what? You won't be able to do *anything* to avoid it, even knowing about it.

On the other hand, if I go for Microsoft I can imagine a bunch of reasons why I would want to install (or get some other way) an XP (or a Windows 2000, or 98 or NT), the major one being that's the product I paid for, and I don't want to pay for an unneeded (funtionality-wise) upgrade. But this is *NOT* the case for Linux distributions. Why the heck would I want to go with "a fresh install of Red Hat from the same time period" when I can have "a fresh install of Red Hat from *this* time period" with no cost implication?

It seems at first glance that comparing XP with a Linux from that days is a fair comparation, but it is not, because Linux is free and open source, so you really don't need to go with the older product because that was the license you bought!

And know what? Your compartion ends up so lamely because Microsoft products are made that way *by design*. Think about it next time you are going to buy another Microsoft license.

Re:Nothing to worry about

[robertjw]

First, no one ever said Linux was invulnerable, just inherently more secure.

Second, I actually read the story. There are three methods of access to the compromised machines listed in the article:
"sniffing passwords, cracking passwords from other compromised systems, or by triggering vulnerabilities in remotely accessible services."

Windows is vulnerable to both sniffing passwords and cracking passwords from other systems, so the only Linux specific problems are in the remotely accessible services. The article lists two specific Linux exploits that were used to access these systems, do_brk() and mremap(). I then read the security alerts for these two exploits. do_brk() is specifically vulnerable to attacks by rsync and mremap() appears to only be useful for local permissions escalation (meaning a password has already been cracked).

Having never worked in a University, I don't know how hard security is to maintain, but in my environment I don't run rsync on any machine that is accessible to unauthorized personnel. Looks like this could easily be attributed to poor system administration. A good firewall would have taken care of all of these problems without having to patch the kernel.