Ongoing Linux/Solaris Compromise Epidemic

An anonymous reader writes to point out that Stanford's Information Technology Systems and Services "has written a summary of a series of compromises that have been happening at universities, research institutions, and high performance computing centers, for the last month or more. The attackers are using known vulnerabilities in Linux and Solaris, along with compromised user accounts, to gain access and control of systems, from standalone servers to HPC clusters ... (the attacks are still ongoing)."

Comments

Nothing to worry about

[Rapid Home Offer]

From the article:

The attacker appears to be deliberately targetting machines in academic and high performance computing environments, rather than attacking systems indiscriminately.

I wouldn't worry too much. It's sounds like some guy is trying to boost his SETI@home ranking.

Re:Nothing to worry about

[dustmite]

Uhm, I think it was a joke ..

Re:Nothing to worry about

[Daniel Dvorkin]

No, no, just ignore this. When Windows is being compromised that's cause for gleeful giggles and jokes on slashdot. When Linux is being compromised that's for social misfits to blush about and shamefacedly ignore.

When Windows is being compromised, that's cause for Microsoft to ignore, deny, and lie about the problem, and if that fails, spend a few billion dollars on PR. When Linux is being compromised, that's for knowledgeable programmers to study, work on, and fix the vulnerability.

Check out a good substitute for all your Linux

[Anonymous Coward]

A good substitute for Linux and Sun boxes. My school migrated two years ago, weren't happier ever since.

Here - those guys make a kernel, kickass GUI environment (faster than GNOME and easier to use than KDE) plus some office word editors and educational stuff like encyclopedias and maps.

I'm just glad...

[Anonymous Coward]

I'm running Windows XP!

aQazaQa

Windows is not the only vulnerable OS

[ObviousGuy]

It is important that when we wave our flags and cheer when Microsoft is laid low by the latest security flaw that we not close our eyes to the very real vulnerabilities in the Unix/Linux system. No OS can be fully secured, and it is absolutely mandatory that we remain vigilant to the possibility of a heretofore unknown security hole in our systems, regardless of the system OS.

Assuming that Unix/Linux is invulnerable to security holes is deadly. Though the OS may have more security features and "more eyes" on the code than closed source operating systems, we must not rest on our laurels watching Windows implode while our own house is burning.

Re:Windows is not the only vulnerable OS

[morelife]

You're joking.

All the vulns mentioned have patches/fixes/replacements for the faulty code.

The System Administrators are at fault FOR NOT MAINTAINING THEIR SYSTEMS PROPERLY.

Re:Windows is not the only vulnerable OS

[FrYGuY101]

How does that differ from the worms which get released for Microsoft almost a year after the patch was released? I hear people railing Microsoft all the time for not 'getting it right the first time' when THAT happens...

Re:Windows is not the only vulnerable OS

[morelife]

How does that differ from the worms which get released for Microsoft almost a year after the patch was released? I hear people railing Microsoft all the time for not 'getting it right the first time' when THAT happens...

Wrong. People rail because Microsoft rarely gets it right the first time, and are damned slow and arrogant about fixing security holes. Oh, sorry. They did speed up their response time on security issues after realizing that the public was noticing and they were losing a little market share in IIS.

Re:Windows is not the only vulnerable OS

[jarich]

Perhaps an alternative view...?

The p;roblem, among others, is that we don't have enough real punishment going on for hacking activities.

The internet has become the equivalent to living in a slum. Sure, the property is cheap, but if you don't have bars on your windows, you can count on a break in. And lots of people will tell you it's your own fault for not putting bars on your windows and living in a walled compound with broken glass on the tops of the walls.

I agree that the systems should be patched, but the real problem is that there are communities of thugs who feel at liberty... NO, who ARE at liberty (due to the lack of a cohesive international enforcement) to do what ever they want to you machine.

I vote for real international difficult (I know that's not going to be trivial) and hard jail time when people are caught. And, just like Kevin Mitnick, they should not be allowed to work with computers when they get out.

Re:Windows is not the only vulnerable OS

[EvilTwinSkippy]

I am a religious patcher. Hell, I've almost gotten a fired a few times when patches went wrong. Bosses just don't understand that machines don't just "work". They require constant intervention. The computers, that is, not the bosses.

Now that said, you have an interesting slant on ethics. By that mindset, a burglar is perfectly entitled to break into your apartment because your door could be kicked in. A theif can swipe your radio because, hey, it was only glass between him and what he wanted.

Yes, there is a certain amount to be said for not painting a target on yourself. But regardless of how much you "had it coming" it's still a crime to break into your dwelling, steal your property, or damage your person or posessions. System intrusion is a crime, and a matter for law enforcement.

Re:Windows is not the only vulnerable OS

[bebing]

Wow this got modded up to +5 while there were only 55 replies to the article, that's fast. Either you're popular or there are a lot of pissed off MS fans out there.

It is important that when we wave our flags and cheer when Microsoft is laid low by the latest security flaw that we not close our eyes to the very real vulnerabilities in the Unix/Linux system.

Is there really flag waving and cheering going on? Perhaps joking and laugher. Also Linux vs. Microsoft(leaving Unix out for now) is not comparable to say Rocky vs. Apollo Creed, but David vs. Goliath. Microsoft does not need you to defend them, they have billions of dollars and a monopoly. We do have to stick up for Linux because we are Linux, and there is nothing close to a monopoly or billion dollar bank accounts. Now that I think about it maybe cheering is ok when the bully takes a blow to the chin, it happens in the movie theatres.

Re:Windows is not the only vulnerable OS

[drinkypoo]

I am a religious patcher.

How's the reformation coming?

Re:Windows is not the only vulnerable OS

[SemperFiDownUnda]

Most companies don't get it right the first time. If they did there would never be patches would there!

People do like to slam MS about holes that have known fixes for them along with newly discovered holes

I agree that MS have tighten up about security because of market share but this doesn't change the fact that some people will look at a situation like this in the linux world and point fingers at the admin for not having things up to date but in the MS world they'll blame MS first not the admin that haven't kept up with patches and procedures.

Re:Windows is not the only vulnerable OS

[RT Alec]

There is a well founded fear many Windows admins have about MS patches. They tend to break things. Patch Win2k, and MS-SQL does not work upon reboot. Or that third party medical charting software suddenly does not work.

Windows is very complex (many would say "too complex"), and certainly suffers from the "integration" of its parts. Therefore, unintentional side effects of patches are envitable. With Unix(ish) systems, the descrete parts can be patched, well, descretely. You can patch Sendmail, or MySQL, or OpenSSL all by itself (although sometimes you must recompile applications that depend on shared libraries, such as OpenSSL).

Re:Windows is not the only vulnerable OS

[Anonymous Coward]

We all know no patch has ever caused any problems with any server.(heavy sarcasm)

I work with a large organization with hundreds of servers and no patch gets install until the patch is tested to make sure it does not break the business app. That means setting up a lab with as close to production setup as possible, install the patch and try to run some realistic tests to confirm that things work. If everything checks out then you can update that server. Repeat process for each application. Don't forget the months of negotiation to get the time to patch/reboot the server for the upgrade.

I have been waiting 4 months to do patches because the users refuse to let the server to be shutdown for even a few minutes a year. They want mainframe uptime on PC budgets. It is a case of the golden rule, and I don't have the gold.

Not ever unpatched system is the fault of bad administrators.

Re:Windows is not the only vulnerable OS

[Anonymous Coward]

The problem with patching is that it's not reasonable to take some slab of code that's been put on the 'Net by the software manufacturer and throw it on the computer.

Why not?

Well, what happens if that system just happens to be the payroll system, for example? What happens if the patch just manages to break the system so that the fortnightly payroll run doesn't happen? What happens when that money, which you expected to be in your bank account, doesn't appear? What happens when your mortgage provider goes to pull out your fortnightly mortgage repayment, and finds that there's no money in there to grab?

It isn't as simple as "Here's a patch, you're now secure as long as you apply it." We're talking real-world systems, with real-world conflicts and requirements. If you step outside the known and tested, you're liable to break things.

In other words: have a second system which you can throw patches onto and pound away on for a week or two, to make sure that those patches don't break anything important. Then throw the patches onto the live, production system. Doing it any other way could cause serious problems.

Sometimes, it's a case of having a choice: either you're secure, or your business is functioning. This is not a choice that I would want anybody to have to make, but you need to know that that choice is entirely possible, every time a new patch is released from your vendor, whether that vendor be Microsoft, Sun, IBM, HP, SGI, Apple, or Linus. Note that I'm not talking about deliberately (or through slacking off) avoiding application of patches; I'm talking about verifying that the patches still let you function as a business.

Or, in other words: IT exists to serve the business. The business does not operate to serve IT. Most of the time, there is no conflict between the two, but when there is, you need to make damn sure that the right one wins.

Re:Windows is not the only vulnerable OS

[jlittle]

Its hard to sit idly by as some statements like this are made. As an individual who has helped out with these systems and further helps maintain processes to keep systems up to date (almost all linux boxen involved use apt-get updates to state up to date pretty religiously), it has generally been the case the successfully attacked systems have come by unique approaches to each machine. Most have been indirect attacks via local root exploits using compromised user names from other institutions. Passwords are generally cracked and not sniffed, with almost all exploits using recently announced local root exploits on systems otherwise protected from direct access to the internet. In a nutshell, these are focused, diligent attempts using a wide arsenal of previously known and unknown attacks (yes, some locally compromised systems were truly patched!) to gain as much resources as possible. It pains me that the people here are apt to paint with such a wide paint brush, when in fact these types of attacks are generally successful whenever there is a keen focus to gain resources and its not your run-of-the-mill script kiddie.

I'll also answer that our redhat-based distros used to use libsafe, but newer 9-based versions have incompabilities with that library (matlab and other standard tools don't work with it, and it hasn't been updated for the new nptl stuff). Also, libsafe doesn't help against the kernel exploits (it didn't help). grsecurity is hard to deploy uniformly, especially when you rely on stock vendor kernels because of other vendor requirements.

In other words

[Rosco P. Coltrane]

a variety of local exploits, including the do_brk() and mremap() exploits on Linux

In other words, Stanford doesn't keep its Linux boxes up to date. These exploits have been fixed. Linux too requires maintenance and patching, not just Windows.

Re:In other words

[winkydink]

Maintaining a large, heterogenous environment (where administrative control may be decided by political or monetary reasons) is not easy to do. This may explain why you see so many really bright sysadmins at .edu's, but even they have difficulty breaking the political & financial layers.

Re:In other words

[randyest]

Actually, it's users who are not following rules (assuming they have rules against using insecure telnet, which I'm sure they do):

The attacks start with the compromise of an unprivileged local user account. Usually this is because the attacker's captured the password from somewhere else: it's been sniffed off the network (through the use of insecure protocols like telnet), it's been collected when the user signs on to or from another compromised machine, it's been harvested from the password file on a compromised system.

So, we have user passwords as the source, which users freely give away by (1) using telnet instead of SSH, (2) just being very uninformed or gullible users, enough to plug in his/her unix password to a web form, and (3) once-removed version of (1) or (2) since these are just obtained from other compromised machines.

(1) and (2) are arguably the same problem, so that boils down to: users breaking rules -- surprise! But, that's easy to say, but hard to fix without more power . What to do? Seriously? Fine users for breaking rules?

Re:In other words

[ameoba]

A cluster is almost always _not_ a heterogenous environment. On top of that, the individual cluster machines should not have, nor do they need, access to the network as a whole. Compared to things like computer labs, HPC clusters should be the easiest thing to secure, since you -can- firewall the hell out of them.

There's no excuse, when putting up a several hundred node cluster to not get an extra machine through which it needs to be accessed that is not part of the cluster. That machine can trivially be kept secure & the cluster can then be updated as is convenient (IE - not replacing the kernel in the middle of a 3-week long computation; even at that, tho, anything that's going to take 3wk should be able to checkpoint itself without loosing much).

Re:In other words

[KrispyKringle]

I don't think they mean clusters as in MOSIX, etc. The term seems to be used frequently in academia to refer to a group of machines, with load balancing between them, used for services like shell access, web and mail serving, etc. Additionally, individual servers are being attacked as well. Many schools have a very, shall we say, fragmented IT infrastructure; I'm at a medium-sized private university (about 10,000 undergrads, perhaps) with four different undergraduate schools and perhaps twice as many graduate schools. Each has its own IT department. The larger ones are well-run, but some of the smaller ones aren't even on the newsgroup of which all the IT departments are supposed to belong because they can't figure out how to use the news server (or so it's been said, at any rate). Point is, academia has some great admins, and some psych professors running servers out of their classrooms.

Academic computing is the epitome of *available* computing, in the sense that availability is the highest priority. Financial institutions may prioritise (or at least, should prioritise) security and a good administration over availability, but by its nature, academic computing involves disparate infrastructures, various levels of admins with various goals, and so forth. All students, faculty, and staff need access; frequently, granting loose, unsecure access is simply more efficient for the time being than making things secure. Such is life.

Re:In other words

[modecx]

At my university nearly everyone used telnet to check their mail, and FTP on the big computer (ran AIX, probably still does). It's really quite stupid, especially when Free software exists for pretty much all platforms under the sun to easiy mitigate that risk.

I once approaced one of the computer dorks at the lab about making PuTTY available to everyone on the lab computers, explaining packet sniffing (what's worse is that most of the individual labs were hubbed), and he turned me into the administration for hacking, and they froze my account. I wrote a letter to the network admins and CS staff, and got my account back explaining this--that I hadn't attempted sniffing passwords, and that I was just illustrating a point. But that's what you get for trying to do the right thing. No good deed goes unpunished, as they say.

So don't doubt that at many universities around the world there's passwords--and all sorts of other good stuff floating around in plaintext--ripe for sniffing.

Admins just need to turn off telnet and FTP where applicable, and force their users to use other methods. That's what it comes down to.

If you read to the VERY end of the article...

[oldosadmin]

It says that good passwords are a good defense.
We know this.
No more default last 4 digits of SSN as a password.
Make them use something more secure! And disable telnet, for goodness sakes.
Inconvieience (sp?) your students in order to secure your system. It's all fun and games until someone uses a rootkit to play with GPAs.

IMO all of these attacks are related

[bersl2]

going back to the back-door insertion attempt on the Kernel, the rooting of gnu.org's ftp server, the compromise of Debian's servers... it's the same people doing this.

Just a feeling.

Washingtonpost.com has the complete story

[tsu doh nimh]

on just how widespread this attack really is. The story IS HERE

Hmm, doesn't seem very unusual.

[mtnharo]

Someone is sniffing passwords off the network (telnet or http sessions probably) or cracking badly chosen ones, and then using privilage escalation vulnerabilities that have been known and patched for quite a while. However, this kind of thing can work at an academic environment or other large network, since it is often not possible to upgrade the kernel on every single system without proper testing. Still, come things can be done to prevent this kind of attack.

Don't send passwords in plain text on the network, and enforce proper password policies (8 char minimum, numbers, letters and symbols etc).

Re:Hmm, doesn't seem very unusual.

[achurch]

enforce proper password policies (8 char minimum, numbers, letters and symbols etc).

I've always been against this, or at least the more anal implementations of it, in that forcing people to choose hard-to-remember passwords typically leads to writing the passwords down--often in obvious places--which makes the problem worse instead of better. Good encryption (e.g. ssh instead of telnet) and good security measures (e.g. shadow passwords) are much more effective than draconian policies that don't achieve their ends anyway.

(And as for numbers and symbols making passwords less crackable--admit it, how many of you use 1337speak to make up the number/symbol quota?)

Re:Hmm, doesn't seem very unusual.

[Spoke]

(And as for numbers and symbols making passwords less crackable--admit it, how many of you use 1337speak to make up the number/symbol quota?)
Doh, how did you know my password was 1337speak? I better change now that you've posted it on Slashdot!

Note to self

[UnknowingFool]

Change Linux root password from 1234 to something harder to guess

Re:Note to self

[RussDavisDotCom]

No Worries. I've already changed it for you.

Sloppy work all around

[fastpage]

What gets me is that you can tell the white hats and black hats are both lazy.

If the sysadmins had actually patched their servers with the appropriate security patches the "hackers" would have never gotten in, in the first place. If you read the counter measure section this isn't anything new that they shouldn't be doing every day and enforcing.

If you look at the section entitled Evidence of compromise you can see that the people breaking into the systems are leaving a pretty big trail to follow. In my job, when customers start complaining that their servers are working quite right, when you take a look at whats going on you can see a root kits been installed. The whole idea of a root kit is to cover your tracks. If these guys did a better job you'd never know you were hacked. Its quite sad really. Laziness is the biggest security problem if you ask me.

Been hitting Caltech too

[Anonymous Coward]

Servers were down much of last week. The ITS website has a few brief details.

Re:Attacks against universities?

[Anonymous Coward]

I'm running a live cd distro based on Damn Small Linux. Is this the coming thing to prevent attacks and viruses from getting anywhere?
Nothing is written to a hard drive with this OS.
If so, how would this apply to the story on these attacks? How would anyone "gain control" of my computer under these circumstances.
BTW, Damn Small has a limit of 50 Mb, mine runs a little over 60 MB, and I put Mozilla Firefox and Wvdial in the remaster, as well as some office applications from the Debian list of over 8000 items.

Yeah, so?

[ameoba]

The entire (up to date) Windows lab here gets compromised & backdoored to hell and everyone just says "Have it working by tommorrow". A Linux cluster gets compromised and they issue a press-conference.

HPC Clusters?

[JessLeah]

Isn't that an oxymoron? Cray Canada's CTO says so. Then again, Borland's CTO said "OS X is my favorite Linux distribution.", so maybe CTOs aren't so smart about Technology after all ;)

My opinion

[weekendwarrior1980]

I dont think we will ever have a fully secure box, these vulnerabilies will continue to pop up occassionally and there's nothing we (the developers) can do about that. It is just a testimony of the fact that we are imperfect beings and sooner or later we will have our errors exposed. It is not a bad thing, in the evolutionary way of dealing things, this (finding and sorting out bugs) could probably be a good thing. Having said that, I think developers do have control over how they respond to these problems, like coming up a problem that doesn't just band-aid the wound hoping to find a cure for in the future. Also developers have control over how fast they respond. On both criterias, open source peer reviewing is winner over closed sourced development. One tends to promote security through openness and and in the other security through obscurity like think MSFT( Read comments from a MSFT bigwig who said the only reason MSFT servers are compromised because the vulnerabilities are announced).

they wanna know WHAT?

[ChipMonk]

If you believe your Unix computer has been affected by these intrusions, please contact the Information Security Services office (650-723-2911 or security@stanford.edu). Please include the name or IP address of the affected machine, as well as any compromised userIDs.

Never mind the compromised machines. Let's try social engineering instead. I know! We'll make a security alert, get it on Slashdot, and the poor trusting souls will beat a path to our POP3 account!

Seriously, you might as well just hand them your hard drive and credit card number.

Libsafe protects against buffer overflow exploits

[tjmather]

Does anyone use Libsafe This library protects against buffer overflow vulnerabilities, and is very easy to install (basically you just install the RPM and you're done)

If more sysadmins installed this, perhaps we wouldn't have problems with so many Linux compromises? Of course it's no substitute for patching, but seems like a good additional security measure.

This is from the gnu.org software directory

The exploitation of buffer overflow and format string vulnerabilities in process stacks are a significant portion of security attacks. 'libsafe' is based on a middleware software layer that intercepts all function calls made to library functions known to be vulnerable. A substitute version of the corresponding function implements the original function in a way that ensures that any buffer overflows are contained within the current stack frame, which prevents attackers from overwriting the return address and hijacking the control flow of a running program.

The true benefit of using libsafe is protection against future attacks on programs not yet known to be vulnerable. The performance overhead of libsafe is negligible, it does not require changes to the OS, it works with existing binary programs, and it does not need access to the source code of defective programs, or recompilation or off-line processing of binaries.

Imagine...

[Odin's Raven]

From the Stanford article:

Stanford, along with a large number of research institutions and high performance computing centers...

And further down...

...the compromised user account is typically used to run a password decoding application called John the Ripper...

To paraphrase a cliche without any attempt at humor:

Imagine a Beowulf cluster running John the Ripper.

/me runs and hides in cellar...

Now, wait a moment ...

[JMZorko]

Just an observation, but this story has the "Security" icon, while the story about Windows critical flaws has the "Bugs" icon. Both stories deal with bugs or "vulnerabilities" that compromise security on the affected machines.

Now, my opinion of MS is not that great, but this just seems wrong.

Regards,

John

Re:Now, wait a moment ...

[c0dedude]

Not really. The vulnerablities in Linux/Solaris have been fixed, thus it is a security issue. The vulnerablities in Windows are still current, thus it is a bug.

Re:Now, wait a moment ...

[CAIMLAS]

This is why you should at least try to pay attention; reading the article would help, too.

This article is about incompetent admins and actual security breaches using exploits that have had fixes for ages. Thus, security. The windows item was on patches for actual bugs and didn't mention any specific exploit instances: thus, bugs.

It all makes sense now, doesn't it?

academic machines?

[dj245]

article: The attacker appears to be deliberately targetting machines in academic and high performance computing environments, rather than attacking systems indiscriminately.

I can see why they would want to target academic boxen if they wanted high-powered computers to do some serious slaved number crunching. If they are just going to launch a DDoS attack or send a bunch of spam though, academic computers are not the best. Most academic sysadmins have fairly limited budgets, and spend a fair amount on bandwidth. As such, they rule their bandwidth with an iron fist in many cases. The Admins at my particular college have bandwidth flags on certain ports and a global flag of somewhere around 1gb/day over 3 days. Break that, and the admin gets very interested in what you are doing with your boxen.

I'm sure other colleges have similar schemes, and I've heard of many colleges which are even more strict with their bandwith (200mb/day limit, etc). These academic boxes may make good targets because of their relatively user intervention and user experience, but they don't have that great of a pipe on them, relatively speaking. If it was me, I would have gone after servers that also run wireless access points. Hard to tell where the bandwidth goes in some cases with those.

Re:Libsafe protects against buffer overflow exploi

[EvilTwinSkippy]

On gentoo I compile everything with -fstack-protector. A nifty new feature in GCC that compiles it into all me binaries.

I still use libsafe. It is the greatest thing since sliced bread. Ok, that and distcc. Distcc and rsync... and ssh... DOH!

this just in...

[medelliadegray]

PC's get compromised if security patches are not applied!

and in other news...
cheerio's get soggy in milk

Sad Mind

[Neo-Rio-101]

I was looking at one of the Solaris vulnerabilities, and I saw "sadmind".

I thought it was some kind of nasty name for a hacking daemon - until I found out that sadmind was the "Solaris ADMIN Daemon"

Re:Attempts easy to guess passwords

[Anonymous Coward]

From "/var/log/messages" on a 64-processor cluster at our university (unrelated to the parent post):

Apr 12 09:51:24 xxx sshd[32583]: Illegal user alias from 210.166.208.97
Apr 12 09:51:24 xxx sshd[32583]: Failed none for illegal user alias from 210.166.208.97 port 34243 ssh2
Apr 12 09:51:26 xxx sshd[32583]: error: Could not get shadow information for NOUSER
Apr 12 09:51:26 xxx sshd[32583]: Failed password for illegal user alias from 210.166.208.97 port 34243 ssh2
Apr 12 09:51:35 xxx sshd[32587]: Illegal user info from 210.166.208.97
Apr 12 09:51:35 xxx sshd[32587]: Failed none for illegal user info from 210.166.208.97 port 34695 ssh2
Apr 12 09:51:35 xxx sshd[32587]: error: Could not get shadow information for NOUSER
Apr 12 09:51:35 xxx sshd[32587]: Failed password for illegal user info from 210.166.208.97 port 34695 ssh2
Apr 12 09:51:41 xxx sshd[32598]: Illegal user backup from 210.166.208.97
Apr 12 09:51:41 xxx sshd[32598]: Failed none for illegal user backup from 210.166.208.97 port 35292 ssh2
Apr 12 09:51:41 xxx sshd[32598]: error: Could not get shadow information for NOUSER
Apr 12 09:51:41 xxx sshd[32598]: Failed password for illegal user backup from 210.166.208.97 port 35292 ssh2
...

The attempted logins appear to be in the exact same order, so it's safe to say the attack was done with a script. The attacking IP address also starts with "210" and resolves back to "ns.himanainu.jp" (not necessary the attacker's machine, but rather a compromised host).

Strategic issues

[Animats]

We're seeing more attacks that seem to be Phase I of something big. Somebody is going to considerable trouble to prepare for something. But what?

I see a day coming when, in one day, half the computers in the US have their disks erased.

The Washington Post has more coverage

[Doofus]

Washington Post has more coverage in this article, Hackers Strike Advanced Computing Networks.

Win 95 to the rescue!

[CaptainPinko]

Heh, I'm running Windows 95. I figure by now the hackers are just bored of hacking me.

Security through boredom, my new secret weapon take th^454&*%2^$^^^B

Re:Does anyone on the inside...

[drmerope]

Yeah, I've been involved in some of the staff discussions at one of the compromised institutions. The vulnerabilities listed seem old because these attacks have been ongoing for a while now. Some of those vulnerabilities were actually discovered originally in relations to this situation. What's important to realize is that this situation is very unlike what's happened to windows machines recently. Most of the Windows intrusions have been remote exploits via services. We've been facing primarily local-root exploits. These people are breaking into accounts--usually by password sniffing, key-stroke logging, etc from other compromised machines. Those accounts are then used to launch various known (and previously unknown) local-root exploits. These people appear to be after other systems for an unknown purpose rather than just "games" or DoS attacks. Most of the targeted institutions have substanial DARPA/government research contracts. It's reasonable that these attacks are being used to steal information. The focus has not been on High Performance Clusters but rather on interactive clusters. These people are after information not computing power.

A few things to try.....

[Mercury2k]

Here is a list of some things that I feel are worth considering:

1. Patch your system! As soon as a patch comes out, get it applied and reboot if you have to! Also, stay up to date on security issues by subscribing to mailing lists that are related to the software your using. One good general purpose site is cert.org. Keep in mind that while mailing lists are great ways of being notified, they arent fool proof. If your subscription expires and you dont know about it, you wont be exactly up to date in the community now will you?

2. Use grsecurity. This is a kernel patch that is briefly lagged behind official Linux kernel versions. It has many great features for protecting against stack attacks/buffer overflows. ie: Those latest greatest scripts your local script kiddie just downloaded wont likely do anything against you since special addresses are randomised. It can also hide files on your computer such as intergrity checkers so nobody except you know they exist. Plus it can stop insert code into a running kernel by making kernel memory readonly (which btw, would have prevented at least one of the attacks they mentioned).

3. Install a filesystem intergrity checker. Aide, integrit and tripwire all come to mind and essentially all do the same thing but with different config file syntax. Besides, how can you tell if a file is changed if you dont actually check? Also, dont forget to hide the existence of this program using something like grsec's gradm filesystem ACL util and be careful of automating checks in the crontab!

4. Read a good linux securing article. One such article I have read is called Securing & Optimizing Linux: The Ultimate Solution. It will teach you how to lock a system down a fair bit and how to remove unused/unneeded services from your computer.

5. Watch those logs! Log files provide a wealth of information, but administrators rarely check them (well, not all). If you dont know what a log entry means, research it, or else you may be looking at an attack and not even realise it. Now I know some of you are thinking I am nuts considering just how many logs even a small system generates, but there are tools to help you. One way is to use a program called swatch (a perl script). It can parse existing and old archived log files using a perl regex syntax and trigger actions based on found text. Start by configuring the system to ignore any log entries that are known to be friendly and show you everything. Then slowly eliminate each friendly entry one at a time. What will be left is a list of purely evil enteries :). Next configure swatch to alert you upon recieving such messages! Of course you can always use perl or even grep -v to parse logs, but for repeated use I think a specialised tool would save you some trouble in the long run.

Now I know I could go on forever with suggestions, but I think that these few things should give anyone a kick in the right direction. I hope this has been helpful.