Intrusion Cleanup Forces Delay For GNOME 2.6

An anonymous reader writes "Looks like the GNOME site (both web and FTP) is back up and running again (from a replacement system). The restoration work is still going on, and dynamic content does not work yet. Bugzilla should be up by tomorrow (it is already in testing mode). More details are available in this announcement. Kudos to the GNOME sysadmin team for such a rapid recovery." However, blurzero writes "GNOME 2.6 was scheduled to be released sometime today, however after evidence of possible intrusion on the web server, the release has been delayed by one week, until March 31st." Update: 03/24 14:08 GMT by T : An anonymous reader points to this story on the delay at ZD Net Australia.

Comments

Must've been a real bugger

[James A. M. Joyce]

Intrustion cleanup is a real bastard to carry out with any degree of success. There's really no way to prove that there isn't just one more subtle little backdoor hiding in the system, in your repository or in your /home area. This is a case where an ounce of prevention is better than a pound of cure. It's too late, here, unfortunately, so they should probably have rolled back to a backup on another set of boxes. (Just my two cents.) How well would TripWire have worked in this kind of situation? Or is that ineffective against an all-out rooting?

Re:Must've been a real bugger

[tobechar]

I am personally disappointed in having to wait another week, however I completely respect the Gnome team on their tireless efforts. :)

I definatly agree with the idea of rolling back to a backed up copy of their site, but perhaps they do not know how long someone was able to access their systems?

Gnome team, take all the time you need. :)

Re:Must've been a real bugger

[Anonymous Coward]

They have TireWire and it didn't work.
TripeWire never works.
I've seen TW failing and being exploited in several installations.
Since the release of wirecutter TripWire has become fucking useless.

Re:Must've been a real bugger

[Penguinisto]

It takes some work, but there is one way to insure a completely clean system: Re-installation of the OS from media, or a backup from a time known before the break-in.

Either way, you only have to check the backup server data itself against (externally backed-up) MD5 checksums, and ask developers to re-commit any changes made during the suspect time.

Now try and do that to a mail server, and the fecal matter hits the air-handler. But, with data that is relatively static by comparison, it takes work, but isn't too much of a trial.

$0.98 in change, please :)

Re:Must've been a real bugger

[Storm]

Intrustion cleanup is a real bastard to carry out with any degree of success. There's really no way to prove that there isn't just one more subtle little backdoor hiding in the system, in your repository or in your /home area.

Basically, what you generally do is to rebuild from scratch, then carefully check and restore your repository.

How well would TripWire have worked in this kind of situation? Or is that ineffective against an all-out rooting?

This is why the authors of the host-based IDS recommend that you keep your database on media that is read-only or kept off of the machine. At that point, it becomes an administrative problem.

• How do you write the updated database to read-only media on a remote box?
• When on a shared box that is not your own, especially with a development box, what changes are valid?
• Who/how many admins do you need or use for the boxes?

You could use something like Samhain, which automates a lot of the detection of changes, and supports a management console.

Remember, if it were easy, anybody could do it. Microsoft has tried this approach to system administration, and look how successful its been. :)

Re:Must've been a real bugger

[ArsonSmith]

Re-installation of the OS from media

What if the OS has a vulnribility and the attacker can get back in without issues?

a backup from a time known before the break-in

What if the attacker had installed the back door months before hand? You may not have a valid backup.

Re:Must've been a real bugger

[ArsonSmith]

Of course even a reinstall still leaves the original hole open that the attacker used in the first place.

Re:Must've been a real bugger

[DarkOx]

The proper responce (in the majority of cases) is, image the compromised file system. Reinstall the production system from source media and patches. Get the system back in production but change all the passwords, ssl keys etc, give it some other ip then where your dns points and only let the people who *absolutely* need it know how to get at it. Remember time is money and getting back in production fast is important, even if its more limited production. Now analyize that filesystem image and figureout what happend. Go to the porduction system and patch the hole. Move to full production. This is almost always my policy, thankfully I have only had to evoke it once.

Dammit...

[thames]

now I have to go to two geek parties in one week

Re:Correlation?

[BRSloth]

Actually, if you check the GNOME-Announces list, you will see that every package was already updated to work with GNOME 2.6. They just want to double check everything.

Boy, that was a close call

[El Cubano]

"GNOME 2.6 was scheduled to be released sometime today, however after evidence of possible intrusion on the web server, the release has been delayed by one week, until March 31st."

That could have been disasterous had they been forced to delay until April 1. Imagine all the jokes that would have ensued.

Awwww man!

[chendo]

Now we have to wait one WHOLE week?

Maybe the KDE team did this to slow Gnome down... :)

By the way, I've tried CVS metacity with FD.O's Xserver..... funky stuff. Translucency when you move windows! Although it chews a fair bit of CPU (when moving the window itself, that is, as just holding the window still doesn't chew CPU), it should be fixed when we finally get HW acceleration. I was able to get MPlayer to play a video in the background, hover a window over it and watch it through it. ub3r cool stuff.

Re:Awwww man!

[bbuchs]

Do you have any notes or tips you could post on the process? I'd like to give it a shot, but haven't had much luck as of yet.

Well, there is one difference I appreciate...

[Penguinisto]

With GNOME and most other F/OSS projects, at least you get honest, up-front answers and timely announcements of intrusion attempts and such.

If only MSFT (and more importantly, proprietary software companies that aren't so much in the spotlight) were as forthcoming about break-ins.

Re:Well, there is one difference I appreciate...

[Penguinisto]

" What does Microsoft have to do with this? You fucking dumb jackass."

Well kiddo, it's not just MSFT truth be known (hence my mention of "more importantly, other proprietary companies..." )

Most proprietary companies are too worried about "customer confidence" to actually be honest with their customers. Back when a group of russians had 3 months' unlimited access to Windows' source code, it took outright proof in public before MSFT would admit to such a thing. ...and that's just MSFT; I wonder how many times Adobe's servers have been compromised? It would be nice to know that P-shop and Acrobat (or worse, the free reader?) wasn't quietly trojaned-up and sleeping on my 'dows boxen.

Now, what about the break-ins we don't know about? How were they handled? How can a proprietary software company, let alone its customers, be sure that there aren't any nasty suprises hidden in their products?

...and therein lies the crux of my argument - open-source companies are specific, honest, and, well, eopn about what goes on security-wise.

It's damned refreshing to be a customer who is treated like an adult, and not lied to, or kept in the dark about the products I use.

Does this answer your question?

Re:Well, there is one difference I appreciate...

[Call Me Black Cloud]

It would be nice to know that P-shop and Acrobat (or worse, the free reader?) wasn't quietly trojaned-up and sleeping on my 'dows boxen.

How does a public disclosure ensure the binaries are secure?

How can a proprietary software company, let alone its customers, be sure that there aren't any nasty suprises hidden in their products?

How? Probably the same way everyone else does it. The OS model does not have a monopoly on practices used to ensure code integrity.

I suppose

[AnonymousCowheart]

I suppose this will get modded as a flame bit, but a lot of people were cheering when Bill Gate's credit card number got stolen just wondering how those people felt now? I know there was no "real" damage in that case, and in this case the server was offline, but still something to consider. Maybe these people were also "trying to help" by showing a server insecurity.

Ya know...

[oldosadmin]

It makes you nervous about the big megacorps -- when their website is compromised -- do they even know... or care? I've never seen M$ shut down for a day because of a website compromise, although it must have happened several times.

Re:Ya know...

[HeghmoH]

A megacorp that will be losing enormous amounts of money for every minute of web site downtime will not be running their site on a single server. They most likely have a physically distributed cluster which can't all be compromised in the same attack, and hot swaps ready to go in case they all somehow get compromised as well. They don't have to take their site down because of an attack, whereas a comparatively small nonprofit effort has no choice.

Re:It's just a hoax

[marcello_dl]

Your hypothesis would be conceivable for a closed source project where bosses get pissed off when the product is not delivered on schedule, I don't think that Gnome developers have this kind of pressure.

Also, this attack reminds me of the one to the Debian servers, because it occurred just before a Woody release. Let's wait and see what the Gnome team has to say about it.

Running IIS?

[Peter_Pork]

A rumor is circulating that Gnome was using an unpatched IIS... I wish they would run Linux, it is much more secure, believe me.

Dumb Cracker?

[gscott]

According to Waugh, the GNOME Web servers that are hosted by Red Hat were compromised by "a dumb cracker who probably didn't realise what they got into".

Seems like he was smart enough to hack their system.

Re:Dumb Cracker?

[stevey]

It would be interesting to learn how the compromise had occurred.

I'm guessing that all the important services would have been up to date (ssh/rsync/apache/etc) - so that leaves a password/ssh keycompromise, or some scripting flaw..

I hope we find out once the cleanup has been completed.

Re:Dumb Cracker?

[Sgt_Jake]

"a dumb cracker who probably didn't realise what they got into"

They meant a white guy from Alabama - he was looking for 'gnome-porn'. ?!

Re:Dumb Cracker?

[Fluffy the Cat]

The machine in question isn't run by Red Hat admins.

Re:Dumb Cracker?

[JamesHenstridge]

What Jeff meant is that the cracker didn't seem to be targetting Gnome specifically. They'd have just as likely broken into any other vulnerable box.

Re:Dumb Cracker?

[FU_Fish]

According to Waugh, the GNOME Web servers that are hosted by Red Hat were compromised by "a dumb cracker who probably didn't realise what they got into".

Seems like he was smart enough to hack their system.

So the dumb cracker was really a smart cookie?

Re:Dumb Cracker?

[Too Much Noise]

Not really. Here's a scenario for you (the debian-style):

• cracker compromises a 3rd-party machine and gets the ssh tokens for a legitimate user.
  
• cracker logs into the server - no particular preference, that server just happened to be one of those he gained access to by sniffing on ssh logins from his initial machine
  
• cracker logs in as a legitimate user on the server (impossible to detect at this stage) and acquires, in some way or another, root access (like a nice, untraceable pam exploit)
  
• cracker tries to secure root acces and triggers an allert in the logs (this being the 'dumb' step)
  

the problem is, you can't trace the initial attack vector. It can be done by any script kiddie who compromises a machine that some developper uses. However, if it's not a mere script kiddie (and covers his tracks successfully), chances are that even a competent sysadmiin can fail to discover it. Yeah, I know about read-only/remote IDS databases, remote logs, backups and so on. It's a nice overhead when you're handling a large farm and you still have to make sure the data is on a secure machine. Do you do it for all your servers? (besides, at this level of complexity you need a full-time job - at least experienced hackers will see it coming and maybe leave you alone).

that said, whoever was the sysadmin for that box picked it up - kudos for that! And if the 'dumb cracker' line means what it says (from the logs, etc) then here's hoping that it was indeed just a lone incident.

Ack. Insightful?

[adamofgreyskull]

Something bad happens to someone we like. Bummer.
Something bad happens to someone we don't like. Haw Haw.

Why do people make such a big fucking deal out of double standards? Should I feel equally angry toward someone who kills a stranger as I would if they'd killed a relative? No.

Re:Ack. Insightful?

[dasmegabyte]

Well, it depends. Do you purport to be a moral and logical person? Do you believe in the protection of personal freedoms?

If so, then even if you don't KNOW or LIKE the victim, you should still support punishment of the criminal. Otherwise, you're encouraging elitism. Or do you want to live in a world where crimes against the unpopular are cheered and go unpunished?

I lived in a similar world called "Middle School," and I wouldn't want to go back.

Intrusion Method Same Of Gnu.org Intrusion?

[Goo.cc]

From what I have read, intrusion details have not been released yet but I wonder if the Gnome server was compromised the same way the gnu.org server was last year. If so, that would be disappointing.

Still, I am happy to see that this will not push the next version of Gnome back very much. It is really starting to look nice to me and I am a Mac OS X user.

On the other hand.

[Rhesus Piece]

As much as not being able to run Gnome 2.6 today makes me want to sit on my bed and weep, I am really grateful that the Gnome team is more concerned with releasing a secure product than with releasing when they said they would. This is one of those advantages of non-commerical software that we always cheer about in action. Rock on.

Deja Vu

[Anonymous Coward]

This event immediately brought thoughts of Half-Life 2 to mind.

I bet in a week the source code for GNOME 2.6 will be all over the Internet, free for anyone to take, read, and use!

MOD PARENT DOWN

[Anonymous Coward]

No post with "M$" in the body contains anything of value.

Could it be??

[goldspider]

Could it be that having competant, diligent system admistrators is more important than using the "right" server platform?

Re:Confidence ?

[prisoner-of-enigma]

How do you know the MD5 wasn't made after the intruder got in? It wouldn't be very valuable then, would it?

The point is, after a breakin you must determine when the breakin occured, because everything after that is suspect. The problem is it can sometimes be very difficult -- or impossible -- to determine when the breakin happened. Then you're really, really screwed.

Re:Linux on the desktop? Fair question, on topic.

[Theatetus]

No, dumbass, the difference is that closed-source companies keep it a secret (or doesn't know in the first place) when their servers are compromised while Gnome and Debian are very up-front about it.

If you think this kind of thing hasn't happened to Microsoft, Oracle, etc., you're wrong. They just like to keep it quiet.

Impatient bastard!

[Anonymous Coward]

Which one of you dirty bastards couldn't wait 1 day for the source? Whoever is running GNOME 2.6 right now, stand up and speak! Impatient Bastard!

Probably a SCO advocate

[cgreuter]

This sort of thing is exactly what I'd expect from freedom-hating closed-source advocates. No doubt, some SCO fan went and did this in retaliation for the Linux developers' attempts to preserve their intellectual property rights.

There is a dark side of the commercial software community and now we are beginning to see it emerge.

(Warning: this article contains sarcasm.)