Ubuntu Servers Hacked

An anonymous reader noted that "Ubuntu had to shutdown 5 of 8 production servers that are sponsored by Canonical, when they started attacking other systems. Canonical blames the community, saying they were community hosted, and were poorly maintained. However, kernel upgrades couldn't be done because of poor backwards compatibility with the very hardware that Canonical had sponsored! While people point fingers at each other it is pretty clear that both sides are equally to blame, the community administrators for practicing bad security practices, such as using unencrypted FTP transfers with accounts, not properly maintaining the system. However Canonical should have been well aware of what they are hosting. The question remains, if any of the files distributed to users have been compromised. A major blow for Canonical though who are attempting to enter the business market with Ubuntu Server."

Comments

I would like to read a report

[QuantumRiff]

Since this is a community based, open source project, I would love in the near future (after the investigation and cleanup are done) to read about how they determined that the machines were compromised, what the attackers did, and more importantly, how Ubuntu cleaned them up...

This could really help the community as a whole, and I know I would enjoy reading it..

Re:I would like to read a report

[gmack]

It's important to note that the servers may not have been actually rooted. There is a large number of ssh dictionary breakin attempts on every machine I administrate on several completely different ip blocks. The worst hit is usually my personal server that tended to get hit with several thousand attempts per hour(enough that legitimate logins were a problem) before I installed countermeasures. Even now the countermeasures are locking out 5 to 8 hosts per day.

They have managed to get user accounts on a few occasions and most of the time they never even attempt to gain root. They just start scanning for new hosts.

I'm now running a python script called DenyHosts to find and lockout dictionary attacks. "apt-get install denyhosts" for debian users. Even on much more liberal settings than the default it's lowered my cpu load considerably and locks out attacks in the first minute rather than the hour it would otherwise take me to notice.

Re:I would like to read a report

[discord5]

Unless we're going to be composing a Linux Administration HOWTO: Best of Bloopers.

I could fill about a 100 pages on my own from stupid things I've done and stupid things I've seen coworkers/customers do.

The funniest one is still one where one of my coworkers nuked /lib on a fairly important machine unintentionally because he just loves his spacebar:

rm -f /home/user/project /lib/*

Upon which of course by he proceeded to ask everyone "Hey, suppose I deleted something like /lib, is there a way to get it back?", followed by 10 people laughing, followed by a minute of silence as soon as we realized what machine he just did that on. He never got a root password for an important server after that incident. In hindsight, that was a funny incident, and a valuable lesson to us all (we all became paranoid of rereading what we just typed).

Yes, we had backups... Yes, tape drives are still slow

The real test

[ZachPruckowski]

The real test is how they react to this, and how they clean up their mess. Everyone screws up, but what separates good people from bad is how they react to problems and screw-ups.

It sounds like that part at least is still underway, with a meeting (FTA) in "#ubuntu-locoteams on Tuesday, August 14, 2007 at 2:00PM UTC". Seeing as that's yesterday, we should probably reserve judgement a day or two to see how they respond.

Re:sftp

[Anonymous Coward]

sftp and scp STILL do not allow anything like a REGET operations. Whenever anyone mentions this they got shot down in flames.

Turns out the whole reason for the attack was...

[bealzabobs_youruncle]

to replace the horrid orange and brown default themes.

I used to be an ardent Ubuntu supporter but since Dapper and the wider adoption there has been too much emphasis on making things more Windows-like and less on best practices throughout the Ubuntu community (note I said the community, not the developers). Stuff like Automatix and the general feeling that any script that or line of code that is posted on the Ubuntu forums is guaranteed safe has led to lax standards. I've brought this up a couple times and any valid discussion quickly descends into a flame-fest and the mods (rightly so) lock it down.

The Ubuntu community has bent over backwards so far to prove they can include everyone they lost site of many of the things that make Linux a better choice for many people; time to get back to fundamentals and best practices, the sooner the better. Stop worrying about besting Windows at every silly thing (ahem, desktop transparency), stop trying to include aunt Tilly (who is never going to "switch" anyway) and remember that some things take more effort but are often worth it.

Breaks happens all the time

[Pecisk]

It is just became obvious recently that open source publishes their breaks as they are, because they can't actually hide anything. I bet breaks in coorporation servers are so frequent that is common practise to be silent about them.

In mean time, there is a tradeoff between having one, LTS release which has rather old kernel with old drivers and new one, which has 18 month support but has everything up to date, including also unstable stuff of course. But in fact it doesn't even mather, because admin is who in charge.

So Linux is more secure than Windows? You bet. Then why such break-ins happens? Because of lazy or hobbist admins who have no time or maybe not enough knowledge to lock down server to protect it from attacks. To lock down such Windows server/workstation is much harder because of "black box" mentality such software has. But it is also possible.

So in resume - those are admins who are gulty persons here. Ubuntu Dapper and Feisty are secure enough releases to keep them locked down without causing trouble for services. And ohh, be careful to which persons you give access to and have good password management system.