Ubuntu Servers Hacked
An anonymous reader noted that "Ubuntu had to shutdown 5 of 8 production servers that are sponsored by Canonical, when they started attacking other systems. Canonical blames the community, saying they were community hosted, and were poorly maintained. However, kernel upgrades couldn't be done because of poor backwards compatibility with the very hardware that Canonical had sponsored! While people point fingers at each other it is pretty clear that both sides are equally to blame, the community administrators for practicing bad security practices, such as using unencrypted FTP transfers with accounts, not properly maintaining the system. However Canonical should have been well aware of what they are hosting. The question remains, if any of the files distributed to users have been compromised. A major blow for Canonical though who are attempting to enter the business market with Ubuntu Server."
Spambuntu
You keep using that word. I do not think it means what you think it means.
This isn't the only Linux distro security breach being disclosed recently. One of Gentoo's web applications was compromised and they are investigating it:
http://bugs.gentoo.org/show_bug.cgi?id=187971
This is just a transitional feature designed to make Windows users more comfortable using Ubuntu.
Creationist Textbook Stickers Declared Unconstitutional by CowboyNeal
Since this is a community based, open source project, I would love in the near future (after the investigation and cleanup are done) to read about how they determined that the machines were compromised, what the attackers did, and more importantly, how Ubuntu cleaned them up...
This could really help the community as a whole, and I know I would enjoy reading it..
What are we going to do tonight Brain?
Ubuntu made a boobootu
Politics is Treachery, Religion is Brainwashing
The real test is how they react to this, and how they clean up their mess. Everyone screws up, but what separates good people from bad is how they react to problems and screw-ups.
It sounds like that part at least is still underway, with a meeting (FTA) in "#ubuntu-locoteams on Tuesday, August 14, 2007 at 2:00PM UTC". Seeing as that's yesterday, we should probably reserve judgement a day or two to see how they respond.
administrators, but:
who the hell places such exposed servers like these on the net without applying security patches and following simple rules? yeah, the freaking old hardware, compab problems, i sure understand that. but then make a fuss 'bout it. threat to stop maintaining the hardware if the networks cards aren't changed. if that REALLY is the only problem with the hardware which prevented updates, then i just don't understand how the hell this could happen. NICs, even though this would be no consumer hardware, aren't that expensive. if my employees servers were hacked because i did not mind telling him that some crappy piece of hardware prevented me from keeping uptodate with security, i would kindly be removed from my desk. period.
it amazes me that people even use the plain old ftp protocol for anything important. sftp has been around forever.
Bruce
Bruce Perens.
had these been windows servers we would have heard cries of a flaky operating system being the problem. in this case, since they're linux servers, we hear that the fault lays on the administrators of the boxen for not hardening the systems?
It's like NT all over again. God only knows what bad things they can do with that.
Friends don't help friends install M$ junk.
Linux systems are only as secure as the admins who manage them.
:)
And for bonus "hate" points, even MS servers can be secure if they are admined probably. Don't worry though, I have my flame suit on.
The price is always right if someone else is paying.
As one of the people affected by this issue, I'd like to give some clarification on this. Firstly, the servers affected were Local Community (LoCo) Team servers, of which I maintain ubuntu-us.org While I'm personally annoyed that the site is down (given it was on the front page of Digg last week), these servers are far from "production" servers; they host LoCo team resources and websites. I'd like to know what "compromised" software would have been downloaded by users, given that these servers did not host user repositories, and for the most part hosted news pages, blogs, and localized documentation. The issues were twofold: the servers were not upgraded past breezy, leaving them open to vulnerabilities after Breezy's EOL; LoCo team users were running an array of web applications (Drupal, Wordpress, Mediawiki, etc), but not updating their systems with new security patches. Top that with ftp logins and no ssh keys, and you have yourself a problem. Canonical is moving the installs to their facilities, retrieving the data, and building the installs (including the aformentioned web applications) from scratch, assuming that everything has been compromised. Hopefully in the next few days this will all be over.
It has nothing to do with dumbing it down for Windows users making it insecure, although I admit, this case is again a demonstration that the bigest secuirty hole on a computer is the lump of carbon/hydrogen/oxygen located between the keyboard and the chair.
They got arrogant, cocky and lazy. They let their security slip on things a Windows uers wouldn't use or care about (ex. FTP vs SFTP, from a user perspective, the difference is minimal).
Does your reality distortion field go so far as to say that Windows is causing functional breakage in Linux now? Geeze. Lemme guess, you are gonna add global warming, wars, AIDS, ebola and the common cold to the list as well, right?
Heck, fairly certain that ftp wasn't active by default on my last install of Ubuntu (I know SFTP was though).
FTP vs SFTP - maintainer arogance/incompetance
Kernel couldn't be upgraded given the hardware supplied by Ubuntu's owning company - the companies own problems
Heh, compatability with new hardware is part of the reason i started using Gentoo... even though Ubuntu uses new software, i've always had at least some problems getting either Broadcom or Nvidia network-cards working on generic-distro kernels. Were they using custom-made kernels, or the stock one?
Well, if they _did_ get broken into all the time, then that would be pretty embarrassing. The last thing they would want to do is publicize the fact, so it only makes sense that they would cover it up and say nothing about it.
Since nobody has _ever_ said anything about frequent break-ins, it's clear that they must be happening.
Why am I the only person who can see how obvious this is?
I wonder if the tone would be so even headed if this was a recent MS operating system.
Firstly these servers were not "Canonical Hosted" as the anonymous readers suggests. They were hosted in a DC which Canonical paid for, but the community maintained them. So Canonical system admins had very little to do with them.
My site - http://screencasts.ubuntu.com was one of them that was affected, so I was of course concerned that there might be some data loss. I only use SCP to copy files up to the site, and logon with my ssh key, so don't think that all Ubuntu community members are using FTP, weak passwords and really old software, it only takes _one_ though to naff it up for everyone else.
The Canonical system admins (on top of the work they already do) migrated the services from those servers to their own DC very quickly. My site went down on Tuesday and was back by Friday. For free hosting and oodles of bandwidth, I'm happy with that downtime - for a community site.
Sir, somewhere in the fully-indexed and data-mined future, your descendants will be publicly shamed and ridiculed because of your post.
I suppose they'll have no choice but to flee to deeper waters.
Rich And Stupid is not so bad as Working For Rich And Stupid.
Please mod this -1, I don't agree with him.
Okay, so your assertion of fact was really just an enormous assumption. Thanks for the clarification.
"Ubuntu had to shutdown 5 of 8 production servers that are sponsored by Canonical, when they started attacking other systems."
In Soviet Russia, server attack you?
Beware: In C++, your friends can see your privates!
It's a lot harder to remotely install a PCI card than it is to complain about it on an internet message board.
And to think, the only reason I post here is so I can be taken seriously by the people who really count.
Another dream shattered!
Well, I heard that Ubuntu isn't very good at that either...
I used to be an ardent Ubuntu supporter but since Dapper and the wider adoption there has been too much emphasis on making things more Windows-like and less on best practices throughout the Ubuntu community (note I said the community, not the developers). Stuff like Automatix and the general feeling that any script that or line of code that is posted on the Ubuntu forums is guaranteed safe has led to lax standards. I've brought this up a couple times and any valid discussion quickly descends into a flame-fest and the mods (rightly so) lock it down.
The Ubuntu community has bent over backwards so far to prove they can include everyone they lost site of many of the things that make Linux a better choice for many people; time to get back to fundamentals and best practices, the sooner the better. Stop worrying about besting Windows at every silly thing (ahem, desktop transparency), stop trying to include aunt Tilly (who is never going to "switch" anyway) and remember that some things take more effort but are often worth it.
How insecure is it to leave a system accessible to Windows users on any front?
I won't give an gnu/linux account to any windows user because a minimum of 25% of them are part of a keylogging botnet. They are liable to access my machines from windoze and things go downhill from there, even if they use a better client. A system is only as strong as it's weakest link.
Ubuntu itself is dangerous because it includes non free software like Adobe Flash, but this should not be of concern to business users. These dangers are orders of magnitudes smaller than those faced by windoze users, but Ubuntu needs more shelter and care than Debian itself. No gnu/linux system is in danger of being auto-rooted like a windoze machine. Business users should continue their move to gnu/linux systems like Ubuntu.
Friends don't help friends install M$ junk.
It is just became obvious recently that open source publishes their breaks as they are, because they can't actually hide anything. I bet breaks in coorporation servers are so frequent that is common practise to be silent about them.
In mean time, there is a tradeoff between having one, LTS release which has rather old kernel with old drivers and new one, which has 18 month support but has everything up to date, including also unstable stuff of course. But in fact it doesn't even mather, because admin is who in charge.
So Linux is more secure than Windows? You bet. Then why such break-ins happens? Because of lazy or hobbist admins who have no time or maybe not enough knowledge to lock down server to protect it from attacks. To lock down such Windows server/workstation is much harder because of "black box" mentality such software has. But it is also possible.
So in resume - those are admins who are gulty persons here. Ubuntu Dapper and Feisty are secure enough releases to keep them locked down without causing trouble for services. And ohh, be careful to which persons you give access to and have good password management system.
user@ubuntubox:~$ stfu This server is going down for shutdown NOW!
I've seen lots of dolphins but none of them had CAT5 coming out of their ass.
May contain traces of nut.
Made from the freshest electrons.
Thousands of Windows machines get exploited every day, and there's barely a word said about it. 3 Linux machines are exploited, and it's "OH MY GOSH!!111". I don't know whether this is a good thing, a bad thing, or, my best guess, both.
If there's anyone I hate more than stupid people, it's intellectuals.
I don't think you know what he thinks it really means. I think he want's to use hacking as a generic term, for doing stuff as in "I hacked together a working PC form all the junk in my basement" or "I hacked that new feature into my existing code.", and so the poster and many people who like using the word hacker for themselves but don't want others to immediately associate themselves with criminal hackers, tried to coin a new term for those people, "crackers". And while that term never caught on people who want you to call criminal hackers crackers, will always make issue of calling them hackers in the hope that one day they might call themselves hackers, with out any of the negative connotations.
Interestingly enough many people who take that position try to use defend their strictly non-criminal activity use of the word by citing the famous MIT non computer hacks. The irony of this of course is that many of those involved minor criminal activity like breaking and entering.
Indeed. I have to question the security of a software company which not only leaves it's source code in public FTP, but, after others discover this mistake, ASKS THEM TO MIRROR IT!
It boggles the mind.
If there's anyone I hate more than stupid people, it's intellectuals.
I've seen this hundreds of times, but never bothered with it.
You made a good argument, but when you use terms like "Windoze" you lose credibility.
People who can't see though my wording probably won't believe the argument anyway. Brainwashing is strangely dehumanizing like that. The victims lose their sense of humor as well as reason. The term "windoze" implies both of those losses and that people who continue to use it are asleep at the wheel.
Friends don't help friends install M$ junk.
You can back up your policy in the packet filter.
In iptables, look up osf and --genre.
For pf, look up osfp.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
"The complete lack of evidence is the surest sign that the conspiracy is working."
- Jack Handey
When someone hacked MS and got a copy of their source code it was headline news.
I am surprised no one reports how oftem Linux source code is taken from company servers, they must get hacked constantly compared to MS.
Wow, sent an e-mail as suggested when clicking on "use classic" banner, and got a fast response that addressed my msg
Is there a similar sort of problem in Windows that was fixed 10 years ago and is now something you have to go out of your way to subject yourself to?
Ten years ago Linux was barely 1.0. The problem wasn't fixed as long ago as you pretend it was.
Most Windows problems tend to be about what the system will do by default, not what sort of ways you can screw yourself up if you really try hard and insist on ignoring decades of other people's mistakes.
The defaults have not been an issue since before that flawed kernel was released. Why do zealots insist on making themselves look stupid by not even being familar with that which they critisize?
I've never seen a paid individual make a stupid mistake like this. The captain of the Exxon Valdez was a volunteer with the Red Cross on a humanitarian mission. The Challenger and Columbia were piloted by kids from space camp. The original Tacoma Narrows bridge was designed by volunteers with Habitat for Humanity.
On the other hand, we all know that segregation & apartheid were both ended by paid professionals. If you want something big done right, only paid professionals can do it.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
Do you have a specific complaint, or is just it that the uncool kids are getting into the clubhouse? If you think the interface has gotten oversimplified, switch to kubuntu.
Done with slashdot, done with nerds, getting a life.
An individual or a group can try to make a term mean one thing or another thing, however until popular support for that definition is accepted it's still just wishful thinking.
As long as I can recall, in the world of computers and main stream media, a "hacker" is a person attempting to circumvent security measures for nefarious purposes (i.e. a Black Hat). Does this mean that you can't tilt at windmills? No. Just keep in mind that you may never win that battle. Can't hurt for trying, though, right? I mean, it's not like anyone is being arrested for being a "hacker" or anything. Oh...wait...
Again, I am pointing at the community more than the developers, who have provided a great distro that has provided a much needed kick in the pants to other distros to improve their usability. Fedora is my favorite example, and my distro of choice again, since they had to face some stiff competition to stay relevant.
Ubuntu was about a clean interface with best of breed apps, solid documentation and a community that balanced ease of use with best practices. When someone wandered into the forums with a "noob" question we avoided the "RTFM newb-sauce" stuff and helped them, as well as re-enforcing best practices and linking where to get better information. We didn't point them to untested scripts or recommend subverting security for ease of use, but that is a regular event these days. Shuttleworth wanted "free as in speech" software that was "free as in beer" for everyone, but now to court Windows users he considers installing binary blobs and distributing closed source software? The "Unofficial Ubuntu FAQ" used to handle this stuff very well while not polluting (or introducing possible legal issues) to the distro. I recall Shuttleworth at Debian conferences with his hat in his hand explaining how he wants to help and work with the community, but if you mention this on the Ubuntu forums you have people suggesting that they don't need Debian or the GNU tools? This is an ignorant and arrogant user base that needs to be educated, and in some instances policed.
The original intent of Ubuntu was great, it just needs to get back on course. I much prefer apt to yum, I hope this wakes up the right people and I will gladly give Ubuntu a shot again.
On my company's server, I solved the attempted SSH break-in problem by disabling password logins via SSH altogether. Only publicKey logins are allowed. The break-in attempts have completely stopped (or at least they are turned away so quickly that there's not even a security log message for them).
Only a few computers have my public/private key pair on them (the private key is encrypted, of course), and I keep an extra copy on a USB thumb drive in case of emergency. If someone needs access to the server, I can use one of the existing logins to install their public key so that they can login.
I highly recommend this solution to anyone who can manage it. It's much more straightforward than trying to maintain blacklists.
--Stuart
I propose we coin a new term, "hatters." However, that might make them mad.
GLaDOS for President 2016! "Well here we are again. It's always such a pleasure." -- GLaDOS, 2011
Automatix in particular is a fantastic story of why I avoid forums. Automatix began life as a bash script under a different title by someone other than "arnieboy", and shared by a sticky forum thread. A marginal step up from guides telling you what commands to run to enable various things, etc. Based on a fundamental misunderstanding of copyright, licensing and the GPL, Automatix was born as a fork of this script, featuring numerous dubious personalizations that might be okay for arnieboy to accept but aren't good suggestions (such as enabling a root account). The forum admins have regularly played an active role, playing favorites amongst the various tools. Automatix at one point had it's own 3rd party project sub forum, where apparently traditional Ubuntu Code of Conduct did not apply ("his forum, his rules"). Eventually automatix was blamed for the failed upgrade of a number of users, and some people took to abusing a "popular searches" front page widget to advertise the phrase "automatix sucks", which was eventually fixed by telling the software that "automatix" was too common a word to search for, I think at the author's request.
As things stand now, Automatix has it's own forum and remains mostly antagonistic towards criticism. It's functionality has been largely dupplicated though it still serves a purpose, to commit copyright infringement via w32codecs etc. Ubuntu has tools that function very similar to Automatix' normal behavior, and in some cases improve upon it. The codec detection stuff in totem is helpful, as you don't need to know about Automatix to learn how to make things work, though it doesn't install w32codecs. And the most significant, repeated complaint has not been solved: Automatix has scheduled for themselves a single week with which to test all bugs and upgrade flaws -- they plan to release one week before gutsy is published.
A number of forum posts relating to this history have gone missing, which I disagree with. The proper thing to do in the face of misconduct is confront it and denounce it, not hide it by deletion. You might have the right to be offended by what people say, but not the right to erase history. Instead of the forums, use mailing lists and IRC when you feel like being sociable with other linux users, and launchpad's bugs and answers services if you have a problem.
I Browse at +4 Flamebait
Open Source Sysadmin