Slashdot Mirror
Kernel.org Attackers Didn't Know What They Had
Trailrunner7 writes "The attack that compromised some high-value servers belonging to kernel.org — but not the Linux kernel source code — may have been the work of hackers who simply got lucky and didn't realize the value of the servers that they had gotten their hands on. The attackers made a couple of mistakes that enabled the administrators at kernel.org to discover the breach and stop it before any major damage occurred. First, they used a known Linux rootkit called Phalanx that the admins were able to detect. And second, the attackers set up SSH backdoors on the compromised servers, which the admins also discovered. Had the hackers been specifically targeting the kernel.org servers, the attack probably would've looked quite different."
A few blog posts in the wake of the attack have agreed with the initial announcement; while it was embarrassing, the integrity of the kernel source is not in question.
They didnt want to harm the kernel.
Read radical news here
...and didn't realize the value of the servers that they had gotten their hands on...
....I don't see any mention of what the phrase refers to. Is this dramatization or intentionally excluded information?
Curious.
not know what they had cracked and how useful it was?
"I don't know, therefore Aliens" Wafflebox1
They must have gotten their hands on the kernel source code, I found it posted online!
but how do we know someone more sophisticated didn't already break in and mess with the code undetected?
Why would the attacks have to look different? Because if somebody wanted to mess with the source, they'd be more sophisticated and use more sophisticated exploits? Like Kaspersky pointed out, if they wanted to mess with the source code, a lot of what they did would have been unnecessary, but whatever initial exploit they used would have still worked! I think the real point is here 'they got in'. Better attackers just mean they wouldn't have discovered the break-in as quickly, and actual damage might have been done. Whether or not the attackers knew what they had is immaterial: the real message here is kernel.org needs to wake up and get serious about security, if any random script kiddie can root them.
"These people look deep within my soul and assign me a number based on the order in which I joined" --Homer re:
Did you bother to read either of the two links in the summary about that very topic?
Basically, the nature of Git makes it very unlikely that someone could insert malicious code into the kernel via kernel.org without someone noticing.
Well you could start by thousands of tin foil hats who are most likely looking at and scrutinizing every line of the code. Though wouldn't it be safe to assume there are backup copies off of the server? Isn't it possible they just either did an md5sum comparison or loaded the backups over the online versions to be safe.
I think the truth is that failers trying to save their asses and trying to make themselves heroes here.
Given that they attackers hacked the server a minimum of 17 days before it was detected, I'm not sure I'm going to buy into a story that makes the attackers sound clueless and the server admins smart and on the ball.
#DeleteChrome
It wasn't until they got into the machines that they realized the Kernel wasn't written in Javascript. "Dammit!"
The thought of hanging myself at my student loan organization doesn't bug me as much when I think it might make a differ
How they got root access after logging in. Was it something simple like a sudo? Was it a known, unpatched kernel vulnerability? Or, was it some new vulnerability current kernels are susceptible to? Last I read, they logged in under a user account, then they got root access.
I was concerned about the fact that a high profile like kernel.org site was rooted, but knowing it didn't take a sophisticated and highly knowledgeable penetration team but just a group of bumbling script kiddies makes it all better.
While the distributed nature of git sure makes it difficult for the kernel source to be compromised, I think that's a bit optimistic (I confess I only read one of those two links, though).
It looks like they argue that since every commit is hashed and everyone has got a full copy, it would be really difficult for someone to alter any single byte without raising suspicion. However, it should be noted that one or several developer accounts got compromised (which later resulted in privilege escalation to root).
How do you know they did not commit something to a developer's tree, then push it upstream? That is, since they got at least one of the developers (if not many, which could complicate things further by "distributing" the harmful change over several different commits from different people) how do you know its not a *legit* commit?
The developer who got himself (or at least his passwords) compromised would need to find out that one of his commits isn't really made by him. That could be not easy to spot, especially for others if the exploit is well hidden. (off-by one bugs that result in remote exploitation come to mind - not trivial but certainly possible and difficult to detect when writing C code)
Of course, code by those developers could be audited, but I that is far from saying: "bah, we got compromised but we sure as hell know the source isn't compromised", which is how it sounds to me.
The first way: Haha, these skiddies didn't have what it takes to effectively hide their cracking.
The second way: Skiddies were able to crack kernel.org using automated cracking tools just Windows, no evil genius required.
AntiFA: An abbreviation for Anti First Amendment.
So - Billy Random Hacker aka Snotnose Scriptkiddie just happens to have authorization via git to change the source code? Then - why didn't he come in through the front door, and change the source code? It makes no sense, I tell you! It's like, I have a safe deposit box at my local bank, and I want to change the contents of that box, so I break in at night to do so. When, all along, all I needed to do, was to walk in during business hours, and inform any bank officer that I need access to my box.
What am I missing here, that seems so obvious to you and so many others?
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
they should run linux on their servers, it's more secure! oh wait... whoops. ho hum, i suppose it's no good saying that they should run a FreeBSD Bastion Server, is it?
Another way to look at it: the fact that the administrators found out and admitted getting hacked says a lot about the ability of the administrators.
I would rather trust these guys than someone who claims to have never been hacked ever.
Its not like they get hacked all that often, which sure would make them look bad.
(I confess I only read one of those two links, though)
Did you read the second one? Because it deals with the specific situation you mention, with a hypothetical about Linus' account getting compromised. I'm not an expert on Git, but according to the article, the developer in question would be informed by Git that a commit had been made that didn't match their personal repository the next time they tried to submit their changes.
This wouldn't have happened if they ran closed source Microsoft Windows Server :)
It's a joke guys kidding :P
Question, how would OpenBSD prevent them from getting into the server with compromised username and password? Or from running arbitrary code once they do so?
So they found the SSH Frontdoors, but did the admins find the rest?
Privacy is terrorism.
Now Linux is popular enough to have rootkits. This must be the year of Linux on the desktop.
The second link deals with kernel.org being compromised, I am talking more about a legit commit making its way to the kernel tree. i.e. What happens if Linus Torvalds get compromised and someone uses *his* repository to add malicious stuff, which will eventually get pushed upstream?
That is, what's preventing a legitimate and trusted developer from introducing malicious code? That could happen as soon as a developer is compromised, which if I'm not mistaken is exactly what happened and how kernel.org got compromised (which in turn might have exposed credentials from more devs)
Or so they think.
If you have weak passwords, no OS is going to help you. The only thing that helps you there is something like DenyHosts or fail2ban. Not even OpenBSD prevents stupid.
Yes, well everyone knows those kernel.org sysops are a bunch of pushover newbies. Im sure you can do way better with the scope and size of the systems they deal with.
I'd recommend Windows instead of OpenBSD. Sure Windows has had its problems in the past but now with Windows 7 and Norton it's practically impossible to hack my systems. Even the Lunix box I run feels safer when the Windows computer is on the network. I imagine it's the firewall in Windows 7 reaching out in to the network to protect all the computers in my office.
I've been supporting Lunix and Windows at the enterprise level for many years now. I think its finally time to move away from Lunix. Linus really needs to ask himself where he wants this to go? The kernal is hacked up, probably with viruses hidden in there (we can't be sure). Sorry, I have to say bye bye to Lunix.
...they were just script kiddies who knew one single method, and thought it would be cool to try it on kernel.org.
...if I was in charge of damage control at kernel.org. Just sayin'.
If a Windows server is hacked, hackers had luck.
They were lucky it wasn't Linux.
Is this a joke or are you completely clueless?
He's being funny, in case you can't tell.
Disturbingly they seem to have considered not wiping and reinstalling.
It appears that the chief kernel.org system administrator is so naive about security that he doesn't even realize the absolute necessity of a full wipe and reinstall after compromise of such an important site. It also appears that there was no routine booting from read only media to check system files and startup scripts for changes. And no daily rootkit scan. If it was me, I would trash the motherboard for fear of BIOS or other firmware contamination. Exploits living on the firmware of network cards and other places have been demonstrated.
It wasn't brute-forced. A user with commit privs had his work laptop trojanned. Yes, I actually was reading kernel.orgs emails when it happened. and all these other articles.
C|N>K
Git records when changes are made (it has to to work and by whom it would be very easy to detect and edit if they changed it through the metaphorical front door and could be changed back. What is being suggested is that someone hid the changes (which would require manual access to the git files).
My understanding is this would not be too hard, but apparently it is?
null
They need to get their MCSE certification updated ;)
I still haven't managed to figure out if the tarball you download from the main page has been compromised.. Yes GIT saved everybody and all, but they seem to not want to say anything about the front page tarball, makes me curious
Track IP - Remotely track the IP address of a machine via email or MySQL.
I'm curious, once inside an OpenBSD server as normal user, what rootkit would they use instead of Phalanx to elevate privileges? The OpenBSD teams has expended a lot of effort to combat such a thing.
This is why I never comment until the mods have had a chance to let me know whether I should be laughing.
but how about that "privilege escalation' business that Linux couldn't prevent?
oh boy, then we can have millions of DVD with compromised code on them that everyone thinks is the golden standard. you are a genius.
Thats the interesting one, hopefully we will get full details about that one soon.
So when you find a backdoored SSH and a Linux rootkit on your server you might only be seeing the tools from one team who got lucky.
echo '[q]sa[ln0=aln80~Psnlbx]16isb572CCB9AE9DB03273snlbxq' |dc
Well if you manage to trojan Linus's machines and inject code into his local machine then of course there is no direct protection against that, and how could there be? However since there are many people who's only hobby/job is to track the changes to the kernel source and some of them would probably raise a question why Linus was committing that code to that part of the kernel (since everything that is commited is also discussed on lkml before). Also note that devs with direct commit access to the main kernel.org repository is not large, everyone else has to have their changes go by these people making it harder to sneak bad code into the kernel. Not saying that it couldn't be done but the detection rate is probably higher than the people interested in such an attack is willing to gamble with (notice that this has happened once against the cvs history of the Linux kernel but was detected by Linus due to his use of bitkeeper).
What is being suggested is that someone hid the changes (which would require manual access to the git files). My understanding is this would not be too hard, but apparently it is?
What do you mean by "git files"? If you mean the files tracked by git, then yes, it is very hard. The two links provided in the summary explain how git uses cryptographic hashes to verify the current files and history. Alternatively, you might mean the git program itself. The attackers could conceivably have swapped in a modified git binary to ignore hash mismatches. But this would be discovered when anybody on a non-compromised machine ran git fsck, or recompiled git (using a compiler from a non-compromised machine). So this is hard to do silently as well.
It is bad, but there is a mitigation. It requires two steps in stead of just one to get root access. Given the fact that you usually try to layer your security and have logging/accounting and tripwire type of alarms set up, you have a bigger chance of catching intruders before they get access to anything really dangerous.
If you admin thousands of systems, used by many more users, you will get compromised accounts, on a fairly regular base. Those accounts in general will be used to try and get root access. By setting up logging, accounting and various other tools, you tend to get a lot of the compromised accounts to trigger an alarm before they get root, or run their code as user. With remote root vulnerabilities, you get none.
Any privilege escalation is something to be serious about, but crying wolf that local exploits are just as bad as remote, will make less people take you serious.
I was promised a flying car. Where is my flying car?
If you are posting the seminal Trusting Trust, you should also post Countering Trusting Trust to balance it. It is possible to escape the trojaned compiler problem through the use of double diverse compilation.
http://www.edithex.com/
The cloud is really good for some things...
With Chkrootkit having seen its last update sometime 2009 and RK Hunter also being on the backburner, how does one even check these days for rootkits and other nasties like it? Suggestions?
Still could be some dumbass with sudo, either set to use the same (weak) password, or they were logging in directly as root (which I sure hope not, but see above about fixing stupid).
The major distributions are safe but some doofus at somewhere like Cisco or Belkin (or more likely their Chinese contractor) may have obliviously downloaded a compromised tarball and shipped it on a million routers.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
Let this be a lesson that there is no such thing as a "safe" operating system. I find it rather amusing that they're downplaying this breach instead of really questioning how one would unknowingly hack kernel.org, which should have been sufficiently protected given the material it hosts.
Thanks for correcting me!
'the firewall in Windows 7 reaching out in to the network' part was hilarious.
Read radical news here