Slashdot Mirror


Kernel.org Attackers Didn't Know What They Had

Trailrunner7 writes "The attack that compromised some high-value servers belonging to kernel.org — but not the Linux kernel source code — may have been the work of hackers who simply got lucky and didn't realize the value of the servers that they had gotten their hands on. The attackers made a couple of mistakes that enabled the administrators at kernel.org to discover the breach and stop it before any major damage occurred. First, they used a known Linux rootkit called Phalanx that the admins were able to detect. And second, the attackers set up SSH backdoors on the compromised servers, which the admins also discovered. Had the hackers been specifically targeting the kernel.org servers, the attack probably would've looked quite different." A few blog posts in the wake of the attack have agreed with the initial announcement; while it was embarrassing, the integrity of the kernel source is not in question.

38 of 183 comments (clear)

  1. How could the attackers... by Nutria · · Score: 2

    not know what they had cracked and how useful it was?

    --
    "I don't know, therefore Aliens" Wafflebox1
    1. Re:How could the attackers... by Anonymous Coward · · Score: 2, Informative

      Any random machine with SSH enabled gets around 2-3 brute force attempts per day from automated zombies. Most likely kernel.org was breached in an automated attack. I've protected my server with Denyhosts which will add your IP address to the firewall after 5 failed passwords... indefinitely. kernel.org should either install it or switch to public key authentication. Having user/password authentication on Internet without any protection is real stupid.

    2. Re:How could the attackers... by Dahamma · · Score: 4, Informative

      2-3? Mine sometimes gets hundreds. It's pretty ridiculous.these days.

    3. Re:How could the attackers... by beuges · · Score: 2

      This argument doesn't make sense. From what I've read, a kernel dev with kernel.org access had his machine trojanned, and the attackers got to kernel.org in that way. That's a far cry from script kiddies trying SSH ports on a bunch of random IP addresses. It sounds like quite a lot of effort to specifically get to the kernel.org network. Whether they managed to do some unknown damage, or access some other data whose relevance is as yet unknown, or maybe they just did it for the reputation of having hacked kernel.org doesn't matter - it does seem to be a targetted attack and the attackers would definitely have known what they had.

      These types of stories are actually more harmful than anything else - instead of calming people down by downplaying the severity of the incident, they're creating the impression that kernel.org was taken down by a bunch of script kiddies doing random port-scans and dictionary attacks, which in turn makes the kernel.org admins look pretty foolish. Not good PR at all.

    4. Re:How could the attackers... by smash · · Score: 2

      Not using SSH key pairs for accounts on a public system like this sounds pretty brain damaged to me.

      --
      I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
  2. Re:and after reading the articles.... by Samantha+Wright · · Score: 5, Informative

    Here is what it's referring to. CS graduates are expected to recognize instances of it instinctively.

    --
    Bio questions? Ask me to start a Q&A journal. Computer analogies available for most topics!
  3. Nonsense! by Anonymous Coward · · Score: 5, Funny

    They must have gotten their hands on the kernel source code, I found it posted online!

  4. Re:Or maybe by Anonymous Coward · · Score: 2, Interesting

    Or maybe, just maybe, the hackers wanted to appear that they didn't realize what they had gotten their hands on. I'm not trying to cause any tinfoil hats to come out, but I would still check everything.

  5. Re:Wishful thinking? by MimeticLie · · Score: 4, Informative

    Did you bother to read either of the two links in the summary about that very topic?

    Basically, the nature of Git makes it very unlikely that someone could insert malicious code into the kernel via kernel.org without someone noticing.

  6. Re:sure THESE guys didn't touch the kernel... by Anonymous Coward · · Score: 3, Informative

    Because in git, everything has got a hash checksum that is not forgeable (try to get something that compiles, does specifically something malicious, and that has the same checksum and size as before).

    And then there are thousands of copies of the entire thing around. I also have one. A simple comparison for equality will work.

  7. Spin by 93+Escort+Wagon · · Score: 4, Insightful

    Given that they attackers hacked the server a minimum of 17 days before it was detected, I'm not sure I'm going to buy into a story that makes the attackers sound clueless and the server admins smart and on the ball.

    --
    #DeleteChrome
    1. Re:Spin by microbee · · Score: 4, Insightful

      Yeah, just admit failure and do better next time. No need to blog about how a trivial issue it was.

    2. Re:Spin by Rogerborg · · Score: 4, Interesting

      We totally hadn't detected any intrusion!

      Uh... then we did.

      But we totally haven't detect any meddling with the sources

      Uh...

      --
      If you were blocking sigs, you wouldn't have to read this.
  8. Re:and after reading the articles.... by Samantha+Wright · · Score: 5, Insightful

    That's pretty much it. Malicious control over the master copy of the kernel source means you can bake a rootkit into everything everywhere with enough clever code. All it takes is one generation of bad files to silently patch all successive copies during compilation, and you've got the stuff that cypherpunk nightmares are made of.

    --
    Bio questions? Ask me to start a Q&A journal. Computer analogies available for most topics!
  9. Feeling better by ChatHuant · · Score: 5, Insightful

    I was concerned about the fact that a high profile like kernel.org site was rooted, but knowing it didn't take a sophisticated and highly knowledgeable penetration team but just a group of bumbling script kiddies makes it all better.

  10. Two ways to look at this... by CajunArson · · Score: 4, Insightful

    The first way: Haha, these skiddies didn't have what it takes to effectively hide their cracking.

    The second way: Skiddies were able to crack kernel.org using automated cracking tools just Windows, no evil genius required.

    --
    AntiFA: An abbreviation for Anti First Amendment.
  11. Re:The motive doesn't matter. It's time for action by LordLimecat · · Score: 3, Insightful

    Question, how would OpenBSD prevent them from getting into the server with compromised username and password? Or from running arbitrary code once they do so?

  12. Re:Wishful thinking? by Superken7 · · Score: 2

    The second link deals with kernel.org being compromised, I am talking more about a legit commit making its way to the kernel tree. i.e. What happens if Linus Torvalds get compromised and someone uses *his* repository to add malicious stuff, which will eventually get pushed upstream?

    That is, what's preventing a legitimate and trusted developer from introducing malicious code? That could happen as soon as a developer is compromised, which if I'm not mistaken is exactly what happened and how kernel.org got compromised (which in turn might have exposed credentials from more devs)

  13. Re:and after reading the articles.... by jimicus · · Score: 4, Informative

    I'm not a programmer, but I'm not entirely ignorant either... which leaves me with a question... Assuming that the Kernel was compromised, and the scenario you describe came into being. Isn't it just a matter of examining the Kernel code until you find the naughty bits and expunge them? Or are you basing your nightmare on this infiltration not being detected?

    Not at all - the paper describes how one could write a custom C compiler which would automagically insert malicious code when it saw a particular pattern.

    With a properly crafted attack you couldn't even compile your own "known clean" compiler - the attack takes advantage of the fact that most modern C compilers can't be compiled without an existing C compiler of some sort. Provided the existing compiler is the malicious one, all it needs to do is insert its own malicious payload as part of the compiler compilation process and every subsequent compiler on that system is equally malicious even though the source code is perfectly clean.

  14. Re:I'd Still Like To Know... by beuges · · Score: 4, Informative

    From what I read, one of the guys with a kernel.org login (HPA, I believe) had his personal machine infected by a trojan. The attackers were then able to login to kernel.org impersonating him. They then used a local-only exploit to get root.

    This is why a local-only exploit is just as bad as a remote exploit. If your machine connects to a network, it has the potential to be compromised by a local-only exploit, by first exploiting a flaw in a completely unrelated program which is accessible remotely. In this case, the "flaw" was the compromised user account. It could have been a buffer overflow in an ftp or web server, which doesn't allow for privilege escalation on its own, but allows arbitrary code to be run as the current user... all the attacker has to do is make that arbitrary code trigger the local-only exploit, and your local-only exploit is now a remote one.

    It's sad that so many people on slashdot keep playing local exploits down, or keep saying things like 'well it doesn't matter if my linux mail program has a flaw - the worst that can happen if I open a dodgy attachment is they wipe out my user directory, the rest of the system is safe'. Nothing is further from the truth. It's harder, yes, but not impossible to chain a bunch of vulnerabilities together so that your local-only exploit becomes remotely accessible.

    This is why Linus doesn't like to classify bugs as security bugs vs other bugs. All bugs are potentially security bugs.

  15. Re:The motive doesn't matter. It's time for action by LordLimecat · · Score: 3, Insightful

    Yes, well everyone knows those kernel.org sysops are a bunch of pushover newbies. Im sure you can do way better with the scope and size of the systems they deal with.

  16. Re:and after reading the articles.... by Samantha+Wright · · Score: 2

    That's exactly the idea. While such a move would eventually be discovered, the amount of damage that could be done before that happened could be substantial, and the amount of work required to (a) find the bad code in a copy of the source tree and, especially, then (b) detect bad binaries would be pretty staggering. It should also be stressed that this isn't just some fanciful nightmare scenario as Kahabut suspects; Ken Thompson did it to the Unix kernel long ago.

    --
    Bio questions? Ask me to start a Q&A journal. Computer analogies available for most topics!
  17. Re:The motive doesn't matter. It's time for action by Slashdot+Assistant · · Score: 5, Funny

    I'd recommend Windows instead of OpenBSD. Sure Windows has had its problems in the past but now with Windows 7 and Norton it's practically impossible to hack my systems. Even the Lunix box I run feels safer when the Windows computer is on the network. I imagine it's the firewall in Windows 7 reaching out in to the network to protect all the computers in my office.

    I've been supporting Lunix and Windows at the enterprise level for many years now. I think its finally time to move away from Lunix. Linus really needs to ask himself where he wants this to go? The kernal is hacked up, probably with viruses hidden in there (we can't be sure). Sorry, I have to say bye bye to Lunix.

  18. Which is exactly what I would say... by rocket+rancher · · Score: 2

    ...if I was in charge of damage control at kernel.org. Just sayin'.

  19. Re:sure THESE guys didn't touch the kernel... by John+Hasler · · Score: 3, Informative

    Because in git, everything has got a hash checksum that is not forgeable...

    And every commit is signed. When the Debian kernel maintainers fetch a new kernel (from Git, not a tarball) they verify the signature on every new commit. The integrity of the Linux kernel does not depend on anything as brittle as a sacred master copy.

    --
    Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
  20. Re:Obviously by Larryish · · Score: 2

    If a Windows server is hacked, hackers had luck.

    They were lucky it wasn't Linux.

  21. Kernel.org may not bother with clean wipe by mindbuilder · · Score: 2

    Disturbingly they seem to have considered not wiping and reinstalling.

    System is being verified from backups, signatures, etc. As of right
    now things look correct, however we MAY take the system down soon to do
    a full reinstall and for more invasive checking.

    (emphasis added) John 'Warthog9' Hawley
    Chief Kernel.org Administrator http://pastebin.com/BKcmMd47

    It appears that the chief kernel.org system administrator is so naive about security that he doesn't even realize the absolute necessity of a full wipe and reinstall after compromise of such an important site. It also appears that there was no routine booting from read only media to check system files and startup scripts for changes. And no daily rootkit scan. If it was me, I would trash the motherboard for fear of BIOS or other firmware contamination. Exploits living on the firmware of network cards and other places have been demonstrated.

  22. Main website tarball by David89 · · Score: 2

    I still haven't managed to figure out if the tarball you download from the main page has been compromised.. Yes GIT saved everybody and all, but they seem to not want to say anything about the front page tarball, makes me curious

    --
    Track IP - Remotely track the IP address of a machine via email or MySQL.
  23. Re:The motive doesn't matter. It's time for action by rubycodez · · Score: 2

    I'm curious, once inside an OpenBSD server as normal user, what rootkit would they use instead of Phalanx to elevate privileges? The OpenBSD teams has expended a lot of effort to combat such a thing.

  24. Re:and after reading the articles.... by F.Ultra · · Score: 2

    Not really, you have to remember that the complete code as it was before these naughty bits was introduced is stored on multiple locations including Linus's own hard drive so extracting a diff between the two versions would point at all these changes immediately.

  25. You can't conclude that from the data .. by AftanGustur · · Score: 2
    Usually advanced attacks are done by multiple teams. I.e. the Analysis, Compromise and exploit parts might all by done by different teams, and there may even be multiple teams for each task.

    So when you find a backdoored SSH and a Linux rootkit on your server you might only be seeing the tools from one team who got lucky.

    --
    echo '[q]sa[ln0=aln80~Psnlbx]16isb572CCB9AE9DB03273snlbxq' |dc
  26. Re:Or maybe by FatdogHaiku · · Score: 4, Funny

    They didnt want to harm the kernel.

    Possibly out of respect for the 11 Secret Herbs and Spices?

    --
    You have the right to remain sentient. If you give up the right to remain sentient, you will be elected to public office
  27. Re:Wishful thinking? by F.Ultra · · Score: 2

    Well if you manage to trojan Linus's machines and inject code into his local machine then of course there is no direct protection against that, and how could there be? However since there are many people who's only hobby/job is to track the changes to the kernel source and some of them would probably raise a question why Linus was committing that code to that part of the kernel (since everything that is commited is also discussed on lkml before). Also note that devs with direct commit access to the main kernel.org repository is not large, everyone else has to have their changes go by these people making it harder to sneak bad code into the kernel. Not saying that it couldn't be done but the detection rate is probably higher than the people interested in such an attack is willing to gamble with (notice that this has happened once against the cvs history of the Linux kernel but was detected by Linus due to his use of bitkeeper).

  28. No, it's not just as bad by dutchwhizzman · · Score: 3, Informative

    It is bad, but there is a mitigation. It requires two steps in stead of just one to get root access. Given the fact that you usually try to layer your security and have logging/accounting and tripwire type of alarms set up, you have a bigger chance of catching intruders before they get access to anything really dangerous.

    If you admin thousands of systems, used by many more users, you will get compromised accounts, on a fairly regular base. Those accounts in general will be used to try and get root access. By setting up logging, accounting and various other tools, you tend to get a lot of the compromised accounts to trigger an alarm before they get root, or run their code as user. With remote root vulnerabilities, you get none.

    Any privilege escalation is something to be serious about, but crying wolf that local exploits are just as bad as remote, will make less people take you serious.

    --
    I was promised a flying car. Where is my flying car?
  29. Detection?? by Anonymous Coward · · Score: 2, Interesting

    With Chkrootkit having seen its last update sometime 2009 and RK Hunter also being on the backburner, how does one even check these days for rootkits and other nasties like it? Suggestions?

  30. A real risk here. by John+Hasler · · Score: 2

    The major distributions are safe but some doofus at somewhere like Cisco or Belkin (or more likely their Chinese contractor) may have obliviously downloaded a compromised tarball and shipped it on a million routers.

    --
    Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
  31. Re:Or maybe by fatphil · · Score: 2

    You my laugh, but I have at least twice been approached by agencies who are looking for a colonel developer.

    --
    Also FatPhil on SoylentNews, id 863