Slashdot Mirror


Fedora Infrastructure Compromised

Trailrunner7 writes "The infrastructure of the Fedora Project was compromised over the weekend and an account belonging to a Fedora contributor was taken over by an attacker. However, Fedora officials said they don't believe that the attacker was able to push any changes to the Fedora package system or make any actual changes to the infrastructure. The attack appears to have targeted one specific user account, which had some high-value privileges. The attacker was able to compromise the account externally, and then had the ability to connect remotely to some Fedora systems. The attacker also changed the account's SSH key, Fedora officials said."

78 of 115 comments (clear)

  1. Believe? by amicusNYCL · · Score: 2

    However, Fedora officials said they don't believe that the attacker was able to push any changes to the Fedora package system or make any actual changes to the infrastructure.

    What do you mean you "don't believe"? You don't have logs?

    --
    "Our two-party system is like a bowl of shit looking at itself in a mirror." - Lewis Black
    1. Re:Believe? by syntap · · Score: 4, Insightful

      Logs can be faked. How about a bitwise comparison to the known-good package system?

    2. Re:Believe? by 0racle · · Score: 2

      Perhaps this is an early release of the information and given the amount of time they have spent in researching the issue they don't believe anything was actually done, but a more thorough investigation is still needed.

      --
      "I use a Mac because I'm just better than you are."
    3. Re:Believe? by 0racle · · Score: 1

      And yep, that's pretty much what the article says.

      --
      "I use a Mac because I'm just better than you are."
    4. Re:Believe? by new+death+barbie · · Score: 2

      would you trust the logs if you had them?

      --

      It's supposed to be completely automatic, but actually you have to press this button.

    5. Re:Believe? by 0123456 · · Score: 5, Informative

      That's why any secure system should be sending logs to a remote machine as well as /var/log.

    6. Re:Believe? by amicusNYCL · · Score: 1

      Well, I would update my local repository, and then check the change logs to see what's different. So yeah, I would trust that.

      --
      "Our two-party system is like a bowl of shit looking at itself in a mirror." - Lewis Black
    7. Re:Believe? by WetCat · · Score: 2

      Excluding logs before and in the exact time of break-in, while attacker hasn't put his stealth instruments to the victim machine yet.

    8. Re:Believe? by h00manist · · Score: 1

      Logs can be faked. How about a bitwise comparison to the known-good package system?

      Fake information can be even more useful, if you detect it's fake of course.

      --
      Build your own energy sources from scratch. http://otherpower.com/
    9. Re:Believe? by timeOday · · Score: 2
      A compromised server could store the packages unchanged but modify them on the way out.

      The critical piece as I see it is the distribution of the checksums. If package maintainers and end users agree on the checksums (and neither of their systems is initially compromised), then everything should be fine. Or am I overlooking something?

    10. Re:Believe? by jd · · Score: 1

      If you're using Nulfs2 for the filesystem the logs are on, then you can determine if data within the logs have been altered.

      Alternatively, if the logging daemon writes events to a logger on another machine, then logs could only ever be appended to and never altered.

      In this day and age, it seems pitiful that anyone would use a setup where the logs could be faked.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    11. Re:Believe? by Beardo+the+Bearded · · Score: 1

      That was my first thought as well: "Uh, didn't they do a diff and look for the parts that changed?"

      --

      ---
      ECHELON is a government program to find words like bomb, jihad, plutonium, assassinate, and anarchy.
    12. Re:Believe? by Darinbob · · Score: 1

      This is why you should periodically save your game.

    13. Re:Believe? by sleepy_weasel · · Score: 1

      Yep... Make the devs change all passwords, wipe the affected system, and re-install. Or, they can do that thing were you put important data on a non-volatile media and put it somewhere in case you lose a system...

      --
      It's all damned lies and statistics!! I mean 47% of all people use statistics to back up their arguments.
    14. Re:Believe? by 0123456 · · Score: 1

      The logs sent to the other machine wouldn't be trustworthy either...

      Only after the attacker has rooted the box and been able to block further logging of his activities. You can easily edit the log files on a machine and remove old entries showing you doing bad things, but you can't do that wih a remote machine that you have no access to.

    15. Re:Believe? by davester666 · · Score: 1

      Is there a 'known-good' Fedora system?

      --
      Sleep your way to a whiter smile...date a dentist!
    16. Re:Believe? by Anonymous Coward · · Score: 5, Informative

      Logs can be faked. How about a bitwise comparison to the known-good package system?

      As a fedora dev account holder, I got the notification email. The filesystem was compared with a previous 'good' snapshot to determine what changes were made.

    17. Re:Believe? by Chapter80 · · Score: 5, Funny

      However, Fedora officials said they don't believe that the attacker was able to push any changes to the Fedora package system or make any actual changes to the infrastructure.

      What do you mean you "don't believe"? You don't have logs?

      Thankfully, I am on Windows, so I don't have to wonder whether hackers are conducting malicious activity.

    18. Re:Believe? by arth1 · · Score: 2

      Alternatively, if the logging daemon writes events to a logger on another machine, then logs could only ever be appended to and never altered.

      This is why there's still a market for dot matrix printers, especially those with a dip switch that disables reverse paper feed.
      Good luck erasing or modifying that audit trail remotely.

    19. Re:Believe? by BitwiseX · · Score: 1

      Logs can be faked. How about a bitwise comparison to the known-good package system?

      Sorry, I'm busy remodeling my house..

    20. Re:Believe? by jd · · Score: 1

      I accidentally set fire to a dot matrix once - the paper somehow wrapped itself from the output back in, causing the motor to jam and ignite.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    21. Re:Believe? by Coren22 · · Score: 1

      Did you get the printer on fire error?

      http://en.wikipedia.org/wiki/Lp0_on_fire

      --
      APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?
    22. Re:Believe? by eyrieowl · · Score: 1

      nicely done.

    23. Re:Believe? by emocomputerjock · · Score: 1

      Look at his UID, he probably wrote that error.

    24. Re:Believe? by David+Greene · · Score: 1

      Pshaw!

      --

    25. Re:Believe? by HomelessInLaJolla · · Score: 1

      man ps

      I_WANT_A_BROKEN_PS
          Force obsolete command line interpretation.

      --
      the NPG electrode was replaced with carbon blac
    26. Re:Believe? by Penguinclaw · · Score: 1

      However, Fedora officials said they don't believe that the attacker was able to push any changes to the Fedora package system or make any actual changes to the infrastructure.

      What do you mean you "don't believe"? You don't have logs?

      Yeah No spin doctors, we're probably all administrators or more. What exactly has happened?

    27. Re:Believe? by Penguinclaw · · Score: 1

      I'm sure these guy's know all the security protocols as this is Fedora. But what needs to be made open is any security breach that may affect users and the exact nature of this breach. As we all know Fedora is a particularly security conscious OS. This needs to be nailed now!

    28. Re:Believe? by Penguinclaw · · Score: 1

      That makes me warm inside!

    29. Re:Believe? by evilviper · · Score: 1

      There's no need to get superstitious about it. A dot matrix printer is expensive and requires maintenance and consumable. it may be particularly difficult to erase the logs, but you're no better off than hooking up a standalone PC via null modem cable, or even an ethernet cable with a resistor on the TX line so it registers as live , but can't send packets, not even a basic handshake... Make that your syslog server and you're set. Your eats won't bleed during peak load either...

      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  2. again? by russlar · · Score: 1

    didn't something very similar happen last year, too?

    --
    Anybody want my mod points?
    1. Re:again? by bsDaemon · · Score: 1

      I think last year it was CentOS that got hit, not Fedora. Also, the nature of the attack was different and I believe some packages were compromised, or at least the repo signing keys.

    2. Re:again? by 0123456 · · Score: 1

      I think last year it was CentOS that got hit, not Fedora. Also, the nature of the attack was different and I believe some packages were compromised, or at least the repo signing keys.

      It was Red Hat, wasn't it?

      http://www.redhat.com/security/data/openssh-blacklist.html

    3. Re:again? by just_another_sean · · Score: 1

      That little gem can be blamed on Debian actually.

      I love Debian so this one came as a real blow; but they did a good job disclosing it and worked with any and all who wanted to participate to develop the blacklist package.

      --
      Creationist Textbook Stickers Declared Unconstitutional by CowboyNeal
  3. Re:Yay, Open Source! by wagnerrp · · Score: 3, Insightful

    No, they have to Virtual Desktop in.

  4. Re:Rivalry by Anonymous Coward · · Score: 1

    really low

  5. Re:Yay, Open Source! by oodaloop · · Score: 2, Insightful

    And what, interject some bad code? How would anyone know?

    --
    Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
  6. Re:Rivalry by oodaloop · · Score: 2

    1 in 7. As long we're just making up wild accusations, let's just make up some wild numbers.

    --
    Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
  7. Re:Yay, Open Source! by buzzsawddog · · Score: 1

    Well we know that thing can't get to much worse... What would they do? Improve the system?

  8. Re:Yay, Open Source! by Anonymous Coward · · Score: 2, Interesting

    Actually, even Microsoft employees working remotely have to jump through so many hoops for a flaky VPN connection that there is no way anyone could get in for long enough to do significant damages. A Microsoft employee recruiting in colleges on the East coast showed me their system. You can't use a standard VPN client - even the built-in Windows one. It uses SmartCards and multiple passwords for authentication and disconnects if the card even shifts a bit in the card reader.

    It would probably be easier to steal a physical device than to get into their network from outside a Microsoft office.

  9. Not a professional job by Mathinker · · Score: 2

    The first action the intruder took, changing the SSH password, set off an automatic email notification, which is how the compromise was detected. Pretty stupid.

    A pity that the clueless black hats eventually learn, tho. Not that this means that open-source is totally helpless. In the past, malevolent software updates have been caught. If this becomes widespread, it just means that the development is slowed by the necessity for peer review.

    1. Re:Not a professional job by localman57 · · Score: 1

      slowed by the necessity for peer review.

      What credible software organization feels that peer review isn't necessary? Automated testing only gets you so far...

    2. Re:Not a professional job by Mathinker · · Score: 1

      > it is not necessary as it is assumed that software package people will not
      > be introducing security holes into software

      And we've seen how one can be bitten by this assumption, badly (Debian SSH patch-of-entropy-death).

    3. Re:Not a professional job by Mathinker · · Score: 1

      I'd guess that most open-source projects are one- or two-developer deals, max (actually, if you look at SourceForge, you'd end up saying that most projects are zero-developer deals!). However, the most-used projects are much better "staffed", which might mean that there is more of a chance that the people in charge of vetting the commits have some specialized training to catch malevolent changes (and also that more than one set of eyeballs might be looking at every commit).

      In the end, it comes down to a matter of trust. Once a developer has gained a reputation for being trustworthy via having made significant contributions, my guess is that it becomes easier for him to slip bad code into a project. So I suppose the black hats might eventually be a net gain for the open-source movement. On the other hand, I'd guess that organized crime might find it expedient to pay key developers a personal visit, to deliver the famous "offer you cannot resist" --- and once that happens, people might be scared off from contributing.

  10. Re:Rivalry by shadowknot · · Score: 1

    I flashed on that thought but then looking at the numbers on Distrowatch the top 2 are Ubuntu and Mint (a, IMHO, crappy Ubuntu derivative) then Fedora. Unless they want to bump Fedora below OpenSUSE what would be the point?

  11. Re:Rivalry by cronius · · Score: 1

    What are the odds that you are just trolling?

    --
    Life is Reality
  12. Developing story by h00manist · · Score: 1

    These things take time to analyze. Surely they will be finding out more things.

    --
    Build your own energy sources from scratch. http://otherpower.com/
  13. Article and headline are completely wrong by MSG · · Score: 5, Informative

    The infrastructure was not compromised. One user's password appears to have been compromised and changed. That account did not have "high value privileges".

    1. Re:Article and headline are completely wrong by timepilot · · Score: 2

      Or digging further:

      # Push access to packages in the Fedora SCM.
      # Ability to perform builds and make updates to Fedora packages.

      Which I would qualify as high-value.

    2. Re:Article and headline are completely wrong by I8TheWorm · · Score: 2

      Being able to change source code and that code getting pushed into builds which the RE group releases would suggest that it's a bit of a high value account.

      I argued for some time at a previous company that they were out of compliance with Sarbanes-Oxley and segregation of duty rules. The reason was the network admins had access to source code repositories (VSS, StarTeam, and TFS). Since network admins did pushes to QAS and PRD, they could feasably alter source code and push it to production.

      The reality, though, is a developer can put in malicious code and, as long as the rest of the dev team doesn't catch it, it can make it to production regardless. Network admins don't typically have any way of knowing anything about the code that's being pushed.

      All of that makes SOX pretty weak from an IT standpoint. But the end result is this: anyone who can push code to a repository has access to do terrible things.

      --
      Saying Android is a family of phones is akin to saying Linux is a family of PCs.
    3. Re:Article and headline are completely wrong by MSG · · Score: 1

      As far as I know, everyone gets those rights, subject to ACLs on each package that govern who can make changes.

    4. Re:Article and headline are completely wrong by MSG · · Score: 1

      Too big for this case.

      No, they're not. The infrastructure itself was not compromised. One account's password was, and that account did not have privileges to affect the infrastructure.

  14. Re:I need to hand in my geek card... by h00manist · · Score: 1

    I only recently discovered what the hell does "fedora" mean apart from a Linux distro.

    --
    Build your own energy sources from scratch. http://otherpower.com/
  15. Some history by snookiex · · Score: 2
    • Debian: [1], [2]
    • Ubuntu: [1]
    • Gentoo: [1]
    --
    Open Source Network Inventory for the masses! Kuwaiba
    1. Re:Some history by snookiex · · Score: 1

      I could swear I read something about Slackware some years ago, but I've never heard about SuSE/OpenSUSE servers compromised. It's not that they're insignificant, but they simply kept themselves secured or no one ever realized they were compromised. Besides OpenSUSE is among the top five Linux distros I can't get your point.

      --
      Open Source Network Inventory for the masses! Kuwaiba
  16. Re:This never would have happened... by 0123456 · · Score: 1

    This never would have happened if they were running Lin...

    Most likely it's a weak SSH password: SSH and VNC with crappy passwords seem to be the two most common ways to get into Linux machines these days... just open port 22 and watch a million Chinese hackers testing out a bazillion ssh passwords on your machine.

    If Windows actually supported SSH then it would be just as vulnerable.

  17. Re:Yay, Open Source! by undecim · · Score: 4, Funny

    IIRC, something like that has happened before. The attacker managed to get RDP access to one of Microsoft's servers where they keep source code. However, when authorities were able to trace the connection back to his house, they entered to find he had died of a simultaneous heart attack, aneurysm, and stroke, with the Windows kernel source code open on his screen.

    --
    The Internet has given stupid people the resources of intelligent people.
  18. Re:This never would have happened... by 0123456 · · Score: 3, Insightful

    P.S. Of course if they were serious about security in the first place they wouldn't even allow logins with passwords and would require public key authentication instead.

  19. The actual email in case anyone wants the facts by seifried · · Score: 5, Informative

    http://lists.fedoraproject.org/pipermail/devel-announce/2011-January/000746.html

    Summary: Fedora infrastructure intrusion but no impact on product integrity

    On January 22, 2011 a Fedora contributor received an email from the Fedora Accounts System indicating that his account details had been changed. He contacted the Fedora Infrastructure Team indicating that he had received the email, but had not made changes to his FAS account. The Infrastructure Team immediately began investigating, and confirmed that the account had indeed been compromised.

    At this time, the Infrastructure Team has evidence that indicates the account credentials were compromised externally, and that the Fedora Infrastructure was not subject to any code vulnerability or exploit.

    The account in question was not a member of any sysadmin or Release Engineering groups. The following is a complete list of privileges on the account:

    • SSH to fedorapeople.org (user permissions are very limited on this machine).
    • Push access to packages in the Fedora SCM.
    • Ability to perform builds and make updates to Fedora packages.

    The Infrastructure Team took the following actions after being notified of the issue:

    • 1. Lock down access to the compromised account
    • 2. Take filesystem snapshots of all systems the account had access to (pkgs.fedoraproject.org, fedorapeople.org)
    • 3. Audit SSH, FAS, Git, and Koji logs from the time of compromise to the present. Here, we found that the attacker did:
      • Change the account's SSH key in FAS
      • Login to fedorapeople.org

      The attacker did not:

      • Push any changes to the Fedora SCM or access pkgs.fedoraproject.org in any way
      • Generate a koji cert or perform any builds
      • Push any package updates

    Based on the results of our investigation so far, we do not believe that any Fedora packages or other Fedora contributor accounts were affected by this compromise.

    While the user in question had the ability to commit to Fedora SCM, the Infrastructure Team does not believe that the compromised account was used to do this, or cause any builds or updates in the Fedora build system. The Infrastructure Team believes that Fedora users are in no way threatened by this security breach and we have found no evidence that the compromise extended beyond this single account.

    As always, Fedora packagers are recommended to regularly review commits to their packages and report any suspicious activity that they notice.

    Fedora contributors are strongly encouraged to choose a strong FAS password. Contributors should *NOT* use their FAS password on any other websites or user accounts. If you receive an email from FAS notifying you of changes to your account that you did not make, please contact the Fedora Infrastructure team immediately via admin@fedoraproject.org.

    We are still performing a more in-depth investigation and security audit and we will post again if there are any material changes to our understanding.

    --

    Jared Smith

    Fedora Project Leader

  20. Re:I need to hand in my geek card... by VGPowerlord · · Score: 1

    You can't have my Fancy Fedora. :|

    --
    GLaDOS for President 2016! "Well here we are again. It's always such a pleasure." -- GLaDOS, 2011
  21. Re:This never would have happened... by mlts · · Score: 2

    Exactly. For example, any machines I have which have to have an Internet facing ssh port are definitely not going to be accepting passwords. Tools like ssh-guard are nice, but it isn't hard for a determined attacker to just keep coming from different IP ranges. To add a little bit of security, port knocking is a nice ability to have, just so an attacker doesn't see an open port to start having fun with.

    What would be ideal is if OATH support would advance to the point where I can just enter my username, then my password and then the random key from a SecurID or other token. This way, an attacker would have to go from passively looking at passwords as they float by to actively MITM-ing the connection.

  22. Re:Yay, Open Source! by Mitchell314 · · Score: 2

    Careful what you wish for. They could permanently fuse that damn paperclip to the desktop.

    --
    I read TFA and all I got was this lousy cookie
  23. Re:I need to hand in my geek card... by jd · · Score: 1

    Yeah, turns out the mercury used was driving the hatters mad.

    --
    It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
  24. Re:I need to hand in my geek card... by MoriaOrc · · Score: 1

    My first thought was "How will Bogart cope with a compromised fedora infrastructure?!"
    Then I remembered Bogart was dead.
    Then I remembered Fedora is a Linux distro.

  25. Fedora infrastructure intrusion by doperative · · Score: 2

    "The Infrastructure Team took the following actions after being notified of the issue:

    1. Lock down access to the compromised account

    2. Take filesystem snapshots of all systems the account had access to

          (pkgs.fedoraproject.org, fedorapeople.org)

    3. Audit SSH, FAS, Git, and Koji logs from the time of compromise to the present

    Here, we found that the attacker did:

            * Change the account's SSH key in FAS

            * Login to fedorapeople.org

          The attacker did not:

            * Push any changes to the Fedora SCM or access pkgs.fedoraproject.org in any way

            * Generate a koji cert or perform any builds

            * Push any package updates .."

  26. Re:I need to hand in my geek card... by foobsr · · Score: 1

    I only recently discovered what the hell does "fedora" mean apart from a Linux distro.

    You encountered a red fedora?

    CC.

    --
    TaijiQuan (Huang, 5 loosenings)
  27. The good news is... by Anonymous Coward · · Score: 1

    I will be releasing Fedora 15 Desktop Edition next week. Standby for download links.

  28. Nah, Security by obscure VC software by The+O+Rly+Factor · · Score: 2

    Nah, MS uses Team Foundation Server for all of their version control. I have not ever met another single person or company that also uses TFS, nor have I really seen any good documentation on how to use it.

  29. Re:This never would have happened... by dAzED1 · · Score: 1

    or someone made the social-engineering mistake of registering at another site with an email address, and then using the same password at that site as the login for their email address (and in this case, the rest of their fedora resources) uses....ie, could have been an extremely good password. If I have a random 400 character password and then post it on a public internet page, it's not the password that is weak.

  30. Re:Yay, Open Source! by suomynonAyletamitlU · · Score: 1

    to find he had died of a simultaneous heart attack, aneurysm, and stroke, with the Windows kernel source code open on his screen.

    And people say that it's selfish of MS to keep it to themselves...

  31. Why would he change the key? by pseudorand · · Score: 1

    If you compromised an account, why would you change the key, an action that would quite likely trigger some sort of alert (as it did). Wouldn't you just silently look around until you knew what you wanted to do with it and then do all your damage at once before they could cut you off?

    1. Re:Why would he change the key? by Ignacio · · Score: 1

      Because, as they say, "you can't get there from here". Changing the key is the only way to get any further into the system, unless you already have the private key and passphrase.

  32. Re:Yay, Open Source! by jd2112 · · Score: 1

    Are you sure he didn't die laughing?

    --
    Any insufficiently advanced magic is indistinguishable from technology.
  33. Re:I need to hand in my geek card... by Chapter80 · · Score: 1

    I only recently discovered what the hell does "fedora" mean apart from a Linux distro.

    Red Hat Fedora
    Apple MacIntosh (McIntosh Apples)
    Sun SOLARis

    There are a lot of plays on words out there in the tech field.

  34. Re:Yay, Open Source! by undecim · · Score: 1

    Are you sure he didn't die laughing?

    That's an excellent alternative ending.

    --
    The Internet has given stupid people the resources of intelligent people.
  35. Re:I need to hand in my geek card... by angus77 · · Score: 1

    Apple MacIntosh (McIntosh Apples)

    Bonus points for noticing the difference in the spelling!

  36. Weakest Link by bill_mcgonigle · · Score: 1

    The web UI where one uploads SSH public keys, however, uses a password. It was that password that the attacker changed.

    See also: weakest link.

    --
    My God, it's Full of Source!
    OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  37. Re:Rivalry by mrdoogee · · Score: 1

    Nah, if that was true, he would have installed Ubuntu on the system he compromised.