Slashdot Mirror


Package Managers As Achilles Heel

An anonymous reader writes "Researchers from the University of Arizona have released a study that takes a look at the security of ten popular package managers. They were able to show all ten were vulnerable to attacks from a mirror or man-in-the-middle that allow an attacker to (along with other things) crash the system or obtain root access. Furthermore, the researchers created a fictitious administrator and company name and were able to lease a server and get it listed as an official mirror for all the distributions they tried (Ubuntu, Debian, Fedora, CentOS, and OpenSUSE). This raised the question: What keeps you up at night, the thought of attacks on your package manager or previously discussed and patched vulnerability in DNS?" justin samuel (one of the Arizona researchers) also points out a synopsis on CERT's blog.

35 of 263 comments (clear)

  1. Answer by Anonymous Coward · · Score: 5, Funny

    What keeps you up at night, the thought of attacks on your package?

    Yes.

    1. Re:Answer by adminstring · · Score: 5, Funny

      You must have kittens...

      --
      My truck is like a series of tubes.
    2. Re:Answer by jd · · Score: 5, Funny

      The correct modern spelling is kitteh, and you apparently solve that problem by supplying them with elebenty cheezburgers.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    3. Re:Answer by mrbluze · · Score: 4, Funny

      Iran's missile keeps me up at night more than my Ubuntu's package manager.

      That depends on whether you live next door to one launchpad or the the other.

      --
      Do it yourself, because no one else will do it yourself. [beta blockade 10-17 Feb]
  2. I better update now by Krneki · · Score: 5, Funny

    ... oh wait!

    --
    Love many, trust a few, do harm to none.
  3. verify against other sites by speedtux · · Score: 4, Interesting

    I think a lot of these risks could be reduced if people downloaded from one site and verified against one or more other sites. Furthermore, if the checksums were verified over SSL, some attacks would be harder.

    Right now, verifying packages against a site other than the one they were downloaded from seems cumbersome with apt--or does anybody know of a simple command line to do so?

    1. Re:verify against other sites by speedtux · · Score: 4, Insightful

      This doesn't really make it more secure.

      That's the kind of simplistic black-and-white view of security that is responsible for so many security problems.

      Of course it makes it more secure if I verify against multiple sites and over SSL: it protects against many of the attacks described in that paper, so it will be harder for an attacker to mount a successful attack.

  4. Depends on bugs in old software by xZgf6xHx2uhoAj9D · · Score: 5, Informative

    Just in case anyone thought (like me) that the vulnerabilities they're talking about might let an attacker install arbitrary software just through the package manager, this doesn't seem to be the case.

    The attack might go like so:

    1. the attacker finds some really old version of some software that they know to be buggy. E.g., they find OpenSSH version negative 5 or something, which had a terrible buffer overflow bug in it. This software has already been signed (years ago) by the package authorities
    2. the attacker then sets up a mirror with only the broken version of OpenSSH on it
    3. when a hapless Linux user goes to update OpenSSH, your mirror replies saying "the newest version is negative 5. See! I even have a 5 year-old certificate for it"
    4. your client says "...oh...okay"
    5. you install the old, buggy version of OpenSSH
    6. the attacker has your IP address and knows that you downloaded (and presumably installed) the old version of OpenSSH
    7. the attacker haxx0rz you

    The simple fix is to change the client so that it never regresses (e.g., never installs software older than what it already has installed).

    1. Re:Depends on bugs in old software by 99BottlesOfBeerInMyF · · Score: 4, Informative

      The simple fix is to change the client so that it never regresses (e.g., never installs software older than what it already has installed).

      That's a start, but then they can just keep you at the current version and when a new buffer overflow is found and an exploit created, then they hack you. Better validation of mirrors and package managers that check multiple repositories and compare the results are probably a more complete fix.

  5. Re:Compile from source yourself! by speedtux · · Score: 4, Insightful

    How does compiling from source help? Trojans can be introduced in source code just as easily as in binaries.

  6. Sounds real and exploitable.. by moreati · · Score: 5, Informative

    Until I RTFA, I was ready to dismiss this as 'failing to understand signed packages. Wrong, they understand package signatures all too well.

    The basic attacks seems to be.

    1. Obtain old, signed packages.
    2. Become a mirror for debian|fedora|ubuntu|$distro.
    3. Wait for vulnerabilities to be found in some package.
    4. Do not serve the updated packages, continue to serve the vulnerable version.
    5. Log IPs of machines downloading from your mirror.
    6. Root them.

    This works because some package manager software will download and use package metadata even if it's older than what's cached.

    One long term solution would be to sign package metadata and serve it only from one central location, over https/sftp. There may be others.

    Alex

    1. Re:Sounds real and exploitable.. by jmorris42 · · Score: 5, Insightful

      > One long term solution would be to sign package metadata and serve
      > it only from one central location, over https/sftp.

      Even that won't help. The authors got so caught up in the complex exploiting they didn't notice the BIG implication of their work. The problem can't be fixed with tech, crypto or anything but https connects to known to be trusted mirror operators.

      Follow along as I demonstrate. Spamgang wants zombies so they install a massive mirror farm for all of the major distros. They run it perfectly, fully updated with upstream as fast as their phat pipe can get it, perfectly signed metadata, packages and everything offered by http or https. Then they wait.

      Sooner or later another remote root bug, in openssh for example, will hit and they are ready. Thousands of machines either automatically connect or their owners see the story here on /. and hit the update button. They download that signed, correct metadata and sure enough their machines realize they need that new openssh package and ask the mirror for it. And are 0wned a few milliseconds later.

      Because in the act of requesting the package all those machines just told the spamgang that a specific IP is a) running openssh, b) it is the vulnerable version and c) that host is currently connected to the network and very likely has the vulnerable software running. So in the time it takes the updated package to transfer, unpack and install they have ample time to get in and install a rootkit. The beauty is that the victim will patch the hole and thus prevent anyone else from getting the zombie.

      Wait a random time before beginning to use the new zombies to help prevent people from getting wise to what is happening and the spamgang could likely get away with it for years.

      --
      Democrat delenda est
    2. Re:Sounds real and exploitable.. by drinkypoo · · Score: 4, Informative

      It's easier just to firewall all new connections until updates are complete.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  7. Re:Compile from source yourself! by TikiTDO · · Score: 5, Insightful

    Unfortunately it's not nearly as easy to type:
    wget http://blahblah/dep1.tar.gz
    wget http://foobar/dep2.tar.gz (404 Sigh)
    wget http://woobar/dep2.tar.gz
    wget ftp://wowowow/dep2.1.tar.gz ... ... x20

    Followed by:
    tar -zxvf files.tgz x 20

    THEN followed by:
    times ./configure && make && make install x20 IN THE RIGHT ORDER

    Plus however long it takes you to debug the eventual compile problems because one of those obscure libraries that only one program uses was installed earlier with the wrong version.

    All said and done, I'd rather take my chances with package managers, thanks.

  8. Re:Compile from source yourself! by skeeto · · Score: 4, Insightful

    If you are downloading from a mirror, someone could apply similar methods to insert malicious code in your version of the source. And checking all of the source code is no trivial matter, especially if you are looking for an intentionally obscure bit of code.

    Also, you have to trust your compiler, which you *had* to get from someone else. Your compiler may be inserting malicious code.

  9. Re:Vendors sign with keys. by kwalker · · Score: 5, Informative

    What package manager silently downgrades packages?

    I can see a package mirror (maliciously) refusing to stock updates, but yum at least picks a mirror at random by default. Apt didn't last I saw, but if you picked your own mirror, you already trust them.

    --
    ... And so it comes to this.
  10. Re:Compile from source yourself! by 77Punker · · Score: 5, Funny

    You've got him all wrong. He's a twelve year old with a PhD that reads and understand every line of code before he allows it to execute on his own hardware.

  11. Re:Hmmm by Anonymous Coward · · Score: 4, Informative

    Your firewall won't stop this type of attack... RTFA.

  12. Re:Neither by HappySmileMan · · Score: 4, Funny

    I bet your girlfriend doesn't keep you up at night either... It's ok, someday you'll find someone who loves you

  13. Comment removed by account_deleted · · Score: 5, Funny

    Comment removed based on user account deletion

  14. Re:Neither by neokushan · · Score: 4, Funny

    I'm almost certain you didn't.

    --
    +1 IDisagreeSoHeMustBeATrollOrAnAstroturferOrAShill
  15. Slackware by FPCat · · Score: 5, Funny

    Good thing I'm running Slackware!

  16. The actual vulnerability by jmorris42 · · Score: 5, Interesting

    > The article actually discusses attacks that can be made by a malicious mirror...

    Yes a mirror can keep you from getting a security update. But if you don't contact that mirror every day you will eventually get a good mirror and update, and since none of the package managers will downgrade automatically this is a mostly theoretical exploit.

    Yes if a really BIG bug hits somebody could keep some subset of machines from updating, and since they would also KNOW the ip of each vulnerable host it could be very bad. That is the part that worries me, hell, they could even deliver the update from their perfectly up to date repo of signed packages, signed metadata AND perfectly in sync with the distro prime mirror.... and root your ass while the update is in flight. This gets to the real security vuln involved, telling an untrusted entity exactly which version (sorta) of a package you are running.

    --
    Democrat delenda est
    1. Re:The actual vulnerability by nabsltd · · Score: 4, Informative

      There is considerably more metadata than the filename in all package systems.

      Although I generally only use RPM, I have seen enough of others to know that each package has information inside it that says things like "supplies foo-2.8.0.so". For example:
      $ rpm -q --provides openssh
      config(openssh) = 4.3p2-24.el5
      openssh = 4.3p2-24.el5

      This sort of metadata is part of the headers that are downloaded by the package manager to determine whether the system needs the package, and what other packages might be needed to fulfill dependencies. So, the system probably won't even download the file just because it was renamed. But, even if it does download it, when it tries to install it, you would get an error because a newer version of the capability (RPM term) named "openssh" is already installed.

      You could always force the install, but no package manager does this sort of downgrade as an out of the box default

  17. Re:So, Linux is not more secure? by sammyF70 · · Score: 5, Insightful

    The short answer : yes

    The longer answer : every OS is vulnerable one way or another. The difference lies mostly in the response and the response time by the vendors.

    Linux : take the debian ssh disaster a few month ago as example. I read about it at Google News, head over here to check how the linux bashing was coming along, and while I was reading, the "update available" icon appeared. A few minutes later and the vulnerability was no more.
    Admitedly, it took a *VERY* long time to find out about the problem in the first place, but the response time from then on was very short, and the update contained concise information about the whole mess.
    Today's vulnerability will probably take a bit longer to be fixed, as it requires some primordial changes in the way packet manager work to be fixed. But I'm rather sure people are already looking for a solution (you know .. people who actually CAN fix this kind of problems, not your average /. reader)

    Apple Mac : when Apple admits that there is a vulnerability in their products, they take their dear sweet time to fix it. As a matter of fact, Apple just released a security fix for Apple TV, covering vulnerabilities dating back to, at least, January 2008 (at which time it was fixed for OSX, but NOT for Apple TV). I can't comment on how detailed the security fixes are, as I don't own apple products

    Microsoft : the Zero Day initiative still lists 12 issues concerning Apple product, classified as "high severity", but the oldest item is a Microsoft vulnerability dating from September 2006 (more or less quoted verbatim from the iWire article I'll link to a bit later). Microsoft updates are particularly obscure in their descriptions, and, if I remember correctly, they are sometimes even applied without asking the user first, and have a bad habbit of breaking other stuff.

    So, is Linux 100% secure? No, and it will never be. But at least the devs react in a timely manner, and they don't just install something without telling you what it is or that they are patching at all. Therefore it is better secured than Apple and Microsoft products whose vulnerabilities are often left open, for the sake of obscurity I suppose.
    "Superiority" is a highly subjective term, so I won't even start to thread on this subject. It is for me, but your mileage might vary

    Apple TV fix article

    Zero Day Initiative upcoming advisories

    --
    "DRM is like the Ford Pinto: it's a smooth ride, right up the point at which it explodes and ruins your day."-C.Doctorow
  18. Re:Compile from source yourself! by Anonymous Coward · · Score: 5, Funny

    Noob, he doesn't need source. I'm a 13 year old PhD and I assure you even I can read binaries.

  19. Re:Neither by MisterSchmoo · · Score: 4, Funny

    Neither keep me up at night, I have a girlfriend and things to get done.

    Perhaps you should consider changing your girlfriend, mine can always keep me up.

    (for those without sense of humour up is used as erected)

    Do you often find yourself having to explain your erections?

  20. Re:Vendors sign with keys. by alrj · · Score: 4, Informative

    Ok, so foo-1.1 is uploaded on security.debian.org.

    What are the official mirrors for security.debian.org ?
    None. And if you decide to use some mirror anyway, try at least to select one you trust.

    End result : everyone gets the patched foo-1.1

  21. Package are already *signed* by DrYak · · Score: 5, Insightful

    No need for anything as complicated as verifying packages accross several source :

    most modern package managers use key-signing on package.
    You can't setup a bogus repository and start serving malware to unaware users - those packages will fail the key check.

    For that to work, the crackers would also have to find a way to inject their own keys into the ISO of the distribution, and they'll have to find a way doing it that still pass the checksum of the peer-2-peer client with which the users downloaded the ISO.
    That might be possible with older p2p protocols relying on older weak hash, like eDonkey2k whose MD4 has known collisions, or even older protocols lacking checksums. (Some companies working for the media corporation back then used such possibility to pose as peer on the network and poisoning it by injecting bogus packets).
    But distribution currently rely on bittorrent which use SHA-1 hash and it's (currently) much harder to find a way to inject tampered data and have the resulting file still pass the checks.

    Another solution would be to trick the users into accepting a new set of keys to get onto the fake repository. For this, this repository will have to pose not as a mirror (as proposed in the TFA) but as an additional 3rd party repository hold functionalities not available in the original source (this is something that would benefit from the harsh imaginary-properties laws as most distro can't provide packages processing some media, and users have to rely on 3rd party repositories for this).

    Besides, the summary is misleading. They didn't actually setup a bogus mirror that served maliciously crafted files, or otherwise injected malware (that would be impossible given the signatures).
    They simply setup a mirror, that wasn't up to date on purpose, potentially exposing computer to exploit due to only older versions of software being updated.

    As actual legit mirror may lag behind the release, it is nevertheless preferable to always add the original repositories to the list of source : thus get the files from the mirror if available there, or straight from the original website if not replicated everywhere.

    This work around is even useful when there's no malicious intent.

    --
    "Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
    1. Re:Package are already *signed* by Zero__Kelvin · · Score: 5, Interesting
      Actually, what they are saying is that they can set up a mirror with older packages that have known flaws in them and in effect downgrade you from having the latest security fixes to having one with vulnerabilities.

      The packages are still signed with an a valid key. They are just older packages rather than the latest ones.

      I have to give more thought to think if this will work, but I doubt that this has not occured to anyone else. I certainly thought of it before. Most likely the package managers have a way to keep this from happening already.

      Their entire Proof of Concept seems to be:
      1. We asked to be added as a mirror
      2. We succeeded without the distributions doing a cavity search
      3. A11 y0ur L1nux are b3l0ng t0 us!

      I wouldn't panick until I see a CERT advisory, or as someone else pointed out, at least one real world incedent.

      --
      Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
    2. Re:Package are already *signed* by justin+samuel · · Score: 4, Interesting

      Just to be very clear about this one point because it is sensitive: the public mirrors we setup and had listed as official repositories were kept up-to-date and we only served current metadata and packages. We served nothing that was out-of-date (beyond normal mirror update lagtime between rsyncs with the upstream mirrors) and we served nothing that had been modified.

      All tests we performed related to what a malicious mirror could do (e.g. serving out-of-date metadata, serving modified metadata in the case of package managers that don't sign metadata, etc.) were performed using separate mirrors and clients of our own. Nothing we did with our mirrors put users at risk.

  22. Re:Compile from source yourself! by shadwstalkr · · Score: 5, Funny

    Also, you have to trust your compiler, which you *had* to get from someone else.

    Nah, I wrote my own compiler directly in machine code. I didn't trust my keyboard manufacturer either, so I tapped out Morse code on a homemade key. I made the BIOS out of coconuts, but that was just because that douchebag Gilligan said it couldn't be done.

    -Roy Hinkley

  23. Re:Well, sign the catalog then. by Zero__Kelvin · · Score: 4, Interesting

    Actually, it is much easier than that. Simply include the metadata for each package in the signature calculation. Every package already has an increasing number scheme. Once this is done, the filenames cannot be changed, and the package manager simply must not allow numbers to go down instead of up.

    I just verified with my distribution, and the package metadata is not used when calculating the signature, so this is a valid vulnerability.

    --
    Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
  24. AFAICT - Debian is not vulnerable by default by Sun · · Score: 4, Insightful

    As far as I can tell, Debian is not vulnerable in its default install mode.

    a - Debian will never automatically downgrade a package.
    b - Security problems (as opposed to mere changes) get published in the Debian security repository.
    c - The Debian installer automatically adds the debian security mirror to the end of your sources file.

    So, you can create a mirror, easily inject it into the list of official Debian mirrors, and freeze the packages offered on it. You can do all that easily. It won't help you. If you take the CERT example, of the vulnerable openssl package, the fix was pushed through the security repository, and that is added by default to all new Debian installations. The attack would fail.

    Furthermore, even if your mirror offered a security mirror as well, I, personally, always keep the Debian source as well, lower in the list. This way, if the local security mirror is up to date, I take stuff from there. If it's not, I take it from the Debian server. This means that even if you deviate from the default behavior, you can still be not vulnerable.

    Shachar

  25. Re:Well, sign the catalog then. by Zero__Kelvin · · Score: 4, Informative

    I guess keeping the "Interesting" mods is OK so I won't request that they be removed (though I would understand if someone throws a few "overrateds" at me), however I misspoke. Someone else pointed out that the package files, such as yada-yada-2.6.7.myarch.rpm have the important metadata in them already, it is signed, and changing the filename doesn't change the dependency information therin, so there is no vulnerability, and this is much ado about nothing. I stand corrected, and should have stayed with my original post on the matter.

    --
    Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun