Slashdot Mirror


Debian Bug Leaves Private SSL/SSH Keys Guessable

SecurityBob writes "Debian package maintainers tend to very often modify the source code of the package they are maintaining so that it better fits into the distribution itself. However, most of the time, their changes are not sent back to upstream for validation, which might cause some tension between upstream developers and Debian packagers. Today, a critical security advisory has been released: a Debian packager modified the source code of OpenSSL back in 2006 so as to remove the seeding of OpenSSL random number generator, which in turns makes cryptographic key material generated on a Debian system guessable. The solution? Upgrade OpenSSL and re-generate all your SSH and SSL keys. This problem not only affects Debian, but also all its derivatives, such as Ubuntu." Reader RichiH also points to Debian's announcement and Ubuntu's announcement.

141 of 670 comments (clear)

  1. stupid stupid stupid by spikedvodka · · Score: 5, Insightful

    Who did this? You don't remove the seeding... stupid

    did I mention stupid?

    this is how some of the old video games were "broken" despite using "random" numbers, the seed was always the same... leading to the same sequence of events

    --
    I will not give in to the terrorists. I will not become fearful.
    1. Re:stupid stupid stupid by Omnifarious · · Score: 4, Insightful

      Who did this? You don't remove the seeding... stupid

      That's what I was thinking too. What kind of idiot makes that sort of change?

    2. Re:stupid stupid stupid by penfold69 · · Score: 5, Funny

      That's funny, I use the exact same seed on my luggage.

      --
      Beer Coat: The invisible but warm coat worn when walking home after a booze cruise at 3 in the morning.
    3. Re:stupid stupid stupid by moderatorrater · · Score: 2, Interesting

      I also remember some games being broken because you could save, and if the battle didn't go how you wanted it to, you could reload and try again. Nowadays, games tend to save the generator's seed so that things go the same way.

      The point is that encryption, proper randomization, etc are HARD and prone to a lot of errors. That's why you commit changes upstream, so that the people who did the work in the first place are able to review the change and let you know if you changed something vital to the security.

    4. Re:stupid stupid stupid by mikael · · Score: 4, Interesting

      There was a lawsuit story some time ago about a Las Vegas casino vs. a regular punter, over the use of a blackjack game. The punter had found a cabinet unit with a frazzled RNG, so it kept dealing the same sequence of cards every time. The punter managed to memorize the sequence, thus guaranteeing a win every time. Of course, the casino wasn't too happy about this.

      --
      Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
    5. Re:stupid stupid stupid by John+Meacham · · Score: 2, Informative
      --
      http://notanumber.net/
    6. Re:stupid stupid stupid by Yvanhoe · · Score: 2, Interesting

      This could be very smart.
      Planting a vulnerability that has gone undiscovered for 2 years and that gave the keys to a lot of SSH servers. Genius !

      --
      The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
    7. Re:stupid stupid stupid by SiliconEntity · · Score: 5, Informative

      The patch that broke it was checked in by Kurt Roeckx [kroeckx@debian.org]. Don't know if he broke it or if he was just the gatekeeper for checkins. See:

      http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/rand/md_rand.c?rev=141&view=diff&r1=141&r2=140&p1=openssl/trunk/rand/md_rand.c&p2=/openssl/trunk/rand/md_rand.c which shows the change that introduced the bug; and its parent node:
      http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/rand/?rev=141#dirlist which shows the maintainer responsible.

      From looking at this patch, I think this is what happened. valgrind complained about a rather unusual coding convention in ssleay_rand_bytes. This is a function that returns random data into a buffer. However, before writing into the buffer, it reads from the buffer and incorporates the old contents into the internal random state. valgrind complained about this use of an output buffer for input. Normally you would never want to use potentially uninitialized data like this, but in this case it is OK as all that is being done is the data is being folded into the random state. In the worst case, this can't hurt, and maybe it will help, if the old data had some randomness.

      Anyway, valgrind complained about it, and the maintainer commented out the use of the buffer. That would actually be OK, it is not a big deal. But the implementor made a mistake, and also commented out another similar usage, in a different function, ssleay_rand_add. This was a huge mistake, as the purpose of ssleay_rand_add is to add randomness into the random state. In that function, buf is an INPUT buffer, and adding it into the random state is perfectly legitimate, in fact it is the whole purpose of the function. But apparently because it looked similar to the questionable usage in ssleay_rand_bytes, the maintainer commented out the code in ssleay_rand_add at the same time. (I don't know whether valgrind also complained about this second usage, but if so, it was mistaken. I think it's more likely that the maintainer just got fooled and over-generalized from the valgrind complaint.)

      So the whole thing was an attempt to clean up code and remove warnings, but the fix accidentally broke a crucial piece of functionality, rendering ssleay_rand_add completely non-functional. So any attempt to add randomness into the RNG state, such as for seeding purposes, is ineffective. The random state ends up with much less variability, and therefore all the crypto is weak. As Bruce Schneier points out, bad crypto looks much the same as good crypto, so it took this long to notice it.

      Hats off to the reviewer who picked up on the problem. Don't know who it was, but the same Kurt Roeckx [kroeckx@debian.org] checked in the fix.

    8. Re:stupid stupid stupid by Dekortage · · Score: 4, Funny

      Exactly what I was thinking. But it could be interpreted multiple ways: (a) it was criminals; (b) it was terrorists; (c) it was Microsoft.

      --
      $nice = $webHosting + $domainNames + $sslCerts
    9. Re:stupid stupid stupid by bk2204 · · Score: 5, Informative

      It's not seeding too often that's the problem, it's seeding with predictable data and expecting that data to be random. The time is very predictable, and contains very little entropy (randomness).

      If you seed very often with data containing a lot of entropy (for example, radioactive decay), then there's no problem. It's also not a problem to add the time in the mix if you mark it as having almost no entropy.

    10. Re:stupid stupid stupid by Beat+The+Odds · · Score: 2, Insightful

      Interestingly, another source of compromised security is seeding too often. http://lists.debian.org/debian-security-announce/2008/msg00152.html is an example of a vulnerability caused by seeding the random number generator with the current time right before use.

      That's not a case of seeding too often, that's a case of using a crappy seed

    11. Re:stupid stupid stupid by Yvanhoe · · Score: 4, Funny

      How is that mutually exclusive ?

      --
      The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
    12. Re:stupid stupid stupid by ivanjager · · Score: 5, Informative
      Wow, that was still a really stupid patch. There was an #ifndef PURIFY there for a reason. It's because the openssl authors knew that line would cause trouble in a memory debuger like Purify or Valgrind. http://en.wikipedia.org/wiki/IBM_Rational_Purify

      Seriously guys, take a few minutes to understand the code before hacking on it. If it was written by a n00b, you need to understand it to make sure they didn't do other stuff wrong. If it was written by someone competent (which is probably the case for openssl), then you need to make sure you're not the one making the mistake. In this case, to make it work with valgrind, the patch should have been to debian/rules, simply changing the way Configure was called.

    13. Re:stupid stupid stupid by Anonymous Coward · · Score: 5, Insightful

      Lesson#1: It's best to not change code you do not understand without getting it reviewed by people who (are supposed to) understand the code.

      Lesson#2: If you write code that deliberately does weird things like wanting to read unitialised memory, PUT A COMMENT so that people other than you have a fighting chance with your code.

    14. Re:stupid stupid stupid by speedbiker · · Score: 2, Interesting

      Wow, that was still a really stupid patch. There was an #ifndef PURIFY there for a reason. It's because the openssl authors knew that line would cause trouble in a memory debuger like Purify or Valgrind. http://en.wikipedia.org/wiki/IBM_Rational_Purify
      Actually, Kurt did ask on openssl-dev whether this change is acceptable, and did not get any answer that this would be a no-no (his mail and subsequent answers can be seen here).
    15. Re:stupid stupid stupid by this+great+guy · · Score: 2, Interesting

      A similar story is described in the book The Art of Intrusion by Kevin Mitnick. Mitnick interviewed someone who claimed to be part of a group of 4 friends who bought a slot machine, reverse engineered the firmware, the game application, the pseudo random number generator, etc. They found a flaw in the PRNG, were able to determine its internal state by watching the cards dealt so far, and were able to develop an algorithm able to predict the outcome of the game.

      Then they built a small portable device to run their algorithm on-site, in casinos using the exact same slot machine model. It is rumored they won a lot of money this way and took enough precautions to never being caught (I guess they spread their attack over a large number of casinos/cities and multiple years).

      Of course the story is anonymized so can't be verified. And it is in Mitnick's interest to have exciting stories in his book. But IMHO the attack looks totally plausible. Game developers usually have no security background, especially regarding how to design a secure PRNG.

  2. Updated advisory from Ubuntu by Anonymous Coward · · Score: 5, Informative

    Ubuntu got an updated advisory at http://www.ubuntu.com/usn/usn-612-2

    1. Re:Updated advisory from Ubuntu by Anonymous Coward · · Score: 5, Informative
    2. Re:Updated advisory from Ubuntu by this+great+guy · · Score: 2, Informative

      Actually Ubuntu released 3 advisories for this vuln, because the vulnerable code is included in 3 packages. Their permanent links are: OpenSSL, OpenSSH and OpenVPN.

  3. The big question is.. by Idaho · · Score: 4, Insightful

    whether this can possibly be claimed to be an accident *dons tinfoil hat*.

    But seriously, removing the code that seeds a random number generator? I can hardly imagine making such a change by accident. I may just lack a sufficiently colorful imagination, though.

    (or, before resorting to conspiracy theories, we should probably ask ourselves first, "can this possibly be explained by simple stupidity?"

    --
    Every expression is true, for a given value of 'true'
    1. Re:The big question is.. by rhavenn · · Score: 2, Insightful

      (or, before resorting to conspiracy theories, we should probably ask ourselves first, "can this possibly be explained by simple stupidity?" Very easily. Some coder who didn't understand what he was doing and couldn't figure out the code decided to just remove it.
    2. Re:The big question is.. by Anonymous Coward · · Score: 2, Interesting

      What business do they have going that deep within the code? I can understand modifying certain things to make the system more integrated (such as modifying the locations of files), but *WHY* on earth would you fuck around with the cryptographic algorithm? Stupid, stupid, stupid! And I'm not even totally opposed to the idea that this was intentionally done, either.

      This is going to make me seriously reconsider using Debian in the future. I always thought the OpenBSD guys were assholes. Now they're laughing at us.

    3. Re:The big question is.. by sciencewhiz · · Score: 5, Informative

      It was reading from initialized memory to for the seed value, leading to valgrind warnings. See the original Debian bug report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516

    4. Re:The big question is.. by Anonymous Coward · · Score: 2, Insightful

      If it was reading from "un"initialized memory, then OpenSSH was "crusin' for a bruisin'" as my father would say. How long before some OS decides "hey, to prevent process B from 'inheriting' key material, passwords, and other random goodies from process A simply by alloc()ing the space process A used to take up, let's zero out the block of memory that we allocate for each process"? A seed of nothing but zeros, every single time!

      If the memory was initialized, then it wouldn't be an issue for valgrind, and the seed would still be whatever the memory had been initialized to.

    5. Re:The big question is.. by gowen · · Score: 2, Informative

      And the ridiculous thing is, in that thread you quote, at least one person is telling the maintainer that they've done a seriously wrong thing, but the Debian OpenSSL maintainer would rather mindlessly eliminate the valgrind warnings than actually think about the reasons for those warnings.

      (Oh, and you meant "UNinitialized" memory.)

      --
      Athletic Scholarships to universities make as much sense as academic scholarships to sports teams.
    6. Re:The big question is.. by Goaway · · Score: 4, Informative

      Apparently, OpenSSL was using uninitialized memory as a source of randomness. This made valgrind complain about the program, and some enterprising programmer decided to fix it by clearing the memory before use.

    7. Re:The big question is.. by IkeTo · · Score: 4, Informative

      I think the problem is not the removal of the use of uninitialized memory as source of randomness. Instead it is the wrong way it is done, removing nearly all sources of randomness as a result. By the way, the fixed package still don't use the uninitialized memory as source of randomness.

    8. Re:The big question is.. by 0xABADC0DA · · Score: 3, Insightful

      Actually it looks like valgrind was just confused about whether the buffer was initialized. The openssl code for the edg swaps a pointer to be an uninitialized buffer right before calling read with it... maybe this kind of code confused valgrind.

      It isn't easy to find where the function they changed is actually called... it gets added to an array of function pointers and then a static variable is set to it, and then there are various macros that can obtain it. I can see how valgrind could be confused, but in any case there is a function that takes a buffer and adds it as randomness and they nuked it. They totally screwed up because they assumed that valgrind was perfect at figuring out what memory was initialized.

      There is no simple excuse like 'it was uninitialized wanyway', the debian openssl team just blew it, big time.

    9. Re:The big question is.. by Goaway · · Score: 2, Informative

      No, *this* bug. Nothing should ever depend on the value of uninitialized memory, and if clearing it produced a security flaw then there was one there already. Unless it was actually cleared after being filled from /dev/random or something, but then there shouldn't have been a valgrind warning in the first place. Uninitialized memory was only being used as an extra source of randomness. There was a compile-time flag to turn this off to make debuggers stop complaining.

      However, Debian didn't use this flag, but instead tried to comment out the code themselves. And they commented out too much, breaking the whole thing completely.
    10. Re:The big question is.. by makomk · · Score: 2, Insightful

      Nope, that's not quite right. Basically, there were two separate bits of code with two nearly-identical lines, one of which stirred in uninitialised data, and another which was used to add actual randomness. The Debian maintainers incorrectly commented out both lines, rather than just the one they needed to.

  4. Re:It will be fixed by gQuigs · · Score: 5, Insightful

    This problem isn't something you can just update your system to fix. This means the basis for all remote authentication on your Debian/Ubuntu machines is compromised until you go and fix it manually.

  5. Ubuntu Gutsy already updated... by psyopper · · Score: 3, Insightful

    Got up this morning, booted the machine and got a software update first thing: OpenSSH (et al) updates for my Ubuntu Gutsy install. Then I show up over here and see why. Presumably Feisty and Hardy got them as well - they are listed on the Ubuntu announcement.

  6. OSS, only as good as the last developer? by MosesJones · · Score: 5, Insightful

    First off I'm a big OSS supporter, yada, yada

    But the point here is that the freedom that OSS gives you does require you to trust the whole distribution chain. In this case there was an added muppet who did something they shouldn't have thus rendering everything downstream insecure. OSS is great but it required great developers, given that it has take well over a year to get the advisory out it shows that the many eyes piece didn't work here, mainly because the eyes were looking at the original source not the botched packaging job.

    The "easy to use" Linux box in the house uses Ubuntu and has this issue and like many people I didn't even think to check that the OpenSSL wasn't the REAL OpenSSL it was OpenSSL with muppet extensions. Maybe there needs to be some form of extension that warns that a package has been modified from its original source code and that the modification was done by "K. Frog" so you can determine whether to trust that package or look back to the source.

    Or some sort of voting system on contributors (how very Web 2.0) so you can see how the people who touched your package were rated with the biggest weighting being given to the last person through the code (hand edited by Linus = 5 stars, hand edited by James Gosling = 5 stars, hand edited by the bloke who wrote clippy = 2 stars, hand edited by a bloke who removed a seed generator = 0 stars).

    Having the code is great, but this makes me want to know much more about who last edited that code.

    --
    An Eye for an Eye will make the whole world blind - Gandhi
    1. Re:OSS, only as good as the last developer? by Anonymous Coward · · Score: 4, Funny

      Or some sort of voting system on contributors (how very Web 2.0) so you can see how the people who touched your package were rated

      I give anyone who touches my package 5 stars!!!

    2. Re:OSS, only as good as the last developer? by peragrin · · Score: 4, Insightful

      Ah so if the same thing happened from MSFT but no one noticed it does that mean closed source is better.

      No software is perfect. when F/OSS screws up everything including the exact versions of the software where the bug began, until it is fixed is known. You know what/where/when/how, and most of the time why it happened.

      With closed source software your considered lucky if you get a patch in a timely fashion.

      Personally i would rather know what happened and when too.

      --
      i thought once I was found, but it was only a dream.
    3. Re:OSS, only as good as the last developer? by Hatta · · Score: 3, Insightful

      It's debian. ALL your packages have been patched for debian. It's pretty much the only thing I don't like about debian, all the patching. You can get the original source, and you can get the patches, but yeah it would be nice if it were a little more transparent.

      --
      Give me Classic Slashdot or give me death!
    4. Re:OSS, only as good as the last developer? by atomic-penguin · · Score: 3, Insightful

      The great thing about the rpm package management system on Fedora and Red Hat is, that the source rpms always include the pristine, original sourcecode as provided by upstream. Yeah, source debs have that too.
      --
      /^([Ss]ame [Bb]at (time, |channel.)){2}$/
    5. Re:OSS, only as good as the last developer? by Graftweed · · Score: 2, Insightful

      Agreed.

      The parent asked how can we be sure we're using the real OpenSSL? Simple, just use a distro that maintains a policy of minimal interference with upstream code. Slackware comes to mind, and not much else unfortunately.

      Any distro that's liberal about patching upstream code is liable to suffer from security issues, as just exemplified by this story, and also to reduce the value of bug reports to the upstream projects, since they have to go through an extra triage to make sure the bug isn't specific to that distro.

  7. Re:Many eyes make all bugs shallow by sterwill · · Score: 2, Insightful

    Who told you that? Why did you believe it? Please cite your sources.

  8. What's the hurry? by n0dna · · Score: 5, Funny

    It was accidentally introduced in 2006... so that's what, another 14 years before it gets moved into 'stable'?

    *grin*

  9. Re:It will be fixed by JackieBrown · · Score: 2, Interesting

    As far as the bug linked for changes upstream (#477454,) is that seriously the best example of Debian not sharing changes back upstream?

    It seems more like an attempt to show that a maintainer was being petty which is not what the article was about.

    Does anyone one really think that this change would have been accepted upstream (or even have had a place for it there?)

  10. Re:It will be fixed by Archangel+Michael · · Score: 4, Insightful

    It shouldn't need fixing in the first place.

    Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu).

    --
    Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
  11. Re:It will be fixed by rhavenn · · Score: 5, Insightful

    I'm sure the problem will be fixed if the developers acknowledge that the problem exists. Not a big worry. No, but it shouldn't have been changed in the first place. Debian needs to stick their ego up their ass sometimes and just let the people who wrote the software do the coding vs. sticking their own code in in places they don't fully understand. This and their attitude of licensing and not reporting changes back upstream is a stupidly annoying habit.

    note: When I have to run Linux instead of a BSD it's Debian and/or Kubuntu all the way since the benefits outweigh the negatives, but it's still an annoying habit of theirs.
  12. Re:Many eyes make all bugs shallow by n0dna · · Score: 3, Interesting

    http://en.wikipedia.org/wiki/Linus's_law

    I understand he's not unheard of in this arena.

  13. Re:You stupid god damned open sourcers by clang_jangle · · Score: 5, Insightful

    Windows here I come. At least someone would be accountable for shit work like this.


    Yeah, like all those times when MS cut checks for all their customers whose computers were compromised! Oh, wait...
    --
    Caveat Utilitor
  14. Re:It will be fixed by Omnifarious · · Score: 5, Informative

    I'm sure the problem will be fixed if the developers acknowledge that the problem exists. Not a big worry.

    Yes, it is a big worry because any keys generated with this package are now potentially suspect. This means that anybody who's used Debian or a Debian derived distribution like Ubuntu needs to go back and destroy all host and personal keys generated since 2006. All of those keys are potentially guessable.

    And that's a real vulnerability. Early versions of Netscape's SSL implementation (the first SSL implementation) were trivially crackable because of just such a vulnerability.

  15. How Frakin stupid can you be? by nweaver · · Score: 5, Funny

    "You fell for one of the classic blunders, the most famous being 'Never get involved in a land war in Asia' but only slightly less well known is 'Don't use poorly seeded pRNGs in cryptographic protocols!' HAHAHAHAHAHHAHAHAHAHHAHAHAHAHAHA!!!!

    --
    Test your net with Netalyzr
    1. Re:How Frakin stupid can you be? by EricR86 · · Score: 4, Funny

      BUTTERCUP: Who are you?

      MAN IN BLACK: I am no one to be trifled with, that is all you ever need know.

      BUTTERCUP: To think -- all that time it was your cryptographic protocol that was poorly seeded.

      MAN IN BLACK: They were both poorly seeded. I spent the morning downloading a patch to build an immunity to keys being guessed.

    2. Re:How Frakin stupid can you be? by soliptic · · Score: 2, Insightful

      One of the OpenSSL developers (or at least, that's what I infer) puts it in even more general terms:

      never fix a bug you don't understand.

      However, I can't help wondering if some fault may arguably lie with the OpenSSL coders. I mean... by his own admission:

      Usually it is bad to have any kind of dependency on uninitialised memory, but OpenSSL happens to include a rare case when its OK...

      (Emphasis mine)

      So, since it's unusually doing something that looks an awful lot like a Cardinal Sin, did this block of code include a prominent:

      /*
      NB: Yes, we are reading unitialised memory
      This is deliberate, NOT A BUG!
      <explanation here>
      */

      I mean, if you're going to write code that basically looks like a duck, walks like a duck and quacks like a duck, and which you know is going to head downstream toward a huge bunch of duck-hunters, it's really a good idea to add a big visible note saying THIS IS NOT A DUCK.

  16. 2 years? by Anonymous Coward · · Score: 5, Interesting

    The seeding was removed and it wasn't noticed for TWO YEARS? In a distro as popular as Debian?

    1. Re:2 years? by jrothwell97 · · Score: 4, Interesting

      Hang on. There was a story here the other day about a bug in BSD that had been carried down to all its descendants (FreeBSD, OS X, NetBSD etc) for twenty-five years without anyone noticing. Two years is nothing.

      --
      Those using pirated Tinysoft signatures(TM) are a real threat to society and should all be thrown in jail.
    2. Re:2 years? by Chemisor · · Score: 2, Funny

      Open Source programmers are getting older, and many eyes are getting blurry.

    3. Re:2 years? by Free+the+Cowards · · Score: 3, Insightful

      Bugs with differing severities have different priorities, though.

      If 99.9% of client code works, then an obscure bug in BSD directory traversal code is pretty unimportant. It won't affect most people, so it won't be found by most people.

      This kind of bug, on the other hand, severely harms the security of everyone using this distro. It's tricky because it's hard to tell the difference between good crypto and crypto which has a critical flaw at the beginning which makes it insecure, which is why it lasted for so long before being found. But still, one would think that crypto software would get more attention and that someone would have noticed that every affected machine is spitting out nearly identical keys.

      --
      If you mod me Overrated, you are admitting that you have no penis.
  17. Re:Of course... by Oxy+the+moron · · Score: 5, Insightful

    Quit being a cry baby and run 'apt-get upgrade' already. It would have taken you less time than to come in here complain.

    ... and regenerate all the keys, yes? It isn't quite as simple as you suggest...

    "All OpenSSH and X.509 keys generated on such systems must be considered untrustworthy, regardless of the system on which they are used, even after the update has been applied."

    --

    Proudly supporting the Libertarian Party.

  18. Re:It will be fixed by Anonymous Coward · · Score: 4, Informative

    Um.... it doesn't matter if it's been patched, you still have to regenerate all your keys.

    There most certainly is something to see here, this is kind of a big deal.

  19. Re:i wondered what was going on by Anonymous Coward · · Score: 3, Informative

    This has nothing to do with the vulnerability being discussed here.

  20. comics by Anonymous Coward · · Score: 5, Funny

    http://www.random.org/analysis/dilbert.jpg
    http://www.xkcd.com/221/

  21. To non-IT people by Thelasko · · Score: 2, Informative

    This is on the SSH server side. If you are a casual desktop user, you don't have to do anything.

    --
    One of our competitors trademarked the term "hypothesis". From now on, we will call them "boneheaded ideas".
    1. Re:To non-IT people by maxume · · Score: 2, Informative

      It affects all https connections, even client https connections. They are probably transient enough to mitigate the issue, but they are impacted.

      --
      Nerd rage is the funniest rage.
    2. Re:To non-IT people by Sun.Jedi · · Score: 5, Insightful

      This is on the SSH server side. If you are a casual desktop user, you don't have to do anything. Yes, a good observation for IM/Email/Youtube/Facebook crowd... but how many others ssh into their home machine? I'd wager the ability to ssh into a home box is one of the better perks to running linux@home.
    3. Re:To non-IT people by jdowland · · Score: 4, Informative

      Uhhm, no. If you generated a key using the affected packages (ssh-keygen), as a user, your key needs to be replaced.

  22. Re:It will be fixed by Jellybob · · Score: 4, Informative

    Downloading the patch is step one - you also need to regenerate any certificates made with OpenSSL since 2005, since they can't be guaranteed to be secure.

    This has the potential to turn into a huge pain in the arse for Debian based shops, who will need to reissue SSL certificates, SSH keys, and a whole host of other essential elements of their security infrastructure.

  23. Re:It will be fixed by Omnifarious · · Score: 2, Informative

    i was sure that netscape problem was due to the US crypto export regs (the silly encryption=munitions thing) that limited the encryption to 40-bit keys.

    Then you are mistaken. Read the linked to paper. N bit security doesn't protect you much when you can guess N-10 bits of the N bits.

  24. Re:It will be fixed by 2short · · Score: 5, Insightful

    Basic cryptographic services have been compromised for a year and your analysis is to assume on faith that it's open source so it will be fixed, so no problem?

    If someone stole your crypto keys and has had them for a year...

    How thoroughly might they have compromised your system by now?
    How many passwords might they have stolen that you use on other systems?
    What else might they have done that will give them access in the future even after you fix this?

    Just regenerate your keys and no problem? The problem that guessable keys are generated will undoubtably be fixed asap, if not already. The problem that this has been the case for the last year will not be, and is a big worry.

  25. Too early by sakonofie · · Score: 5, Funny
    I realized I probably should be legally required to have a morning cup of coffee before thinking because I am an idiot otherwise.

    I wake up and what do I see first thing? That there is a problem with Debian's OpenSSH package and the /. article links to the following code snippet:

    def init(pipeline, librarian):
              gst.debug_set_default_threshold(gst.LEVEL_ERROR)
    - if gst.element_make_from_uri(gst.URI_SRC, "file://", ""):
    + if gst.element_make_from_uri(
    + gst.URI_SRC,
    + "file:///Sebastian/Droge/please/choke/on/a/bucket/of/cocks", ""):
                      global playlist
                      playlist = PlaylistPlayer(pipeline or "gconfaudiosink", librarian)
                      return playlist

    Now I am thinking, "What exactly is going on here? Is choking on a bucket of cocks not a good source of randomness?"
  26. A great filter by Free+the+Cowards · · Score: 4, Insightful

    Anyone who posts to this story saying that this is no big deal or telling other people to stop whining should simply be banned from Slashdot for life. If you cannot be bothered to read the article and you cannot be bothered to understand just what a serious vulnerability this is but you insist on insulting those who do, why should you be allowed to continue to post your ignorant bleating?

    --
    If you mod me Overrated, you are admitting that you have no penis.
    1. Re:A great filter by Free+the+Cowards · · Score: 4, Insightful

      Your SSL-enabled web sites spent two years being vulnerable to masquerade or man-in-the-middle attacks. Your SSH servers spent two years being vulnerable to same. Your SSH key pairs spent two years being easily crackable by anyone who happened to notice this vulnerability but didn't tell the world. I'm not entirely sure, but I think that SSL and SSH sessions either to or from the affected OSes may be crackable, and if so this would include traffic that was recorded at any time during the vulnerable period and can now be analyzed using knowledge of this bug to find out what data they carried.

      This is a big deal. Maybe it doesn't affect you very much. It doesn't affect me at all, I've never run these distros. But you can bet right now that there are a lot of heavy Linux users out there going through a lot of trouble. This is going to be a bonanza for certificate authorities as everyone who generated SSL keys with these distros is going to need to purchase a new cert.

      --
      If you mod me Overrated, you are admitting that you have no penis.
  27. Surely this is not the only source of entropy! by argent · · Score: 4, Interesting
    Going to http://www.links.org/?p=327 I read...

    OpenSSL happens to include a rare case when its OK, or even a good idea: its randomness pool. Adding uninitialised memory to it can do no harm and might do some good, which is why we do it.
    Uninitialised data doesn't seem to be a good source of randomness to depend on, since depending on where it happens you may consistently end up with a buffer that previously contained all zeroes (or some default memory test pattern), the same part of the same shared library header, or a series of stack frames that for whatever reason happen to be the same frames on every run.

    In fact I'd expect that separate runs of the same program with the same parameters and environment would leave the same junk on the stack every time.

    So I would hope that they have some better source of entropy than unpredictable uninitialized data of dubious randomness.

    So if this is really a serious problem, then it seems to me there's already a serious problem in OpenSSH.
    1. Re:Surely this is not the only source of entropy! by mmontour · · Score: 2, Insightful

      Why not just use /dev/random and stop the stupid games with reinventing the wheel? Probably because the rate at which it dribbles out random bits is too slow for most real-world applications (unless you happen to have it hooked up to a hardware RNG).
    2. Re:Surely this is not the only source of entropy! by argent · · Score: 2, Insightful

      The uninitialised memory there isn't the only source they use, they just use it if it is there because it doesn't matter, and at worst doesn't make the data less random.

      Yes, I get that, you're just restating what I quoted in slightly different words. My point is that it looks to me that at best it doesn't matter because it's not a good source of randomness.

      They then call a randomisation routine. The broken patch commented out both calls.

      Both calls were to the same routine, MD_Update, that added part of the uninitialized buffer to the message digest being built in two different places.

      Don't mess with code you don't understand, especially when it is so important.

      I'm not messing with it, I'm questioning it. If that code is important, then I'd like an explanation of why they think this is a good sort of randomness to depend on.

      If they don't want to use /dev/random, and they don't want to make the user stop and enter some random text (I always faithfully enter gibberish for OpenSSH initialization, don't you?), then they could pick a better fallback than a chunk of data whose randomness depends unpredictably on the execution environment.

      Heck, you can do a lot better by deliberately taking advantage of known variations in the environment: they'd do better hashing /var/log/messages, particularly as soon after a boot as this seems likely to happen, because then at least they'd get a buffer that actually depends on the details of the hardware installation, the IP address, the speed of the processor, the amount of memory, the distro, the enumeration of the PCI bus, the file systems, programs installed, the time and date, and so on... not predictable or reproducible (even if you cloned the hardware and faked the time and IP address network load will change the contents of timestamps unpredictably).

    3. Re:Surely this is not the only source of entropy! by Chris+Burke · · Score: 3, Insightful

      Uninitialised data doesn't seem to be a good source of randomness to depend on, since depending on where it happens you may consistently end up with a buffer that previously contained all zeroes (or some default memory test pattern), the same part of the same shared library header, or a series of stack frames that for whatever reason happen to be the same frames on every run.

      Well yeah, especially because a byte of "uninitialized" memory, like all memory regardless of how many times it has been initialized, is much more likely to contain the value zero than any other. That'd be like using a 6-sided die as a source of randomness when the '1' side was ten times the area of the '6' side.

      However as mentioned, that's not the only or main source of randomness, and getting rid of that randomness was not the bug. It was getting rid of other sources of randomness in the process, because they -resembled- the function that used uninitialized memory.

      --

      The enemies of Democracy are
  28. The issue is a bit more complicated... by Anonymous Coward · · Score: 2, Insightful

    Hmm... no, it shouldn't have been changed. There is absolutely no excuse for touching crypto code when you don't *fully* understand it.

    But the fact that changing the use of NON-INITIALISED memory causes most of the seed of a RNG to go away really is worrisome... unless that memory is actually initialized with a seed BUT in a way Valgrind and others can't track. Which means it is sloppy coding on OpenSSL.

    You cannot trust unitialized memory, PERIOD. And it will take looking deep into OpenSSL code for me to tell if it was intialized in a way valgrind couldn't see, or that OpenSSL was working due to just plain blind luck (or lack of an attacker).

    Either way, upstream prepared the trap, and a debian maintainer sprung it.

    Why do YOU think we are not calling for blood on the guy who did the change? Because code that breaks like that is a travesty by itself.

    1. Re:The issue is a bit more complicated... by Free+the+Cowards · · Score: 3, Informative

      This has been explained elsethread, but just in case anyone wanders into this one....

      The problem is not the fact that the uninitialized memory seeding was removed. The problem is that the removal was done in an incompetent and destructive manner. Rather than remove only the uninitialized memory seeding, the maintainer managed to remove all seeding.

      Reading uninitialized memory as part of random seeding isn't very useful but it's not bad either, since at worst the data is predictable, and OpenSSL mixes it with lots of other randomness. But when you botch your change and completely destroy any randomness present in the system, this is not the fault of the software you were trying to "fix".

      --
      If you mod me Overrated, you are admitting that you have no penis.
  29. Re:i wondered what was going on by sumdumass · · Score: 5, Informative

    No, the random text your put in can be random or not. What they were talking about is a number generator that was used or applied to an algorithm that decided the prime of your key in so that the same algorithm wouldn't be used on two machines on the first then second keys. Without the random number generator, you could theoretically guess the algorithm used to generate the key based of the actions of another install and then decipher the key to gain access to whatever it was protecting.

    To simplify and generalize it, if every machine uses X+1*256 to get a 256 bit key equal to 768, then you could reverse that and know X would =2 (3*256=768) and fake the credentials. The random number generator should change that to X+R*256 which make reversing the key harder because you can only solve to X+R=3 now. In practice, it will be a really larger number and a lot different process though. I can't say that I fully understand it but that simplification should show the difference well enough to give an Idea of where the problem is.

  30. Re:It will be fixed by illumin8 · · Score: 2, Interesting

    This means that anybody who's used Debian or a Debian derived distribution like Ubuntu needs to go back and destroy all host and personal keys generated since 2006. All of those keys are potentially guessable.
    Great... just when I had mostly convinced the PHBs in management that yes, open source software was trustworthy, and that yes, good developers write Linux, and that no, you don't always get what you pay for, and answered all the hundreds of questions about "why do you think they would develop this software for free?" Now some jackass that shouldn't have even been touching this code fucks around with OpenSSL... I'm not trolling, but maybe open source isn't ready for the enterprise. If some coder did this at a company at least I'm pretty confident they'd get their ass fired, but with open source it's basically "whoops, my bad."
    --
    "When the president does it, that means it's not illegal." - Richard M. Nixon
  31. Re:It will be fixed by MSG · · Score: 4, Insightful

    I think the point was that if the Debian maintainer had submitted his patch upstream, he'd have been told exactly how stupid it was.

  32. Re:It will be fixed by eean · · Score: 4, Informative

    Check out this bug report and tell me they weren't just being arrogant:
    http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516

    Says the first comment to the patch, in regards to what is being reversed:
    "What it's doing is adding uninitialised numbers to the pool to create random numbers."

    OpenSSL having more-random numbers vs. Valgrind looking good. And Valgrind won?

    The consensus in the bug report seems to be not to do it, but then someone adds the patch anyways.

  33. Re:It will be fixed by imbaczek · · Score: 5, Informative

    Not entirely true. Keys generated before the patch made it to the repos are safe - and I think quite a lot of debian boxes are old enough (I know I've got one.) There's a link in the advisory to a tool that checks if the keys are vulnerable.

    This doesn't change the fact that this is a really serious fuckup. Debian lost quite some credibility in my eyes.

  34. Don't underestimate corporate arse-covering by zooblethorpe · · Score: 4, Insightful

    If some coder did this at a company at least I'm pretty confident they'd get their ass fired, but with open source it's basically "whoops, my bad."

    I, on the other hand, strongly suspect that any similar mistake at a major software corporation would in all probability be quietly ignored, if it were even noticed at all -- and if it were instead deemed enough of a public relations risk to warrant dealing with, the company would likely just silently push an update to correct the problem for future users, leaving anyone using extant keys with their arses hanging in the breeze.

    But maybe I'm just being overly cynical. :-\

    Cheers,

    --
    "What in the name of Fats Waller is that?"
    "A four-foot prune."
    1. Re:Don't underestimate corporate arse-covering by Omnifarious · · Score: 2, Insightful

      I think you are completely correct and that this is exactly what would happen in most cases with software that came out of a corporation. I don't think you're being cynical at all, just realistic.

  35. Re:interdiff by lukas84 · · Score: 2, Insightful

    Never attribute to malice that which can be adequately explained by stupidity.

  36. Re:Why open source doesn't work for business by ledow · · Score: 3, Informative

    Hello troll,

    You have no recourse unless you have PAID to have one, like with almost everything else in the world. The GPL specifically states that it's not a warranty or guarantee that things will work.

    However, if you were using it in a business, I would hope that you either a) hired your own programmers to work with the code or b) bought a support package/liability clause from someone like Red Hat. In which case it would be down to your programmers or Red Hat respectively. But we're not talking about Red Hat. Or any of the other big-name, support-contract-and-some-sort-of-indemnity-clause-included distros. They don't have this problem, presumably because they are not that stupid.

    We're talking about Debian. Got a support contract / liability agreement with Debian? No? Bad luck. It's a bit like asking Microsoft to accept responsibility for your pirate copy of Windows, then, isn't it?

  37. You are funny by FranTaylor · · Score: 2, Insightful

    Read the end-user agreements for your commercial software and you will see that you have no right to sue them for anything beyond the purchase price of the software.

  38. Bug found & fixed -- "many eyes" seems to work by zooblethorpe · · Score: 3, Insightful

    And we were told this sort of bug could NEVER happen in an open source operating system. Seriously, this is a major cockup.

    First off, we were not told any such thing -- or at least I've never run across any such claims myself. Secondly, this would actually seem to substantiate the many-eyes theory, as the bug was found and summarily corrected, rather than never found, or found and swept under the rug, or found first by black-hats and exploited, all of which seem quite common in the proprietary software world.

    The many-eyes model does not guarantee zero bugs, nor does it guarantee speedy bug hunting. However, it would seem to guarantee that bugs will be found eventually, and also that they will be dealt with in some productive way once they are found. This model aims for the opposite of security through obscurity, that hoary old chestnut of proprietary development, by instead ensuring security by knowing exactly what things do. That also means that folks need to go through all changes -- i.e., many eyes actually need to go over the code. Open source ensures that this is possible, but actual people still need to put in the time and effort.

    Cheers,

    --
    "What in the name of Fats Waller is that?"
    "A four-foot prune."
  39. Re:Of course... by travalas · · Score: 3, Informative

    On ubuntu.... according to the vulnerability description.. Once the update is applied, weak user keys will be automatically rejected where possible (though they cannot be detected in all cases). If you are using such keys for user authentication, they will immediately stop working and will need to be replaced (see step 3). OpenSSH host keys can be automatically regenerated when the OpenSSH security update is applied. The update will prompt for confirmation before taking this step.

  40. Re:It will be fixed by truthsearch · · Score: 5, Funny

    My wifes ssh key is "Debian compromised". Talk about an easy key to guess! At least throw in some numbers or something. ;)
  41. Degree of Compromise? by ewhac · · Score: 5, Interesting
    Does anyone have any back-of-the-envelope calculations as to how badly this compromises existing keys? That is to say, about how long is the brute-force lifetime shortened? If it's been shortened from the age of the known universe to 300 hours, then that's a problem I need to address fairly immediately. OTOH, if it's been shortened to one-quarter the age of the known universe, then I'm not going to deal with this before I've had more coffee...

    Schwab

    1. Re:Degree of Compromise? by multi+io · · Score: 5, Informative

      If I read the published "weak key detector" script correctly, Debian OpenSSHs will always generate one out of a fixed set of 262148 possible keypairs. Do the math yourself. If you know those keys, this is a 5-minute brute force attack.

  42. Is it good enough now? by Animats · · Score: 2, Insightful

    Has someone actually checked the random output for cryptographic soundness?

    The proper source of cryptographic-quality random bits in Linux is "/dev/random". Reading uninitialized memory is not a good source of entropy - it might be initialized to some constant value, especially if you're compiling with a debug allocator or running under a virtual machine monitor.

    OpenSSL makes substantial efforts to get a good random starter. Unless someone did something so stupid that OpenSSL didn't use /dev/random, it should still work. OpenSSL is supposed to have a check for bad random seeds. Was that bypassed, or doesn't it work, or what?

    Obtaining a good source of randomness is hard. Computers are rather deterministic. Historically, there have been major failures in this area. See "Venona" where the USSR was generating "one-time pads" by having people type random digits on typewriters. Arlington Hall, NSA's predecessor, cracked that. Humans aren't random enough. True random number generation requires special hardware, like a noise diode or a radiation source.

    "Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin." -- John von Neumann

  43. correct me if I'm wrong by Deanalator · · Score: 4, Interesting

    From this log: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
    It looks like openssl is pulling in "entropy" from uninitialized memory, causing valigrind to complain. The debian maintainer "fixed" this issue by memsetting the buffer to zero.

    My question is, wouldn't I see the same behavior using grsecurity to scrub deallocated memory? From what I am seeing, this looks much more like the fault of the openssl team.

    1. Re:correct me if I'm wrong by hackstraw · · Score: 3, Interesting

      It looks like openssl is pulling in "entropy" from uninitialized memory, causing valigrind to complain. The debian maintainer "fixed" this issue by memsetting the buffer to zero.

      If you look further down the code, it then fills up the buffer with stuff from some random device (/dev/random /dev/urandom, or other things found in ./configure).

      AFAIK, all modern Linux implementations have /dev/urandom and isn't that enough entropy for a seed?

      I only spent a minute looking that the code and the diff in the parent's post, but this is looking not to be a bug to me. But, like I said that I didn't look at the code that long, but I don't see where memsetting the memory makes any difference if its all being overwritten.

    2. Re:correct me if I'm wrong by annodomini · · Score: 4, Informative

      No, I don't think you're quite correct. The original patch in that thread is not what actually was applied. There are two functions, one for seeding the pseudo-random number generator, and one for getting random data out of it. The one for getting random data out was actually mixing in some uninitialized data into the entropy pool. This is not particularly harmful, but not particularly helpful either, and it caused Valgrind to complain. So, the maintainer removed that line, but also removed the line that mixes in the value provided to seed the random number in the first place (maybe it was being called with unitialized data at some point, too).

      See my comment here for more details: http://reddit.com/info/6j7a9/comments/c03zxko

  44. Re:Why open source doesn't work for business by Experiment+626 · · Score: 2

    So who's accountable for damages as a direct result of such a problem? If I were using such software to run my business, and this sort of security problem became more than just a threat, what sort of recourse do I have? Which programmer do I get to sue?

    As opposed to the closed source world? If a security bug were discovered in Windows, would Microsoft would compensate you for your damages instead of just disavow any liability in a EULA?

  45. What does "guessable" mean here? by Kent+Recal · · Score: 2, Insightful

    Can someone explain what "guessable" means in this context?

    Does that mean someone can now generate a "master-key" and ssh into root@any-debian-box that allows public key authentication?
    What does a realistic attack scenario look like?

    1. Re:What does "guessable" mean here? by Yvanhoe · · Score: 4, Informative

      From what I undestood, with my limited cryptographic knowledge and my total reluctance to dig into all the details of the sources, what happens is that when you asked your debian box to generate a key, it did so without integrating enough random data in its choice. That means that a brute force attack will not have to try every key to find yours but only a small subset, depending on the knowledge it has about your machine and its state when it generated the keys.
      There is no such thing as a master key, but it would mean that several different servers may have generated the same "random" keys and that a clever attacker could create a dictionary of plausible keys (or, more plausibly, an algorithm to generate them) and try all of them on your server with a good chance to have one of them working. The dictionary would probably be huge however. A good measure would be to block an IP on SSH after a reasonnable number of failures (like 100)

      --
      The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
    2. Re:What does "guessable" mean here? by Kent+Recal · · Score: 2, Interesting

      Thanks to you and the other guys who responded and confirmed.
      I have now patched my firewall and ssh config to (hopefully) limit any bruteforcing to one attempt per minute: /etc/sshd/sshd_config
      MaxAuthTries = 1 /root/bin/firewall
      iptables -I INPUT -p tcp --dport 22 --syn -m limit --limit 1/minute -j DROP

      Ofcourse what remains is the question of how many attempts would be realistically needed to break a weakened key?
      I'm eager to see the first proof-of-concept exploit.

    3. Re:What does "guessable" mean here? by LordSnooty · · Score: 2, Informative

      SSHDFilter may help in this regard. Everyone should be taking steps to limit brute-force ssh attacks even without this current issue.

    4. Re:What does "guessable" mean here? by elronxenu · · Score: 2, Informative

      What you are doing here is dropping only one packet per minute (with a maximum burst of 5 by default). The opposite of what you intended.

      To make it work the way you intended, you need two rules:

      iptables -I INPUT 1 -p tcp --dport 22 --syn -m limit --limit 1/minute -j ACCEPT
      iptables -I INPUT 2 -p tcp --dport 22 --syn -j DROP

      The limit rule runs first, accepting an average of 1 syn packet per minute. If the rate is exceeded then the rule fails and the DROP rule runs, which drops all further attempts to connect to ssh.

  46. Re:It will be fixed by profplump · · Score: 3, Insightful

    It's not arrogance. Or it's at least not arrogance alone. As a distro maintainer I can tell you that upstream providers generally do not care about distro-specific patches. Even if you have a patch that would be useful on more than on distro, there's often a reluctance to inherit code to support any installation method other than source tarballs.

    That's not to say that upstream providers are unduly arrogant either -- if you're happy with the way your build/install system works, why would you want to maintain patches for some other system that you don't even use? It's extra code to keep up, and requires more testing on more platforms, and it it doesn't make your core package any better.

  47. Re:Of course... by goofy183 · · Score: 4, Informative

    But that doesn't fix all of the machines that my public key (generated on Ubuntu) resides on that do not run Ubuntu. So yes fixing Ubuntu machines is easy, it is a PITA to have to go and re-upload my public key to all of these hosts.

  48. Re:It will be fixed by MikeyO · · Score: 4, Informative

    I think the point was that if the Debian maintainer had submitted his patch upstream, he'd have been told exactly how stupid it was.


    Actually, here is where the Debian maintainer brought this up with upstream, and he was not told how stupid it was, in fact the Debian maintainer says, "the pool might receive less entropy." and upstream says, "If it helps with debugging, I'm in favor of removing them."
  49. Re:It will be fixed by Aaron+Denney · · Score: 4, Informative

    Actually, any DSA key *used* on a box with the bad SSL packages may be compromised:

    From http://wiki.debian.org/SSLkeys

    Additionally, some DSA keys may be compromised in the following situations:

            * key generated with broken openssl = bad
            * key generated with good openssl and used to ssh from a machine with bad ssl = bad
            * key generated with good openssl and used to ssh from a machine with good ssl = good

    This is because the random numbers used during the signature process must also be good.

  50. Re:It will be fixed by dondelelcaro · · Score: 2, Informative

    The consensus in the bug report seems to be not to do it, but then someone adds the patch anyways.

    The package maintainer first checked with upstream about the best way to resolve this. In retrospect, it's clear that upstream didn't either understand what was being asked, or what the code was doing. In any event, another Debian Developer, Luciano Bello, later found the problem and resolved it.

    --
    http://www.donarmstrong.com
  51. Re:Of course... by Just+Some+Guy · · Score: 4, Informative

    So yes fixing Ubuntu machines is easy, it is a PITA to have to go and re-upload my public key to all of these hosts.

    Especially when they won't work anymore:

    Once the update is applied, weak user keys will be automatically rejected where possible (though they cannot be detected in all cases). If you are using such keys for user authentication, they will immediately stop working and will need to be replaced (see step 3).

    The GP poster said the same thing, but this is for the benefit of other readers who were skeptical that any such policy was in place.

    So, basically, once you upgrade, you'll have no apparent way to access your other machines [1] to upload your new key. That's just spiffy!

    [1] Uninstall openssh-blacklist first.

    --
    Dewey, what part of this looks like authorities should be involved?
  52. Re:It will be fixed by dotancohen · · Score: 5, Funny

    My wifes ssh key is "Debian compromised" Say thank you that your wife is at least that far. If I dare say ssh in the wife's presence, I get a vase thrown at me and she screams for me to shut the fuck up.
    --
    It is dangerous to be right when the government is wrong.
  53. Re:It will be fixed by IamTheRealMike · · Score: 4, Interesting

    He would have been told, but who says he would have cared? This happens all the time, although usually it doesn't result in a giant fucking security hole. I was warning people this sort of thing would happen more than two years ago back when I was working on autopackage. It was entirely predictable.

    Having distributors randomly change source code as they package it is fundamentally broken. It cannot ever work. It's not just OpenSSL that's been broken like this. Wine often went through periods in which the Debian packages were completely hosed in subtle ways. It would look like a bug in Wine, but was in fact due to parts of the software "going away", because the Debian packager felt they weren't important enough to be a part of the base install. Pointing out that their change was broken didn't help. The result was many, many hours wasted tracking down non-bugs. Then you go through a giant waste of time trying to get the bug in the packages fixed, and just as you finally get it sorted out, some other distribution introduces some other bug into their packages.

    So I got sick of this, and started telling people to avoid their distros packages. Some distributors (especially Debian) took it personally and started childish slanging matches. But really, who cares how "integrated" a program is, if it could have had arbitrary bugs silently introduced? It's just a busted model of software distribution.

    Imagine if Microsoft reserved the right to modify any software for Windows in any way it saw fit! Yet that's exactly what Debian (and Fedora and Mandrake and Ubuntu) said to me - they reserve the right to make any modifications they like to the software they ship, and if upstream don't like it, tough luck.

    I wonder if they'll re-evaluate that policy in light of this disaster. Probably not.

  54. Re:It will be fixed by fwarren · · Score: 3, Insightful
    I guess the real question is how many compromised keys are out there?

    Most users do not generate nor use ssh/ssl keys or certificates.

    The real questions are

    • How many clueless users are actively using these keys/ceritificates?
    • How many cluefull users are using these keys and think they are not an issue or that their keys are ok and won't fix it?
    • How many cluefull users are going to fix their keys/certificates?
    • Is someone able to develop a workable exploit?
    • Will it be worth anyones time with a working exploit to scan the internet for compressible systems?
    --
    vi + /etc over regedit any day of the week.
  55. Kurt Roeckx (debian) mail to openssl-dev list by omz · · Score: 3, Interesting

    Found this post at openssl-dev list by Kurt Roeckx ( AFAIK the Debian OpenSSL Team member that made this RNG-clean patch )

    http://www.mail-archive.com/openssl-dev@openssl.org/msg21156.html

    Extract:

    "What I currently see as best option is to actually comment out those 2 lines of code. But I have no idea what effect this really has on the RNG. The only effect I see is that the pool might receive less entropy. But on the other hand, I'm not even sure how much entropy some unitialised data has.

    What do you people think about removing those 2 lines of code?

    Kurt

    "

    BTW, i thought that Debian had some kind of policies about testing each package before committing changes in testing/stable branches. Also, the following paper, contributed by another poster, says interesting things about touching cryptographic code, we have to learn from this experience and have tighter policies !

    " In a narrow sense, the security flaw we found in the Netscape browser serves merely as an anecdote to emphasize the difficulty of generating cryptographically strong random numbers. But there's a broader moral to the story. The security community has painfully learned that small bugs in a security-critical module of a software system can have serious consequences, and that such errors are easy to commit. The only way to catch these mistakes is to expose the source code to scrutiny by security experts.

    Peer review is essential to the development of any secure software. Netscape did not encourage outside auditing or peer review of its software-and that goes against everything the security industry has learned from past mistakes. By extension, without peer review and intense outside scrutiny of Netscape's software at the source-code level, there is simply no way consumers can know where there will be future security problems with Netscape's products. "

    http://www.cs.berkeley.edu/~daw/papers/ddj-netscape.html"

    1. Re:Kurt Roeckx (debian) mail to openssl-dev list by grumbel · · Score: 3, Informative

      i thought that Debian had some kind of policies about testing each package before committing changes in testing/stable branches. This is a common myth, but Debian doesn't do any explicit testing, they pack things into unstable and then just let them go into testing/stable if no bug reports come in. There is no standard code audit or anything going on.
  56. Ongoing problem for host keys? by Urban+Garlic · · Score: 2, Interesting

    Doesn't this mean that all the "etch" host keys are vulnerable, too, on an ongoing basis?

    SSH host keys are generated at sshd install time, and that package is included on the netinstall CD-roms, which means every time I re-use my 4.0r1 netinstall CD to set up a new box, I generate a new set of vulnerable host keys before I get to the security-updates step.

    Hopefully, they'll advance the next point release and a 4.0r2 netinstall CD will do better.

    --
    2*3*3*3*3*11*251
    1. Re:Ongoing problem for host keys? by Serious+Callers+Only · · Score: 2, Interesting

      I believe the security update will automatically prompt you to regenerate the host key (it did for me at least on ubuntu), so it shouldn't really be an issue.

      If the host keys are vulnerable on a server, doesn't this just leave you open to a man in the middle attack anyway? That's not as serious as generating keys you use to login with, in which case you need to regenerate those as soon as possible. Is it possible to break into a server just by guessing the host or server keys?

  57. Passwords may be compromised, too by Krischi · · Score: 4, Informative

    It is even worse than this. Passwords that were used on a server with a weak DSA key may be compromised, as well:

    From http://wiki.debian.org/SSLkeys

    1. Re:Passwords may be compromised, too by embsysdev · · Score: 2, Informative

      Thanks, that was very informative. Also, here is the text from a Gutsy upgrade:

      Vulnerable host keys will be regenerated

          Some of the OpenSSH server host keys on this system were generated with a version of OpenSSL that had a broken random number generator. As a result, these host keys are from a well-known set, are subject to brute-force attacks, and must be regenerated.

          Users of this system should be informed of this change, as they will be prompted about the host key change the next time they log in. Use 'ssh-keygen -l -f HOST_KEY_FILE' after the upgrade has changed to print the fingerprints of the new host keys.
          The affected host keys are: /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_dsa_key

          User keys may also be affected by this problem. The 'ssh-vulnkey' command may be used as a partial test for this. See /usr/share/doc/openssh-server/README.compromised-keys.gz for more details.

  58. Re:It will be fixed by IamTheRealMike · · Score: 4, Insightful

    The problem is that the email is rather misleading, from what I understand. It talks about two places causing the problem even though only one of them actually was making valgrind complain (and you can see, purify already picked up on the problem some time before). The trick is that his patch, not included in the email, removed both calls. Which was busted. So if he'd actually sent a full patch, proposed it for inclusion and had it properly code reviewed, the mistake would probably have been caught.

  59. Re:Why open source doesn't work for business by Minwee · · Score: 3, Insightful

    It's a bit like asking Microsoft to accept responsibility for your pirate copy of Windows, then, isn't it?

    Or a bit like asking Microsoft to accept responsibility for the copy of Windows that you paid for.

    Look through the Windows license agreement some time and see how many times phrases like "THERE IS NO WARRANTY OR CONDITION OF ANY KIND", "YOU ARE NOT ENTITLED TO ANY DAMAGES" and "IN NO EVENT SHALL MICROSOFT OR ITS SUPPLIERS BE LIABLE" show up.

    The only difference with Microsoft is that if Windows breaks you don't get to keep both pieces.

  60. Re:It will be fixed by eean · · Score: 3, Informative

    Ben Laurie of OpenSSL/Apache puts it into some context:
    http://www.links.org/?p=327#comment-176642

    Obviously some of the OpenSSL devs probably should've been like "zOMG, SITUATION FUBAR", but it wasn't a formal code review being requested, more of a "hey, what do you think of this and this?" and the patch was never submitted to upstream.

  61. Some important information by xtronics · · Score: 3, Informative

    You will want to download this tool to check each system:

    http://security.debian.org/project/extra/dowkd/dowkd.pl.gz

    To use it:

    $ perl dowkd.pl user

    $ perl dowkd.pl host localhost

    See http://lists.debian.org/debian-security-announce/2008/msg00152.html

    Why isn't such a scan isn't part of the package update? http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=481106

    1. Re:Some important information by xtronics · · Score: 2, Informative

      To change-out server keys:

      Had to change out all the other machines.

      if you

      $ cd /etc/ssh
      $ rm ssh_host*
      $ wajig reconfigure openssh-server

      changing user keys - become user and ssh-keygen

      You may have to fiddle with knownhosts and authorized_hosts as documented many places..

  62. 262148 possible keypairs? by multi+io · · Score: 5, Informative
    reading the weak key detector script...

    ...it looks like the affected Debian OpenSSHs can generate only 262148 (or something) distinct keypairs. Can this be true? If it is, this is fucking serious.

    1. Re:262148 possible keypairs? by multi+io · · Score: 3, Informative

      From the same script:

      Note that the blacklist by dowkd may be incomplete; it is only intended as a quick check. With a statement as vague as that, there could be several orders of magnitude more potential keys. I have an affected system here, and when I use its ssh-keygen to generate a few hundred keys, the script reports them all as "weak"[1], i.e. they are all in that set of 262,000. Looks like the script is at least quite good at what it does, and/or if there are more potential keys, they are rarely generated? When I do the same on a Fedora box here, none of them are reported, as expected.

      260,000 would be bad though. At one test per second, that's just over 3 days. Of course, if you blacklist hosts after a certain number of failed authorization attempts, you can greatly increase the difficulty of exhausting the keyspace for a potential cracker.

      ...unless the attacker can obtain the public key directly, which he should be able to do frequently (it's called "public key" for a reason). For example, you can query a host for its public host (SSH) key without a problem. So, unless I'm missing something, you could just obtain that key and match it locally against those 262,000 keys (takes a minute at most, if you have all those keys) in order to find out the private key.

      [1]
      $ for i in `seq 1 500`; do ssh-keygen -N '' -t rsa -f key$i; done
      $ ./dowkd.pl file key*.pub | grep 'weak key' | wc -l
      500
      $

  63. Re:It will be fixed by KutuluWare · · Score: 2, Informative

    I think you misread his statement. He didn't say your "machine" was compromised. He said that the underlying authentication mechanism was compromised, which is quite different. If I were to leave my car doors unlocked, the engine running, and the alarm disabled, then the security system on my car would be compromised; but until someone came along to actually hop in and drive away with it, the car itself would still be working properly.

    Similarly, the fact that there is a problem with the OpenSSH package means that the security mechanism that is guarding access to your system is no longer as secure as it was previously thought -- that makes it compromised. The individual systems may or may not have been compromised themselves by way of the compromised security mechanism, but continuing to use OpenSSL knowing it (the key gen algorithm, specifically) is less secure than it should be is just asking for trouble.

  64. Re:It will be fixed by IamTheRealMike · · Score: 4, Insightful

    No, he posted a question openssl-dev, which is a mailing list for people writing software that uses OpenSSL. This OpenSSL developer doesn't read it and that's similar with most open source project - developers often don't read mailing lists for end users.

    What's more, that mail doesn't contain a patch. It contains a misleading question with two lines posted in isolation. An actual patch, submitted for an actual code review, would almost certainly have revealed the problem via context.

    You don't change crypto code of all things based on an idle question on a mailing list populated mostly by users. What's next, changing the kernel scheduler based on a conversation in #kernel-newbies?

  65. Re:It will be fixed by IamTheRealMike · · Score: 2, Insightful

    Right, that's why Debian renames Firefox. I remember them not being very happy about being forced to do that at the time. But seriously, are all open source projects supposed to use trademarks to force distributors hands? It seems like a bad precedent.

  66. Re:It will be fixed by Sir_Real · · Score: 2, Informative

    No, but it shouldn't have been changed in the first place. Debian needs to stick their ego up their ass sometimes and just let the people who wrote the software do the coding vs. sticking their own code in in places they don't fully understand. WTF? This bug is the result of a Debian guy saying "Hey, this doesn't look right, can you fix it?" The PACKAGE MAINTAINER (Not the DEBIAN GUY) "fixed" it for Debian (though for obvious reasons that fix won't propogate any further) and got us here. Speaking of "don't fully understand..." Would you maybe read .... prior to ....

    ah fuck it
    this is slashdot
  67. Re:It will be fixed by noahm · · Score: 4, Informative

    No, he posted a question openssl-dev, which is a mailing list for people writing software that uses OpenSSL

    Incorrect, no matter what Ben claims. According to http://www.openssl.org/support/, openssl-dev is for the developers of openssl itself. To quote the list description: Discussions on development of the OpenSSL library. Not for application development questions!

    So yes, perhaps Kurt could have been more explicit in his description of what he was trying to do, but he was definitely using the appropriate address to reach the developers.

    noah

  68. Re:Does SSH not use /dev/random? by vrmlguy · · Score: 5, Informative

    The problem is, OpenSSL needed a buffer in which to XOR several sources of randomness, including /dev/random. OpenSSL didn't bother initializing the buffer, because doing so just eats CPU cycles, and a possibly random buffer is marginally better than a non-random one. However, the Debian patch removed *all* of the randomness except for the PID, leaving the user with one of just 262148 (or so) distinct keypairs. The tool that checks if your keys have been conpromised has a list of all of those keypairs and checks if your keypair appears.

    --
    Nothing for 6-digit uids?
  69. Once again, shoddy code leads to errors by metamatic · · Score: 2, Insightful

    But the implementor made a mistake, and also commented out another similar usage, in a different function, ssleay_rand_add. This was a huge mistake, as the purpose of ssleay_rand_add is to add randomness into the random state.

    Given that the code is so shoddily written that neither of the functions has a comment to say what it's supposed to do, I think I can forgive him for that.

    --
    GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
  70. To update the host keys on servers by vrillusions · · Score: 2, Informative

    Since they didn't mention how to do this in the security advisory, or give you a warning message when you apply the update that you should regen your keys here's how to regenerate the host keys. Either log in as root or put sudo before each of these (this should work remotely, I didn't have problems but YMMV)

    rm -f /etc/ssh/ssh_host_dsa_key
    rm -f /etc/ssh/ssh_host_rsa_key
    ssh-keygen -b 1024 -N '' -f /etc/ssh/ssh_host_dsa_key -t dsa
    ssh-keygen -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key -t rsa
    /etc/init.d/ssh restart

    You'll then be asked about the key changing next time you log in.

    I just spent all day updating servers. Doesn't sound so bad till you realize all these servers also had client keys used for remote backups that had to be recreated and verified, etc.

  71. Re:It will be fixed by SiliconEntity · · Score: 3, Informative

    According to http://www.openssl.org/support/, openssl-dev is for the developers of openssl itself. To quote the list description: Discussions on development of the OpenSSL library. Not for application development questions!
    Right, and further, one of the participants in the discussion was Ulf Moeller, who is one of the main crypto developers in OpenSSL. Unfortunately nobody apparently bothered to check whether the two lines proposed for removal were both involved in this questionable behavior of folding uninitialized data into the random state. Actually, only one of them was being used that way, the other was adding very important data to the random state. The OpenSSL people never caught the fact that the proposed change to the two lines was going to kill the security of the system, even though they had all the information available to do so.

  72. Don't forget the authorized_keys files by halsathome · · Score: 2, Informative

    Regenerating keys is fine, but the important step is clearing out the authorized_keys file that contains the old keys :-). At the same time take the opportunity to clear out the keys belonging to that long-gone co-worker.

    If you try logging in with a new key where an old one is expected, you will get an error and remove the old one from authorized_keys. When the client is never in use however, the hole might stay in there for a long time ...

  73. I have an eCommerce site by an.echte.trilingue · · Score: 3, Interesting

    I am going to hijack this thread because I have an eCommerce site hosted on Debian and this made me nervous.

    According to the reps at Thawte, if you are using a third party ssl cert (thawte, verisign, etc), this does not affect you. According to them, this is only dangerous for people who have generated their own SSL certs for their sites from the ground up (probably a minority, I would guess). If anybody has any information to the contrary, please let me know.

    --
    weirdest thing I ever saw: scientology advertising on slashdot.
    1. Re:I have an eCommerce site by Mysteray · · Score: 3, Interesting

      https/ssl/tls/ssh etc use the certificates to agree on a conventional "session key" every time you connect. Session keys are typically chosen with strong random numbers... but if either the client or server is using the OpenSSL from Debian Etch, there's nearly no randomness in it.

      Even if your certs are fine, individual transactions are likely vulnerable.

      If someone has taken packet captures, they will likely be able to decrypt them retro-actively.

  74. Ubuntu Dapper 6.06 LTS fine, too by zzatz · · Score: 2, Informative

    As with Debian Sarge, Dapper uses a version that predates the dangerous patch.

  75. Re:It will be fixed by BooRolla · · Score: 5, Funny

    Don't go bragging. A lot of people know the key to rooting your wife's box.

  76. Re:It will be fixed by dissy · · Score: 2, Insightful

    I'm not trolling, but maybe open source isn't ready for the enterprise. Well, if you want 'enterprise' you should be using real enterprise software, like a solution from IBM, Sun Microsystems, or RedHat Linux.
    These companys will sell you a contract that DOES give your business recourse when something goes wrong.

    Most businesses don't need enterprise software though, so they stick with linux, bsd, windows, and mac os.

    Great... just when I had mostly convinced the PHBs in management that yes, open source software was trustworthy, and that yes, good developers write Linux, and snip Also it's worth pointing out, this one bug has zero to do with linux or opensource. OpenSSL IS secure. One guy broke the package and introduced this problem.
    It is still a bad problem, and there are alot of debian users, but just compared to all the linux distros out there, it is still only a percentage and under 100% (by Far under 100)

    What you are saying is basically that because I personally can download an opensource program, change (aka break) it, and give it to someone, that opensource in whole is broken, untrustworthy, and bad.

    Clearly that is just stupid.

    I suppose if you don't trust openssl anymore, for a problem that effects only debian users, compared to MS's bugs that give admin access to anyone on the internet from win NT4 up to Vista SP1, then that is your call. (I am curious how you might justify that to your PHB though)
    http://www.microsoft.com/technet/security/bulletin/MS08-001.mspx

    Thats akin to saying "I no longer trust the guy at the store since he shorted me $0.03 in change, so i'm gunna just trust the crackwhore at the corner to by the same stuff from."

  77. The human kind of idiot by Chuck+Chunder · · Score: 2, Insightful

    Frankly the guy probably doesn't need ragging on (in fact people should be checking he's ok).

    A small coding mistake has had rather nasty and rather public consequences.

    What does bear scrutiny is the process. In such a critical package should it really come down to just one guy? On such a critical package should Ubuntu not have their own man double checking everything rather than just accepting changes from Debian.

    You can't make people infallible, but you improve the system to better mitigate such fallibility.

    --
    Boffoonery - downloadable Comedy Benefit for Bletchley Park
  78. Re:It will be fixed by AigariusDebian · · Score: 4, Informative

    I cann't claim to 100% understand the situation but after glancing trough the logs of the discussions and of the patches the conclusion I came to was this - OpenSSL used supposed randomness of the uninitialized memory as an added source of entropy (interesting hack, but not an example of good coding as such). Valgring caught that problem and the Debian maintainer during a cleanup fixed it. Making such a fix can be considered a preventive step against possible attack vectors by poisoning the uninitialized memory. He took it up to upstream, they did not raise red flags, but did not quite merge the âclean upâ(TM) patch either. It fell through the cracks.

    The problem is that in the same file, in another function all other sources of entropy were being merged into the pool of randomness using exactly the same code line as the one code line flagged by Valgrind. The maintainer assumed that the second code line has a similar function to the first and commented that one as well. AFAIK that also did not show up in the emails to the upstream list.

    So we have:

            * Upstream using clever hacks that rely on uninitialized memory having some randomness to it
            * Upstream using same code and same variable names to describe different things
            * Upstream having no comments in the code explaining the two things above
            * Maintainer slightly over-generalizing a change
            * A bug slipping trough the cracks in the review processes
            * Another Debian Developer discovering the bug and recognizing its significance despite all of the above
            * Debian project coming out and admitting all of the above and scrambling to get fixes out to its users ASAP

    I am impressed by the swift action of the people involved in fixing this. And while I think everyone can find some lesson be learned here, I think this is another good example of free software in action. And I hope that in the aftermath of this we will find ways to prevent this from happening in the future without stifling our progress.

    http://www.aigarius.com/blog/2008/05/14/too-similar-to-be-different/

  79. Re:It will be fixed by Archangel+Michael · · Score: 2, Informative

    Why stop with just compiling source?

    Why not read every line, of every piece of code you compile, to make sure it wasn't tampered with?

    Why not write your own code, code you review yourself and compile and maintain for your own use?

    Unless you write, review, compile every piece of software source yourself, you have to trust someone to do at least one of them. At that point, you trust someone to do it for you, and you're already where your compaint lies. Its just that you don't realize it yet.

    Compiling code you've never looked at is just as risky as trusting that someone to write the code for you. In fact, I suspect that the Debian Source would have compiled with the same problem, had you compiled it yourself.

    So no, compiling from source (Debian) wouldn't have solved this problem, unless you could realize and fix the changes from the Debian Source and the Master Source yourself. But you didn't, someone else did.

    Your point is kind of moot, isn't it?

    --
    Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
  80. Check those CA contracts... by sunbird · · Score: 2, Interesting
    Look on your CA website for the contract that governs your CA. It should have a clause on revocations. For example, this is the .pdf for ipsCA's contract (.pdf, only in Spanish) and it clearly provides that if there's a problem with the certificate that makes it untrustworthy, you get a new one for free.

    This is page 45-46, which talks about the reasons why a certificate may be revoked:

    Los Certificados deberán ser revocados cuando concurra alguna de las circunstancias siguientes:
    *snip*
    Por cualquier causa que razonablemente induzca a creer que el servicio de certificación haya sido comprometido hasta el punto que se ponga en duda la fiabilidad del Certificado.

    Rough translation: The certificate may be revoked when ... there's something that makes you reasonably believe that the certificate has been compromised to the point that it is not reliable.

    And then on page 47 it says:

    La revocación del Certificado por causa no imputable al Suscriptor originará la emisión de un nuevo Certificado a favor del Suscriptor por el plazo equivalente al restante para concluir el período originario de validez del Certificado revocado.

    Rough translation: If the revocation of the certificate is not the fault of the subscriber (that's you), then you get a new certificate with the same validity period as the old one.

    Obviously, you have to check with the contract governing your CA, but you might find something similar, so check with your CA before paying for new certs.

  81. MOD PARENT DOWN by jareds · · Score: 2, Informative

    This is dangerously misleading! Blocking IPs after a reasonable number of failures stops only one particular attack against SSH. It prevents someone from logging into a user account that has a compromised-Debian-generated SSH public key by brute force, trying all possible compromised-Debian key pairs. There are a number of other attacks.

    The most serious involve the SSH host key. The public host key is given to anyone who attempts to connect to the machine, whether they succuessfully authenticate or not. This must be the case -- giving out the public key is part of establishing an encrypted connection, and you must have encryption before you send your password or something. An attacker can connect ONCE to your server and, from the public host key, lookup the corresponding private key offline in microseconds. They can then decrypt any SSH communications they can sniff where that server is the server. You have little more security than telnet. They can perform a man-in-the-middle attack on the server. Even if a user is carefully using public-key authentication with something other than a compromised-Debian-generated key rather than password authentication, if they connect to a server with a compromised-Debian-generated host key, they might be connecting to a MITM attacker who can then perform any action on the server as said user.

    Public-key authentication is still seriously vulnerable even if you limit the number of attempts. Normally, I can generate a key pair on machine A and set up machines B and C to allow me to log in with that public key, and a root attacker on machine B or C will not be able to access the other machine with my identity. If machine A is a compromised-Debian machine, this no longer holds. Anyone with access to the public key can immediately obtain the private key and impersonate me. Moreover, I normally wouldn't take the same precautions with my public key as my private key, but with a compromised-Debian-generated key pair they are practically equivalent. In general, there is no basis to rely on the public key being secret, which you implicitly recommend doing.

    Any rational and informed person will immediately regenerate all key pairs generated with a compromised-Debian machine, and remove such public keys from any authorized_keys files in which they appear. They would also do well to consider the Ubuntu strategy of rejecting known weak public keys for public-key authentication.