Slashdot Mirror
Debian Bug Leaves Private SSL/SSH Keys Guessable
SecurityBob writes "Debian package maintainers tend to very often modify the source code of the package they are maintaining so that it better fits into the distribution itself. However, most of the time, their changes are not sent back to upstream for validation, which might cause some tension between upstream developers and Debian packagers. Today, a critical security advisory has been released: a Debian packager modified the source code of OpenSSL back in 2006 so as to remove the seeding of OpenSSL random number generator, which in turns makes cryptographic key material generated on a Debian system guessable. The solution? Upgrade OpenSSL and re-generate all your SSH and SSL keys. This problem not only affects Debian, but also all its derivatives, such as Ubuntu." Reader RichiH also points to Debian's announcement and Ubuntu's announcement.
Who did this? You don't remove the seeding... stupid
did I mention stupid?
this is how some of the old video games were "broken" despite using "random" numbers, the seed was always the same... leading to the same sequence of events
I will not give in to the terrorists. I will not become fearful.
Ubuntu got an updated advisory at http://www.ubuntu.com/usn/usn-612-2
whether this can possibly be claimed to be an accident *dons tinfoil hat*.
But seriously, removing the code that seeds a random number generator? I can hardly imagine making such a change by accident. I may just lack a sufficiently colorful imagination, though.
(or, before resorting to conspiracy theories, we should probably ask ourselves first, "can this possibly be explained by simple stupidity?"
Every expression is true, for a given value of 'true'
This problem isn't something you can just update your system to fix. This means the basis for all remote authentication on your Debian/Ubuntu machines is compromised until you go and fix it manually.
Got up this morning, booted the machine and got a software update first thing: OpenSSH (et al) updates for my Ubuntu Gutsy install. Then I show up over here and see why. Presumably Feisty and Hardy got them as well - they are listed on the Ubuntu announcement.
First off I'm a big OSS supporter, yada, yada
But the point here is that the freedom that OSS gives you does require you to trust the whole distribution chain. In this case there was an added muppet who did something they shouldn't have thus rendering everything downstream insecure. OSS is great but it required great developers, given that it has take well over a year to get the advisory out it shows that the many eyes piece didn't work here, mainly because the eyes were looking at the original source not the botched packaging job.
The "easy to use" Linux box in the house uses Ubuntu and has this issue and like many people I didn't even think to check that the OpenSSL wasn't the REAL OpenSSL it was OpenSSL with muppet extensions. Maybe there needs to be some form of extension that warns that a package has been modified from its original source code and that the modification was done by "K. Frog" so you can determine whether to trust that package or look back to the source.
Or some sort of voting system on contributors (how very Web 2.0) so you can see how the people who touched your package were rated with the biggest weighting being given to the last person through the code (hand edited by Linus = 5 stars, hand edited by James Gosling = 5 stars, hand edited by the bloke who wrote clippy = 2 stars, hand edited by a bloke who removed a seed generator = 0 stars).
Having the code is great, but this makes me want to know much more about who last edited that code.
An Eye for an Eye will make the whole world blind - Gandhi
Who told you that? Why did you believe it? Please cite your sources.
... this had to be discovered four days *after* I finished setting up my new Ubuntu 8.04 server... *grumble*
Proudly supporting the Libertarian Party.
Yes it's already fixed. I was in the middle downloading the update when when I saw this article.
Preparing to replace openssh-server 1:4.6p1-5ubuntu0.2 (using .../openssh-server_1%3a4.6p1-5ubuntu0.3_i386.deb) ... /var/lib/dpkg/tmp.ci/templates has a duplicate field "template" with new value "ssh/vulnerable_host_keys". Probably two templates are not properly separated by a lone newline. /var/cache/apt/archives/openssh-server_1%3a4.6p1-5ubuntu0.3_i386.deb (--unpack):
Template #4 in
dpkg: error processing
subprocess pre-installation script returned error exit status 255
And then:
openssh-server: Depends: openssh-client (= 1:4.6p1-5ubuntu0.2) but 1:4.6p1-5ubuntu0.3 is installed
Anyone else seeing this?
It was accidentally introduced in 2006... so that's what, another 14 years before it gets moved into 'stable'?
*grin*
As far as the bug linked for changes upstream (#477454,) is that seriously the best example of Debian not sharing changes back upstream?
It seems more like an attempt to show that a maintainer was being petty which is not what the article was about.
Does anyone one really think that this change would have been accepted upstream (or even have had a place for it there?)
So this is how linux is going to replace Windows on the desktop? By creating custom functionality that break RFC and common sense? Some things never change, do they?
It shouldn't need fixing in the first place.
Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu).
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
note: When I have to run Linux instead of a BSD it's Debian and/or Kubuntu all the way since the benefits outweigh the negatives, but it's still an annoying habit of theirs.
http://en.wikipedia.org/wiki/Linus's_law
I understand he's not unheard of in this arena.
Yeah, like all those times when MS cut checks for all their customers whose computers were compromised! Oh, wait...
Caveat Utilitor
I'm sure the problem will be fixed if the developers acknowledge that the problem exists. Not a big worry.
Yes, it is a big worry because any keys generated with this package are now potentially suspect. This means that anybody who's used Debian or a Debian derived distribution like Ubuntu needs to go back and destroy all host and personal keys generated since 2006. All of those keys are potentially guessable.
And that's a real vulnerability. Early versions of Netscape's SSL implementation (the first SSL implementation) were trivially crackable because of just such a vulnerability.
Need a Python, C++, Unix, Linux develop
"You fell for one of the classic blunders, the most famous being 'Never get involved in a land war in Asia' but only slightly less well known is 'Don't use poorly seeded pRNGs in cryptographic protocols!' HAHAHAHAHAHHAHAHAHAHHAHAHAHAHAHA!!!!
Test your net with Netalyzr
The seeding was removed and it wasn't noticed for TWO YEARS? In a distro as popular as Debian?
Um.... it doesn't matter if it's been patched, you still have to regenerate all your keys.
There most certainly is something to see here, this is kind of a big deal.
This has nothing to do with the vulnerability being discussed here.
i was sure that netscape problem was due to the US crypto export regs (the silly encryption=munitions thing) that limited the encryption to 40-bit keys.
upon the advice of my lawyer, i have no sig at this time
http://www.random.org/analysis/dilbert.jpg
http://www.xkcd.com/221/
This is on the SSH server side. If you are a casual desktop user, you don't have to do anything.
One of our competitors trademarked the term "hypothesis". From now on, we will call them "boneheaded ideas".
Downloading the patch is step one - you also need to regenerate any certificates made with OpenSSL since 2005, since they can't be guaranteed to be secure.
This has the potential to turn into a huge pain in the arse for Debian based shops, who will need to reissue SSL certificates, SSH keys, and a whole host of other essential elements of their security infrastructure.
i was sure that netscape problem was due to the US crypto export regs (the silly encryption=munitions thing) that limited the encryption to 40-bit keys.
Then you are mistaken. Read the linked to paper. N bit security doesn't protect you much when you can guess N-10 bits of the N bits.
Need a Python, C++, Unix, Linux develop
Who was the member of the security services who acted as package maintainer?
I cannot believe that such a change would have gone by unnoticed if it had been properly documented, and I cannot believe anyone given the trust of the OpenSSL distribution would make such a mistake.
This was an intentional act of sabotage.
There were two issues: one was that the "export version" had 40-bit encryption which was relatively mathematically weak. Everyone knew that it could be brute forced with enough horsepower, and eventually someone demonstrated that -- though it still cost a lot more computer time (100 HP minis for a year?) than you'd probably want to spend for a single CC number.
:)
The second issue was that the full-strength domestic version used a crappy RNG source. That narrowed the possible keyspace by some tremendous amount. What remained could be brute forced on a desktop PC in a few hours.
Only as good as the implentation
Basic cryptographic services have been compromised for a year and your analysis is to assume on faith that it's open source so it will be fixed, so no problem?
If someone stole your crypto keys and has had them for a year...
How thoroughly might they have compromised your system by now?
How many passwords might they have stolen that you use on other systems?
What else might they have done that will give them access in the future even after you fix this?
Just regenerate your keys and no problem? The problem that guessable keys are generated will undoubtably be fixed asap, if not already. The problem that this has been the case for the last year will not be, and is a big worry.
How very...reasonable...of you.
So, you're luvin' Windows, then?
.there is enough of everything for everyone.
I wake up and what do I see first thing? That there is a problem with Debian's OpenSSH package and the
Now I am thinking, "What exactly is going on here? Is choking on a bucket of cocks not a good source of randomness?"
I honestly don't think this is much more than a mistake, like one of those who surfaces every two years. I wouldn't go as far as blaming OSS or any other instance for that matter. No matter what you feel, you cannot deny that Debian usually has some of the most comprehensive focus on security in the scope of popular Linux Distributions.
It's all fun & games until someone loses the game.
My second question was the more important one. Any answer?
Anyone who posts to this story saying that this is no big deal or telling other people to stop whining should simply be banned from Slashdot for life. If you cannot be bothered to read the article and you cannot be bothered to understand just what a serious vulnerability this is but you insist on insulting those who do, why should you be allowed to continue to post your ignorant bleating?
If you mod me Overrated, you are admitting that you have no penis.
In fact I'd expect that separate runs of the same program with the same parameters and environment would leave the same junk on the stack every time.
So I would hope that they have some better source of entropy than unpredictable uninitialized data of dubious randomness.
So if this is really a serious problem, then it seems to me there's already a serious problem in OpenSSH.
Or any other distro, or Mac, just because he doesn't like the way Debian does things doesn't automatically mean he's sleeping with bill Gates.
Can you show me the succeful suites since wga came out?
Hmm... no, it shouldn't have been changed. There is absolutely no excuse for touching crypto code when you don't *fully* understand it.
But the fact that changing the use of NON-INITIALISED memory causes most of the seed of a RNG to go away really is worrisome... unless that memory is actually initialized with a seed BUT in a way Valgrind and others can't track. Which means it is sloppy coding on OpenSSL.
You cannot trust unitialized memory, PERIOD. And it will take looking deep into OpenSSL code for me to tell if it was intialized in a way valgrind couldn't see, or that OpenSSL was working due to just plain blind luck (or lack of an attacker).
Either way, upstream prepared the trap, and a debian maintainer sprung it.
Why do YOU think we are not calling for blood on the guy who did the change? Because code that breaks like that is a travesty by itself.
Well, Debian tries hard to build its reputation for stability and security and sensibleness in general, but arrogantly modifying source code without sending it upstream for verification seriously undermines this.
And I'm not sure that it could be any more serious than when dealing with encryption.
The rule was formulated and named by Eric S. Raymond in his essay "The Cathedral and the Bazaar". It was not very hard to find after wikipedia article.
Extreme Programming - Redundant Array of Inexpensive Developers
No, the random text your put in can be random or not. What they were talking about is a number generator that was used or applied to an algorithm that decided the prime of your key in so that the same algorithm wouldn't be used on two machines on the first then second keys. Without the random number generator, you could theoretically guess the algorithm used to generate the key based of the actions of another install and then decipher the key to gain access to whatever it was protecting.
To simplify and generalize it, if every machine uses X+1*256 to get a 256 bit key equal to 768, then you could reverse that and know X would =2 (3*256=768) and fake the credentials. The random number generator should change that to X+R*256 which make reversing the key harder because you can only solve to X+R=3 now. In practice, it will be a really larger number and a lot different process though. I can't say that I fully understand it but that simplification should show the difference well enough to give an Idea of where the problem is.
"When the president does it, that means it's not illegal." - Richard M. Nixon
There is a great little story in the link. The best part of the story is where the mod says, "Although this abuse is unfortunate, I fail to see what makes this bug release critical." I nearly fell out of my chair laughing.
I think the point was that if the Debian maintainer had submitted his patch upstream, he'd have been told exactly how stupid it was.
OpenSSH is hardly "every cryptography program".
Hmm, I misread your question. Actually many eyes DO make bugs shallow. If this is not true, why bother with peer review in scientific journals? That there is a bug in a code doesn't mean it's harder for bugs to slip through. It's harder, but not impossible. Show me some big bug-free code written by one programmer and not checked by anyone else.
Extreme Programming - Redundant Array of Inexpensive Developers
Check out this bug report and tell me they weren't just being arrogant:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
Says the first comment to the patch, in regards to what is being reversed:
"What it's doing is adding uninitialised numbers to the pool to create random numbers."
OpenSSL having more-random numbers vs. Valgrind looking good. And Valgrind won?
The consensus in the bug report seems to be not to do it, but then someone adds the patch anyways.
True enough.
I considered, "So, you're luvin' whichever operating system you prefer, then?" as a punch line, but it didn't have quite the same effect. I'll try to be a little more vendor/ideology neutral when crafting nerd jokes in the future.
.there is enough of everything for everyone.
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
I thought /dev/random used several sources of randomness. Does SSH not use /dev/random?
Not entirely true. Keys generated before the patch made it to the repos are safe - and I think quite a lot of debian boxes are old enough (I know I've got one.) There's a link in the advisory to a tool that checks if the keys are vulnerable.
This doesn't change the fact that this is a really serious fuckup. Debian lost quite some credibility in my eyes.
I, on the other hand, strongly suspect that any similar mistake at a major software corporation would in all probability be quietly ignored, if it were even noticed at all -- and if it were instead deemed enough of a public relations risk to warrant dealing with, the company would likely just silently push an update to correct the problem for future users, leaving anyone using extant keys with their arses hanging in the breeze.
But maybe I'm just being overly cynical. :-\
Cheers,
"What in the name of Fats Waller is that?"
"A four-foot prune."
"Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix will be obvious to someone."
First of all, this was Debian doing it and not releasing their patches upstream, that debatebly cancels out the large enough co-developer bases.
It also uses the word "Almost"... maybe you misread it as "Always"? otherwise there's NO way you could claim that it in any way implies something could "NEVER" happen.
Never attribute to malice that which can be adequately explained by stupidity.
I'm glad I keep port 22 firewalled, except from specific hosts. The largest problem is the CA certificates though...
Are encrypted file systems affected?? cryptsetup for example?
Not "Or any other distro" in this case, this is a 'debian way' result.
As the other guys have stated, but stating it myself for clarity:
1) They shouldn't have messed with the NON-BROKEN Source (which every other distro uses), without testing it better.
2) I use whatever works, currently I have Ubuntu on my kids computers, Windows and SUSE and Mac OS and BSD at work. I'm more of a utilitarian; computers are nothing more than tools. Right tool for the job, makes all the difference in the world.
3) Reputations are hard to come by. Good ones are hard to maintain. I'm sorry for Debian, because they lost a tad bit of credibility on this one, in my book.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
Hello troll,
You have no recourse unless you have PAID to have one, like with almost everything else in the world. The GPL specifically states that it's not a warranty or guarantee that things will work.
However, if you were using it in a business, I would hope that you either a) hired your own programmers to work with the code or b) bought a support package/liability clause from someone like Red Hat. In which case it would be down to your programmers or Red Hat respectively. But we're not talking about Red Hat. Or any of the other big-name, support-contract-and-some-sort-of-indemnity-clause-included distros. They don't have this problem, presumably because they are not that stupid.
We're talking about Debian. Got a support contract / liability agreement with Debian? No? Bad luck. It's a bit like asking Microsoft to accept responsibility for your pirate copy of Windows, then, isn't it?
Read the end-user agreements for your commercial software and you will see that you have no right to sue them for anything beyond the purchase price of the software.
I just updated again and they fixed the package.
First off, we were not told any such thing -- or at least I've never run across any such claims myself. Secondly, this would actually seem to substantiate the many-eyes theory, as the bug was found and summarily corrected, rather than never found, or found and swept under the rug, or found first by black-hats and exploited, all of which seem quite common in the proprietary software world.
The many-eyes model does not guarantee zero bugs, nor does it guarantee speedy bug hunting. However, it would seem to guarantee that bugs will be found eventually, and also that they will be dealt with in some productive way once they are found. This model aims for the opposite of security through obscurity, that hoary old chestnut of proprietary development, by instead ensuring security by knowing exactly what things do. That also means that folks need to go through all changes -- i.e., many eyes actually need to go over the code. Open source ensures that this is possible, but actual people still need to put in the time and effort.
Cheers,
"What in the name of Fats Waller is that?"
"A four-foot prune."
On the other hand, I met, dated and got Married since 2006. My wifes ssh key is "Debian compromised". Looks like I have some work to do....
vi +
Tonights forecast: Dark. Continued dark throughout most of the evening, with some widely-scattered light towards morning
If that's not tongue in cheek, you need to think about that question a little more. Which programmer forced you to use his free open-source software?
It is a problem of OpenSSL, not OpenSSH. The former has a much larger set of applications, including, e.g., telnet-ssl and Apache SSL module.
Actually, looking closer, the real problem was not that they removed the uninitialized memory use (which was just there because it couldn't hurt with a little extra randomness), but that they also removed OTHER sources of randomness due to incredible stupidity.
Developers: We can use your help.
Schwab
Editor, A1-AAA AmeriCaptions
Has someone actually checked the random output for cryptographic soundness?
The proper source of cryptographic-quality random bits in Linux is "/dev/random". Reading uninitialized memory is not a good source of entropy - it might be initialized to some constant value, especially if you're compiling with a debug allocator or running under a virtual machine monitor.
OpenSSL makes substantial efforts to get a good random starter. Unless someone did something so stupid that OpenSSL didn't use /dev/random, it should still work. OpenSSL is supposed to have a check for bad random seeds. Was that bypassed, or doesn't it work, or what?
Obtaining a good source of randomness is hard. Computers are rather deterministic. Historically, there have been major failures in this area. See "Venona" where the USSR was generating "one-time pads" by having people type random digits on typewriters. Arlington Hall, NSA's predecessor, cracked that. Humans aren't random enough. True random number generation requires special hardware, like a noise diode or a radiation source.
"Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin." -- John von Neumann
From this log: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
It looks like openssl is pulling in "entropy" from uninitialized memory, causing valigrind to complain. The debian maintainer "fixed" this issue by memsetting the buffer to zero.
My question is, wouldn't I see the same behavior using grsecurity to scrub deallocated memory? From what I am seeing, this looks much more like the fault of the openssl team.
As opposed to the closed source world? If a security bug were discovered in Windows, would Microsoft would compensate you for your damages instead of just disavow any liability in a EULA?
OK, this is as good a place as any.
FUCKING IDIOT NOOB ASSHOLES!!!!!1!
Fight hunger. Filet a politician and send him to a 3rd world country of your choice.
Can someone explain what "guessable" means in this context?
Does that mean someone can now generate a "master-key" and ssh into root@any-debian-box that allows public key authentication?
What does a realistic attack scenario look like?
I don't know, go into the SVN or CVS repo and find out yourself. Luckily you're not using closed source where you don't even know if the code does it's work nor do you know the programmer.
And I heard the 'we get to sue x if we buy from x' but when was the last time you sued Microsoft because of your system being part of a botnet? Or Oracle because their database corrupted. Heck, it's difficult enough to sue or set responsible PeopleSoft or SAP for the defective product they (after 3 years of implementation and about 200 contractors) don't/can't deliver, imagine something you DID get.
Custom electronics and digital signage for your business: www.evcircuits.com
It's not arrogance. Or it's at least not arrogance alone. As a distro maintainer I can tell you that upstream providers generally do not care about distro-specific patches. Even if you have a patch that would be useful on more than on distro, there's often a reluctance to inherit code to support any installation method other than source tarballs.
That's not to say that upstream providers are unduly arrogant either -- if you're happy with the way your build/install system works, why would you want to maintain patches for some other system that you don't even use? It's extra code to keep up, and requires more testing on more platforms, and it it doesn't make your core package any better.
Read you EULA, you can't sue your vendor either.
As a maintainer of some upstream code, I can tell you that I am always happy to receive patches that make life easier for one distribution / OS as long as they don't break anything for anyone else. We want your users to be our users too, so please help us make life easier for them.
I am TheRaven on Soylent News
I think it's worse than that. Basically, it looks like they're disabling the addition of truly random data to the random pool, full-stop, by commenting out the line that does it.
Actually, here is where the Debian maintainer brought this up with upstream, and he was not told how stupid it was, in fact the Debian maintainer says, "the pool might receive less entropy." and upstream says, "If it helps with debugging, I'm in favor of removing them."
Actually, any DSA key *used* on a box with the bad SSL packages may be compromised:
From http://wiki.debian.org/SSLkeys
Additionally, some DSA keys may be compromised in the following situations:
* key generated with broken openssl = bad
* key generated with good openssl and used to ssh from a machine with bad ssl = bad
* key generated with good openssl and used to ssh from a machine with good ssl = good
This is because the random numbers used during the signature process must also be good.
Yeah fuck all these people compiling stuff for me, its not like im too lazy to compile and patch my own system!
IranAir Flight 655 never forget!
Its a shame that this isnt always the case though, for example the 25 year bsd bug got reported by samba but it got bumped back at samba instead of looking into it.
Some maintainers upstream/distro are just hard to work with, it seams premature to label all debian maintainers arrogant tho, just as it would be premature to claim that all BSD maintainers are arrogant.
IranAir Flight 655 never forget!
The host keys on my two file servers, generated while they were still running sarge, were OK according to the perl script provided by Debian to verify whether a key is weak or not. If you were previously running sarge and have upgraded to etch but haven't issued new keys after upgrading, you're fine (as least as far as your host keys go). The host keys on my recently installed etch workstation were detected as being weak, however.
Gentoo - the distro that lets me do what I want, without forcing things on me. (Apart from removing XMMS with little notice. Grr.
Get your own free personal location tracker
The package maintainer first checked with upstream about the best way to resolve this. In retrospect, it's clear that upstream didn't either understand what was being asked, or what the code was doing. In any event, another Debian Developer, Luciano Bello, later found the problem and resolved it.
http://www.donarmstrong.com
It is dangerous to be right when the government is wrong.
He would have been told, but who says he would have cared? This happens all the time, although usually it doesn't result in a giant fucking security hole. I was warning people this sort of thing would happen more than two years ago back when I was working on autopackage. It was entirely predictable.
Having distributors randomly change source code as they package it is fundamentally broken. It cannot ever work. It's not just OpenSSL that's been broken like this. Wine often went through periods in which the Debian packages were completely hosed in subtle ways. It would look like a bug in Wine, but was in fact due to parts of the software "going away", because the Debian packager felt they weren't important enough to be a part of the base install. Pointing out that their change was broken didn't help. The result was many, many hours wasted tracking down non-bugs. Then you go through a giant waste of time trying to get the bug in the packages fixed, and just as you finally get it sorted out, some other distribution introduces some other bug into their packages.
So I got sick of this, and started telling people to avoid their distros packages. Some distributors (especially Debian) took it personally and started childish slanging matches. But really, who cares how "integrated" a program is, if it could have had arbitrary bugs silently introduced? It's just a busted model of software distribution.
Imagine if Microsoft reserved the right to modify any software for Windows in any way it saw fit! Yet that's exactly what Debian (and Fedora and Mandrake and Ubuntu) said to me - they reserve the right to make any modifications they like to the software they ship, and if upstream don't like it, tough luck.
I wonder if they'll re-evaluate that policy in light of this disaster. Probably not.
Certainly a black eye on Debian here - it's not the distro I use and it's a little less likely that it will be in the future because of this. (So many distros, so little time...)
Why is there an "insightful" mod and why isn't it "-1"? If I wanted insight, I wouldn't be reading
Mod parent up.
How do you know uninitialized = random or even different?
You could end up reading some BIOS junk that's the same even after cold reboots.
Most users do not generate nor use ssh/ssl keys or certificates.
The real questions are
vi +
It appears to have already been patched on my Debian etch system (both kernel and libraries). Does anyone know how to force all my ssl/ssh keys to refresh? Yes, I know it'll break all the old connections, but I'd much rather know it was secure.
Is there a way to suppress specific valgrind uninitialized memory warnings without having to clear the buffer (and potentially mess up other code)? Some of my code has the same problem - valgrind whines when I encrypt a block of uninitialized memory, even though the numbers are intended to be random.
"Screw Sun, cross-platform will never work. Let's move on and steal the Java language." - Visual J++ Product Manager
Yeah. It would have a beautiful change for someone who sells certs for a living to make. All that extra $$$ from folks who have to buy new certs.
I wonder what the cost from this screw up will be for certs alone, let alone man hours changing keys on machines and distributing them to the appropriate clients, and whatever hacks may have happened in the past two years.
Found this post at openssl-dev list by Kurt Roeckx ( AFAIK the Debian OpenSSL Team member that made this RNG-clean patch )
http://www.mail-archive.com/openssl-dev@openssl.org/msg21156.html
Extract:
"What I currently see as best option is to actually comment out those 2 lines of code. But I have no idea what effect this really has on the RNG. The only effect I see is that the pool might receive less entropy. But on the other hand, I'm not even sure how much entropy some unitialised data has.
What do you people think about removing those 2 lines of code?
Kurt
"
BTW, i thought that Debian had some kind of policies about testing each package before committing changes in testing/stable branches. Also, the following paper, contributed by another poster, says interesting things about touching cryptographic code, we have to learn from this experience and have tighter policies !
" In a narrow sense, the security flaw we found in the Netscape browser serves merely as an anecdote to emphasize the difficulty of generating cryptographically strong random numbers. But there's a broader moral to the story. The security community has painfully learned that small bugs in a security-critical module of a software system can have serious consequences, and that such errors are easy to commit. The only way to catch these mistakes is to expose the source code to scrutiny by security experts.
Peer review is essential to the development of any secure software. Netscape did not encourage outside auditing or peer review of its software-and that goes against everything the security industry has learned from past mistakes. By extension, without peer review and intense outside scrutiny of Netscape's software at the source-code level, there is simply no way consumers can know where there will be future security problems with Netscape's products. "
http://www.cs.berkeley.edu/~daw/papers/ddj-netscape.html"
Doesn't this mean that all the "etch" host keys are vulnerable, too, on an ongoing basis?
SSH host keys are generated at sshd install time, and that package is included on the netinstall CD-roms, which means every time I re-use my 4.0r1 netinstall CD to set up a new box, I generate a new set of vulnerable host keys before I get to the security-updates step.
Hopefully, they'll advance the next point release and a 4.0r2 netinstall CD will do better.
2*3*3*3*3*11*251
It is even worse than this. Passwords that were used on a server with a weak DSA key may be compromised, as well:
From http://wiki.debian.org/SSLkeys
Only regenerate if you made your keys post 2006 though. Any made prior to the removal of the random seed would be fine.
The problem is that the email is rather misleading, from what I understand. It talks about two places causing the problem even though only one of them actually was making valgrind complain (and you can see, purify already picked up on the problem some time before). The trick is that his patch, not included in the email, removed both calls. Which was busted. So if he'd actually sent a full patch, proposed it for inclusion and had it properly code reviewed, the mistake would probably have been caught.
I was wondering, as soon as I read the GP, what you do if a system doesn't have /dev/random? And there is the crux of the problem. If you want to write portable code that could potentially be deployed on any OS with a compiler/runtime for your language, you are going to have to either include a random number generator of some sort with your code, or use one provided by the standard system libraries for your language, e.g the rand() function in the C library - (but I get the impression that people don't feel the C rand() function is sufficiently random).
/dev/random exists, and use it, else use the other random source you fall back on. But, at that point, either your random number generator is strong enough, or it isn't. Assuming you ship a sufficiently strong random number generator with your software, why even bother with some non-portable system dependent feature like /dev/random?
/dev/random? Because it's not portable.
Now, you might do a test, to see if
And, I bet the questions I ask provide the answer to the original GP's question - why not use
Or a bit like asking Microsoft to accept responsibility for the copy of Windows that you paid for.
Look through the Windows license agreement some time and see how many times phrases like "THERE IS NO WARRANTY OR CONDITION OF ANY KIND", "YOU ARE NOT ENTITLED TO ANY DAMAGES" and "IN NO EVENT SHALL MICROSOFT OR ITS SUPPLIERS BE LIABLE" show up.
The only difference with Microsoft is that if Windows breaks you don't get to keep both pieces.
My point is, the question isn't "Is someone able to develop a workable exploit?", but rather "Was someone able to develop a workable exploit?" And the answer is: we don't know.
(you're hearing my head repeatedly hitting the tabletop)
Ben Laurie of OpenSSL/Apache puts it into some context:
http://www.links.org/?p=327#comment-176642
Obviously some of the OpenSSL devs probably should've been like "zOMG, SITUATION FUBAR", but it wasn't a formal code review being requested, more of a "hey, what do you think of this and this?" and the patch was never submitted to upstream.
Though, as has been pointed out elsewhere, the Debian package maintainer did raise this issue with the openssl developers, and was basically told "go ahead, it's OK."
You will want to download this tool to check each system:
http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
To use it:
$ perl dowkd.pl user
$ perl dowkd.pl host localhost
See http://lists.debian.org/debian-security-announce/2008/msg00152.html
Why isn't such a scan isn't part of the package update? http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=481106
No kidding. I've used Gentoo for years; when I got a new computer a few months ago, I was considering switching to something else, perhaps something Debian-based, out of a combination of wanting to try something different and the fact that I use Debian at work.
I ended up sticking with Gentoo. Why? It has by far and above the most complete package repository out there. Most other distros are spineless and leave out things like libdvdcss, lame, win32codecs, and mad. Gentoo, on the other hand, has everything.
I support the Center for Consumer Freedom
Not everyone is a cryptography expert. What needed to be done is someone to specifically state: "We shouldn't do this. Here's why:"
What appeared to have happened was a bunch of technical jabber completely skipping the security aspects, except for the one you quoted - which you must admit looks quite harmless and irrelevant - unless you understand fully what it means.
The ideal solution was that the maintainer(s) of the OpenSSL packages should have been watching for that kind of thing - after all, random patch submitters are not wrong in assuming the maintainer is a subject-matter-expert.
I don't fault the patch submitter. If you just plain don't know, theres nothing to really hold against you when there is supposed to be someone checking what you submit. I point fingers at the maintainer(s), but humans are only humans and mistakes happen. The important part is that it IS recoverable (pain-in-the-ass though it is) and, hopefully, it won't happen again.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
'Cause Windows never have remote exploiting bugs, right? Yeah, not trolling.
Ok, you are really not trolling. You are for a +5 Funny. Sorry, now I understand.
Rethinking email
You have got to be kidding?
I have never even heard of someone getting fired over a bug, and I run into a lot more problematic ones monthly. Are you seriously suggesting that people are getting fired over things like this? Care to drop a couple of links on us to back that theory up?
As a sibling poster mentioned, most vendors of proprietary software would probably shut up about a thing like this, and let the next "upgrade" silently fix the problem (leaving weak keys intact).
May we live long and die out
I think you misread his statement. He didn't say your "machine" was compromised. He said that the underlying authentication mechanism was compromised, which is quite different. If I were to leave my car doors unlocked, the engine running, and the alarm disabled, then the security system on my car would be compromised; but until someone came along to actually hop in and drive away with it, the car itself would still be working properly.
Similarly, the fact that there is a problem with the OpenSSH package means that the security mechanism that is guarding access to your system is no longer as secure as it was previously thought -- that makes it compromised. The individual systems may or may not have been compromised themselves by way of the compromised security mechanism, but continuing to use OpenSSL knowing it (the key gen algorithm, specifically) is less secure than it should be is just asking for trouble.
If you mod me Overrated, you are admitting that you have no penis.
Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu). Volunteers not maintaining the free product that you've decided to trust your security with properly? Get over it.
Though there are concerns with naming conventions. Mozilla seemingly foresaw this problem and demanded that non-official releases be named differently to avoid confusion in the event that the package maintainers introduce bugs. This is why we have Iceweasel on Debian instead of Firefox.
No, he posted a question openssl-dev, which is a mailing list for people writing software that uses OpenSSL. This OpenSSL developer doesn't read it and that's similar with most open source project - developers often don't read mailing lists for end users.
What's more, that mail doesn't contain a patch. It contains a misleading question with two lines posted in isolation. An actual patch, submitted for an actual code review, would almost certainly have revealed the problem via context.
You don't change crypto code of all things based on an idle question on a mailing list populated mostly by users. What's next, changing the kernel scheduler based on a conversation in #kernel-newbies?
Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu). The huge distaste in your mouth is probably the CROW you should be chewing on since the Debian people did NOT SCREW THIS UP. This was the fault of the maintainer.
Right, that's why Debian renames Firefox. I remember them not being very happy about being forced to do that at the time. But seriously, are all open source projects supposed to use trademarks to force distributors hands? It seems like a bad precedent.
I wonder since when content of stack became "random" as to serve a seed.
In all my years of system and application developer, I'd say picking something random from stack as good as picking some constant - because all code paths thru program would leave 1-2 unique stack states. No entropy there for you.
Even address of stack contains more entropy that its content - since most OSs now randomize stack base address.
Once in a while, I have to call the OpenSSL guy an incompetent and give kudos to Debian guys removing another piece of cruft from code.
All hope abandon ye who enter here.
ah fuck it
this is slashdot
Good or bad, the point is that there are legitimate concerns with packaging up binaries from a modified source tree. A lot of people blasted Mozilla for their actions--I wonder what these people think now that there's an example of this sort of thing.
The only way it could be better (for Mozilla) is if people blamed OpenSSL for the problem--which someone did a little lower in the comments, but I don't think that it's the general consensus.
The "many eyes make bugs shallow" statement shows a lack of understanding of the nature of the problem. Using your corollary of peer review, the purpose isn't to get a lot of eyeballs looking at the article but to get knowledgeable experts to consider the article. The same applies to code. If your reviewers don't understand the code, their eyeballs are worthless.
One area where your comparison falls down is that in peer review there is a certain accountability that the peer review is actually happening. In open source code, you don't know that anyone is actually reviewing the source. Unless you can point at someone who has substantial incentive to do the code review, then it didn't happen.
Incorrect, no matter what Ben claims. According to http://www.openssl.org/support/, openssl-dev is for the developers of openssl itself. To quote the list description: Discussions on development of the OpenSSL library. Not for application development questions!
So yes, perhaps Kurt could have been more explicit in his description of what he was trying to do, but he was definitely using the appropriate address to reach the developers.
noah
I stand corrected. That's a pretty odd mistake for Ben to make.
Still, I remain pissed off - I think I narrowly avoided this problem (my Debian is old enough that it predates the patch) but this is still the latest example of a series of problems that Debian has introduced into software its packaged (including software I've worked on).
And when was the last time you certified that such things have not been part of your favorite closed source operating system for the last 20 years? Please provide the code for your favorite closed source operating system's RNG. I'm sure you feel very comfortable with the security of that code since you can review it and make sure the RNG is air tight and seeded properly.
Still waiting on that code...
Just callin' it like I see it.
Well, not necessarily. The most serious Debian packaging bug I've investigated myself was when they decided that the regedit utility was not an important part of Wine and separated it into an optional (not installed by default) wine-utils package. Simple change to the package definitions.
Except that regedit is not optional. It's often invoked by installers, which expect it to be there. You can't install Windows without regedit, so these installers typically never check if it worked or not. They just assume it did, and the programs then assume the registry keys exist (because if you installed the software, they must exist) and crashed when they didn't. It took a long time to figure this one out, and even longer to get them to fix it.
Imagine if Microsoft reserved the right to modify any software for Windows in any way it saw fit! Yet that's exactly what Debian (and Fedora and Mandrake and Ubuntu) said to me - they reserve the right to make any modifications they like to the software they ship, and if upstream don't like it, tough luck.
First, you should realize that Microsoft DOES modify software that they distribute with windows. Even third party packages the bundle with windows under license.Second, you should read and try to understand the GPL. Distros are perfectly within their right to modify packages. Some do a good job of this, and some don't.
You need only review the history of Debian to realize that users are always at risk of another personality spat hozing their distro of choice. Its happened time and time again. The place is infested with swollen heads, sycophants, and drama queens.
This is why you should CHOOSE a distro based on integrity and quality, as well as history.
Introducing security flaws for the convenience of packagers is unconscionable.
Sig Battery depleted. Reverting to safe mode.
Now Amazon may have some work to do...
vi +
also the README in the source code in quesiton says, "Development is coordinated on the openssl-dev mailing list (see http://www.openssl.org/ for information on subscribing)."
noah
Every Debian-based distro generates SSH host keys upon installation, and turns on sshd by default. So every naively-installed Debian installation out there with a weak key is potentially vulnerable.
A stop-gap solution is to turn off sshd, until you can get the keys regenerated.
Schwab
Editor, A1-AAA AmeriCaptions
The possible answers are a) they already have b) someone will do it or c) no one is able to and the point is moot.
Yes, we don't know if (a) has happened, we don't know if (b) will happen. So there is no way to guarantee that (c) is the case and we are all safe.
So we have to act like it has been compromised and change all of our affected keys/certs...and leave the paranoid to comb though things looking for proof it has already happened.
vi +
Right, I agree, but I said "right to modify any software for Windows in any way they saw fit", not just any software they distribute. Debian has always been clear that the only supported way to get software onto their system is via their repositories, and that they reserve the right to patch software in their repositories as they see fit. It'd be sorta like Windows only accepting software from the Microsoft Download service, and Microsoft insisting on being able to modify the code of, say, Skype.
No, the random text your put in can be random or not.
Yes, I'm aware of that. I didn't make myself clear. The prompt that used to appear as part of the config process for the openssl package asked you to press a bunch of random keys in order to help seed the random number generator. The prompt was very similar to whats described on this page:
https://search.thawte.com/support/ssl-digital-certificates/index?page=content&id=SO2632
which says:
Generating 1024 bits of randomness:
Generating 1024 random bits based on measuring the time interval between your keystrokes. Please enter random text on your keyboard.
Given that the code is so shoddily written that neither of the functions has a comment to say what it's supposed to do, I think I can forgive him for that.
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
Things that you don't think about tend to fall apart behind your back. SSH/SSL is a best-of-breed cryptographic and authentication solution. It's so good, most of us didn't need to think about it.
Even excellent tools need maintenance now and then. And part of the maintenance overhead for crypto and authentication is to change your keys every so often.
So everyone has to suffer a Flag Day all at the same time, update their software and change out their keys. But I think the result will be more secure hosts and a more secure network, even for machines and networks which didn't "need" the change. Which, IMHO, is a good thing.
Schwab
Editor, A1-AAA AmeriCaptions
What I mean is there's an implicit assumption on the part of some OSS advocates that because the source is open, and thus viewable by anyone, that a large number of experts do choose to look at it and audit it. Well, that is usually not the case actually. People just happily use the software as is and never think to check and see if there are any problems with it, even if they have the technical expertise to do so.
That's the downside to the free, open nature of OSS is that while it ALLOWS anyone to look at it, it doesn't REQUIRE any one to. So maybe you have something that a bunch of experts are interested in and it is heavily audited. However you also maybe have something that nobody but the original programmer ever bothered to look at. Also the openness means that you can have something like what happened here: An original version that is audited but a derivative that isn't.
Not at all the same thing. I think your comparison is somewhat flawed.
;-)
Microsoft used to distribute HyperTerminal by Hilgraeve. The Microsoft version was obviously a hack, but you could upgrade to the full version by going direct to the developer, and it integrated smoothly with windows.
Similarly, there are non-debian controlled repositories for Debian. (The fact that Debian does not reach out and control (support) those 3rd party repositories is clearly a Good Thing(tm).) They don't prevent you from using third party repositories.
All distros do the same thing. OpenSuse cripples mp3 capabilities (for legal reasons they say), but they provide links on their site pointing to 3rd party repositories that provide un-crippled versions of the same package.
As for this concept of "the only supported way", I have to take issue of that as well. Have you ever heard of anyone getting "support" from Debian? Ubuntu maybe, but Debian?
I don't want to appear to be defending Debian for this bone headed decision, just their right to be bone-headed.
Sig Battery depleted. Reverting to safe mode.
They didn't depend on it: They needed a buffer to do some XORing to, and there was no need to initialize it. So they didn't initialize. That choice offered a potentially minor increase in entropy, and saved the admittedly minor computational cost of initializing the memory.
Ah, the light dawns. All that faffing about the "benefit" of using the uninitialised data was a smoke screen. The problem is that they saw an opportunity for a minor optimization that made the code a lot less clear, and didn't explain why they were doing it. They didn't even put in a "you're not expected to understand this" line.
They screwed up by writing dodgy code, and the Debian blokes screwed up by commenting out the code instead of initializing buf[]. They loaded the gun, and Debian shot themselves in the foot with it. Both sides should be embarrassed, not self-righteous.
Since they didn't mention how to do this in the security advisory, or give you a warning message when you apply the update that you should regen your keys here's how to regenerate the host keys. Either log in as root or put sudo before each of these (this should work remotely, I didn't have problems but YMMV)
rm -frm -f
ssh-keygen -b 1024 -N '' -f
ssh-keygen -b 2048 -N '' -f
You'll then be asked about the key changing next time you log in.
I just spent all day updating servers. Doesn't sound so bad till you realize all these servers also had client keys used for remote backups that had to be recreated and verified, etc.
You still have to distribute those new keys manually. An update on my server isn't going to install the new key on my laptop.
Chernobyl 'not a wildlife haven' - BBC News
echo sshd: ALL >>
echo sshd: your.home.ip.address >>
Get your own free personal location tracker
That you, dear?
It is dangerous to be right when the government is wrong.
Hi, I made a quick study of openssl random generator http://mag.entropy.be/blog/ - it looks really bad, debian implementation generated only about ~200k random numbers.
This should have been in the advisory. Under the circumstances I don't want to assume that I know what I'm doing.
Fortunately, you renew for several years AND the last time you renewed was not while this bug was affecting Debian! With your method you'd be really screwed if it were.
How do you figure this is the result of the "debian way"? This is the result of a user error, and it just so happens that he's making this mistake on a package that will be redistributed to thousands of users amongst various redistributions of Debian, and Debian itself. The "Debian way" you're referring to is a set of guidelines on how they operate, which by no means does it force him to make modifications to source code -- the option is simply there for him to do so, and that's based on FOSS and the GPL.
Regenerating keys is fine, but the important step is clearing out the authorized_keys file that contains the old keys :-). At the same time take the opportunity to clear out the keys belonging to that long-gone co-worker.
...
If you try logging in with a new key where an old one is expected, you will get an error and remove the old one from authorized_keys. When the client is never in use however, the hole might stay in there for a long time
You sure? I seem to recall this being shoved in my face on every Debian install I've ever done. The default motd. "Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law."
mirrorshades radio -- darkwave, industrial, futurepop, ebm.
That's not the accountability I'm talking about. No matter how much you don't like Microsoft, or their software, they want you to like it. Every time you don't, they lose money.
When a Microsoft programmer makes a mistake, they risk losing their job. They risk class-action suits. There are many ways to be held accountable.
But when a hobbyist programmer writes something, and it doesn't work, they've lost nothing.
Nothing to lose. It's that simple. I'm not saying they have nothing to gain by doing it properly. I'm saying that they have nothing to lose by failing miserably.
I write custom software. Of course even my clients can't sue me for huge damages and bugs. But I'm held very accountable for any problems. I risk losing the client, which is incredibly significant to me -- dollar-wise. I risk losing the reference, the reputation, and my entire business.
Tell me, while we're speaking of security, what stops a malicious person from "contributing" to the open source package, with the intension of destroying it? I know that there are plenty of checks, just like Wikipedia, but clearly this one made it through.
I can be absolutely certain that any malicious code in Windows is from someone who works for Microsoft. That's a much smaller group, and that group is well scruitinized.
No, she's great, and carrying our second bundle of joy. I exercise every opportunity to joke about her, though. Any woman who lets me run Ubuntu (or any other linux distro) at home is great. I've even got her brainwashed to the point where she's recruiting her girlfriends to Firefox from IE.
It is dangerous to be right when the government is wrong.
No one's forced me use anything, that's why I create much of my own.
But tell me, this particular person who caused them bug, will they lose their job? Their annual bonus? Anything at all?
First, if it were bad enough, Microsoft would be sued class-action style. Second, the programmer responsible likely loses his job. That's the big thing. Just because you don't publicly see the issue being dealt with, doesn't mean that nothing happened.
There's a damn good reason to not screw up -- you can lose your job. But that bets that your actions are linked to your job.
... for the past couple of years, or a webhosting firm (generating ssl certs), etc. etc.
I really can't believe somebody would be mucking with the OpenSSL code at all: it's a crypto rng, if you haven't written it or don't understand it fully it's definitely one of those things to LEAVE ALONE.
-- the cake is a lie
I hate this stupid community. Every time I mention anything about not liking open source -- a.k.a. open liability -- no one reads the point. And then I can't even respond to them all, because there's a limit.
I can't sue microsoft for a bug in windows just like I can't sue apple for a scratch in my ipod. But we're not talking about consumer worlds here. I'm talking about the business world.
When a commercial solutions provider implements a solution for me, I can sue them. Second, Microsoft and Apple get class-action suits all the time.
But all of that is still not what I'm talking about. I'm saying simply this. When a Microsoft or Apple programmer screws up, they risk losing their job. When a volunteer and often anonymous programmer screws up, they lose absolutely nothing. And what's more, where's the security in open source if a programmer can cause these sorts of problems? Forget security in the end product, what about security in the non-existent office?
The first thing that I tell my clients is that their database can't be any more secure than the lock on the office door.
Again, as everywhere else, someone loses their job. And class-action exists for a reason.
I am going to hijack this thread because I have an eCommerce site hosted on Debian and this made me nervous.
According to the reps at Thawte, if you are using a third party ssl cert (thawte, verisign, etc), this does not affect you. According to them, this is only dangerous for people who have generated their own SSL certs for their sites from the ground up (probably a minority, I would guess). If anybody has any information to the contrary, please let me know.
weirdest thing I ever saw: scientology advertising on slashdot.
As with Debian Sarge, Dapper uses a version that predates the dangerous patch.
Don't go bragging. A lot of people know the key to rooting your wife's box.
In defense of said muppet. This has got to be a case of poor code commentation causing a maintenance problem.
The muppet was responding to a warning put out by an analysis tool (Valgrind) that some uninitialized memory was being read from, any programmer will tell you that's not a fantastic idea in general but it happens that in this case it was not only ok, but necessary.
But the muppet came along and "fixed" the problem.
Now this can only mean that there was inadequate code commenting (or, that the muppet didn't read it, in which case I retract my defense, but not the reasoning) around this unusual piece of code. If a piece if code is doing something decidedly strange, they there NEEDS to be a comment regarding it, preferably informing the reader of what is happening, or at least saying here be dragons or similar.
All it needed was: /* Reading some uninitialized memory to get a randomization seed. */
NZ Electronics Enthusiasts: Check out my Trade Me Listings
I'd like a definitive answer too. SSH keys might be important to a lot of people but as they are usually within an organisation revocation can be managed. I'm not sure that is the case with keys that are used publically.
Personally if the problem is with not particularly random (ie easily recreatable) keys then I don't see how the fact the key is signed by a third party makes any difference whatsoever.
If someone can regenerate your key then they can use that regenerated key with your certificate (which is publicly available from your webserver).
In fact even if you regenerate a new key/certificate, if someone has already captured your certificate then they can do this unless the users browser somehow finds out that your old certificate has been revoked.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
These companys will sell you a contract that DOES give your business recourse when something goes wrong.
Most businesses don't need enterprise software though, so they stick with linux, bsd, windows, and mac os. Great... just when I had mostly convinced the PHBs in management that yes, open source software was trustworthy, and that yes, good developers write Linux, and snip Also it's worth pointing out, this one bug has zero to do with linux or opensource. OpenSSL IS secure. One guy broke the package and introduced this problem.
It is still a bad problem, and there are alot of debian users, but just compared to all the linux distros out there, it is still only a percentage and under 100% (by Far under 100)
What you are saying is basically that because I personally can download an opensource program, change (aka break) it, and give it to someone, that opensource in whole is broken, untrustworthy, and bad.
Clearly that is just stupid.
I suppose if you don't trust openssl anymore, for a problem that effects only debian users, compared to MS's bugs that give admin access to anyone on the internet from win NT4 up to Vista SP1, then that is your call. (I am curious how you might justify that to your PHB though)
http://www.microsoft.com/technet/security/bulletin/MS08-001.mspx
Thats akin to saying "I no longer trust the guy at the store since he shorted me $0.03 in change, so i'm gunna just trust the crackwhore at the corner to by the same stuff from."
Hashing newly received random data with the existing contents of a buffer is a completely standard cryptographic idiom.
That's not the cleverly stupid bit. But I've already pointed that out.
You say, I don't care, it's not important so go ahead and fix it.
You're pointing at the wrong bloke: I didn't say "fix it", I said "document it".
If you take a gun and aim it at your foot and pull the trigger, and it goes off, I'll give you a Darwin award (albeit with a nervous laugh). But if the guy who loaded the gun doesn't at least feel bad about leaving a loaded gun around, well, he's taking the Darwin awards too seriously. This kind of thing isn't "funny haha", it's "funny sheesh" (as Daffy Duck says, or is that Sylvester?).
Looks like the person asking the question didn't provide enough context. Commenting out the second "MD_Update(&m,buf,j);" is perfectly safe; commenting out the first one is dangerous, but unless you look at the surrounding context this isn't obvious. (If you do look at the context it's in, it's fairly clear that removing it is a very risky idea and potentially affects seeding of the RNG from sources other than uninitialised buffers.)
The OpenSSL developers are right to say that not using uninitialised data as an entropy source is safe, but didn't spot the implications of the suggested code removal. Some people did suggest building with -DPURIFY, though (which does the right thing).
That's what I thought originally, but the whole business about the randomness of the stack frame is a red herring. The randomness in the stack frame isn't being used for anything, it's just that there's no reason to care what was in the buffer, so it didn't matter that it wasn't initialized.
I think that is a silly optimization, but it didn't hurt. What hurt was not documenting it, leaving a big old tiger trap with an illusionary bug as bait for a careless bug-hunter to fall into.
The waffling about the minor improvement in randomness from the stack frame is just blowing smoke to distract attention from the fact that they didn't write "this is not a bug" on the bait they accidentally left by the hole. It's "oops" all around this time.
I was trying to explain what happened by putting you in the place of the OpenSSL developers.
I'm not nearly obdurate enough to write security software. You gotta be a hard boy for that stuff, so I can't wear those shoes no matter what.
OK, so the code didn't confuse them, and the bloke waffling about the minor advantage of using the stack frome for extra randomness was deliberately blowing smoke, that's fine, that was a side issue anyway. The main issue is that you put a note up saying "OH HAI, I'M NOT A BUG!" next to code that might look like a bug (especially when you've already to #ifdef it out to get it to pass a bug checker).
I've seen way too much of this cleverly stupid stuff over the past 30 years, I've even done some of it back in the day when I was younger and less familiar with the traps that lay in wait for the stupidly clever. And I wouldn't have even commented about it except that this fella decided to explain how the original code was clever enough to put a tail on and call it a weasel. It ain't. That's all.
Comment removed based on user account deletion
Comment removed based on user account deletion
Frankly the guy probably doesn't need ragging on (in fact people should be checking he's ok).
A small coding mistake has had rather nasty and rather public consequences.
What does bear scrutiny is the process. In such a critical package should it really come down to just one guy? On such a critical package should Ubuntu not have their own man double checking everything rather than just accepting changes from Debian.
You can't make people infallible, but you improve the system to better mitigate such fallibility.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
I don't know why people are tagging this "Informative" when it's obviously a joke. Stupid /.'ers
Well, what makes the difference is the security team, Microsoft don't go around patching other people's software. I read a bit of the discussion, and it ended mostly like this:
Firefox has a trademark and they won't let anyone use it to stop scumware builds, particularly on Windows that doesn't use repositories.
Debian has a security team that refuses to be prevented from making security patches, otherwise they'd get very little done.
Compromises were tried like the security team uploading patches to firefox for validation, but as a matter of policy Debian wouldn't commit to waiting for those. Firefox couldn't accept temporary hotfixes during a security crisis, the requirements on code approval were absolute. Firefox was looking into giving them some sort of trademark license, but since it'd only apply to Debian they couldn't take it under the DFSG #8. Obviously granting a free-for-all license would nullify the whole point of the trademark and was out of the question. And since the issue had been raised and trademarks must be defended, Firefox had no other choice than insisting that Debian stop using it.
I think it would be a terrible result if every upstream package trademarked themselves to prevent downstream patches. Imagine the Linux(TM) Apache(TM) MySql(TM) PHP(TM) stack running Gnome(TM) and KDE(TM) with applications like Firefox(TM) and OpenOffice(TM) all like that, being a distro would be almost meaningless since they'd all be exactly the same and bound on hands and feet. Distros aren't in general trying to be idiotic, and you can point to issues like this to see why it's a really bad idea. Still I think it would be much saner to continue letting downstream make patches under the GPL and leave trademarks out of it. In a free market, distros that make stupid decisions downstream should be taken care of by natural selection.
Live today, because you never know what tomorrow brings
You are.
If you read all the comments following this rant, you will discover that the person who created the offending patch tried to check it with the OpenSSL devs by posting the patch to the openssl-dev mailing list.
Unfortunately that list is not for OpenSSL devs, instead it is for users of OpenSSL. Therefore only other clueless users saw the patch. To reach the OpenSSL devs one needed to use the openssl-team mailing list instead of openssl-dev.
IMO, this problem was due to a communication problem and it is hard to blame just one person for that. If I had to place blame, I would say the fault was with the poorly chosen names for the OpenSSL mailing lists.
We don't see the world as it is, we see it as we are.
-- Anais Nin
And praise be for that. When I looked at Slashdot this morning I thought was going to have a very crappy day today.
I administer at least 8 multi-year wildcard certificates.
If we hadn't gone with the LTS a lot of my time (and thousands of dollars) would have been wasted.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
The Lynx browser uses OpenSSL!! So you text-only browsing folks would be toast if you didn't pay attention.
Although as was already mentioned, most other browsers do not use OpenSSL. But, that doesn't mean the issue isn't important to casual webbrowser users, though!
Had your favorite browser used OpenSSL (and as a casual user, you wouldn't know), your HTTPS communications could have been compromized. So it is vitally important to be concerned about things like this even if you aren't a SSH user or generate your own CA's or keys for any reason. Dismissing the problem for casual users because they don't actively generate keys would be wrong.
And there's always the risk of getting MITM'd when talking to someone using a weak key, so you'd want to make sure you don't talk to people with weak keys (the code should take care of this too, but it might not always be possible to detect, so you should ask your bank or whoever if their keys might be bad).
In the end it is the casual users who get bitten the worst by things like these, simply because they aren't aware that a problem might affect them.
Verisign
Considering how much they charge for certificates it's not hard to imagine them making millions out of people buying new certificates.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
In this case there was an added muppet who did something they shouldn't have ...
OSS is great but it required great developers
Steady on. This guy just made a coding mistake. Even the best coders do that from time to time.
How would you feel about people who know nothing about you going on about what a muppet you are for FOSS work you did, probably for free, at who knows what time of day and under who knows what circumstances. Way to encourage people to contribute.
This guy probably owes the community an apology of the "whoops, truely sorry, feel sheepish" variety, but doesn't deserve to be called a muppet or a bad developer.
Do you really wonder why Linux hasn't taken off if this is the respect that can be expected from the community?
given that it has take well over a year to get the advisory out it shows that the many eyes piece didn't work here, mainly because the eyes were looking at the original source not the botched packaging job.
Many eyes increases your chances, it doesn't give you an iron-clad guarantee.
These posts express my own personal views, not those of my employer
whooooosh
Some CA's allow users to have their certs re-issued for free a limited number of times.
I cann't claim to 100% understand the situation but after glancing trough the logs of the discussions and of the patches the conclusion I came to was this - OpenSSL used supposed randomness of the uninitialized memory as an added source of entropy (interesting hack, but not an example of good coding as such). Valgring caught that problem and the Debian maintainer during a cleanup fixed it. Making such a fix can be considered a preventive step against possible attack vectors by poisoning the uninitialized memory. He took it up to upstream, they did not raise red flags, but did not quite merge the âclean upâ(TM) patch either. It fell through the cracks.
The problem is that in the same file, in another function all other sources of entropy were being merged into the pool of randomness using exactly the same code line as the one code line flagged by Valgrind. The maintainer assumed that the second code line has a similar function to the first and commented that one as well. AFAIK that also did not show up in the emails to the upstream list.
So we have:
* Upstream using clever hacks that rely on uninitialized memory having some randomness to it
* Upstream using same code and same variable names to describe different things
* Upstream having no comments in the code explaining the two things above
* Maintainer slightly over-generalizing a change
* A bug slipping trough the cracks in the review processes
* Another Debian Developer discovering the bug and recognizing its significance despite all of the above
* Debian project coming out and admitting all of the above and scrambling to get fixes out to its users ASAP
I am impressed by the swift action of the people involved in fixing this. And while I think everyone can find some lesson be learned here, I think this is another good example of free software in action. And I hope that in the aftermath of this we will find ways to prevent this from happening in the future without stifling our progress.
http://www.aigarius.com/blog/2008/05/14/too-similar-to-be-different/
Why stop with just compiling source?
Why not read every line, of every piece of code you compile, to make sure it wasn't tampered with?
Why not write your own code, code you review yourself and compile and maintain for your own use?
Unless you write, review, compile every piece of software source yourself, you have to trust someone to do at least one of them. At that point, you trust someone to do it for you, and you're already where your compaint lies. Its just that you don't realize it yet.
Compiling code you've never looked at is just as risky as trusting that someone to write the code for you. In fact, I suspect that the Debian Source would have compiled with the same problem, had you compiled it yourself.
So no, compiling from source (Debian) wouldn't have solved this problem, unless you could realize and fix the changes from the Debian Source and the Master Source yourself. But you didn't, someone else did.
Your point is kind of moot, isn't it?
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
Ubuntu has released an OpenSSL fix already, it seems.
``Tension, apprehension & dissension have begun!'' - Duffy Wyg&, in Alfred Bester's _The Demolished Man_
Don't get me wrong, I'm not attacking Debian at all. I'm saying that the poster I replied to is infact wrong by saying 'any other distro' in this case as the bug is caused by a Developer who changed the package so it conforms to the guidelines, and made a mistake.
If some coder did this at a company at least I'm pretty confident they'd get their ass fired
How does firing a coder who made a mistake fix your company's problem after the fact? People make such mistakes despite such incentives. What's more the practice of firing someone for making a single coding mistake is stupid. If you want that level of security you need lots of peer review.
All this would do is make someone feel good that someone got the boot for it. Your keys are still compromised.
These posts express my own personal views, not those of my employer
This is where the R comes in on the simplified example I gave. X is what you or a text file or whatever inputs, R is what the random number generator creates separate from X and then it is applied to an algorithm like in the X+R*256=key. I guess the most convenient way of illustrating this is where you type a pass phrase into a wireless router and can use the same pass phraze on the wireless clients to generate the same wep key in order for it to access the wireless router's services. With a random number generator applied like with openSSL, you would never get the same keys with the same pass phraze even when reentered on the same device. And of course this does no good if your not familiar with setting up wireless routers but it is the closest and easiest illustration I could think of. Without the random generator code, it would be like using WEP which we know can be broken pretty easily.
If my memory serves me correct, the random number generator can or does use static input from key presses as well as mouse movements and other activities to create the "randomness". It depends on the how the generator is designed. I think most linux boxes can even take randomness from the PCI bus so playing a music file or transferring a large file between drives or partitions could create it too. But it is important to understand that it will still perform without entering anything into a prompt and using a supplied file or something. That is what I mean by It can be random or not, it should work just the same whether it has been entered manually or from a text file somewhere. You will likely still need some activity to cause the random process, normal processes and stuff will cause it too. It just takes a lot longer then if you were to open a free cell game or a text file and typed stuff into it.
NSA has a much easier time inserting backdoors into Windows than in open source. And what about the backdoor to Firebird database that was only discovered after Borland open sourced it?
:). See Debian advisory for upstream link.
If you just want blood, you will NOT get it from Microsoft if their random number generator sucks. And it did, for a long while. The most damages you are entitled to is a refund of your purchase price. Want more?? Better get insurance or lucky with jury. And if you do get lucky, and if I were microsoft, I would sure require you and anyone connected with you to NEVER use the software every again (remove you explicitly from licensing in future)
The error here was 2 lines of code that upstream said were ok to remove (at least some people upstream
Then you are telling them a wrong thing. A database can be much more secure than the lock on the office door. Encrypt the filesystem and require passkey to startup the filesystem (NOT on the machine!). That way, you can't just go into the office, and steal it (the data). So since you apparently don't know that, and if someone steals their box and takes their data, can they sue you for not encrypting the disk??
;-)
In a class action, you can't get more than the product value anyway. So the point is moot. And people suing Apple or Microsoft are idiots in the first place.
Sure, you can sue the debian developer, or debian, or whoever the hell you want. And no, you will not get ANY money back and just get a huge public backlash.
Finally, I don't think you will win any modpoints calling the slashdot community stupid! Does that make you stupid as well?
Really? Absolutely certain is a very strong phrase! You're absolutely certain that every single line of code that winds up in a released product out of Microsoft has been peer reviewed by someone with a good understanding of the software, and a manager or someone has personally verified that the developer whose name is against that code in their source control system did in fact make the change? For every single bit of code in every single product, every single time any kind of release is made? And that the code in source control is in fact the code that was used to compile the release build, and no nasty hacker with knowledge of the processes got into any of the systems involved to plant their own code?
I'm sure they have stringent checks and plenty of security in various forms in place, but I'm not absolutely certain that they will always work 100% of the time.
Besides, I'm not sure that really matters. The change was made by a Debian developer, not some random Joe, and so is equivalent to someone at Microsoft deciding to make the change. Given it was a "simple fix" to make the valgrind output look nice, I don't see any particular reason why this kind of mistake couldn't happen at a commercial shop, even Microsoft. In fact, I'd suggest that a place like Microsoft would have very strict requirements that every build passes a variety of automated tests of the valgrind variety, as they release a crapload of software and it's essential they maintain same base level of code quality. That would increase the likelihood of a seemingly innocuous change in order to comply with code auditing programs resulting in problems, which may not be identified by the user test scripts.
Besides which, as others have pointed out in related threads, firing a talented and productive developer because they made a mistake will eventually result in you having very few talented and productive developers left. Also, all of this "accountability" is an effective way of rewarding those who cover up their mistakes, and punishing those who admit them. Both on an individual level (blame shifting etc.), and on a corporate level (not announcing critical security fixes if the flaw isn't already generating a lot of bad publicity).
Sure, if you say "Many blind eyes make bugs shallow" it will not hold. The premise of this statement is that there ARE some eyes which are actually watching.
Extreme Programming - Redundant Array of Inexpensive Developers
around 7.30 CET DST the ssh server keys were regenerated on our Ubuntu 7.10 boxes, which we noticed because the ssh client complained.
While it is good that this has increased security our first reaction was to wonder if we had been hacked.
One wonders, is there a good way to recover from a security gaff like this?
These companys will sell you a contract that DOES give your business recourse when something goes wrong. Read such a contract. It almost always boils down to "best efforts".
If you were exploited by a 0-day bug in code that was in OpenSSL at source (rather than any patches your distro maintainer had added), everyone else was also vulnerable but you happened to be the unlucky one who was exploited, I suspect the contract wouldn't be terribly useful.
This is why you don't rely on one thing as your sole source of security. In this case, you'd probably want to configure SSH to deny root logins, set up something like DenyHosts to thwart bruteforce attempts and maybe run some sort of intrusion detection system - preferably connected directly to a firewall so detected intrusion attempts can be automatically blocked.
Every Debian-based distro generates SSH host keys upon installation, and turns on sshd by default.
Except for Debian. The standard install (with none of the optional extra packages) does not include OpenSSH.I don't know about a full blown certificate agency, but where I'm working it shouldn't be too much work, since we use CF Engine to distribute certificates out to our servers from a central repository.
It would still be nice if we didn't have to fix problems caused by some idiot with no understanding of crypto though.
If my CA issued me with an insecure certificate, then I'd expect them to be reissuing for free whatever there usual policy is.
Sure as hell I wouldn't be paying for the priviledge of having a known problem fixed - especially since it doesn't actually cost them anything more then a process or two to regenerate a certificate.
It was unimportant and could be commented out at whim without breaking anything.
* Upstream using same code and same variable names to describe different thingsThe code was MD_Update(&m,buf,j);. If you look at variable names like "m", "buf", and "j" and conclude that every distinct use of those names describes the same thing, you need your head examined.
* Upstream having no comments in the code explaining the two things aboveRegarding the first thing, it was under an ifdef and at least had an overly terse comment noting why. Regarding the second, I'd hate to have to wade through drivel explaining that calling MD_Update with different buffers does different things. That would just drown out code and useful comments.
I'm not an OpenSSL developer, the code is poorly commented, etc., but it's obvious that it's poorly commented and it's obvious that "buf" does not necessarily mean the same thing each time it's a paramater name, regardless of whether more descriptive names are better.
I don't believe such machines would be remotely vulnerably to a direct assault, but a weak host key would make logins to that machine vulnerable to a MITM attack.
Please read the other replies in this thread. I am getting tired of answering this.
How a cascade of small failings lead to a major disaster.
OpenSSL leave in an "uninitialized memory read" because it "doesn't matter and might help". This is a bug. It does matter. The fact that all compilers will deal with this bug in a way that is OK is not sufficient in security critical code. Once you reach this point _anything_ can happen. If the compiler decides to flag this memory as read only and never modify it again then that is allowed. I can't imagine why a compiler or system would do something like this but it's allowed to. The OpenSSL code is non-portable and fragile.
And many people will be using this library in places where the memory is guaranteed to be initialized a particular way. Either this doesn't matter (in which case it doesn't matter if the uninitialized memory is initialized to a known state at the start) or it does matter in which case OpenSSL is broken on platforms where this memory is not random.
Debian spots this uninitialized read and tries to fix it. The proposed update is passed upstream but the email is ambiguous as to what exactly is required.
OpenSSL team do not jump on this. If the email went to the wrong address then the right address should have been provided (and the email forwarded as a courtesy). If this was an update to sensitive, critical code (which it was) then someone should have said "hold on there, this is sensitive code and introducing bugs here is really serious. We need a precise patch, we need time to evaluate the patch, we need time to work out how to fix the underlying uninitialized memory bug."
I haven't looked but has the underlying bug in OpenSSL been fixed even now? We've got an "undefined behaviour" bug in what we now know to be a critical path in OpenSSL. IMO Debian are doing the correct thing in fixing this bug if upstream hasn't fixed it.
Have Debian logged a bug with upstream? (Unfortunately, I suspect if they did this now it would "go political" - but I hope I'm wrong)
At the end of the day lots of people got things wrong. I feel particularly sorry for the Debian developer who appears to have done everything he reasonably could but still got caught out.
YMMV.
Tim.
God said, "div D = rho, div B = 0, curl E = -@B/@t, curl H = J + @D/@t," and there was light.
The bad version could only generate 2^16 unique keys for each length of key. The only source of entropy for those keys was the process ID.
I stand corrected - it was indeed 2006 looking at the advisory again.
While not a security argument, I would say that the main argument against the patch was that you shouldn't change runtime behavior just to make debugging easier. Which I think is a good idea, especially for downstream. If they had followed this idea, the patch would never have been committed.
The problem isn't with people generating their own certificates, but rather with people generating their own keys. Certain kinds of hosting generate your key for you, but you shouldn't really be using them. (And you'd have to know if they were using a compromised version of OpenSSL.)
It's easy enough to test for this; you should have a private key for your SSL cert. Run "openssl-vulnkey commercial.key" (or whatever your key's filename is) on a patched machine in order to test it. The one that my organization is using, for instance, comes up compromised.
(Coming up not-compromised wouldn't have meant that the key was definitely in the clear, but coming up compromised means that you definitely need to replace it. Enjoy paying for your incredibly expensive bits yet again.)
Laws do not persuade just because they threaten. --Seneca
This is page 45-46, which talks about the reasons why a certificate may be revoked:
Rough translation: The certificate may be revoked when ... there's something that makes you reasonably believe that the certificate has been compromised to the point that it is not reliable.
And then on page 47 it says:
Rough translation: If the revocation of the certificate is not the fault of the subscriber (that's you), then you get a new certificate with the same validity period as the old one.
Obviously, you have to check with the contract governing your CA, but you might find something similar, so check with your CA before paying for new certs.
Very, very wrong...
Also on the client side you generate certificates, i.e. if you generated certificates for doing password-less login to a machine...
OpenSSL used a variety of sources of randomness all mushed together, including /dev/random and whatever else the developers could think of. One of these methods of randomness was looking into uninitialized memory space. Because a debugging tool (valgrind) about that, the Debian guy commented it out (with an okay from the upstream mailing list). However, he made a rather ham-handed mistake and removed all sources of entropy (with the exception of the current process ID, which counts only enough to hide this bug rather than make it crushingly obvious) rather than just the one.
So, now it's time to buy stock in Verisign, in Thawte, in whoever's going to be scoring a huge windfall off of this.
Laws do not persuade just because they threaten. --Seneca
The SSL key my workplace uses to secure our websites is vulnerable. Now, of course we're getting it revoked and replaced with a non-vulnerable one, but that doesn't stop the problem. How many web clients actually check CRLs? Given that CRLs are a terribly stupid idea and nobody uses them anyway, how many web clients actually check the embedded OCSP URLs in certs that support them? (It's under Preferences/Advanced/Encryption/Verification in Firefox 2, and it's disabled by default.)
An as-yet-undetermined set of SSL websites are vulnerable (until their certs expire) to being hijacked and masqueraded-as--with that shiny yellow bar at the top, no less. And if there are any root CA keys out there that are vulnerable to being guessed? This problem is going to be with us for years.
Laws do not persuade just because they threaten. --Seneca
A patched machine will reject logins using blacklisted keys. Of course, this means that when you ssh into one of them and update the packages, you'd better have your new, fixed keys installed, or else you'll be locked out of your server.
I suppose you'd better hope you don't have security updates automatically installing themselves.
Laws do not persuade just because they threaten. --Seneca
The CA has to add your cert to their CRL and make that list widely available until the cert expires. They also may have to run an OCSP responder to let users know that it's been revoked. (The fact that pretty much nobody actually uses these mechanisms doesn't change anything.)
Additionally, it's almost certain that this isn't the CA's fault; people generally generate their own RSA keys, then send a CSR to the CA to get them signed. It's not the CA's fault if the original key was bad, and there's nothing they could have done about it at the time in any case.
Laws do not persuade just because they threaten. --Seneca
Your system, when an SSH server (the package on Debian or Ubuntu is called "openssh-server") is installed, it generates host keys (that is, secret keys that identify your computer uniquely). This update will regenerate the keys for you, though you can do it manually (as root: "rm /etc/ssh/ssh_host*; dpkg-reconfigure openssh-server") if you need to for some reason. Note that regenerating your keys will not fix anything unless you install the update first, because the generated keys will still be weak.
If you or anyone else logs into your machine via SSH, they'll get a warning about the host keys having changed. If you log into anyone else's machine via SSH, you'll need to make yourself new keys (by running ssh-keygen) and send everyone your new public key.
If you run an SSL website with an affected key, you'll have to get the certificate revoked by the issuing CA, and get a new key signed to make a new cert.
There's a list of applications which may be using weak keys at the Debian wiki. If you're not running any kind of server, it's likely that you're only going to need to replace your SSH keys, as described previously.
Those who administer servers are in for a significantly bumpier ride.
Laws do not persuade just because they threaten. --Seneca
Generating a certificate involves generating a public and private key pair. You provide the public key to the CA (e.g. Thawte) and they give you a certificate which says that the corresponding private key is to be trusted. What matters is where the public/private key pair were generated. If you did that on a vulnerable Debian/Ubuntu machine, your certificate absolutely is vulnerable. The vulnerable act is the generation of the keys, not the signing of the certificate.
--Quentin
"Video bona proboque; deteriora sequor." -- Ovid
I doubt that a malicious mole would have invited this exchange. But perhaps that makes it all-the-more brilliant, hmm?
Seriously, if one wanted to compromise the Debian codebase, there'd be far subtler places to do it. The fact that this gaping hole went undetected for two years suggests that there's probably little need to insert new ones.
How a cascade of small failings lead to a major disaster. OpenSSL leave in an "uninitialized memory read" because it "doesn't matter and might help".
I think you're putting words in their mouth. The OpenSSL developers didn't lazily "leave it in" and dismiss legitimate concerns by saying "it doesn't matter". They wrote it that way intentionally.
This is a bug.
No, it's not a bug. In C, if someone you passes you a buffer of a stated minimum size, there's nothing wrong with you reading from it. There's no such thing as "write only" memory.
Sure, 99.5% of the time, mediocre C code reading such memory indicates an error. Most programs are not in the business of accumulating entropy. Tools like Valgrind pick up on that to warn about it, and as useful as they are, they're no "gold standard" of program correctness.
However, accumulating entropy for random number generation is an entirely valid use of such a technique. The OpenSSL developers knew exactly what they were doing, and this happen to be one of those 0.5% function which legitimately reads allocated-but-uninitialized memory.
The Debian developer appears to have been robotically following "policy" and changed code without fully understanding its consequences.
It does matter. The fact that all compilers will deal with this bug in a way that is OK is not sufficient in security critical code.
The security-critical code behaved correctly before the "fix".
Once you reach this point _anything_ can happen. If the compiler decides to flag this memory as read only and never modify it again then that is allowed. I can't imagine why a compiler or system would do something like this but it's allowed to. The OpenSSL code is non-portable and fragile.
I know OpenSSL to be a portable program by virtue of the fact that it has been ported to many platforms.
Perhaps it is a bit fragile, but considering how much effort has gone into providing so many algorithms with hand-tuned assembly for multiple processor architectures, I'm willing to overlook it a bit. And every common platform except those from Microsoft (and perhaps now Debian), I can expect that someone else has done a reasonable job of packaging it for me.
And many people will be using this library in places where the memory is guaranteed to be initialized a particular way. Either this doesn't matter (in which case it doesn't matter if the uninitialized memory is initialized to a known state at the start) or it does matter in which case OpenSSL is broken on platforms where this memory is not random.
It's not your false dichotomy of "does/doesn't matter". Entropy is accumulated when it is available, and is not decreased when it is not. Since there's not a source of perfect entropy available, we need to take what we can get, whenever we can get it.
Debian spots this uninitialized read and tries to fix it.
It's not a bug, and shouldn't be "fixed".
The proposed update is passed upstream but the email is ambiguous as to what exactly is required. OpenSSL team do not jump on this.
That exchange looks like a beautiful example of miscommunication. I don't think it ever occurred to the OpenSSL developers that the Debian guy was thinking about commenting out all entropy mix-in for the production release!
If the email went to the wrong address then the right address should have been provided (and the email forwarded as a courtesy). If this was an update to sensitive, critical code (which it was) then someone should have said "hold on there, this is sensitive code and introducing bugs here is really serious. We need a precise patch, we need time to evaluate the patch, we need time to work out how to fix the underlying uninitialized memory bug." I haven't looked but has the underlying bug in OpenSSL been fixed even now? We've got an "undefined behavio
This is dangerously misleading! Blocking IPs after a reasonable number of failures stops only one particular attack against SSH. It prevents someone from logging into a user account that has a compromised-Debian-generated SSH public key by brute force, trying all possible compromised-Debian key pairs. There are a number of other attacks.
The most serious involve the SSH host key. The public host key is given to anyone who attempts to connect to the machine, whether they succuessfully authenticate or not. This must be the case -- giving out the public key is part of establishing an encrypted connection, and you must have encryption before you send your password or something. An attacker can connect ONCE to your server and, from the public host key, lookup the corresponding private key offline in microseconds. They can then decrypt any SSH communications they can sniff where that server is the server. You have little more security than telnet. They can perform a man-in-the-middle attack on the server. Even if a user is carefully using public-key authentication with something other than a compromised-Debian-generated key rather than password authentication, if they connect to a server with a compromised-Debian-generated host key, they might be connecting to a MITM attacker who can then perform any action on the server as said user.
Public-key authentication is still seriously vulnerable even if you limit the number of attempts. Normally, I can generate a key pair on machine A and set up machines B and C to allow me to log in with that public key, and a root attacker on machine B or C will not be able to access the other machine with my identity. If machine A is a compromised-Debian machine, this no longer holds. Anyone with access to the public key can immediately obtain the private key and impersonate me. Moreover, I normally wouldn't take the same precautions with my public key as my private key, but with a compromised-Debian-generated key pair they are practically equivalent. In general, there is no basis to rely on the public key being secret, which you implicitly recommend doing.
Any rational and informed person will immediately regenerate all key pairs generated with a compromised-Debian machine, and remove such public keys from any authorized_keys files in which they appear. They would also do well to consider the Ubuntu strategy of rejecting known weak public keys for public-key authentication.
Wow. That's so cute.
You actually seem to believe that you are not only a customer of Microsoft, but that your happiness somehow matters. You probably think that TV stations should support programs you like because you are their customer too. You even used the phrase "class-action suit" as if it was something big and scary.
Microsoft is interested in one thing: Making money. If they can cut corners without losing their multi-bazillion installs every year through Dell, then they will do it. If that happens to piss off a number of self-styled security experts and drives them off to use a third party product, then that just means the corporation has to spend less time and money supporting them. It's not about "doing the right thing" or "making code for code's sake", it's about shipping a whole lot of product and making money off of it.
Given the choice between putting in a one line hack that makes a number of very obvious and potentially embarrassing profiler warnings go away and assigning enough people to make a proper change, a project which could take weeks or months, when nobody is ever going to notice the difference until long after next quarter's financials have come in, and the seven quarters after that, why do you believe that a large corporation is somehow going to be motivated to spend extra money when they don't need to?
Sittin' there on that sack of seeds!
"Having distributors randomly change source code as they package it is fundamentally broken."
This is one of the reasons I switched to BSD, where such things tend not to happen.
In the old ANSI C standard (sorry I don't have the new one on my desk) it's in section A.6.2
Undefined behavior:
The behavior in the following circumstances is undefined:
* The value of an uninitialized object that has automatic storage
duration is used before a value is assigned ($3.5.7).
Tim.
God said, "div D = rho, div B = 0, curl E = -@B/@t, curl H = J + @D/@t," and there was light.
No, it's not a bug. In C, if someone you passes you a buffer of a stated minimum size, there's nothing wrong with you reading from it. There's no such thing as "write only" memory.
Reading uninitialized memory invokes undefined behavior. Once you do that the compiler is allowed to do anything.
* Undefined behavior --- behavior, upon use of a nonportable or
erroneous program construct, of erroneous data, or of
indeterminately-valued objects, for which the Standard imposes no
requirements. Permissible undefined behavior ranges from ignoring the
situation completely with unpredictable results, to behaving during
translation or program execution in a documented manner characteristic
of the environment (with or without the issuance of a diagnostic
message), to terminating a translation or execution (with the issuance
of a diagnostic message).
Emphasis mine.
Tim.
God said, "div D = rho, div B = 0, curl E = -@B/@t, curl H = J + @D/@t," and there was light.
IranAir Flight 655 never forget!
If my memory serves me correct, the random number generator can or does use static input from key presses as well as mouse movements and other activities to create the "randomness".
For the third time, this is what I'm talking about.
the issue isn't the lack of a key for input being physically typed in
I know how public key cryptography works.
The blacklist is active by default, at least on Ubuntu. Add a blacklisted key to an authorized_keys file and try to log in using it. auth.log on the server will have a line like the following:
May 15 08:47:40 ns1 sshd[2989]: Public key 27:26:b8:52:d2:8e:7a:18:3f:8e:81:87:6b:7e:87:e9 blacklisted (see ssh-vulnkey(1))
and the client will read Permission denied (publickey).
Keys do need to be regenerated, but a patched system is not vulnerable to SSH key guessing, no matter what the authorized_hosts file contains. Bad keys will simply stop working. Any SSH remote-login vulnerability is fixed as soon as you dist-upgrade your packages, installing openssh-blacklist.
(Okay, certain keys which have options on them weren't detected by the first version of the ssh-vulnkey tool, but now they are.)
Laws do not persuade just because they threaten. --Seneca
Sure, the ANSI/ISO C standard does not prohibit Valgrind from terminating your process, or gcc from issuing code that reformats your hard drive. However, there's no law that says no programs can be written which do things not explicitly endorsed by that particular document. In reality, systems programming occasionally needs to use platform-specific functionality that comes from the the C language standard does not define. By your definition of "bug" one could not write an multithreaded application, OS kernel, debugger, or use a memory-mapped file.
Opening a text file and typing a report would have the same effect as a box opening for you to press random keys in in case you still want it after installing the fixed code. The lack of that isn't the problem though, it is the inability to use the input from that in processing random numbers that (or any of the other ways it takes cues to generate characters) is.
And BTW, I didn't intend to make it look like I was attempting to school you. It's just that you aren't the only person reading this and it was an attempt to be as clear as possible with the ability for someone else to correct me if I got something wrong. Don't take my thoroughness as an insult. It wasn't intended to be one.
Nice, thanks.
But the buffer passed to those functions is not necessarily in automatic storage, in that function the compiler can't treat it as undefined behavior. At least that is how I see it.
"Video bona proboque; deteriora sequor." -- Ovid
So who got fired for the problem? Who lost their job? Or their bonus?
I'm not saying that I need to get money when something doesn't work to perfection. I'm saying that someone is held accountable not out of my need for revenge, but out of the very simple concept that actions should have consequences -- both negative and positive ones.
I'm not asking for accountability out of a sense of revenge. I'm simply saying that actions benefit from consequences -- negative and positive alike.
Regarding the database, no one ever said stealing the data involved stealing the machine. The database is a live server, running, with data loaded. Not only can someone find the unencrypted data in active memory during decryption, but someone sitting there can simply use the interface directly. Second, any encryption can be broken, especially if you have unlimited time to do so. Stealing the machine and taking it away gives you all the time in the world. And finally, not all data is always encrypted all the time. Even that data is private, confidential, secret, or valuable.
And yes, if I weren't encrypting the credit-card data, and it got stolen, I would be sued by a dozen clients. And not only do I agree with that, but I offer that accountability voluntarily. They have a very simple choice. They can trust that I've done things correctly, or they can go and hire someone to check. If they trust me, they pay me that extra money, instead of someone else.
If your entire life is "who pays in blood for a mistake", then you live a very very sad life.
"I'm saying that someone is held accountable not out of my need for revenge"
All your posts ARE about REVENGE. I think you should crawl under the rock you slithered out of as you do not seem to provide any constructive criticism at all.
It doesn't have to be the compiler.
Imagine a new type of flash ram that can be in a low power, read only state or a high power read/write state per cell.
It also has the property that the low or high power state is initially selected based on the first access. Once the low or high power state is chosen it can only be changed again by going though a complicated procedure.
The compiler/system is allowed to assume that if it returns a block of memory from malloc etc that the first access will be a write. If you do make a read first then you will switch the memory into it's low power, read only state.
Or say the memory is paged. The first write operation causes a new page to be allocated from the swap file but if you start with a read then that causes the paging mecanism to always return 0xFF for that memory.
I'm not saying these are likely setups, but the C standard allows them.
Tim.
God said, "div D = rho, div B = 0, curl E = -@B/@t, curl H = J + @D/@t," and there was light.
I specifically said that it's not about revenge. I said it's about holding people responsible and accountable for their actions and decisions. I wouldn't want to live in a world without consequences -- that's a religious world, where credit and blame are conveniently shuffled off to an imaginary creature.
But if you're saying that you prefer to live in a world where you acn make mistakes willy-nilly and not be held accountable for them, then clearly you do shotty work at best, and are not to be trusted with anything important.
That's not what I said. What I said was that given the choice between making a change that is sure to cause public mayhem, and one that won't, anyone trying to ship product will choose not to anger their entire market. In the case where an employee makes the opposite decision, that employee will be held responsible for going against the company's interests. It's got nothing to do with me as the customer, it has everything to do with us as the market.
I never said you have to fire the person, I said you get to hold them accountable. Please see the difference. Bonusses, vacations, required to stick-around and fix things late, reference letters, reduced salary, demotion, fringe benefits, allowances, assistants, and general trust.
And yes, I'm absolutely certain that the code was written by someone who works for microsoft, doesn't have to be a manager, can be a lowly programmer, but the point is that it's still someone who can be held accountable for mistakes.
..it's not the uninitialized buffers that are the issue. It's that the call to use info from the system entropy pool was *also* commented out, accidentally, in the process.
Here's the basic flow as it was, with *EVERYTHING* else snipped out:
Original:
----------
Add the following to Our_Randomness:
Lots of highly random info using system algorithms designed for the sheer purpose of being random
Some rather random information from some rather random location in memory, causing a warning to issue forth from some compilation utilities
A tiny amount of randomness from the kind-of-random process ID
Generate Key, using Our_Randomness.
----------
Debian seems to have intended to change this to:
----------
Add the following to Our_Randomness:
Lots of highly random info using system algorithms designed for the sheer purpose of being random
A tiny amount of randomness from the kind-of-random process ID.
Generate Key, using Our_Randomness.
----------
That would have been fine.
But instead, it was accidentally changed to:
----------
Add the following to Our_Randomness:
A tiny amount of randomness from the kind-of-random process ID.
Generate Key, using Our_Randomness.
----------
Notice the missing usage of "Lots of highly random info using system algorithms designed for the sheer purpose of being random".
I am not an atomic playboy.
Ah yep since I wrote the comment you replied to I read all this.
I think the point made by one of the comments on the bug report saying something like "let's not change stuff just to make it debug easier" is a valid point, and what happened demonstrates why. I think this is also an issue and that this POV was ignored shows a problem with Debian's patching policy in general.
I never said you have to fire the person
So when you said When a Microsoft programmer makes a mistake, they risk losing their job. you meant they risk misplacing their job, and not being able to find it again? Okay... :)
I said you get to hold them accountable
Yes, and any kind of meaningful accountability is going to have negative consequences for the person or people involved. If a lowly programmer makes a mistake, their manager and their manager's manager are going to held accountable as well, because it's their responsibility to ensure there's procedures in place to prevent mistakes being released and ensure the procedures are always followed.
So either the accountability is such that there's going to be a strong incentive for those involved to hide mistakes whenever possible; or the accountability is going to be on the same level as for open source software and therefore, according to you, either meaningless or non-existent.
Furthermore, you're assuming the person responsible still works at the company. 2 years is a long time for a programmer to spend in one job, and there's plenty of stupid security flaws and other bugs that are way more than two years old.
Here's the things you suggested:
Bonusses, vacations,
It's likely people will be very keen on retaining both of these, and not many people are going to be keen on signing contracts that say the company might deny your vacations on a whim.
required to stick-around and fix things late,
This assumes there's some kind of punishment for not staying 'til late, and can apply just as much to open source software -- so long as you're actually willing to follow through with a meaningful punishment. "Fix it or you're off the project."
reference letters,
A lot of open source coders will use their open source work in their CVs. I'm sure the Debian openssl maintainer used to have that on his CV. Do you think he'll be so keen now, knowing that when any potential employer searches the 'net for info on his involvement they're going to find out he introduced a bug which rendered all Debian-generated keys compromised for 2 years?
Also it's worth noting that this information is public and easy to find. Employers providing reference checks will usually limit themselves to confirming that the individual worked their in a particular position, and not make any comments on their job performance. It's just too dangerous, and the company doesn't benefit from doing so. Even if Microsoft does fire a programmer for writing really bad code, all anyone else will know is that they did in fact work at Microsoft on such-and-such a project for a period of time. Most employers would see that as a good thing. If the person involved thinks the people at Microsoft will actually warn people not to hire them - as unlikely as that is - then they can simply leave their time at Microsoft off their resume. Say they took time off to find themselves, or whatever. Or simply make sure the reference they provide likes them and will say nice things.
So in terms of accountability for future positions, there's virtually none in the commercial world because your blunders aren't going to be publicised for the world to see. You screw up in free software and it's going to be all over mailing lists and tech sites. Any time you apply for a job or contribute to another project, you have to be prepared to be asked "wtf happened with such-and-such a project". For the rest of your life. If anything, there's too much accountability for mistakes in free software.
Using this particular case as an example, the Debian maintainer made a pretty innocent mistake. This could happen to anyone. However, it was both a) in a very important package and b) not noticed for a very long time. Tiny innocent mistake is now a big black mark against his name for the rest of his life. If the
I find all this rather unsuprising.
...
;-)
Quality of some debian packages is really, really low - obviously fanatical adherence to the "bazaar" development model does not
always pay off
What's annoying is sheer arogance of lame folks.
C'mon people. This is unix - not windows
Bah. Posted in response to wrong comment after copy/login/paste. My mistake.
I am not an atomic playboy.
The weakness cuts the cost of brute-forcing down to minutes. There are only 65536 possible values for keys that are generated using this piece of code, and some of that space can be cut down using a little bit of knowledge (the 16 bit seed is the pid of the process, so it's not uniformly distributed. Pretty much anything that is worth the hassle of using ssh/SSL for is worth 20 minutes of an attacker's computing time (effective cost ~0).
my password really is 'stinkypants'