Slashdot Mirror
Debian Bug Leaves Private SSL/SSH Keys Guessable
SecurityBob writes "Debian package maintainers tend to very often modify the source code of the package they are maintaining so that it better fits into the distribution itself. However, most of the time, their changes are not sent back to upstream for validation, which might cause some tension between upstream developers and Debian packagers. Today, a critical security advisory has been released: a Debian packager modified the source code of OpenSSL back in 2006 so as to remove the seeding of OpenSSL random number generator, which in turns makes cryptographic key material generated on a Debian system guessable. The solution? Upgrade OpenSSL and re-generate all your SSH and SSL keys. This problem not only affects Debian, but also all its derivatives, such as Ubuntu." Reader RichiH also points to Debian's announcement and Ubuntu's announcement.
I'm sure the problem will be fixed if the developers acknowledge that the problem exists. Not a big worry.
I remember installing openSSL on debian long ago and being prompted to type some random text or to move the mouse while the key was generated. More recently, when I installed debian or ubuntu, it just magically generated a key. I guess that was the part that was removed.
Who did this? You don't remove the seeding... stupid
did I mention stupid?
this is how some of the old video games were "broken" despite using "random" numbers, the seed was always the same... leading to the same sequence of events
I will not give in to the terrorists. I will not become fearful.
You told me your OS was secure, and you leave in huge HUGE hole like this? For 2 god damned YEARS?
Windows here I come. At least someone would be accountable for shit work like this.
Ubuntu got an updated advisory at http://www.ubuntu.com/usn/usn-612-2
And we were told this sort of bug could NEVER happen in an open source operating system. Seriously, this is a major cockup.
whether this can possibly be claimed to be an accident *dons tinfoil hat*.
But seriously, removing the code that seeds a random number generator? I can hardly imagine making such a change by accident. I may just lack a sufficiently colorful imagination, though.
(or, before resorting to conspiracy theories, we should probably ask ourselves first, "can this possibly be explained by simple stupidity?"
Every expression is true, for a given value of 'true'
Got up this morning, booted the machine and got a software update first thing: OpenSSH (et al) updates for my Ubuntu Gutsy install. Then I show up over here and see why. Presumably Feisty and Hardy got them as well - they are listed on the Ubuntu announcement.
First off I'm a big OSS supporter, yada, yada
But the point here is that the freedom that OSS gives you does require you to trust the whole distribution chain. In this case there was an added muppet who did something they shouldn't have thus rendering everything downstream insecure. OSS is great but it required great developers, given that it has take well over a year to get the advisory out it shows that the many eyes piece didn't work here, mainly because the eyes were looking at the original source not the botched packaging job.
The "easy to use" Linux box in the house uses Ubuntu and has this issue and like many people I didn't even think to check that the OpenSSL wasn't the REAL OpenSSL it was OpenSSL with muppet extensions. Maybe there needs to be some form of extension that warns that a package has been modified from its original source code and that the modification was done by "K. Frog" so you can determine whether to trust that package or look back to the source.
Or some sort of voting system on contributors (how very Web 2.0) so you can see how the people who touched your package were rated with the biggest weighting being given to the last person through the code (hand edited by Linus = 5 stars, hand edited by James Gosling = 5 stars, hand edited by the bloke who wrote clippy = 2 stars, hand edited by a bloke who removed a seed generator = 0 stars).
Having the code is great, but this makes me want to know much more about who last edited that code.
An Eye for an Eye will make the whole world blind - Gandhi
Why are we not just using /dev/urandom for this, which is already seeded with random data?
Do we really need to reinvent the square wheel for every cryptographic program?
... this had to be discovered four days *after* I finished setting up my new Ubuntu 8.04 server... *grumble*
Proudly supporting the Libertarian Party.
Preparing to replace openssh-server 1:4.6p1-5ubuntu0.2 (using .../openssh-server_1%3a4.6p1-5ubuntu0.3_i386.deb) ... /var/lib/dpkg/tmp.ci/templates has a duplicate field "template" with new value "ssh/vulnerable_host_keys". Probably two templates are not properly separated by a lone newline. /var/cache/apt/archives/openssh-server_1%3a4.6p1-5ubuntu0.3_i386.deb (--unpack):
Template #4 in
dpkg: error processing
subprocess pre-installation script returned error exit status 255
And then:
openssh-server: Depends: openssh-client (= 1:4.6p1-5ubuntu0.2) but 1:4.6p1-5ubuntu0.3 is installed
Anyone else seeing this?
It was accidentally introduced in 2006... so that's what, another 14 years before it gets moved into 'stable'?
*grin*
So this is how linux is going to replace Windows on the desktop? By creating custom functionality that break RFC and common sense? Some things never change, do they?
"You fell for one of the classic blunders, the most famous being 'Never get involved in a land war in Asia' but only slightly less well known is 'Don't use poorly seeded pRNGs in cryptographic protocols!' HAHAHAHAHAHHAHAHAHAHHAHAHAHAHAHA!!!!
Test your net with Netalyzr
Funny how 2 years after the fact people make a big fuss over it.
The seeding was removed and it wasn't noticed for TWO YEARS? In a distro as popular as Debian?
This is why I only use software that is cryptographically signed by Windows Update; that way there's someone to sue if critical security flaws are "updated" into my system. Laugh all you want at the Genuine Advantage, I think this story proves it exists.
'Makes it appear to be run by a bunch of half-wits.. I don't understand any level of justification that would make anybody think it was wise to touch SSL in your own distribution.
Such bugs and the thread posted from debian discussions show how far linux has to go to really be viewed in any sort of professional light.
http://www.random.org/analysis/dilbert.jpg
http://www.xkcd.com/221/
This is on the SSH server side. If you are a casual desktop user, you don't have to do anything.
One of our competitors trademarked the term "hypothesis". From now on, we will call them "boneheaded ideas".
Who was the member of the security services who acted as package maintainer?
I cannot believe that such a change would have gone by unnoticed if it had been properly documented, and I cannot believe anyone given the trust of the OpenSSL distribution would make such a mistake.
This was an intentional act of sabotage.
Doh!
Looks like they were a few pairs of eyes short. Again.
Wasn't FOSS suppose to prevent these types of problems in the first place?
I wake up and what do I see first thing? That there is a problem with Debian's OpenSSH package and the
Now I am thinking, "What exactly is going on here? Is choking on a bucket of cocks not a good source of randomness?"
I honestly don't think this is much more than a mistake, like one of those who surfaces every two years. I wouldn't go as far as blaming OSS or any other instance for that matter. No matter what you feel, you cannot deny that Debian usually has some of the most comprehensive focus on security in the scope of popular Linux Distributions.
It's all fun & games until someone loses the game.
Anyone who posts to this story saying that this is no big deal or telling other people to stop whining should simply be banned from Slashdot for life. If you cannot be bothered to read the article and you cannot be bothered to understand just what a serious vulnerability this is but you insist on insulting those who do, why should you be allowed to continue to post your ignorant bleating?
If you mod me Overrated, you are admitting that you have no penis.
In fact I'd expect that separate runs of the same program with the same parameters and environment would leave the same junk on the stack every time.
So I would hope that they have some better source of entropy than unpredictable uninitialized data of dubious randomness.
So if this is really a serious problem, then it seems to me there's already a serious problem in OpenSSH.
I've been getting badsigs since last week every time I try to update Hardy. Looks like I'm not alone, judging by the forums.
Hmm... no, it shouldn't have been changed. There is absolutely no excuse for touching crypto code when you don't *fully* understand it.
But the fact that changing the use of NON-INITIALISED memory causes most of the seed of a RNG to go away really is worrisome... unless that memory is actually initialized with a seed BUT in a way Valgrind and others can't track. Which means it is sloppy coding on OpenSSL.
You cannot trust unitialized memory, PERIOD. And it will take looking deep into OpenSSL code for me to tell if it was intialized in a way valgrind couldn't see, or that OpenSSL was working due to just plain blind luck (or lack of an attacker).
Either way, upstream prepared the trap, and a debian maintainer sprung it.
Why do YOU think we are not calling for blood on the guy who did the change? Because code that breaks like that is a travesty by itself.
This isn't about "desktop linux", this is really a huge problem for server admins.
And what does kerberos have to do with anything?
There is a great little story in the link. The best part of the story is where the mod says, "Although this abuse is unfortunate, I fail to see what makes this bug release critical." I nearly fell out of my chair laughing.
So who's accountable for damages as a direct result of such a problem? If I were using such software to run my business, and this sort of security problem became more than just a threat, what sort of recourse do I have? Which programmer do I get to sue?
The patch was:
--- openssl-0.9.8c/crypto/rand/md_rand.c
+++ openssl-0.9.8c/crypto/rand/md_rand.c
@@ -271,10 +271,7 @@
else
MD_Update(&m,&(state[st_idx]),j);
-/*
- * Don't add uninitialised data.
MD_Update(&m,buf,j);
-*/
MD_Update(&m,(unsigned char *)&(md_c[0]),sizeof(md_c));
MD_Final(&m,local_md);
md_c[1]++;
What is this? Why would Debian ever make such a change? This is either arrogance or malicious.
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
I thought /dev/random used several sources of randomness. Does SSH not use /dev/random?
I, on the other hand, strongly suspect that any similar mistake at a major software corporation would in all probability be quietly ignored, if it were even noticed at all -- and if it were instead deemed enough of a public relations risk to warrant dealing with, the company would likely just silently push an update to correct the problem for future users, leaving anyone using extant keys with their arses hanging in the breeze.
But maybe I'm just being overly cynical. :-\
Cheers,
"What in the name of Fats Waller is that?"
"A four-foot prune."
I'm pretty sure it was getting the memory off the stack
And when main() calls foo() calls bar() calls do_random_seed() the stack is going to be different from the next run when main() calls foo() calls bar() calls do_random_seed()? Last I checked, apache starts with the exact same commandline parameters every time I run apachectl start.
I'm glad I keep port 22 firewalled, except from specific hosts. The largest problem is the CA certificates though...
Are encrypted file systems affected?? cryptsetup for example?
Read the end-user agreements for your commercial software and you will see that you have no right to sue them for anything beyond the purchase price of the software.
I just updated again and they fixed the package.
Affected keys include SSH keys, OpenVPN keys, DNSSEC keys, and key material for use in X.509 certificates and session keys used in SSL/TLS connections. .. so basically every form of cryptography we use in daily business is compromised as we're running debian almost exclusively. WHO THE HELL LET THIS KIND OF FUCKUP TO HAPPEN?!?
I think this is the last straw for us on linux, the last kernel exploit was bad enough. Moving to OpenBSD asap.
First off, we were not told any such thing -- or at least I've never run across any such claims myself. Secondly, this would actually seem to substantiate the many-eyes theory, as the bug was found and summarily corrected, rather than never found, or found and swept under the rug, or found first by black-hats and exploited, all of which seem quite common in the proprietary software world.
The many-eyes model does not guarantee zero bugs, nor does it guarantee speedy bug hunting. However, it would seem to guarantee that bugs will be found eventually, and also that they will be dealt with in some productive way once they are found. This model aims for the opposite of security through obscurity, that hoary old chestnut of proprietary development, by instead ensuring security by knowing exactly what things do. That also means that folks need to go through all changes -- i.e., many eyes actually need to go over the code. Open source ensures that this is possible, but actual people still need to put in the time and effort.
Cheers,
"What in the name of Fats Waller is that?"
"A four-foot prune."
Schwab
Editor, A1-AAA AmeriCaptions
Has someone actually checked the random output for cryptographic soundness?
The proper source of cryptographic-quality random bits in Linux is "/dev/random". Reading uninitialized memory is not a good source of entropy - it might be initialized to some constant value, especially if you're compiling with a debug allocator or running under a virtual machine monitor.
OpenSSL makes substantial efforts to get a good random starter. Unless someone did something so stupid that OpenSSL didn't use /dev/random, it should still work. OpenSSL is supposed to have a check for bad random seeds. Was that bypassed, or doesn't it work, or what?
Obtaining a good source of randomness is hard. Computers are rather deterministic. Historically, there have been major failures in this area. See "Venona" where the USSR was generating "one-time pads" by having people type random digits on typewriters. Arlington Hall, NSA's predecessor, cracked that. Humans aren't random enough. True random number generation requires special hardware, like a noise diode or a radiation source.
"Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin." -- John von Neumann
From this log: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
It looks like openssl is pulling in "entropy" from uninitialized memory, causing valigrind to complain. The debian maintainer "fixed" this issue by memsetting the buffer to zero.
My question is, wouldn't I see the same behavior using grsecurity to scrub deallocated memory? From what I am seeing, this looks much more like the fault of the openssl team.
OK, this is as good a place as any.
FUCKING IDIOT NOOB ASSHOLES!!!!!1!
Fight hunger. Filet a politician and send him to a 3rd world country of your choice.
Can someone explain what "guessable" means in this context?
Does that mean someone can now generate a "master-key" and ssh into root@any-debian-box that allows public key authentication?
What does a realistic attack scenario look like?
clear explanation of incident
So, who's the CIA agent that implemented this "feature"?
The host keys on my two file servers, generated while they were still running sarge, were OK according to the perl script provided by Debian to verify whether a key is weak or not. If you were previously running sarge and have upgraded to etch but haven't issued new keys after upgrading, you're fine (as least as far as your host keys go). The host keys on my recently installed etch workstation were detected as being weak, however.
Facepalm
I don't have any other words.
Mod parent up.
How do you know uninitialized = random or even different?
You could end up reading some BIOS junk that's the same even after cold reboots.
It appears to have already been patched on my Debian etch system (both kernel and libraries). Does anyone know how to force all my ssl/ssh keys to refresh? Yes, I know it'll break all the old connections, but I'd much rather know it was secure.
Is there a way to suppress specific valgrind uninitialized memory warnings without having to clear the buffer (and potentially mess up other code)? Some of my code has the same problem - valgrind whines when I encrypt a block of uninitialized memory, even though the numbers are intended to be random.
"Screw Sun, cross-platform will never work. Let's move on and steal the Java language." - Visual J++ Product Manager
Found this post at openssl-dev list by Kurt Roeckx ( AFAIK the Debian OpenSSL Team member that made this RNG-clean patch )
http://www.mail-archive.com/openssl-dev@openssl.org/msg21156.html
Extract:
"What I currently see as best option is to actually comment out those 2 lines of code. But I have no idea what effect this really has on the RNG. The only effect I see is that the pool might receive less entropy. But on the other hand, I'm not even sure how much entropy some unitialised data has.
What do you people think about removing those 2 lines of code?
Kurt
"
BTW, i thought that Debian had some kind of policies about testing each package before committing changes in testing/stable branches. Also, the following paper, contributed by another poster, says interesting things about touching cryptographic code, we have to learn from this experience and have tighter policies !
" In a narrow sense, the security flaw we found in the Netscape browser serves merely as an anecdote to emphasize the difficulty of generating cryptographically strong random numbers. But there's a broader moral to the story. The security community has painfully learned that small bugs in a security-critical module of a software system can have serious consequences, and that such errors are easy to commit. The only way to catch these mistakes is to expose the source code to scrutiny by security experts.
Peer review is essential to the development of any secure software. Netscape did not encourage outside auditing or peer review of its software-and that goes against everything the security industry has learned from past mistakes. By extension, without peer review and intense outside scrutiny of Netscape's software at the source-code level, there is simply no way consumers can know where there will be future security problems with Netscape's products. "
http://www.cs.berkeley.edu/~daw/papers/ddj-netscape.html"
Doesn't this mean that all the "etch" host keys are vulnerable, too, on an ongoing basis?
SSH host keys are generated at sshd install time, and that package is included on the netinstall CD-roms, which means every time I re-use my 4.0r1 netinstall CD to set up a new box, I generate a new set of vulnerable host keys before I get to the security-updates step.
Hopefully, they'll advance the next point release and a 4.0r2 netinstall CD will do better.
2*3*3*3*3*11*251
It is even worse than this. Passwords that were used on a server with a weak DSA key may be compromised, as well:
From http://wiki.debian.org/SSLkeys
I was wondering, as soon as I read the GP, what you do if a system doesn't have /dev/random? And there is the crux of the problem. If you want to write portable code that could potentially be deployed on any OS with a compiler/runtime for your language, you are going to have to either include a random number generator of some sort with your code, or use one provided by the standard system libraries for your language, e.g the rand() function in the C library - (but I get the impression that people don't feel the C rand() function is sufficiently random).
/dev/random exists, and use it, else use the other random source you fall back on. But, at that point, either your random number generator is strong enough, or it isn't. Assuming you ship a sufficiently strong random number generator with your software, why even bother with some non-portable system dependent feature like /dev/random?
/dev/random? Because it's not portable.
Now, you might do a test, to see if
And, I bet the questions I ask provide the answer to the original GP's question - why not use
$ cat ~/.ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
7
------END RSA PRIVATE KEY------
Yipes.
(you're hearing my head repeatedly hitting the tabletop)
You will want to download this tool to check each system:
http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
To use it:
$ perl dowkd.pl user
$ perl dowkd.pl host localhost
See http://lists.debian.org/debian-security-announce/2008/msg00152.html
Why isn't such a scan isn't part of the package update? http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=481106
I has been using Slakware the last couple of years ...
So Sebastian Droge has been choking on a bucket of "cookies" on my box since then ?
Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu). The huge distaste in your mouth is probably the CROW you should be chewing on since the Debian people did NOT SCREW THIS UP. This was the fault of the maintainer.
I wonder since when content of stack became "random" as to serve a seed.
In all my years of system and application developer, I'd say picking something random from stack as good as picking some constant - because all code paths thru program would leave 1-2 unique stack states. No entropy there for you.
Even address of stack contains more entropy that its content - since most OSs now randomize stack base address.
Once in a while, I have to call the OpenSSL guy an incompetent and give kudos to Debian guys removing another piece of cruft from code.
All hope abandon ye who enter here.
If you're a casual desktop user who uses public key authentication, and you've ever run ssh-keygen on an affected client, you need to consider the resulting key compromised.
Given that the code is so shoddily written that neither of the functions has a comment to say what it's supposed to do, I think I can forgive him for that.
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
Things that you don't think about tend to fall apart behind your back. SSH/SSL is a best-of-breed cryptographic and authentication solution. It's so good, most of us didn't need to think about it.
Even excellent tools need maintenance now and then. And part of the maintenance overhead for crypto and authentication is to change your keys every so often.
So everyone has to suffer a Flag Day all at the same time, update their software and change out their keys. But I think the result will be more secure hosts and a more secure network, even for machines and networks which didn't "need" the change. Which, IMHO, is a good thing.
Schwab
Editor, A1-AAA AmeriCaptions
Seriously, I'm glad I don't put up with this shit from Debian. The place for code development is upstream. If you have a patch, get it upstream and get the approval of the people who actually develop the software so you can understand why Valgrind is complaining the way it is and why it has been left in. Backporting security fixes because of Debian's stupid development cycle I can just about understand (and even that's a problem as upstream moves on to new versions), but hacking on something downstream that few people will see is one reason why I just don't feel too much trust for Debian - and they do it far more than they should usually because of their own silly ideas about what is 'right'. There's just no reason for that patch at all.
I have quite a few OpenVPN keys generated for a wide variety of purposes, and if I'd been using Debian's OpenSSL then I'd be really pissed. There's just no reason for this, and no reason for anyone outside of upstream to commit a patch like that - and there it would have been caught pretty much instantly before a release was even made, or even better, dismissed as the crap that it is.
All that crap you hear from certain people about Debian being 'rock solid' and 'stable' is just that. Crap.
What I mean is there's an implicit assumption on the part of some OSS advocates that because the source is open, and thus viewable by anyone, that a large number of experts do choose to look at it and audit it. Well, that is usually not the case actually. People just happily use the software as is and never think to check and see if there are any problems with it, even if they have the technical expertise to do so.
That's the downside to the free, open nature of OSS is that while it ALLOWS anyone to look at it, it doesn't REQUIRE any one to. So maybe you have something that a bunch of experts are interested in and it is heavily audited. However you also maybe have something that nobody but the original programmer ever bothered to look at. Also the openness means that you can have something like what happened here: An original version that is audited but a derivative that isn't.
For heaven's sake people, OpenSSL wasn't counting on uninitialized memory as a big source of randomness.
It appears that some specific _application_ was passing an uninitialized buffer to OpenSSL. This made valgrind complain when _that_application_ was run. The Debian solution was to basically just patch OpenSSL to comment-out the function call about which valgrind was complaining. Of course, this same line of code was used for adding just about all entropy to the PRNG.
Yeah, I know.
They didn't depend on it: They needed a buffer to do some XORing to, and there was no need to initialize it. So they didn't initialize. That choice offered a potentially minor increase in entropy, and saved the admittedly minor computational cost of initializing the memory.
Ah, the light dawns. All that faffing about the "benefit" of using the uninitialised data was a smoke screen. The problem is that they saw an opportunity for a minor optimization that made the code a lot less clear, and didn't explain why they were doing it. They didn't even put in a "you're not expected to understand this" line.
They screwed up by writing dodgy code, and the Debian blokes screwed up by commenting out the code instead of initializing buf[]. They loaded the gun, and Debian shot themselves in the foot with it. Both sides should be embarrassed, not self-righteous.
How about the /. zealots figure out how to blame MS for their F-up...
Call me a troll, but the bashers just got a slap in the face, and are trying desperately to downplay the security problem that OPEN SOURCE should have caught .
Sorry, busy laughing at the zealots.
Since they didn't mention how to do this in the security advisory, or give you a warning message when you apply the update that you should regen your keys here's how to regenerate the host keys. Either log in as root or put sudo before each of these (this should work remotely, I didn't have problems but YMMV)
rm -frm -f
ssh-keygen -b 1024 -N '' -f
ssh-keygen -b 2048 -N '' -f
You'll then be asked about the key changing next time you log in.
I just spent all day updating servers. Doesn't sound so bad till you realize all these servers also had client keys used for remote backups that had to be recreated and verified, etc.
...when your dev team looks like this: http://www.flickr.com/photos/53246655@N00/581894457/
Hi, I made a quick study of openssl random generator http://mag.entropy.be/blog/ - it looks really bad, debian implementation generated only about ~200k random numbers.
This should have been in the advisory. Under the circumstances I don't want to assume that I know what I'm doing.
Fortunately, you renew for several years AND the last time you renewed was not while this bug was affecting Debian! With your method you'd be really screwed if it were.
Regenerating keys is fine, but the important step is clearing out the authorized_keys file that contains the old keys :-). At the same time take the opportunity to clear out the keys belonging to that long-gone co-worker.
...
If you try logging in with a new key where an old one is expected, you will get an error and remove the old one from authorized_keys. When the client is never in use however, the hole might stay in there for a long time
Here!!! Here!!!
... for the past couple of years, or a webhosting firm (generating ssl certs), etc. etc.
I really can't believe somebody would be mucking with the OpenSSL code at all: it's a crypto rng, if you haven't written it or don't understand it fully it's definitely one of those things to LEAVE ALONE.
-- the cake is a lie
hey,
/* Keep valgrind happy */ /* Use a random entropy pool device. Linux, FreeBSD and OpenBSD /dev/urandom if you can as /dev/random may block
/dev/urandom and /dev/random - as its stated in the comment.
are you guys rly sure, thats insecure? look:
--- openssl-0.9.7e.orig/crypto/rand/rand_unix.c 2003-12-27 16:01:52.000000000 +0000
+++ openssl-0.9.7e/crypto/rand/rand_unix.c 2006-04-19 15:42:32.000000000 +0100
@@ -160,6 +160,9 @@
const char **egdsocket = NULL;
#endif
+
+ memset(tmpbuf, 0, sizeof tmpbuf);
+
#ifdef DEVRANDOM
* have this. Use
that's the patch, up there.
he added zeroing the tmpbuf. ok.
but if you look into the code, next each and every byte of tmpbuf is overwriten with values read fro
so - if it's overwriten, before the buffer is used for anything - whats the big deal?
I am going to hijack this thread because I have an eCommerce site hosted on Debian and this made me nervous.
According to the reps at Thawte, if you are using a third party ssl cert (thawte, verisign, etc), this does not affect you. According to them, this is only dangerous for people who have generated their own SSL certs for their sites from the ground up (probably a minority, I would guess). If anybody has any information to the contrary, please let me know.
weirdest thing I ever saw: scientology advertising on slashdot.
As with Debian Sarge, Dapper uses a version that predates the dangerous patch.
In defense of said muppet. This has got to be a case of poor code commentation causing a maintenance problem.
The muppet was responding to a warning put out by an analysis tool (Valgrind) that some uninitialized memory was being read from, any programmer will tell you that's not a fantastic idea in general but it happens that in this case it was not only ok, but necessary.
But the muppet came along and "fixed" the problem.
Now this can only mean that there was inadequate code commenting (or, that the muppet didn't read it, in which case I retract my defense, but not the reasoning) around this unusual piece of code. If a piece if code is doing something decidedly strange, they there NEEDS to be a comment regarding it, preferably informing the reader of what is happening, or at least saying here be dragons or similar.
All it needed was: /* Reading some uninitialized memory to get a randomization seed. */
NZ Electronics Enthusiasts: Check out my Trade Me Listings
I'd like a definitive answer too. SSH keys might be important to a lot of people but as they are usually within an organisation revocation can be managed. I'm not sure that is the case with keys that are used publically.
Personally if the problem is with not particularly random (ie easily recreatable) keys then I don't see how the fact the key is signed by a third party makes any difference whatsoever.
If someone can regenerate your key then they can use that regenerated key with your certificate (which is publicly available from your webserver).
In fact even if you regenerate a new key/certificate, if someone has already captured your certificate then they can do this unless the users browser somehow finds out that your old certificate has been revoked.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
Hashing newly received random data with the existing contents of a buffer is a completely standard cryptographic idiom.
That's not the cleverly stupid bit. But I've already pointed that out.
You say, I don't care, it's not important so go ahead and fix it.
You're pointing at the wrong bloke: I didn't say "fix it", I said "document it".
If you take a gun and aim it at your foot and pull the trigger, and it goes off, I'll give you a Darwin award (albeit with a nervous laugh). But if the guy who loaded the gun doesn't at least feel bad about leaving a loaded gun around, well, he's taking the Darwin awards too seriously. This kind of thing isn't "funny haha", it's "funny sheesh" (as Daffy Duck says, or is that Sylvester?).
It appears that the Tor network is also compromised by this bug:
http://archives.seul.org/or/announce/May-2008/msg00000.html
IMPACT:
A local attacker or malicious directory cache may be able to trick
a client running 0.2.0.x into believing a false directory consensus, thus
(e.g.) causing the client to create a path wholly owned by the attacker.
Further, relay identity keys or hidden service secret keys that were
generated on most versions of Debian, Ubuntu, or other Debian-derived OS
are also weak (regardless of your Tor version):
http://lists.debian.org/debian-security-announce/2008/msg00152.html
Given that Tor is relied on in some pretty scary places (China, etc.), I wonder if anyone will get killed because of this.
That's what I thought originally, but the whole business about the randomness of the stack frame is a red herring. The randomness in the stack frame isn't being used for anything, it's just that there's no reason to care what was in the buffer, so it didn't matter that it wasn't initialized.
I think that is a silly optimization, but it didn't hurt. What hurt was not documenting it, leaving a big old tiger trap with an illusionary bug as bait for a careless bug-hunter to fall into.
The waffling about the minor improvement in randomness from the stack frame is just blowing smoke to distract attention from the fact that they didn't write "this is not a bug" on the bait they accidentally left by the hole. It's "oops" all around this time.
I was trying to explain what happened by putting you in the place of the OpenSSL developers.
I'm not nearly obdurate enough to write security software. You gotta be a hard boy for that stuff, so I can't wear those shoes no matter what.
OK, so the code didn't confuse them, and the bloke waffling about the minor advantage of using the stack frome for extra randomness was deliberately blowing smoke, that's fine, that was a side issue anyway. The main issue is that you put a note up saying "OH HAI, I'M NOT A BUG!" next to code that might look like a bug (especially when you've already to #ifdef it out to get it to pass a bug checker).
I've seen way too much of this cleverly stupid stuff over the past 30 years, I've even done some of it back in the day when I was younger and less familiar with the traps that lay in wait for the stupidly clever. And I wouldn't have even commented about it except that this fella decided to explain how the original code was clever enough to put a tail on and call it a weasel. It ain't. That's all.
Please explain how the Debian package mantainer is 'not the Debian guy'.
Comment removed based on user account deletion
Comment removed based on user account deletion
Frankly the guy probably doesn't need ragging on (in fact people should be checking he's ok).
A small coding mistake has had rather nasty and rather public consequences.
What does bear scrutiny is the process. In such a critical package should it really come down to just one guy? On such a critical package should Ubuntu not have their own man double checking everything rather than just accepting changes from Debian.
You can't make people infallible, but you improve the system to better mitigate such fallibility.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
Well done. That is why I love you, NSA guys.
P.S. Search for the new one already in place...
The Lynx browser uses OpenSSL!! So you text-only browsing folks would be toast if you didn't pay attention.
Although as was already mentioned, most other browsers do not use OpenSSL. But, that doesn't mean the issue isn't important to casual webbrowser users, though!
Had your favorite browser used OpenSSL (and as a casual user, you wouldn't know), your HTTPS communications could have been compromized. So it is vitally important to be concerned about things like this even if you aren't a SSH user or generate your own CA's or keys for any reason. Dismissing the problem for casual users because they don't actively generate keys would be wrong.
And there's always the risk of getting MITM'd when talking to someone using a weak key, so you'd want to make sure you don't talk to people with weak keys (the code should take care of this too, but it might not always be possible to detect, so you should ask your bank or whoever if their keys might be bad).
In the end it is the casual users who get bitten the worst by things like these, simply because they aren't aware that a problem might affect them.
Verisign
Considering how much they charge for certificates it's not hard to imagine them making millions out of people buying new certificates.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
In this case there was an added muppet who did something they shouldn't have ...
OSS is great but it required great developers
Steady on. This guy just made a coding mistake. Even the best coders do that from time to time.
How would you feel about people who know nothing about you going on about what a muppet you are for FOSS work you did, probably for free, at who knows what time of day and under who knows what circumstances. Way to encourage people to contribute.
This guy probably owes the community an apology of the "whoops, truely sorry, feel sheepish" variety, but doesn't deserve to be called a muppet or a bad developer.
Do you really wonder why Linux hasn't taken off if this is the respect that can be expected from the community?
given that it has take well over a year to get the advisory out it shows that the many eyes piece didn't work here, mainly because the eyes were looking at the original source not the botched packaging job.
Many eyes increases your chances, it doesn't give you an iron-clad guarantee.
These posts express my own personal views, not those of my employer
Had you tested randomness of the memory? That kind of randomnes stinks for a mile. Sorry to say it but it seams that no crypto people can do opensource (or do not want to).
Whem we are about seed sources - may be someone with the academic background could 'backfire' some students to do randomness studies that could be later published under 'open' licence? I remember analysis of the DRAM memories I have done years aga and it was clear that that can not be a seed source (even preprocesed). Also state of the memory in the live system could be hardly treated as random. Only 'true' sources of randomness would be well processed human interraction and to some degree traffic on the net. So why memory was used? Was there a call to make it to some degree less random? Or just there are no people / time& money for the analysis etc.
Worth thinking about when planing a new open source project...
I think the tagging would have been a lot less gracious
apt-get upgrade doesn't even fix the real problem...
/etc/ssh/ssh_host_*" and regenerate them "dpkg-reconfigure openssh-server" --piece of cake...
Why? Who knows--the debian people are even more stubborn idiots than the BSD community, and don't even spit out appropriate warnings. Oh, they'll restart my sshd--that's great. Is it a problem that my crypto-libs were weak--sure. A bigger problem is that all of my ssh sessions were *STILL* insecure even after the patch because the debian people couldn't be bothered put in a three line prompt to regenerate the crippled things.
"rm
Oh--they have it on the wiki, but their damned "security fix" doesn't even fix the ssh service.
Ubuntu has released an OpenSSL fix already, it seems.
``Tension, apprehension & dissension have begun!'' - Duffy Wyg&, in Alfred Bester's _The Demolished Man_
sample bash script + 64k of rsa+dsa default keys, on amd64 target
http://hysteria.cz/sd/debilian/
around 7.30 CET DST the ssh server keys were regenerated on our Ubuntu 7.10 boxes, which we noticed because the ssh client complained.
While it is good that this has increased security our first reaction was to wonder if we had been hacked.
One wonders, is there a good way to recover from a security gaff like this?
Please read the other replies in this thread. I am getting tired of answering this.
The bad version could only generate 2^16 unique keys for each length of key. The only source of entropy for those keys was the process ID.
I know SSL has something to do with https, but that's about it.
The thing is, I was considering trying out Ubuntu (always used Windows) and I know you can download a fix for this key thing, but:
- Are there any keys generated on install?
- How would you regenerate any keys?
- Where are these keys anyway??
Basically, how does this affect a fresh Ubuntu install?
Maybe this explains why so many popular Tor Hidden Server .onion sites went down so quickly. Read torproject.org and the related announcement + the Tor Blog for more info. If you run a tor server, please read! If you're just a tor user, please read!
The problem isn't with people generating their own certificates, but rather with people generating their own keys. Certain kinds of hosting generate your key for you, but you shouldn't really be using them. (And you'd have to know if they were using a compromised version of OpenSSL.)
It's easy enough to test for this; you should have a private key for your SSL cert. Run "openssl-vulnkey commercial.key" (or whatever your key's filename is) on a patched machine in order to test it. The one that my organization is using, for instance, comes up compromised.
(Coming up not-compromised wouldn't have meant that the key was definitely in the clear, but coming up compromised means that you definitely need to replace it. Enjoy paying for your incredibly expensive bits yet again.)
Laws do not persuade just because they threaten. --Seneca
I'm not placing blame on anyone, but let us consider for a moment:
How long would it take a member of a rogue organization, a company such as Microsoft, or an intelligence agency to land a spot into such a role as a code monkey at Debian.org, under the guise of a pro-FOSS person? You do know all three examples above are quite savvy when it comes to infiltration, mafias, corporations, and intelligence agencies do this all of the time. So let us suppose this is what happened here, and considering the wide range of impact with this issue, I believe this is exactly what may have happened.
What checks and balances are in place to weed out potential moles? Any? And would you really know what to look for even if such a policy is in place? Perhaps this question is worthy of an "Ask Slashdot" submission?
This is page 45-46, which talks about the reasons why a certificate may be revoked:
Rough translation: The certificate may be revoked when ... there's something that makes you reasonably believe that the certificate has been compromised to the point that it is not reliable.
And then on page 47 it says:
Rough translation: If the revocation of the certificate is not the fault of the subscriber (that's you), then you get a new certificate with the same validity period as the old one.
Obviously, you have to check with the contract governing your CA, but you might find something similar, so check with your CA before paying for new certs.
Very, very wrong...
Also on the client side you generate certificates, i.e. if you generated certificates for doing password-less login to a machine...
OpenSSL used a variety of sources of randomness all mushed together, including /dev/random and whatever else the developers could think of. One of these methods of randomness was looking into uninitialized memory space. Because a debugging tool (valgrind) about that, the Debian guy commented it out (with an okay from the upstream mailing list). However, he made a rather ham-handed mistake and removed all sources of entropy (with the exception of the current process ID, which counts only enough to hide this bug rather than make it crushingly obvious) rather than just the one.
So, now it's time to buy stock in Verisign, in Thawte, in whoever's going to be scoring a huge windfall off of this.
Laws do not persuade just because they threaten. --Seneca
The SSL key my workplace uses to secure our websites is vulnerable. Now, of course we're getting it revoked and replaced with a non-vulnerable one, but that doesn't stop the problem. How many web clients actually check CRLs? Given that CRLs are a terribly stupid idea and nobody uses them anyway, how many web clients actually check the embedded OCSP URLs in certs that support them? (It's under Preferences/Advanced/Encryption/Verification in Firefox 2, and it's disabled by default.)
An as-yet-undetermined set of SSL websites are vulnerable (until their certs expire) to being hijacked and masqueraded-as--with that shiny yellow bar at the top, no less. And if there are any root CA keys out there that are vulnerable to being guessed? This problem is going to be with us for years.
Laws do not persuade just because they threaten. --Seneca
A patched machine will reject logins using blacklisted keys. Of course, this means that when you ssh into one of them and update the packages, you'd better have your new, fixed keys installed, or else you'll be locked out of your server.
I suppose you'd better hope you don't have security updates automatically installing themselves.
Laws do not persuade just because they threaten. --Seneca
The CA has to add your cert to their CRL and make that list widely available until the cert expires. They also may have to run an OCSP responder to let users know that it's been revoked. (The fact that pretty much nobody actually uses these mechanisms doesn't change anything.)
Additionally, it's almost certain that this isn't the CA's fault; people generally generate their own RSA keys, then send a CSR to the CA to get them signed. It's not the CA's fault if the original key was bad, and there's nothing they could have done about it at the time in any case.
Laws do not persuade just because they threaten. --Seneca
Your system, when an SSH server (the package on Debian or Ubuntu is called "openssh-server") is installed, it generates host keys (that is, secret keys that identify your computer uniquely). This update will regenerate the keys for you, though you can do it manually (as root: "rm /etc/ssh/ssh_host*; dpkg-reconfigure openssh-server") if you need to for some reason. Note that regenerating your keys will not fix anything unless you install the update first, because the generated keys will still be weak.
If you or anyone else logs into your machine via SSH, they'll get a warning about the host keys having changed. If you log into anyone else's machine via SSH, you'll need to make yourself new keys (by running ssh-keygen) and send everyone your new public key.
If you run an SSL website with an affected key, you'll have to get the certificate revoked by the issuing CA, and get a new key signed to make a new cert.
There's a list of applications which may be using weak keys at the Debian wiki. If you're not running any kind of server, it's likely that you're only going to need to replace your SSH keys, as described previously.
Those who administer servers are in for a significantly bumpier ride.
Laws do not persuade just because they threaten. --Seneca
Generating a certificate involves generating a public and private key pair. You provide the public key to the CA (e.g. Thawte) and they give you a certificate which says that the corresponding private key is to be trusted. What matters is where the public/private key pair were generated. If you did that on a vulnerable Debian/Ubuntu machine, your certificate absolutely is vulnerable. The vulnerable act is the generation of the keys, not the signing of the certificate.
--Quentin
This is dangerously misleading! Blocking IPs after a reasonable number of failures stops only one particular attack against SSH. It prevents someone from logging into a user account that has a compromised-Debian-generated SSH public key by brute force, trying all possible compromised-Debian key pairs. There are a number of other attacks.
The most serious involve the SSH host key. The public host key is given to anyone who attempts to connect to the machine, whether they succuessfully authenticate or not. This must be the case -- giving out the public key is part of establishing an encrypted connection, and you must have encryption before you send your password or something. An attacker can connect ONCE to your server and, from the public host key, lookup the corresponding private key offline in microseconds. They can then decrypt any SSH communications they can sniff where that server is the server. You have little more security than telnet. They can perform a man-in-the-middle attack on the server. Even if a user is carefully using public-key authentication with something other than a compromised-Debian-generated key rather than password authentication, if they connect to a server with a compromised-Debian-generated host key, they might be connecting to a MITM attacker who can then perform any action on the server as said user.
Public-key authentication is still seriously vulnerable even if you limit the number of attempts. Normally, I can generate a key pair on machine A and set up machines B and C to allow me to log in with that public key, and a root attacker on machine B or C will not be able to access the other machine with my identity. If machine A is a compromised-Debian machine, this no longer holds. Anyone with access to the public key can immediately obtain the private key and impersonate me. Moreover, I normally wouldn't take the same precautions with my public key as my private key, but with a compromised-Debian-generated key pair they are practically equivalent. In general, there is no basis to rely on the public key being secret, which you implicitly recommend doing.
Any rational and informed person will immediately regenerate all key pairs generated with a compromised-Debian machine, and remove such public keys from any authorized_keys files in which they appear. They would also do well to consider the Ubuntu strategy of rejecting known weak public keys for public-key authentication.
Sittin' there on that sack of seeds!
"Having distributors randomly change source code as they package it is fundamentally broken."
This is one of the reasons I switched to BSD, where such things tend not to happen.
The blacklist is active by default, at least on Ubuntu. Add a blacklisted key to an authorized_keys file and try to log in using it. auth.log on the server will have a line like the following:
May 15 08:47:40 ns1 sshd[2989]: Public key 27:26:b8:52:d2:8e:7a:18:3f:8e:81:87:6b:7e:87:e9 blacklisted (see ssh-vulnkey(1))
and the client will read Permission denied (publickey).
Keys do need to be regenerated, but a patched system is not vulnerable to SSH key guessing, no matter what the authorized_hosts file contains. Bad keys will simply stop working. Any SSH remote-login vulnerability is fixed as soon as you dist-upgrade your packages, installing openssh-blacklist.
(Okay, certain keys which have options on them weren't detected by the first version of the ssh-vulnkey tool, but now they are.)
Laws do not persuade just because they threaten. --Seneca
Microsoft implant?
Why else would anyone be this stupid/clever?
These changes had NOTHING to do with the packaging of the software.
Read this again: The changes made had NOTHING TO DO with the proper packaging of the software.
But it slipped by.
Is anyone out there getting this?
Microsoft has pwned every openssl "protected" box there is - for the past *two years*.
Every Debian box is pwned, Ubuntu is entirely pwned.
OSS? NICE TRY
Look there were two steps.
1) Mix in randomness from strong random sources.
2) Mix in 'randomness' from uninitialized memory.
Because (1) exists (2) was at worst harmless and at best somewhat helpful. Clever hack.
(2) caused an error to display in Valgrind. This was long known and a FAQ issue on the OpenSSL list. In fact, it was so well known that there was alread a handy #ifdef around it so you could easily turn it off with a -Dpurify flag to the compiler if you were doing valgrind testing.
Unfortunately the person who decided to remove it (after being told by the OpenSSL devs to use the purify define) also removed the like which accomplished (1).
I find all this rather unsuprising.
...
;-)
Quality of some debian packages is really, really low - obviously fanatical adherence to the "bazaar" development model does not
always pay off
What's annoying is sheer arogance of lame folks.
C'mon people. This is unix - not windows
The weakness cuts the cost of brute-forcing down to minutes. There are only 65536 possible values for keys that are generated using this piece of code, and some of that space can be cut down using a little bit of knowledge (the 16 bit seed is the pid of the process, so it's not uniformly distributed. Pretty much anything that is worth the hassle of using ssh/SSL for is worth 20 minutes of an attacker's computing time (effective cost ~0).
my password really is 'stinkypants'
Ok, so I'm a tad confused. If I've understood this correctly, then, even though I've upgraded my packages and regenerated all my ssl certificates and ssh keys, it's possible that some random hacker who arbitrarily recorded traffic, say, a year ago, of an ssh conversation between me and a machine that, at the time, had a compromised host key, is going to be able, at their leisure, to read all the passwords I typed in over that session?
That's a bit scary.