Slashdot Mirror
Heartbleed Coder: Bug In OpenSSL Was an Honest Mistake
nk497 (1345219) writes "The Heartbleed bug in OpenSSL wasn't placed there deliberately, according to the coder responsible for the mistake — despite suspicions from many that security services may have been behind it. OpenSSL logs show that German developer Robin Seggelmann introduced the bug into OpenSSL when working on the open-source project two and a half years ago, according to an Australian newspaper. The change was logged on New Year's Eve 2011. 'I was working on improving OpenSSL and submitted numerous bug fixes and added new features,' Seggelmann told the Sydney Morning Herald. 'In one of the new features, unfortunately, I missed validating a variable containing a length.' His work was reviewed, but the reviewer also missed the error, and it was included in the released version of OpenSSL."
Coding and campaign do not mix.
We know the coder, we don't know the reviewer.
hats off to the developer who admits a mistake.
I suspect s/he'll get pilloried in the press and may end up with some lawsuits (?) but I, for one, recognize a person big enough to take responsibility.
"The greatest lesson in life is to know that even fools are right sometimes" - Winston Churchill
The design of the feature looks like a backdoor too. A heartbeat function with a variable length payload, and there is a superfluous field for the payload length, and all running on top of TCP, which already has a keep-alive function? And then the feature contains a "rookie mistake", but still passes review. Yes, we totally believe you. It was a mistake.
He's sorry now, wait until some lawyer shows up on his doorstep with a bill.
Even the NSA wouldn't cause a disaster of this magnitude just to spy on everyone.
If adding features ever improves software it is usually only for a tiny percentage of users. Mostly it just introduces vulnerabilities like this and soaks up cpu cycles and disk space. Of course we have to have progress or coders would get awfully bored doing nothing but fixing bugs and improving performance. Still it surprises me that security software can be modified so quickly and with only one review. If you want to add a feature to anything as critical as SSL, it should be reviewed and tested by numerous experienced coders over a period of many months.
I missed validating a variable containing a length
right.
All I know is the organization I work for has prohibited use of C or C++ for mission critical software for years now. The languages we use would not ALLOW code to execute which tries to copy 64K from a 2 byte sized container.
Part of software engineering is to use the right tool for the right job. When a buffer overrun can destroy the security of the entire internet, you damn well better not be using C as your tool. Or assembly language for that matter.
Do that where I work, and you'd be fired.
Well, maybe this is a blessing. While it's open source, maybe multiple eye's need to look at it for final validation.
While never have worked with open source, I do work with data very often. I re-validate entries a lot and just scan data look for something odd. I am amazed at the amount of errors I find, so I report them and go from that point onwards.
sometimes I work with teams of people and have a kind of race looking to find the most errors, then we go out and the winner only buys 1 round, and does not pay the bill for dinner.
if you see me, smile and say hello.
... so much of the internet depends on for security just one reviewer for a commit seems way way way too little, honestly checking anything into openssl (or gnutls) should be at least a 4-step approval process (submitter -> mantainer for that area -> overall library mantainer -> security officer), for any code that includes buffers/malloc especially if related to user supplied data the final security review should be a panel.
Everybody makes mistakes, everybody can have a 'brown paper bag' coding moment (especially around Christmas/New Year's like it happened in this case), 2 people having a 'brown paper bag' moment at the same time around the holidays is definitely not that unlikely, for something as important as a crypto library on which so many things depend a single reviewer is just not enough.
I do feel for the original developer, and hope that he won't suffer more about this than he already is (any developer worth their salt feels quite bad about bugs they introduce, let alone if they lead to this many problems), we've all made coding mistakes, no matter how experienced we are, so the focus should not be on "who" but more on "what kind of process can we introduce so this does not happen again".
Moving away from C in my opinion would just be a band-aid, other languages don't expose you to this particular bug, that's fine, however for security software choosing a vetting process for what goes in the codebase is a lot more important than choosing what language it's written in, not to mention that it's not that "hard" to write "secure C" especially if one leans on all the various available tools/libraries and writes proper unit tests, in this case for example had the malloc decision not been influenced by performance reasons (on unspecified platforms) this would not have been as big of a deal as it was.
-- the cake is a lie
He and the reviewer that ok'd his code submission should be brought in for questioning.
The bigger problem is coders that think they need to optimize for speed.
Read the horror here: http://article.gmane.org/gmane...
Ugh... premature optimization, the root of all evil. And now also the root of the biggest security hole ever.
Bram Stolk http://stolk.org/tlctc/
He has no reason to. He gave it a best effort shot given the resources. If you can do better, fucking do it.
http://article.gmane.org/gmane...
I glanced at some of the OpenSSL C code, in particular the new code that introduced this bug. No comments, no asserts, no cross-checks, stupid variable names (like "payload" for the size of the payload, "pl" for a pointer to the payload data), no suggestions for how to test this new feature (such as what if the request has the payload size field that's not the same as the actual payload). In an unrelated function I saw an array declared on the stack, getting filled up, and then a pointer to this array getting assigned to a field of an argument to this function, and then a return...
Never stay up rushing commits on New Year's Eve.
I had to post anonymously because I was not sure my credentials are safe on this site
As Linux grows, so will the bugs, exploits, and vulnerabilities.
Don't know much about what and how but many major corporations are struggling to figure out what the risks are and how to remediate.
It is going to be a busy month and a log of developers coders and qa people will be very busy...
And even if these companies could legally sue (jurisdictions notwithstanding), what would the point be?
Take ownership and control of the project.
... Find a way to sue the FSF, get ownership/control, relicense all GNU software.
Hmmm
Seggelderp is a much better name than Heartbleed.
If there is more than one explanation always choose the one that involves a conspiracy.
"As an open-source dev myself, I often wonder why the fuck I do anything useful for others when they'll just turn on me the moment their toys don't work exactly as desired because -- gorsh -- I'm not perfect, though I work very hard to be."
The cost of this bug will probably run into billions of dollars, not to mention an increasing distrust of anything supposedly 'secure' on the Internet. That's hardly 'not working exactly as desired'.
Anyone writing security software should know to never, ever, ever trust any user-supplied input. We have to deal with a lot of user-provided messaging, and that's pretty much rule #1; the only crashes in our software over the last couple of years were due to third-party libraries which don't sanitize and validate user-provided messages before processing them. Dumb fscks.
C is a perfect language for back-dooring code. Without good tools - even for the sake of code reviewers that are trying to defend the code base against good malicious programmers - we won't be able to keep up. Most C code is going to be full of accidental back doors, and possibly a few intentional ones. Something has to be done about machine assisted proof of memory safety, and about ensuring that parse code either follows a spec or stops parsing the input. High level languages with memory safe semantics are essentially constructive proof of memory safety at the cost of performance. You can also go the route of fixing the defaults of what is going to be allowed in important C files, and force programmers to #pragma their way out of violating the constraints so that potential bugs are easy to spot. Or better yet, stop using read() in the raw to ingest input; and force the code to parse input with a well defined grammar (and fail to compile otherwise). You don't necessarily need an abstract garbage collected language that prevents you from cutting yourself. What you really need is by any means necessary to limit security critical code to constructs that the compiler can prove the consistency of. Code review should essentially amount to the reviewer making a list of proof demands, with the review process failing until there are machine checkable proofs of these facts. Every exception to these rules (Supress warnings, or accepting a hand proof, or proof by authority) need to be documented at the line where they happen. That way, when the next back door is discovered, we can easily search for the place that introduced it. This is not theoretical at all. I worked with a guy who was publicly accused of back dooring a Unix. No amount of code review will clear anyone of suspicion; because we use the crappiest tools for proving correctness of even small things.
Was this an old feature, or a relatively now one.
If so how did it make it into distros without being extremely vetted. Given that openSSL is one of the core parts of security?
Poor bastard. Being responsible for one of the worst security breaches ever does not help the CV...
This isn't one site we're talking about, or even a thousand sites, losing millions of dollars each. This is millions of sites losing hundreds to thousands of dollars as they have to upgrade their servers, potentially shut them down until they're upgraded, replace SSL keys, and make users replace passwords.
All because one bozo didn't check user data before using it.
Hmm, considering that the real bug is OpenSSL's malloc, where it was reusing 'freed' memory I think that's the bug that is critical. The developer who put in the TLS support itself may have been under the assumption that since OpenSSL didn't die/crash once implemented that everything was honky-dory, and the reviewer likewise.
That was the purpose of the feature he put in that had the bug. He wrote the RFC, too. For fuck's sake, why do we need to compute maximum transfer unit from the goldang application layer? Isn't that what the transport layer is for?
...thing is, they were by law forbidden to disclose anything to the general public :)
Slashdot: stuff for news, nerds that matter, matter for news, stuff that nerd
Isn't the mantra that open source is wonderful because there's so many people looking at the code there can never be a problem?
I mean keep alive is already done by TCP, a keep alive doesn't need data. Why did this even get into the standard?
It seems like the only purpose for this code was for Robin S. to erect a monument for himself. Again why did this get into the standard?
Many eyes my ass. Just admit that software is gonna have bugs and get off your presumed high horse. Reminds me of the incident years ago about a repository being compromised for quite some time and the claims made that it was fine because they claimed they checked against other sources.
Now he'll be relegated to solely open-source work - the lowest of the low.
The Troll is strong with this one :)
There ain't no such thing as proprietary standards only proprietary formats. Standards are by definition open.
Does anyone else here worry that the press's identifying the programmer will discourage people from contributing to open-source projects? Programmers in corporations are often anonymous, which helps in this kind of situation.
It was probably modded down, because it is as clear a troll as they come. That last paragraph is a dead giveaway.
the press said it's been known since 2012, who knew it, or is the press just bs?
This reinforces a comment I made elsewhere. If you have enterprise class needs, buy enterprise class software.
Suborbital [spaceflight] is the special olympics of spaceflight. - Rei
Uh, yes, if you'd actually bothered to read my comments, you'd see that I do code, and I code safety-related systems that run 24/7/365, which can't fail just because someone sends bad data.
I've also written open source cryptography code in the distance past, and it sure as heck didn't have bugs like this one.
But, hey, just because you're a bad coder, don't take it out on the rest of us.
You're likely to get modded Troll; but this really does remind me a bit of Ford vs. Toyota. For years Ford was fixed in peoples minds as the exploding Pinto company, and Toyota was high quality. Now Toyota isn't what it used to be, and Ford is better... but neither is perfect.
If nothing else this is a good argument against monoculture. We have different systems with different bugs, so it's not a total loss. If the market shares were evenly distributed among 10 different vendors, the black-hat task would be even harder, their impact of success that much less.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
This wasn't a facebook app to make him an extra buck.
He was working on New Years Eve on a C based project that is one of the backbone projects for the entire open source stack.
That is one hard core programming geek. This guy is an employer's dream.
Suborbital [spaceflight] is the special olympics of spaceflight. - Rei
As long as he's saying that, his family is safe from the CIA.
No production code without unit tests. Every possible type or class of input must be tested. All assumptions must be tested. All outputs must be verified for each possible combination of inputs. All failure modes must be exercised. No excuses, just do it.
These kind of mistake are done everyday by pretty much every programmer. Nobody test 100% of the assertions made in their code, or are aware of ALL the subtle way bugs can happen.
Exactly. See this site for more information on enterprise-level quality :-)
Great, buy some closed source software, and you will have no idea when such a bug is found, at least here once it was found, everybody knew, and a fix was made available quickly, no attempt to cover it up or pass it off as nothing important
Most of the Open Source code I've seen has been high quality and I assumed such a high-profile project as OpenSSL would be the same. Having dug out the code myself when this blew up I was shocked at how ropey it is. Magic numbers everywhere, memory handled in a cavalier way, no clear structure. Now I feel bad defending Open Source against FUD shills because I know they can whip out this example.
Nobody cares, and if they do, they don't understand.
There are more bad guys looking for security flaws than good guys doing code review. So, either code review must be done by computers (I don't know how) or the system must make less errors possible. Will we have to write all system software in Java or Ada?
You're right that skills are very important, of course, but the language matters a lot. OpenSSL would have far less bugs if it had been written in Ada with critical sections in Spark and some formal validation, for example.
There is no perfect programming language for all purposes and languages are more or less suited for different purposes. Beware the language aficionado who has an excuse for every deficiency of his favorite language ...
There is no Consideration to the Programmer, so there is no contract and no liability.
http://en.wikipedia.org/wiki/C...
If he's honest, he'd say exactly that. If he's dishonest, he'd either waffle or again say exactly that. If he's planting for some secret service he'd also say exactly that. Only one of the cases I'd take my hat off for. Worse, this guy made his academic career out of "security" protocols but gets blind-sided by a "honest mistake", and only one casual reviewer didn't spot it either. That really isn't good enough, neither for him nor for the openssl project, or any widely-used security-heavy piece of software.
Perhaps more importantly, it shows the dangers of poor protocol design. I'm not too versed in the intricacies of this one, but I can't help but wonder why it needed two length fields, and why you even have one if you're not going to check them.
The trick is to only look at the input once it's assumption-free, ie all necessary assumptions are known and have been checked to hold. That way you don't have to juggle one (or two, or N) length fields and hope you have the right one.
I think our dear honestly mistaken holder of a PhD in computer security has a bit more soul searching to do. As does the openssl team.
No, I'm not advocating more layers upon layers of software, I'm advocating better design in protocol and software*, better code review and code acceptance practices, and understanding things like line coding before tinkering with any protocol, much less a security-sensitive one. Like, oh, openssl.
* There's also Theo's (of TheoBSD fame) observation that openssl bypasses security measures available in certain malloc/free implementations by dint of defaulting on all platforms to their own allocator without such features because on some (other) platforms malloc is "slow". Worse, turning that "feature" off breaks openssl because the team hasn't tested without it. Looks like there's room for improvement with the unit testing practices too.
If these NSA types actually meant that oath they swore, they'd kill all the other assholes around them violating the law, before the killed themselves.
The fact no one has yet is a clear indicator of how they view the Constitution.
Truth isn't Truth - Guliani
Exactly - software is complex and sometimes shit happens. An employer who knows anything about code knows that - and he (+ the reviewer) probably won't make the same mistake again.
In this case, the code is open, so pretty much everyone understands exactly what happens, exactly how bad it is, and how to fix it. If the code wasn't open, the bug could just as well be there, except that it would be less understood, maybe never found (except by exploits), and patched more slowly (or never, or tuesday next month, or when you pay to upgrade to the newest version of the program which includes the buggy library).
Don't drink and code!
The only way malloc could completely protect against the bug would be by putting an unmapped guard page after every malloced block - making every malloced block at least one page long and slowing malloc down by the time needed for all those munmaps. (Probably making malloc slow enough to incite OpenSSL devs to implement their own malloc layer...)
And if someone is willing to make that decision, then the OpenSSL developers should not override it.
Most malloc()s that have these protections have them as an option which can be toggled (the BSDs certainly do). So this policy should not be overruled, espcially if it can reduce security. It should be left to the sysadmins who run the OS and set the malloc() options, and the trade offs they want.
C++ has bounds-checked containers.
This is relevant only to the extent that C++ programmers use bounds-checked containers. A lot of C++ programmers have to use bounds-unchecked containers in order to interface with C libraries that use bounds-unchecked containers, and exploits might take advantage of inadvertent undefined behaviors in this interface code.
Except the biggest advocates of TDD seem to be dictating from the ivory tower rather than writing the tests themselves.
And when they do get stuck writing test code code; their tests are sloppy, poorly maintained, and not up to the standards they expect from everybody else.
In this case, the code is open, so pretty much everyone understands exactly what happens, exactly how bad it is, and how to fix it.
Ahem, sir. Open source can be useful, but it is not a magic bullet like that. Even I can read and understand the C code in OpenSSL, but to see the bigger picture and to understand how this particular software actually works and is arranged is completely different story. I bet that in case of OpenSSL, only under 100 people in the world can "understand exactly what happens, exactly how bad it is, and how to fix it".
The heartbleed vulnerability existed for years in OpenSSL codebase too. We have no idea if someone exploited before it was finally fixed.
The point being that we are starting to see that there are no guarantees that serious bugs will be caught quickly in open source software any better than in closed source software.
Remember also that while the public domain does not see the source for closed software, the company is able to pay for professional code audits which ensures that the code actually gets the eyeballs to go through it very carefully and thoroughly.
in the same way any company using open source can pay for a professional code audit
For this particular bug, seeing the actual code at least helped me to understand what was going on - and undoubtedly helped many who wrote explanations for it. I don't really need to understand the core crypto-algorithms in OpenSSL's core to understand what happened in this case.
Of course, and I hope they do.
See this invention of mine:
http://sourceforge.net/p/sappeurcompiler/code-0/HEAD/tree/trunk/
http://sourceforge.net/p/sappeurcompiler/code-0/HEAD/tree/trunk/doc/manual.pdf?format=raw
Also, no need for GC and the resulting ugly memory consumption and runtime effects. No need for sluggish VM startup.
C is indeed a dangerous anti-pattern of engineering. Nobody can do it right - see the HPUX Ping of Death or StuxNet. All based on C artifacts.
If I had sufficient tine at hand, I would implemt TLS in Sappeur, but alas, I have other responsibilites at this time.
Frank Gerlach
Gäufelden
Germany
Low-level memory safety can be easily achieved using a Pascal-style (on a conceptual, not syntactical level) language with a nice type system.
See my language Sappeur for an example: http://sourceforge.net/p/sappeurcompiler/code-0/HEAD/tree/trunk/doc/manual.pdf?format=raw
But that is just a necessary, not a sufficient condition of security. To be 100% secure, you need complete mathematical proof. The L4 OS people of Uni Dresden make this claim for 20k of kernel code.
Short of complete proofs, it would definitely help to retire C or at least to use it only in exceptional cases. The mafia of C developers will agitate against this, as they always did. Realistically, there is nobody who will be 100% of time be capable of avoiding memory errors. So, C is indeed a dangerous language if written by Homo Sapines. If generated by a kind of compiler, this could be different.
Robin Seggelmann, thank you and the entire OpenSSL Team for your contributions to free open source software. Glad we could find a serious security flaw, that you're helping to find out how it happend and that the OpenSSL crew is so fast in coming up with a fix.
With just about any other development paradigm and folks like MS we'd've waited for weeks for that to happen.
Carry on with the good work, you guys rock!
We suffer more in our imagination than in reality. - Seneca
The fix has the sum 1 + 2 + 16 hardcoded into it. Twice. Why not speed things up by replacing that with a hardcoded 19? (If the sum has significance to a human reading the code, that could be explained in a comment.)
That that is is that that that that is not is not.
Programmers make these kinds of mistakes all the time.
Do you think that one of these days, a mistake like this will have catastrophic consequences?
If yes, should I lay in a supply of freeze-dried food and shotgun shells?
That that is is that that that that is not is not.
We have no idea if someone exploited before it was finally fixed.
hmmmm. So this sort of phenomena only cropped up in the 80's right?
That's ~30 years ago. Shouldn't we start getting a bunch of retired end-stagers willing to divulge who was doing what with the zero-days in their pocket?
Death beds are a great place to learn the truth about what happened 30-40 years ago. For some matters, it's the only place you'll learn the truth. The field of computer security is reaching the end-life of their first generation. Where are the memoirs?
If intelligence agencies where aware of this bug and choose not to report it they, and any politician who knew, should be prosecuted. Because if they knew they consciously exposed their citizens data, financial and personal information to foreign intelligence outfits.
I'm sure there's some contractor somewhere who uses OpenSSL when communicating or protecting government data so that would be prosecution for treason.
Bloomberg reports NSA has been using this exploit for two years. This looks a lot less like an accident now.
Being a developer myself, I believe him completely.
"I missed validating a variable containing a length"
Developers half the time and floundering in the dark, only understanding one small portion of someone else's code. Years were spent developing the whole, and we usually need one change in one piece. When making changes we simple hope the code compiles. Unit Tests help a ton. It increases our understanding and it assures us that we didn't break anything else.
If the change is made and the code compiles and unit tests passed, why would anyone be worried?
This shows that we need more tools that scan code. Can we write a new rule in source code scanners that can detect such a vulnerability and make sure it never happens again?
The part that is better is mostly that open source is ... reviewable.
"And that is probably true; the worst open source code I've seen is much better than the worst proprietary code I've seen."
Not a good point since most proprietary code is not supposed to be "seen" by most.
And finally Goodwins' law kicks in. If nothing else works - try and associate opponent with Hitler and/or nazism.
And also "the winners write history", for second...
Wow there sure is lots of paid shills in here, aren't there...?
Mostly because i am not arrogant enough to believe i am up to the task.
99% of programmers think they are good at what they do.
It's only when the system allows it, that space shuttles start falling from the sky.
"Good enough" wasn't even a question that was asked here.
The problem with most open source is that *I* might be the next guy to fix your code.
Release early and often is as brain dead as it gets.
If they're required to interface with existing C libraries, they're going to run into similar unsafe API wrapping regardless of their chosen language. C++ may interface with C libraries more frequently than others, but that doesn't make the "safer" languages immune.
The process of dissolving a big problem into low-level steps as is required by C programming is mentally brutal. You can't just go "I want to save the text that was deleted and restore it when they hit the undo key." You have to translate that into variables, pointers, structs, mallocs, and glue logic.
Or you could use some tools which abstract away this problems.
- you could use a high level language where some of this porblems don't exist (e.g.: no pointers, and automatic garbage collection).
- or you could stay within the C/C++ world and write wrappers that take care to check everything (for example, almost any moderne tool-kit [Qt, Boost, or even default C++'s std++] will define type that are bound checked automatically [QByteArray or std::string] and smart pointers.
- some of these could even by implemented in plain C.
(But full implementation might require some Macro-ugliness. GTK+-level of ugliness)
Done correctly, such tool can automate the taking care of corner cases that can break the system.
But instead some programmer still decide to use as a simpler syntax for assembler and do everything bare-metal.
In case of OpenSSL, I'm a bit surprised. It has so many helper functions, but nobody has bothered yet to implement buffers with buffer-size checking safetey.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
... His work was reviewed, but the reviewer also missed the error, and it was included in the released version of OpenSSL."
So where is the apology from the Reviewer? They are just as at-fault as the programmer !
Not to mention the people in the user companies that checked it...