Slashdot Mirror
Heartbleed Coder: Bug In OpenSSL Was an Honest Mistake
nk497 (1345219) writes "The Heartbleed bug in OpenSSL wasn't placed there deliberately, according to the coder responsible for the mistake — despite suspicions from many that security services may have been behind it. OpenSSL logs show that German developer Robin Seggelmann introduced the bug into OpenSSL when working on the open-source project two and a half years ago, according to an Australian newspaper. The change was logged on New Year's Eve 2011. 'I was working on improving OpenSSL and submitted numerous bug fixes and added new features,' Seggelmann told the Sydney Morning Herald. 'In one of the new features, unfortunately, I missed validating a variable containing a length.' His work was reviewed, but the reviewer also missed the error, and it was included in the released version of OpenSSL."
Coding and campaign do not mix.
hats off to the developer who admits a mistake.
I suspect s/he'll get pilloried in the press and may end up with some lawsuits (?) but I, for one, recognize a person big enough to take responsibility.
"The greatest lesson in life is to know that even fools are right sometimes" - Winston Churchill
The design of the feature looks like a backdoor too. A heartbeat function with a variable length payload, and there is a superfluous field for the payload length, and all running on top of TCP, which already has a keep-alive function? And then the feature contains a "rookie mistake", but still passes review. Yes, we totally believe you. It was a mistake.
The reviewer was named too.Dr Stephen Henson
Lawyers love EULA's and licenses. The OpenSSL license disclaims all liability
* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
* EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR
* ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
* OF THE POSSIBILITY OF SUCH DAMAGE.
https://www.openssl.org/source...
If you never agreed to that license, you're violating their copyright.
All I know is the organization I work for has prohibited use of C or C++ for mission critical software for years now. The languages we use would not ALLOW code to execute which tries to copy 64K from a 2 byte sized container.
Part of software engineering is to use the right tool for the right job. When a buffer overrun can destroy the security of the entire internet, you damn well better not be using C as your tool. Or assembly language for that matter.
Do that where I work, and you'd be fired.
Have a look for yourself. The reviewer "steve" is Stephen Henson.
Trolling is a art,
... so much of the internet depends on for security just one reviewer for a commit seems way way way too little, honestly checking anything into openssl (or gnutls) should be at least a 4-step approval process (submitter -> mantainer for that area -> overall library mantainer -> security officer), for any code that includes buffers/malloc especially if related to user supplied data the final security review should be a panel.
Everybody makes mistakes, everybody can have a 'brown paper bag' coding moment (especially around Christmas/New Year's like it happened in this case), 2 people having a 'brown paper bag' moment at the same time around the holidays is definitely not that unlikely, for something as important as a crypto library on which so many things depend a single reviewer is just not enough.
I do feel for the original developer, and hope that he won't suffer more about this than he already is (any developer worth their salt feels quite bad about bugs they introduce, let alone if they lead to this many problems), we've all made coding mistakes, no matter how experienced we are, so the focus should not be on "who" but more on "what kind of process can we introduce so this does not happen again".
Moving away from C in my opinion would just be a band-aid, other languages don't expose you to this particular bug, that's fine, however for security software choosing a vetting process for what goes in the codebase is a lot more important than choosing what language it's written in, not to mention that it's not that "hard" to write "secure C" especially if one leans on all the various available tools/libraries and writes proper unit tests, in this case for example had the malloc decision not been influenced by performance reasons (on unspecified platforms) this would not have been as big of a deal as it was.
-- the cake is a lie
The bigger problem is coders that think they need to optimize for speed.
Read the horror here: http://article.gmane.org/gmane...
Ugh... premature optimization, the root of all evil. And now also the root of the biggest security hole ever.
Bram Stolk http://stolk.org/tlctc/
He has no reason to. He gave it a best effort shot given the resources. If you can do better, fucking do it.
http://article.gmane.org/gmane...
They always did, the US and British intel agencies have always been double-agent ridden. I doubt that much of what Snowden revealed was a surprise to Moscow.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
I glanced at some of the OpenSSL C code, in particular the new code that introduced this bug. No comments, no asserts, no cross-checks, stupid variable names (like "payload" for the size of the payload, "pl" for a pointer to the payload data), no suggestions for how to test this new feature (such as what if the request has the payload size field that's not the same as the actual payload). In an unrelated function I saw an array declared on the stack, getting filled up, and then a pointer to this array getting assigned to a field of an argument to this function, and then a return...
we do it just to bug YOU, cold fnord.
we know it pisses you off to have your boys look bad.
DEAL WITH IT. or just leave. either is fine with most of us.
--
"It is now safe to switch off your computer."
Lawyers love EULA's and licenses. The OpenSSL license disclaims all liability
* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED * OF THE POSSIBILITY OF SUCH DAMAGE.
https://www.openssl.org/source...
If you never agreed to that license, you're violating their copyright.
OK, switching from humor to serious. The above can be challenged in court. And being correct/innocent does not necessarily determine the outcome of a case. As a hostile lawyer once explained: the facts of the matter are irrelevant, my client can afford to go to court, you can not. See "pyrrhic victory".
C is a perfect language for back-dooring code. Without good tools - even for the sake of code reviewers that are trying to defend the code base against good malicious programmers - we won't be able to keep up. Most C code is going to be full of accidental back doors, and possibly a few intentional ones. Something has to be done about machine assisted proof of memory safety, and about ensuring that parse code either follows a spec or stops parsing the input. High level languages with memory safe semantics are essentially constructive proof of memory safety at the cost of performance. You can also go the route of fixing the defaults of what is going to be allowed in important C files, and force programmers to #pragma their way out of violating the constraints so that potential bugs are easy to spot. Or better yet, stop using read() in the raw to ingest input; and force the code to parse input with a well defined grammar (and fail to compile otherwise). You don't necessarily need an abstract garbage collected language that prevents you from cutting yourself. What you really need is by any means necessary to limit security critical code to constructs that the compiler can prove the consistency of. Code review should essentially amount to the reviewer making a list of proof demands, with the review process failing until there are machine checkable proofs of these facts. Every exception to these rules (Supress warnings, or accepting a hand proof, or proof by authority) need to be documented at the line where they happen. That way, when the next back door is discovered, we can easily search for the place that introduced it. This is not theoretical at all. I worked with a guy who was publicly accused of back dooring a Unix. No amount of code review will clear anyone of suspicion; because we use the crappiest tools for proving correctness of even small things.
I found your post on slashdot and used it as legal advise.
It turns out you were negligent in checking your facts so I'm suing you for damages.
Hmm, considering that the real bug is OpenSSL's malloc, where it was reusing 'freed' memory I think that's the bug that is critical. The developer who put in the TLS support itself may have been under the assumption that since OpenSSL didn't die/crash once implemented that everything was honky-dory, and the reviewer likewise.
UDP does not have keep alive. TCP over TCP is an inherently broken combo so a VPN would prefer UDP. In crypto, it's necessary to hide nature of packets to make traffic analysis harder, which is probably why there is all that length stuff (did not check RFC, if it explains reasons).
Well, maybe this is a blessing. While it's open source, maybe multiple eye's need to look at it for final validation.
No it's a curse. I have input fuzzing, unit tests, code coverage profiling and Valgrind memory tests. Such a bug wouldn't have slipped past me with both eyes shut -- no seriously! If I fuck up accidentally like this THE COMPUTER TELLS ME SO without ever having to do anything but make the mistake and type make test all. I test every line of code on every side of my #ifdef options, in all my projects. If you're implementing ENCRYPTION AND/OR SECURITY SOFTWARE then I expect such practices as the absolute minimum effort -- I mean, that's what I do, even when it's just me coding on dinky indie games as a hobby. I don't want to be known as the guy who's game was used to compromise users' credentials or data, that would be game over for me.
These ass-hats have just shown the world that they can't be trusted to use the fucking tools we wrote that would have prevented this shit if they'd have just ran them. It's really not acceptable. It's hard to comprehend the degree of unacceptable this is. It reeks of intentional disaster masquerading as coy "accidental" screw up, "silly me, I just didn't do anything you're supposed to do when you're developing industry standard security software". No. Just, no. An ancient optimization that was made default even though it only mattered on SOME slower platforms? Yeah, OK, that's fucking dumb, I can buy it as an accident. However, NOT TESTING BOTH BRANCHES for that option? What the actual fuck? I could see someone missing an edge case in their unit test, but not even using input fuzzing at all? It's not hard, shit, I have a script that generates the basic unit fuzzing code from the function signatures in .H files, you know, so you don't miss a stub...
"Never attribute to malice what can be adequately explained by stupidity." -- The level of stupidity required is unexplainable. How the fuck are they this inept and in charge of THIS project? THAT'S the real issue. This isn't even the fist time OpenSSL shit the bed so bad. In <- this linked example, it was Debian maintainers and not the OpenSSL maintainers fault (directly): Instead of adding an exception to the Valgrind ignore list (which you most frequently must have in any moderately sized project, esp one that handles its own memory management) they instead commented out the source of entropy, making all the SSL connections and keys generated by OpenSSL easily exploitable since it gutted the entropy of the random number generator (which is a known prime target for breakage that's very hard to get right even if you're not evil, so any change thereto needs to be extremely well vetted). Last time the OpenSSL maintainers brazenly commented they "would have fallen about laughing, and once we had got our breath back, told them what a terrible idea this was." -- Except that they silently stopped paying attention to to the public bug tracker / questions and quietly moved to another dev area, making it nearly impossible to contact them to ask them about anything (a big no-no in Open Source dev), but it gives you a better idea about the sort of maintainers these fuck-tards are.
We don't know absolutely for sure, but we're pretty damn close to absolutely certain that OpenSSL and other security products (see: RSA's BSafe) are being targeted for anti-sec by damn near all the powers that be. So, now we find out OpenSSL has an obsolete optimization -- a custom memory pool (red flag goes up right away if you see memory reuse in a security product, that shit MUST be even more throughly checked than entropy-pools, since it can cause remote code execution, memory leaks, and internal state exposure... you don't say?). We find that optimization would have been caught by basic fuzz test with Valgrind, which apparently folks have been using previously according to the comments in the prior S
Not really. That means every application that wants to talk securely would have to add the noise, vs the library that they use adds the noise behind the scenes and the multitude of applications can concern themselves with what they need to do. And if the noise generation needed to be changed for some reason, then you only have to update the library and not the multitudes of applications (some of which may never be updated....).
You're likely to get modded Troll; but this really does remind me a bit of Ford vs. Toyota. For years Ford was fixed in peoples minds as the exploding Pinto company, and Toyota was high quality. Now Toyota isn't what it used to be, and Ford is better... but neither is perfect.
If nothing else this is a good argument against monoculture. We have different systems with different bugs, so it's not a total loss. If the market shares were evenly distributed among 10 different vendors, the black-hat task would be even harder, their impact of success that much less.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
No production code without unit tests. Every possible type or class of input must be tested. All assumptions must be tested. All outputs must be verified for each possible combination of inputs. All failure modes must be exercised. No excuses, just do it.
The above cannot be challenged in court. No court in the Universe holds jurisdiction over this. The contributor didn't sell you OpenSSL, he didn't force you to use it, it didn't tell you to use it, he didn't make any guarantees about its functionality, you have no contract, no warranty, no expectation for it to actually do anything, etc.
You may as well sue someone after walking into their house uninvited, listening to them whistle while they're sitting on the toilet, and hearing a missed note.
Serious question: Why don't you become the new maintainer yourself, if you honestly believe you can do a significantly better job at it than the current person(s)?
I don't do it myself because I can not guarantee that I wouldn't make even worse mistakes. I'm glad there are people out there who are willing to do the job, and I'm in no position to bite their heads off when they mess it up. And you're probably glad that I'm not a maintainer of anything even remotely security-related :-)
Bingo. There's a lot of people who are willing to declare the unpaid volunteers don't know what they're doing...but have absolutely no patience to try and contribute or agitate for change from within. Which really makes me wonder whether they'd have the wherewithall to run a new project, or you know, do anything at all.
Code that gets written gets run.
OpenSSL would be meaningless and not in use if it had been written in ADA. Also, ADA is not at all the "magic bullet" some people make it out to be. In fact, it suffers from unreadable code even more than clean C code. Also remember that this is a library. Unless ADA allows creating libraries with "C" bindings in ADA, it is not even usable for the task. And if you invest the same time on the C side, you get a similar level of quality. Hell, this bug would have been found with some halfway competent fuzzing or completely avoided with mandatory time-of-use checking of all bounds. Which secure coding guidelines can make mandatory, unless you explicitly explain in each instance where you omit it why this is ok.
Sure, languages play an important role, but it always falls back to Assembler-level (or C if you want to be somewhat portable) and there competent people can do all things that imperative languages can do. The problem with most languages is that they restrict too much what you can do and/or foster bloat and/or have intransparent behavior.
I have done Eiffel, Sather, Python, Lua, Pascal, Modula, Haskell, Gofer, Prolog, Lisp, C++, several assembler variants, Basic, Java, JavaScript, Pizza, and some more things. I am back to doing all "library" level things in C and the glue in Python or Lua (or C where that is not an option). Basically all OO languages suck badly, except Eiffel, which unfortunately it too exotic to be useful. Functional languages are nice and compact for some things, but suffer from interfacing problems. (Will have a look at Erlang though, that may be better.) Logical languages have some value in some specific situations, but do not even approach "general purpose". C++ and Java are bloated, intransparent, non-orthogonal atrocities made by people that do not understand OO. These languages typically make things worse unless you have really experienced and capable people. The typical mediocre-to-bad coders just throw the features of these languages around indiscriminately, resulting in an awful mess. IDEs make that worse as suddenly it becomes easy to write more code and distribute it into even more files.
Yes, you need discipline, experience and insight to do things well in C. The current OpenSSL disaster shows all three are missing in the OpenSSL team. Yes, many C programmers suffer from a terminal desire to place efficiency above everything else. But that problem is with the programmer, not the language. And you get just the same stupid mistakes in other languages and they add their own problems.
That is not to say C is a good language. It is (besides assembler) just the only language where code quality rests completely with the coder and that does not stand in the way of the coder. It is the only "native" language if you do not want to do assembler and that gives it a special place and all that cannot work competently with it a clear limitation. At the same time, all approaches to force people to write "good code" by language features and restrictions that force them to do so have failed. There is even bloated, unreadable, unreliable code in Python or Eiffel, and you have to really work at that in these languages.
After being in this for 30 years and having seen and reviewed countless pieces of code by others, I am convinced that language selection is mostly irrelevant. The only thing language can give you is problems in the form of restrictions. If the coder is not top-notch, the code will suck and be insecure, slow, unreliable and unmaintainable, regardless of the language used. If the coder is top-notch, the code will be secure, fast, reliable and easy to maintain, mostly regardless of language used (within the restrictions the language comes with). However a top-notch coder will know or be able to figure out fast what a specific language can do and what its restrictions are and know early whether it is suitable for a specific project or not.
Of course, that said, you are right that people that want to do everything in one specific language (often Java these da
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Instead of adding an exception to the Valgrind ignore list (which you most frequently must have in any moderately sized project, esp one that handles its own memory management)
If you need to add exceptions to get a tool to work... the tool is wrong for the job.
The date that it was added to the OpenSSL codebase is very close to the time when the leaked NSA documents claim that they had a 'major breakthrough' in decrypting SSL. I would imagine that they are not responsible for introducing it, but do have people doing very careful code review and fuzzing on all changes to common crypto libraries, so I wouldn't be surprised if they'd known about it (and been exploiting it) since it was originally released.
I am TheRaven on Soylent News
In this case, the code is open, so pretty much everyone understands exactly what happens, exactly how bad it is, and how to fix it.
Ahem, sir. Open source can be useful, but it is not a magic bullet like that. Even I can read and understand the C code in OpenSSL, but to see the bigger picture and to understand how this particular software actually works and is arranged is completely different story. I bet that in case of OpenSSL, only under 100 people in the world can "understand exactly what happens, exactly how bad it is, and how to fix it".
Robin Seggelmann, thank you and the entire OpenSSL Team for your contributions to free open source software. Glad we could find a serious security flaw, that you're helping to find out how it happend and that the OpenSSL crew is so fast in coming up with a fix.
With just about any other development paradigm and folks like MS we'd've waited for weeks for that to happen.
Carry on with the good work, you guys rock!
We suffer more in our imagination than in reality. - Seneca