Slashdot Mirror
How To Prevent the Next Heartbleed
dwheeler (321049) writes "Heartbleed was bad vulnerability in OpenSSL. My article How to Prevent the next Heartbleed explains why so many tools missed it... and what could be done to prevent the next one. Are there other ways to detect these vulnerabilities ahead-of-time? What did I miss?"
It could have been discovered with static analysis if anyone had the foresight to implement a specific check ahead of time (although it's unknown whether anyone could have thought of checking this specific case before Heartbleed was discovered):
http://blog.trailofbits.com/2014/04/27/using-static-analysis-and-clang-to-find-heartbleed/
about as effective as sunshine and puppies.
Every industry goes through this. At one point it was aviation, and the "hot shot pilot" was the Real Deal. But then they figured out that even the Hottest Shot pilots are human and sometimes forget something critical and people die, so now, pilots use checklists all the time for safety. No matter how awesome they might be, they can have a bad day, etc. And this is also why we have two pilots in commercial aviation, to cross check each other.
In programming something as critical as SSL it's long past time for "macho programming culture" to die. First off, it needs many eyes checking. Second, there needs to be an emphasis on using languages that are not susceptible to buffer overrunning. This isn't 1975 any more. No matter how macho the programmer thinks s/he is, s/he is only human and WILL make mistakes like this. We need better tools and technologies to move the industry forward.
Last, in other engineering professions there is licensing and engineers are held accountable for mistakes they make. Maybe we don't need that for some $2 phone app, but for critical infrastructure it is also past time, and programmers need to start being held accountable for the quality of their work.
It's things the "brogrammer" culture will complain BITTERLY about, their precious playground being held to professional standards. But it's the only way forward. It isn't the wild west any more. The world depends on technology and we need to improve the quality and the processes behind it.
Yes, I'm prepared to be modded down by those cowboy programmers who don't want to be accountable for the results of their poor techniques... But that is exactly the way of thinking that our industry needs to shed.
Don't use C and its variants like C++. C is an extremely unsafe, low-level language that is just one step above assembly language. This makes it great for low-level, performance sensitive programs like OSes, compilers, etc. but the low-levelness also increases bug count for general purpose applications.
Instead use safer languages like Pascal, Eiffel (design by contract), Ada, etc. These languages guard against buffer overflows and don't have the slowness and bloat associated with garbage collected languages like Java and .NET, but are much safer than C. The problem usually is, few people know these languages and they are not portable from one platform to another.
Let's remember the good old bug that plagued (and probably still does) many libraries that read graphic files such as TIFF. The classic scenario was that the programmer was reading the expected image size from the TIFF file header, allocated memory for this size, then read the reminder of the file into said buffer, until end of file. Instead of reading as many bytes as he has allocated. Now for a correct file this would work, however if the file is maliciously crafted to indicate one size in the header while having a much larger real size, you would do a classic buffer overrun. This is pretty much similar to what the SSL programmer did. And no tools were ever able to predict this type of errors, whether TIFF or SSL.
BTW the last famous one with TIFF files was pretty recent:
http://threatpost.com/microsoft-to-patch-tiff-zero-day-wait-til-next-year-for-xp-zero-day-fix/103117
The problem has more to do with the "hey, this is free so lets just take it" attitude of the downstream consumers not willing to pay for anyone to look at the code or pay anyone to write it.
Why would you want the OpenSSL people to be held accountable for something they basically just wrote on their own time since nobody else bothered?
Striking out to solve a problem should NOT be punished (that culture of legal punishment for being useful is part of why knowledge industries are leaving North America).
This problem was caused by a simple missed parameter check, nothing more. Stop acting like the cultural problem is with the developers when it is with the leaches who consumer their work.
Yeah, we'll just rewrite the Internet in Pascal.
Libraries like OpenSSL are built in C in no small part because C can easily be linked into just about any other language out there. Nothing is going to change that.
And idiots can write bad code in any language. It might not be a buffer overflow, but they could still have screwed up in many other ways.
Or you just learn how to code properly. This particular vulnerability wasn't because there was a mistake, it was because they opted to bypass a function that was meant to keep people safe. It's a bit like bolting the fire escapes closed then wondering why everybody died after the fire.
It's astonishing to me that somebody would put code into a production environment that asked for a certain length of response without bothering to do any validation.
What did I miss?
An article before the word "bad."
Slashdot: providing anti-social weirdos a soapbox, since 1997.
If that really worked, there would be no QA dept. for software. Unless you can formally prove your software is correct, you should assume there are bugs. And no one has the time, money or ability to formally prove hundreds of millions of lines of code.
And even more astonishing the head maintainer and merger did not review this change and ask the developer to fix it.
I'm not sure you can auto-generate a C header file but you can create a library (.dll or .o) file from Ada source and call it from C. You have to hand generate the C header file.
Create DLL library in Ada
Yes, I think it's clear the next gen of CPUs really needs to have the machine language removed entirely. What a security hole!
I have no idea why you're maintaining that "Republicans" create these bugs, and I'm, like, a socialist.
> If that really worked, there would be no QA dept. for software.
No, that's just poor reasoning.
Quality must be built-in, not added-on. QA expectations and improvement scope are largely imposed on any QA department, therefore the level of 'quality' reached can never be an absolute bar.
Developers in general need to minimise the vector product of bug count/severity that could be exposed before it gets to QA. This allows the bar to be raised, and focus to be spent on where it should be rather than catching obvious mistakes, or dealing with unnecessary performance/cognitive/configuration complexity.
> programmers need to start being held accountable for the quality of their work.
They are.
But I guess you mean that people who aren't paying for your work, and companies which aren't paying for the processes and professional services necessary for some level of quality, should hold programmers who don't have any kind of engineering or financial relationship with them accountable.
I have personally ported OpenSSL to at least 6 embedded systems, one of which was so proprietary they wrote their own C/C++ compiler. Good luck finding an Ada compiler for that.
his makes it great for low-level, performance sensitive programs like OSes, compilers,
Aaand... performance sensitive like, say... crypto? There isn't much code more performance sensitive than crypto libraries, which is one of OpenSSL's main uses. In fact, there are a whole bunch of native assembler implementations for x86, MIPS, ARM, PPC, etc to achieve that low level performance. Clearly you have never actually looked at the OpenSSL code base...
The US Army will swear that I was once, many moons ago, officially certified in Ada, whether that means anything or not. I never liked it much, even though I did turn in successful code a few times, and I really have a problem with Ada for open source applications - Yes, in theory, Ada has some very strong security functions by design, but it's definitely not going to result in the 'many eyes make all bugs shallow' effect. I actually read your post as deliberately tongue in cheek at first, what with phrases such as 'extremely unsafe'.
But as I think more about it, one of the problems revealed by Heartbleed is open sourcing the target code didn't result in a lot of properly trained eyes passing over that code. I never thought I'd encourage anyone to learn Ada after I got out of the service (just as I never thought I'd encourage anyone to start a cult worshipping many-tentacled, eldritch, blasphemous horrors from beyond space-time as we delusionally try to limit our conceptions of it to preserve our fundamental human sanity, and for much the same reasons), but I have to admit, you may have a damned good argument for Ada there.I don't know if the extensive compile time checking of Ada 2012 could have automatically caught the bug that made Heartbleed possible - the last version of Ada I've really used is 95, but I'd be really interested to hear from someone who's current if they think Ada is just about totally bulletproof against this sort of bug, because even the older versions I recall had some features that would have made it hard to make this sort of mistake.
Who is John Cabal?
Hi dwheeler,
This is a great article. It covers many common software development and testing techniques. But also some "on live system" techniques. It was a pleasure to read, I'll recommend it to various places.
My experience is that people who trust in their language to keep their code bug free inevitably have more bugs in their code. It's amazing how many memory leaks I've found in Java, by programmers who swore such things were impossible. Another entertaining situation is people who manage to get around deadlock detection by creating live-locks.
I think it's clear to everyone who's actually looked at the situation that the problem here wasn't the language, it was the people who were using the language. They would have written bad code in any language.
"First they came for the slanderers and i said nothing."
But as I think more about it, one of the problems revealed by Heartbleed is open sourcing the target code didn't result in a lot of properly trained eyes passing over that code.
My experience is that reading code isn't a very good way to catch bugs, mainly because reviewers tend not to read it as carefully as the person who wrote it. If you want to find bugs, it's more effective to do white/black box testing of some sort.
"First they came for the slanderers and i said nothing."
I saw that a week ago, so I donated $50 USD to the OpenSSL Software Foundation. I figured I would either whine about the problem, or do something. I chose the latter.
I actually agree with both of you. The Open SSL guys gave out their work for free for anybody to use. Anybody should be free to do that without repercussions. Code is a kind of literature and thus should be protected by free speech laws.
However, if you pay peanuts (or nothing at all) then likewise you shouldn't expect anything other than monkeys. The real fault here is big business using unverified (in the sense of correctness!) source for security critical components of their system.
If regulation is needed anywhere, it is there. People who develop safety and security critical stuff should be certified and businesses with a turn over $x million dollars should be required to use software developed only by the approved organisations.
There is nothing in this definition that prevents an open source implementation. In fact, there's an argument to say that any such verified implementation must be open source precisely so it can be inspected. But it is quite a lot of work and people need to be paid to do that work. You can't expect to get this level of quality assurance for free.
it's super effective.
Spent All My Mod Points
If your web server is pushing out lots of https traffic then yes it is performance sensitive.
Excerpt...
* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
* EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR
* ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
* OF THE POSSIBILITY OF SUCH DAMAGE.
This problem was caused by a simple missed parameter check, nothing more. Stop acting like the cultural problem is with the developers when it is with the leaches who consumer their work.
I do not believe you. If this were an isolated case, then you'd be right. But no, this kind of "oops, well now it is fixed" things happens all the time, over and over again. The culture of the programming never improves due to the error - no matter how simple, no matter that it should have been noticed earlier, no matter what.
I am willing to bet that after next hole the excuses will be same "it was simple, now it is fixed, should up" and "why don't you make better, shut up" or just "you don't understand, shut up". And still the cowboy-coding continues.
This was caused partially by unchecked parameter (this should have never happened, there is no excuse for it), partially because the idiots used their own allocator which created the covert channel and prohibited the use of malloc-debug libraries. Libraries which would have found the error - again this should not have happened.
But then, maybe I just should shut up ...
A quote from the "Insane Coding" blog, which in turn quotes from the book "cryptography engineering":
The issues with higher level languages being used in cryptography are:
- Ensuring data is wiped clean, without the compiler optimizations or virtual machine ignoring what they deem to be pointless operations.
- The inability to use some high-level languages because they lack a way to tie in forceful cleanup of primitive data types, and their error handling mechanisms may end up leaving no way to wipe data, or data is duplicated without permission.
- Almost every single thing which may be the right way of doing things elsewhere is completely wrong where cryptography is concerned.
Rigorous testing is helpful, but I think it's the wrong approach. The problem here was lack of requirements and/or rigorous design. In the physical engineering disciplines, much effort is done to think about failure modes of designs before they are implemented. In software, for some reason, the lack of pre-implementation design and analysis is endemic. This leads to things like Heartbleed - not language choice, not tools, not lack of static testing.
I would also go as far as saying if you're relying on testing to see if your code is correct (rather than verify your expectations), you're already SOL because testing itself is meaningless if you don't know the things you have to test - which means up-front design and analysis.
That said, tools and such can help mitigate issues associated with lack of design, but the problem is more fundamental than a "coding error."
"There are a dozen opinions on a matter until you know the truth. Then there is only one." - CS Lewis (paraprhase)
Yes. Also, the problem with Rap is that it is in English. If they just wrote their masogenistic statement in a different language all would be well!
Seriously, please stop with the ridiculous claim that the language is the problem. The problem is that nobody is perfect, no process is perfect, and mistakes will always happen. They will happen far more often when the system is implemented by people who understand so little about software development that they think the language is the problem.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
No doubt. So why didn't YOU take steps to prevent the Heartbleed vulnerability? The same reason everybody else didn't: time. Finding bugs takes time. Sure, you can automate, but that automation also takes time. So we are caught between two desires: 1) the desire to add or improve functionality, and 2) the desire to avoid vulnerabilities. The two desires compete for the amount of time that is available, so it becomes a trade-off.
It's also an arms race. There is real financial incentive for finding vulnerabilities that can be exploited, far more incentive than there is for software authors to prevent exploitation.
Given that it's not FUN to find vulnerabilities, unless you are the guy trying to exploit it, there are always going to be vulnerabilities.