Slashdot Mirror
Heartbleed Coder: Bug In OpenSSL Was an Honest Mistake
nk497 (1345219) writes "The Heartbleed bug in OpenSSL wasn't placed there deliberately, according to the coder responsible for the mistake — despite suspicions from many that security services may have been behind it. OpenSSL logs show that German developer Robin Seggelmann introduced the bug into OpenSSL when working on the open-source project two and a half years ago, according to an Australian newspaper. The change was logged on New Year's Eve 2011. 'I was working on improving OpenSSL and submitted numerous bug fixes and added new features,' Seggelmann told the Sydney Morning Herald. 'In one of the new features, unfortunately, I missed validating a variable containing a length.' His work was reviewed, but the reviewer also missed the error, and it was included in the released version of OpenSSL."
Coding and campaign do not mix.
hats off to the developer who admits a mistake.
I suspect s/he'll get pilloried in the press and may end up with some lawsuits (?) but I, for one, recognize a person big enough to take responsibility.
"The greatest lesson in life is to know that even fools are right sometimes" - Winston Churchill
The design of the feature looks like a backdoor too. A heartbeat function with a variable length payload, and there is a superfluous field for the payload length, and all running on top of TCP, which already has a keep-alive function? And then the feature contains a "rookie mistake", but still passes review. Yes, we totally believe you. It was a mistake.
Even the NSA wouldn't cause a disaster of this magnitude just to spy on everyone.
The reviewer was named too.Dr Stephen Henson
Lawyers love EULA's and licenses. The OpenSSL license disclaims all liability
* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
* EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR
* ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
* OF THE POSSIBILITY OF SUCH DAMAGE.
https://www.openssl.org/source...
If you never agreed to that license, you're violating their copyright.
All I know is the organization I work for has prohibited use of C or C++ for mission critical software for years now. The languages we use would not ALLOW code to execute which tries to copy 64K from a 2 byte sized container.
Part of software engineering is to use the right tool for the right job. When a buffer overrun can destroy the security of the entire internet, you damn well better not be using C as your tool. Or assembly language for that matter.
Do that where I work, and you'd be fired.
Well, maybe this is a blessing. While it's open source, maybe multiple eye's need to look at it for final validation.
While never have worked with open source, I do work with data very often. I re-validate entries a lot and just scan data look for something odd. I am amazed at the amount of errors I find, so I report them and go from that point onwards.
sometimes I work with teams of people and have a kind of race looking to find the most errors, then we go out and the winner only buys 1 round, and does not pay the bill for dinner.
if you see me, smile and say hello.
Have a look for yourself. The reviewer "steve" is Stephen Henson.
Trolling is a art,
... so much of the internet depends on for security just one reviewer for a commit seems way way way too little, honestly checking anything into openssl (or gnutls) should be at least a 4-step approval process (submitter -> mantainer for that area -> overall library mantainer -> security officer), for any code that includes buffers/malloc especially if related to user supplied data the final security review should be a panel.
Everybody makes mistakes, everybody can have a 'brown paper bag' coding moment (especially around Christmas/New Year's like it happened in this case), 2 people having a 'brown paper bag' moment at the same time around the holidays is definitely not that unlikely, for something as important as a crypto library on which so many things depend a single reviewer is just not enough.
I do feel for the original developer, and hope that he won't suffer more about this than he already is (any developer worth their salt feels quite bad about bugs they introduce, let alone if they lead to this many problems), we've all made coding mistakes, no matter how experienced we are, so the focus should not be on "who" but more on "what kind of process can we introduce so this does not happen again".
Moving away from C in my opinion would just be a band-aid, other languages don't expose you to this particular bug, that's fine, however for security software choosing a vetting process for what goes in the codebase is a lot more important than choosing what language it's written in, not to mention that it's not that "hard" to write "secure C" especially if one leans on all the various available tools/libraries and writes proper unit tests, in this case for example had the malloc decision not been influenced by performance reasons (on unspecified platforms) this would not have been as big of a deal as it was.
-- the cake is a lie
The bigger problem is coders that think they need to optimize for speed.
Read the horror here: http://article.gmane.org/gmane...
Ugh... premature optimization, the root of all evil. And now also the root of the biggest security hole ever.
Bram Stolk http://stolk.org/tlctc/
He has no reason to. He gave it a best effort shot given the resources. If you can do better, fucking do it.
http://article.gmane.org/gmane...
posting to undo moderating error, please ignore...
I glanced at some of the OpenSSL C code, in particular the new code that introduced this bug. No comments, no asserts, no cross-checks, stupid variable names (like "payload" for the size of the payload, "pl" for a pointer to the payload data), no suggestions for how to test this new feature (such as what if the request has the payload size field that's not the same as the actual payload). In an unrelated function I saw an array declared on the stack, getting filled up, and then a pointer to this array getting assigned to a field of an argument to this function, and then a return...
Never stay up rushing commits on New Year's Eve.
I don't believe him. He and the reviewer that ok'd his code submission should be brought in for questioning.
You are totally correct. As it is "you" who does not beleive him, governments of the world should work together to bring this man and his reviewer in for questioning.
This is now a matter of urgent and International importance.
Hurry before its too late! AC has spoken!
This is the second example I've seen in this thread where disclaiming negligence for vehicular accidents is compared to disclaiming negligence for software bugs on an unpaid open source project that companies aren't obligated to use.
And even if these companies could legally sue (jurisdictions notwithstanding), what would the point be? This is an individual with limited funds - they'd bankrupt him but wouldn't get enough from him to cover their legal fees.
Still it surprises me that security software can be modified so quickly and with only one review
It's an open source project, who's going to stop them writing the code and making it available?
Sure, but what's the case law on the application of consumer-rights to free stuff?
while the comparison between cars and software is ridiculous. Many places do not permit a EULA or any other agreement to sign away a persons rights to protection from negligence regardless of what the product is, such clauses can be deemed invalid.
The point is that you disclaimers are not always legally binding. The vehicular examples are just examples that many people may find familiar. You're reading too much into them.
Regardless, I agree that he is safe legally. It's not the disclaimer that protecting him though.
I'm sure there are many places that don't permit such a EULA, however are they the jurisdiction where he wrote and published the code? (honest question - I don't know) Jurisdiction matters for such things, which is how some open source projects get away with code that infringes patents in some regions but not others for example.
And even if these companies could legally sue (jurisdictions notwithstanding), what would the point be?
Take ownership and control of the project.
... Find a way to sue the FSF, get ownership/control, relicense all GNU software.
Hmmm
Lawyers love EULA's and licenses. The OpenSSL license disclaims all liability
* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED * OF THE POSSIBILITY OF SUCH DAMAGE.
https://www.openssl.org/source...
If you never agreed to that license, you're violating their copyright.
OK, switching from humor to serious. The above can be challenged in court. And being correct/innocent does not necessarily determine the outcome of a case. As a hostile lawyer once explained: the facts of the matter are irrelevant, my client can afford to go to court, you can not. See "pyrrhic victory".
Unit tests?
Advocates of TDD would stress a complex suite of tests that self-document assumptions and side-effects - especially prior to refactoring code to bug fix and add new features, as specified in the summary.
C is a perfect language for back-dooring code. Without good tools - even for the sake of code reviewers that are trying to defend the code base against good malicious programmers - we won't be able to keep up. Most C code is going to be full of accidental back doors, and possibly a few intentional ones. Something has to be done about machine assisted proof of memory safety, and about ensuring that parse code either follows a spec or stops parsing the input. High level languages with memory safe semantics are essentially constructive proof of memory safety at the cost of performance. You can also go the route of fixing the defaults of what is going to be allowed in important C files, and force programmers to #pragma their way out of violating the constraints so that potential bugs are easy to spot. Or better yet, stop using read() in the raw to ingest input; and force the code to parse input with a well defined grammar (and fail to compile otherwise). You don't necessarily need an abstract garbage collected language that prevents you from cutting yourself. What you really need is by any means necessary to limit security critical code to constructs that the compiler can prove the consistency of. Code review should essentially amount to the reviewer making a list of proof demands, with the review process failing until there are machine checkable proofs of these facts. Every exception to these rules (Supress warnings, or accepting a hand proof, or proof by authority) need to be documented at the line where they happen. That way, when the next back door is discovered, we can easily search for the place that introduced it. This is not theoretical at all. I worked with a guy who was publicly accused of back dooring a Unix. No amount of code review will clear anyone of suspicion; because we use the crappiest tools for proving correctness of even small things.
Was this an old feature, or a relatively now one.
If so how did it make it into distros without being extremely vetted. Given that openSSL is one of the core parts of security?
I found your post on slashdot and used it as legal advise.
It turns out you were negligent in checking your facts so I'm suing you for damages.
Hmm, considering that the real bug is OpenSSL's malloc, where it was reusing 'freed' memory I think that's the bug that is critical. The developer who put in the TLS support itself may have been under the assumption that since OpenSSL didn't die/crash once implemented that everything was honky-dory, and the reviewer likewise.
...thing is, they were by law forbidden to disclose anything to the general public :)
Slashdot: stuff for news, nerds that matter, matter for news, stuff that nerd
I mean keep alive is already done by TCP, a keep alive doesn't need data. Why did this even get into the standard?
It seems like the only purpose for this code was for Robin S. to erect a monument for himself. Again why did this get into the standard?
Now he'll be relegated to solely open-source work - the lowest of the low.
The Troll is strong with this one :)
There ain't no such thing as proprietary standards only proprietary formats. Standards are by definition open.
It was probably modded down, because it is as clear a troll as they come. That last paragraph is a dead giveaway.
This reinforces a comment I made elsewhere. If you have enterprise class needs, buy enterprise class software.
Suborbital [spaceflight] is the special olympics of spaceflight. - Rei
You're likely to get modded Troll; but this really does remind me a bit of Ford vs. Toyota. For years Ford was fixed in peoples minds as the exploding Pinto company, and Toyota was high quality. Now Toyota isn't what it used to be, and Ford is better... but neither is perfect.
If nothing else this is a good argument against monoculture. We have different systems with different bugs, so it's not a total loss. If the market shares were evenly distributed among 10 different vendors, the black-hat task would be even harder, their impact of success that much less.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
This wasn't a facebook app to make him an extra buck.
He was working on New Years Eve on a C based project that is one of the backbone projects for the entire open source stack.
That is one hard core programming geek. This guy is an employer's dream.
Suborbital [spaceflight] is the special olympics of spaceflight. - Rei
No production code without unit tests. Every possible type or class of input must be tested. All assumptions must be tested. All outputs must be verified for each possible combination of inputs. All failure modes must be exercised. No excuses, just do it.
then you would trip on something you were not testing. There is a reason free software is pretty much free, it is imperfect. If you want perfect software, go in the aviation industry, where you might still end up with... http://www.cas.mcmaster.ca/~ba...
These kind of mistake are done everyday by pretty much every programmer. Nobody test 100% of the assertions made in their code, or are aware of ALL the subtle way bugs can happen.
The above cannot be challenged in court. No court in the Universe holds jurisdiction over this. The contributor didn't sell you OpenSSL, he didn't force you to use it, it didn't tell you to use it, he didn't make any guarantees about its functionality, you have no contract, no warranty, no expectation for it to actually do anything, etc.
You may as well sue someone after walking into their house uninvited, listening to them whistle while they're sitting on the toilet, and hearing a missed note.
Navy destroyers are on their way and all the navy seals are already in route. Drones keeps patrolling the perimeters. Just as a back up, President Obama is ready during the operation to launch nukes to counter the threat.
Exactly. See this site for more information on enterprise-level quality :-)
I would be glad about it. At least, he is doing something, not idling on /. pointing fingers.
Great, buy some closed source software, and you will have no idea when such a bug is found, at least here once it was found, everybody knew, and a fix was made available quickly, no attempt to cover it up or pass it off as nothing important
but a unit test wouldn't show this up - nobody would write a test, testing that the function worked, and start looking at external impacts of that function.
For this to be caught using testing, you'd need a fairly good coverage integration test. Unit tests are just too focussed on small things. They prove the unit works as expected, they never consider interaction with other parts of the overall system.
Most of the Open Source code I've seen has been high quality and I assumed such a high-profile project as OpenSSL would be the same. Having dug out the code myself when this blew up I was shocked at how ropey it is. Magic numbers everywhere, memory handled in a cavalier way, no clear structure. Now I feel bad defending Open Source against FUD shills because I know they can whip out this example.
Nobody cares, and if they do, they don't understand.
There are more bad guys looking for security flaws than good guys doing code review. So, either code review must be done by computers (I don't know how) or the system must make less errors possible. Will we have to write all system software in Java or Ada?
You're right that skills are very important, of course, but the language matters a lot. OpenSSL would have far less bugs if it had been written in Ada with critical sections in Spark and some formal validation, for example.
There is no perfect programming language for all purposes and languages are more or less suited for different purposes. Beware the language aficionado who has an excuse for every deficiency of his favorite language ...
https://www.openssl.org/source...
If you never agreed to that license, you're violating their copyright.
You're only violating their copyright if you distribute it. If I legally acquire a copy of a piece of software, I can use it without agreeing to any other stipulations. Depending on jurisdiction, of course, different legal systems may rule in different ways on that point. And I'm not sure what the jurisdiction that this guy lives in has said about it.
The GPL has a specific clause pointing this out, and it's there because the authors of the GPL believe that they have no authority to prevent you from using their software. I agree with them. It always amuses me when GPL'd software contains a clickthrough insisting that you press an "Agree" button, when the licence specifically says that no such agreement is necessary.
There is no Consideration to the Programmer, so there is no contract and no liability.
http://en.wikipedia.org/wiki/C...
If he's honest, he'd say exactly that. If he's dishonest, he'd either waffle or again say exactly that. If he's planting for some secret service he'd also say exactly that. Only one of the cases I'd take my hat off for. Worse, this guy made his academic career out of "security" protocols but gets blind-sided by a "honest mistake", and only one casual reviewer didn't spot it either. That really isn't good enough, neither for him nor for the openssl project, or any widely-used security-heavy piece of software.
Perhaps more importantly, it shows the dangers of poor protocol design. I'm not too versed in the intricacies of this one, but I can't help but wonder why it needed two length fields, and why you even have one if you're not going to check them.
The trick is to only look at the input once it's assumption-free, ie all necessary assumptions are known and have been checked to hold. That way you don't have to juggle one (or two, or N) length fields and hope you have the right one.
I think our dear honestly mistaken holder of a PhD in computer security has a bit more soul searching to do. As does the openssl team.
No, I'm not advocating more layers upon layers of software, I'm advocating better design in protocol and software*, better code review and code acceptance practices, and understanding things like line coding before tinkering with any protocol, much less a security-sensitive one. Like, oh, openssl.
* There's also Theo's (of TheoBSD fame) observation that openssl bypasses security measures available in certain malloc/free implementations by dint of defaulting on all platforms to their own allocator without such features because on some (other) platforms malloc is "slow". Worse, turning that "feature" off breaks openssl because the team hasn't tested without it. Looks like there's room for improvement with the unit testing practices too.
Has that happened in other cases, like when Microsoft had security holes?
Most software is inperfect, also non-free - even what you mention, code in control systems for areospace.
I don't think there is any precedent for claiming that non-free software is more secure than free software.
Consumers are not the ones providing OpenSSL, the vendors downloading and installing it on their vending systems, are. And so, it is the vendors who in fact should be afraid of lawsuits, not Robin or anyone else contributing to OpenSSL. But in any case, anyone is free to sue anyone else, the assumption is that the judging party understands what the usage license for OpenSSL implied. Which they are expected to.
If these NSA types actually meant that oath they swore, they'd kill all the other assholes around them violating the law, before the killed themselves.
The fact no one has yet is a clear indicator of how they view the Constitution.
Truth isn't Truth - Guliani
Exactly - software is complex and sometimes shit happens. An employer who knows anything about code knows that - and he (+ the reviewer) probably won't make the same mistake again.
In this case, the code is open, so pretty much everyone understands exactly what happens, exactly how bad it is, and how to fix it. If the code wasn't open, the bug could just as well be there, except that it would be less understood, maybe never found (except by exploits), and patched more slowly (or never, or tuesday next month, or when you pay to upgrade to the newest version of the program which includes the buggy library).
It always amuses me when GPL'd software contains a clickthrough insisting that you press an "Agree" button, when the licence specifically says that no such agreement is necessary.
In fact, by placing the requirement that someone agrees to the license before using a derived work of the GPL'd software, they are violating the GPL...
I am TheRaven on Soylent News
Don't drink and code!
C++ has bounds-checked containers.
This is relevant only to the extent that C++ programmers use bounds-checked containers. A lot of C++ programmers have to use bounds-unchecked containers in order to interface with C libraries that use bounds-unchecked containers, and exploits might take advantage of inadvertent undefined behaviors in this interface code.
I think you're misstating the mantra, but anyway... Incompetence isn't limited to proprietary software. Does that make you happy?
Kythe
In this case, the code is open, so pretty much everyone understands exactly what happens, exactly how bad it is, and how to fix it.
Ahem, sir. Open source can be useful, but it is not a magic bullet like that. Even I can read and understand the C code in OpenSSL, but to see the bigger picture and to understand how this particular software actually works and is arranged is completely different story. I bet that in case of OpenSSL, only under 100 people in the world can "understand exactly what happens, exactly how bad it is, and how to fix it".
The heartbleed vulnerability existed for years in OpenSSL codebase too. We have no idea if someone exploited before it was finally fixed.
The point being that we are starting to see that there are no guarantees that serious bugs will be caught quickly in open source software any better than in closed source software.
Remember also that while the public domain does not see the source for closed software, the company is able to pay for professional code audits which ensures that the code actually gets the eyeballs to go through it very carefully and thoroughly.
in the same way any company using open source can pay for a professional code audit
For this particular bug, seeing the actual code at least helped me to understand what was going on - and undoubtedly helped many who wrote explanations for it. I don't really need to understand the core crypto-algorithms in OpenSSL's core to understand what happened in this case.
His code was reviewed. The established process was followed. The reviewer and possibly anyone else with access to the code could be viewed as negligent for not preventing this undesirable functionality.
Of course, and I hope they do.
Robin Seggelmann, thank you and the entire OpenSSL Team for your contributions to free open source software. Glad we could find a serious security flaw, that you're helping to find out how it happend and that the OpenSSL crew is so fast in coming up with a fix.
With just about any other development paradigm and folks like MS we'd've waited for weeks for that to happen.
Carry on with the good work, you guys rock!
We suffer more in our imagination than in reality. - Seneca
And by questioning you mean bullets?
(I totally don't condone killing this guy. I am, however, baffled as to why he would admit to being the one who committed this bug, as it wouldn't surprise me if *someone* from some large corporation decided to call up their secret hitman line and get even for the billions of dollars the guy lost them.)
The fix has the sum 1 + 2 + 16 hardcoded into it. Twice. Why not speed things up by replacing that with a hardcoded 19? (If the sum has significance to a human reading the code, that could be explained in a comment.)
That that is is that that that that is not is not.
Programmers make these kinds of mistakes all the time.
Do you think that one of these days, a mistake like this will have catastrophic consequences?
If yes, should I lay in a supply of freeze-dried food and shotgun shells?
That that is is that that that that is not is not.
We have no idea if someone exploited before it was finally fixed.
hmmmm. So this sort of phenomena only cropped up in the 80's right?
That's ~30 years ago. Shouldn't we start getting a bunch of retired end-stagers willing to divulge who was doing what with the zero-days in their pocket?
Death beds are a great place to learn the truth about what happened 30-40 years ago. For some matters, it's the only place you'll learn the truth. The field of computer security is reaching the end-life of their first generation. Where are the memoirs?
Bloomberg reports NSA has been using this exploit for two years. This looks a lot less like an accident now.
The above cannot be challenged in court. No court in the Universe holds jurisdiction over this. The contributor didn't sell you OpenSSL, he didn't force you to use it, it didn't tell you to use it, he didn't make any guarantees about its functionality, you have no contract, no warranty, no expectation for it to actually do anything, etc. You may as well sue someone after walking into their house uninvited, listening to them whistle while they're sitting on the toilet, and hearing a missed note.
They law may not work as logically as you assume.
In the U.S. a person can enter a homeowner's yard uninvited and without the homeowner's knowledge, slip and fall, injure themselves, and successfully sue the homeowner.
There are a lot of AC here, with a lot of talents from coding to network guru to string theorist to quality lawyer. I didn't realize you were one of the coding rockstar variety.
Suborbital [spaceflight] is the special olympics of spaceflight. - Rei
in short, all software is flawed one way or another.
Being a developer myself, I believe him completely.
"I missed validating a variable containing a length"
Developers half the time and floundering in the dark, only understanding one small portion of someone else's code. Years were spent developing the whole, and we usually need one change in one piece. When making changes we simple hope the code compiles. Unit Tests help a ton. It increases our understanding and it assures us that we didn't break anything else.
If the change is made and the code compiles and unit tests passed, why would anyone be worried?
This shows that we need more tools that scan code. Can we write a new rule in source code scanners that can detect such a vulnerability and make sure it never happens again?
If they're required to interface with existing C libraries, they're going to run into similar unsafe API wrapping regardless of their chosen language. C++ may interface with C libraries more frequently than others, but that doesn't make the "safer" languages immune.
The process of dissolving a big problem into low-level steps as is required by C programming is mentally brutal. You can't just go "I want to save the text that was deleted and restore it when they hit the undo key." You have to translate that into variables, pointers, structs, mallocs, and glue logic.
Or you could use some tools which abstract away this problems.
- you could use a high level language where some of this porblems don't exist (e.g.: no pointers, and automatic garbage collection).
- or you could stay within the C/C++ world and write wrappers that take care to check everything (for example, almost any moderne tool-kit [Qt, Boost, or even default C++'s std++] will define type that are bound checked automatically [QByteArray or std::string] and smart pointers.
- some of these could even by implemented in plain C.
(But full implementation might require some Macro-ugliness. GTK+-level of ugliness)
Done correctly, such tool can automate the taking care of corner cases that can break the system.
But instead some programmer still decide to use as a simpler syntax for assembler and do everything bare-metal.
In case of OpenSSL, I'm a bit surprised. It has so many helper functions, but nobody has bothered yet to implement buffers with buffer-size checking safetey.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
... His work was reviewed, but the reviewer also missed the error, and it was included in the released version of OpenSSL."
So where is the apology from the Reviewer? They are just as at-fault as the programmer !
Not to mention the people in the user companies that checked it...