Slashdot Mirror
Should Vendors Close All Security Holes?
johnmeister writes to tell us that InfoWorld's Roger Grimes is finding it hard to completely discount a reader's argument to only patch minimum or low security bugs when they are publicly discovered. "The reader wrote to say that his company often sits on security bugs until they are publicly announced or until at least one customer complaint is made. Before you start disagreeing with this policy, hear out the rest of his argument. 'Our company spends significantly to root out security issues,' says the reader. 'We train all our programmers in secure coding, and we follow the basic tenets of secure programming design and management. When bugs are reported, we fix them. Any significant security bug that is likely to be high risk or widely used is also immediately fixed. But if we internally find a low- or medium-risk security bug, we often sit on the bug until it is reported publicly. We still research the bug and come up with tentative solutions, but we don't patch the problem.'"
I did not RTFA, but I did read the summary. I did not hear his argument, I heard his conclusion repeated with more words.
Yes.
Exploit Chaining means that low risk holes can become high risk hole when combined. Patch them all. Patch them quickly.
If you mod me down, I shall become more powerful than you could possibly imagine.
We still research the bug and come up with tentative solutions, but we don't patch the problem. I can understand the point if it's to save time and money for other things, but if they are going to find a solution to the problem and time/money is already spent, then that is completely wasted if it isn't utilized. Plus you're risking the data by not closing a known hole or bug. Doesnt make sense.
As is too common, the ./ summary doesn't have the relevant portions of the article under discussion, so let me try to summarize the main points of their argument.
1. It is better to focus resources on high risk security bugs.
2. We get rated better (internally and externally) for fixing publically known problems.
3. Hackers usually find additional bugs by examining patches to existing bugs, so a patch could expose more bugs than fixes are available for.
4. If we disclose a bug and fix it, it just escalates the "arms race" with the hackers. Better to keep it hidden.
5. Not all customers immediately patch. So by announcing a patch to previously unknown to the public bug, we actually exponentially increase the chances of that bug being exploited by hackers.
It could work as well as the normal method, but if it catches on, it will mostly be used as an excuse to not do anything until publicly shamed. Call me cynical.
Hax-fu?
Basically ...
.... what the fuck?
#1. If we spend time fixing those bugs, we won't have as much time to fix the important bugs.
Translation: we put in so many bugs that we don't have time to fix them all.
#2. We give priority to any bugs that someone's leaked to the press.
Translation: we only fix it if you force us to.
#3. "Third, the additional bugs that external hackers find are commonly found by examining the patches we apply to our software."
I had to post that verbatim. They're releasing new bugs in their patches.
#4. "Fourth, every disclosed bug increases the pace of battle against the hackers."
Yeah, that one too. The more bugs they fix, the faster the
#5. If we don't announce it, they won't know about it.
Great. So your customers are at risk, but don't know it.
Pre-emptive disclosure works against the typical closed source company.
Option 1:
Exploit is published, patch is delivered really quickly. Sysadmin thinks, "Those guys at company X are on top of it..." PHB will say the same.
Option 2:
Unilaterally announce fixes, make patches available. Sysadmin doesn't bother to read the whole announcement and whines because it makes work she doesn't understand or think is urgent. PHB thinks "Gee company X's software isn't very good, look at all the patches..."
The market for secure software is small even smaller if you add standards compliance. Microsoft is a shining example of how large the market is for insecure, non-standard software.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
Examples:
Not likely to be fixed completely - In some ways, Windows is a security hole
Could be fixed if escalates - password strength and use
Should be fixed - Lack of any authorization requirements etc.
If you remember the Pinto car-b-ques, there is a money factor to think about. Since most standard computing systems are not life-critical, some bugs can be left for later. Some bugs you might know about but they are not in your code such as those shipped with the networking stack of the RTOS that you use for an embedded product. An insecure FTP client on an embedded machine that has no access to other machines or sensitive material is not terribly bad.
On the other hand, if the machine can be compromised and allow the attacker access to other machines... that needs to be fixed.
Support NYCountryLawyer RIAA vs People
Was not it GM, that lost millions of dollars a few years ago in a lawsuit brought by people (and their kin), whose car was rear-ended on a toll plaza and exploded in flames?
GM's arguments, that making the car's fuel-tank more protected was too expensive for the modicum of additional safety that would've provided, were — for better or worth — ignored by the jury...
In other words, you may not deem a security hole to be large compared with the expense of pushing out another patch, but if somebody gets hurt, and their lawyer subpoenas your internal e-mails on the subject, you may well be out of business.
In Soviet Washington the swamp drains you.
The one you hope for: Someone finds and public announces a problem. You team looks "quick to act" deploys a solution.
That other one: Someone exploits the bug to a degree you and your team never considered and your user community is devastated.
all known security bugs should be fixed, but low-risk non-public ones can be low priority. we cant expect any vendor to send a patch each and every time they find a security bug, but once they find one the next version they release damn well better have it patched.
I think developers and companies should think long and hard about how such policies would be received if the end-user were presented with them in plainspeak.
"Welcome, JoeUser. This is WidgetMax v2.0.3. As a reminder, this product may contain security holes or exploits that we already know about and don't want to spend the money to fix because we internally classify them as low-to-medium risk."
I'm not saying it's necessarily wrong -- budgets are finite -- but keeping policies internal because of how they would be viewed publicly is deceiving your customer, full stop. Also, these guys are setting themselves up as final arbiter of what's risky in code exploits. It makes one uncomfortable to think about.
If an automaker or toy manufacturer didn't issue a recall on a minor safety issue immediately, they'd get tons of bad press. But a software company can sit on just about any security bug indefinitely (I'm looking at you, Microsoft) and few people care.
I suspect 2 factors are at work here:
#2 probably won't ever happen industry wide, and until the public understands how much impact software security can have, they won't care.
We do the same thing. Every company has limited resources, and every decision is a usiness priority decision. So the decision is always between new features and old bugs.
Outside of terribly serious security holes, security holes are only security holes when they become a problem. Until then, they are merely potential problems. Solving potential problems is rarely a good idea.
We're not talking about tiny functions that don't consider zero values. We're talking about complex systems where every piece of programming has the potential to add problems not only to the system logic, but also to add more security holes.
That's right, I said it -- security patches can cause security holes.
It is our standard practice not to touch things that are working. Not every application is a military application.
I'll say it again. Not every application is a military application.
Your front door has a key-lock. That's hardly secure -- they are easily picked. But it's secure enough for most neighbourhoods.
So the question with your software is: when does this security hole matter, and how does it matter. If it's only going to matter when a malicious person attacks, then the question comes down to how many attackers you have. And if those attackers are professional, you might as well make their life easier, because they'll get in eventually in one way or another -- I'd rather know how they got in and be able to recover.
How it matters. If it reveals sensitive information, it matters. If it causes your application to occasionally crash, when an operator is near-by, then it doesn't matter.
There are omre important things to work on -- and many of these minor security holes actually disappear with feature upgrades -- as code is replaced.
I could see not waiting till your next regular patch, so as to avoid bringing it to the attention of the hackers.
But the rest of his arguments are pretty crappy.
excitingthingstodo.blogspot.com
Low-risk does not mean easy to fix. Sometimes, a bug might be a very low-risk bug, but demand immense amounts of time to find and fix. For instance, sometimes I might be writing a program, and at some point, it begins crashing unpredictably, but very rarely. I know that there is a bug, but I have no idea what the trigger is, I have no idea which part of the code contains the bug, and I have no idea how to fix it. Since the MTBF is (say) 3 months, and (say) the code is not long-running (like a daemon or a kernel), it is probably not worth finding and fixing the bug.
Now, that's bugs, which is a wider category than security holes. So, suppose that instead of crashing, it very rarely and briefly enters a state in which a particular sequence of bytes sent to it via the net can cause it to execute arbitrary code. Furthermore, suppose the program should never be running as root, so the arbitrary code is nearly useless. This is a low risk security hole, and probably not worth patching.
Could take hundreds of man-hours to find the cause, and perhaps even longer to fix. Probability of ever seeing this exploited is very very low. Should it then be patched?
SIGSEGV caught, terminating
wait... not that kind of sig.
IANAL, but if a company suffers a significant financial loss due to a bug that the vendor knew about but did not patch, does that not open them up for big time law suits?
"Always forgive your enemies; nothing annoys them so much." - Oscar Wilde
The place I worked for was a security company. They had no automated unit testing and never analyzed for intrusions. You'd be shocked to find out how many holes exist on devices people depend on to keep them safe. The employees took it upon themselves (subverted authority) to patch our product. Security problems, even on security hardware, were not "priority" issues.
We too "trained" our coders in the art of secure programming. The problem, of course, is that we were also training them in basic things like what a C pointer is and how to not leak memory. Advanced security topics were over their head. This is the case in 9 out of 10 places I've worked. The weak links, once identified, can't be fired. No... these places move them to critical projects to "get their feet wet."
At the security giant, training only covered the absolute basics: shell escaping and preventing buffer overflows with range checking. The real problem is that only half of our bugs were caused by those problems. The overwhelming majority were caused by poor design often enabled by poor encapsulation (or complete lack of it).
There were so many use cases for our product that hadn't been modeled. Strange combinations of end-user interaction had the potential to execute code our appliance. Surprisingly, our QA team's manual regression testing (click around our U.I.) never caught these issues, but did catch many typos.
I don't believe security problems are inevitable. I've been writing code for years and mine never has these problems (arrogant, but mostly true). I can say, with certainty, than any given minor-version release has had 1,000's of high-quality tests performed and verified. I use the computer, not people... so there's hardly any "cost" to do so repeatedly.
I run my code through the paces. I'm cautious whenever I access RAM directly. My permissions engines are always centralized and the most thoroughly tested. I use malformed data to ensure that my code can always gracefully handle garbage. I model my use cases. I profile my memory usage. I write automated test suites to get as close to 100% code coverage as possible. I don't stop there. I test each decision branch for every bit of functionality that modifies state.
Aside from my debug-time assertions, I use exception handling quite liberally. It helps keep my code from doing exceptional things. Buffer overflows are never a problem, because I assume that all incoming data from ANY source should be treated as if it were pure Ebola virus. I assume nothing, even if the protocol / file header is of a certain type.
Security problems exist because bad coders exist. If you code and you know your art form well, you don't build code that works in unintended ways. Proper planning, good design, code reviews, and disciplined testing is all you need.
They say: if you know about it, you're obliged to fix them. And then you kick your QA department's butts around the corridor several times. If your customers are your software testers, then you're business model is likely corrupt. And while there are a number of coders that will complain that it was the libs, or the other guy's fault, ultimately, a responsible organization takes ownership of their faults, just like humans should.
---- Teach Peace. It's Cheaper Than War.
Besides, aren't there liability issues with knowingly shipping a product with undisclosed defects?
The fix the problem as soon as they discover it. The next release of the product does not have the problem. If the
problem becomes public before the next release, then they immediately issue the patch for it and hope that people
patch.
As long as they release often enough that the fixes are largely in place before the problems are found, I have no
issue with this. It actually seems responsible since it posses less risk to the customers that are slow in patching
their systems that the alternative.
*sigh* back to work...
Are you willing to indemnify your users for any and all losses suffered by them due to a flaw/bug which you knew about but chose not to patch? If not, then patch it!
'The tyrant will always find pretext for his tyranny.' - Aesop's Fables
The problem is that if you've discovered a security hole; chances are someone else has as well. Just because a problem hasn't been reported to your company doesn't mean that it is unknown.
History shows that there are lots of black hats that will sit on security breaches/expploits/bugs/etc and exploit them for their own end rather then reporting them to a company. Breaches in security should be patched as soon as they are discovered. If 1 person found the bug/hole/exploit/whatever, that means another person can find it as well. There's nothing there about having to report it to a vendor once found by either person.
One of these days i'm going to find this 'peer' guy and reset HIS connection!
Not that simple. Developing a patch does not fix a security hole. Releasing a patch does not fix a security hole. Applying a patch fixes a security hole, if all goes well. When you combine the fact that the number of holes in existence is multiplied by the number of installations, with the fact that the development team very rarely has any power over when patches are actually applied, security through obscurity doesn't look so cut-and-dried naïve versus publicizing your holes by releasing patches.
Notice that the person who posted the argument in the article never said they leave holes unpatched to save "a few hours of time". He didn't say they left holes unpatched at all. He said that they prioritized based on severity and publicity—who wouldn't? And he said that patches for unknown holes were developed, tested, and not released until they were needed. This gives them the advantage of not alerting the malicious user community that there are holes they can exploit if they act more quickly than the users who have to apply the patches. But it also means that when the time comes for them to release the patch, they know it's been patiently tested, not kneejerked out the door. I'd rather have no patch for an unknown hole that's not likely to be exploited than a patch that's going to make my software buggy.
The only downside to this is that you have to trust the developers to correctly assess the risk of leaving unknown holes unpatched. If they think a hole is unlikely to be randomly spotted and will only do minimal damage if it is, but script kiddies spot it in a week and manage to get remote root access from it, yes they screwed up by not patching it right away. Mistakes like that can be made the same way the mistakes that created the hole in the first place were made. But you have to make such judgments just to prioritize the holes, regardless of what you intend to do with them. And there certainly isn't anybody more qualified to make those judgments than the developers who discovered the hole.
Yes, yes they should patch them all. Personally it'd eat away at me knowing I could spending a few minutes, hours, or days to fix a vulnerability in my software. I don't think I could take pride in what I do if I just leave crap like this around because I don't have to fix it and don't think it's important unless someone finds it publicly. I'm glad they fix the HIGHs (however they rate this.. who knows?) and the publicly disclosed ones. But why not fix the small ones as you find them? It's a little bit of embarassment every time an issue is found. This is one less piece of embarassment. However, maybe it's the quasi-perfectionist in me, I couldn't imagine not fixing this stuff.
The article and responses miss an important point: patches of any kind are risky! And not just because they might introduce a new security flaw, but more generally because they may break some feature or another. In applications with millions of lines of code, and where the cost of doing a patch release amortized over all customers is millions of dollars, it can make lots of sense to just roll a fix into the next planned upgrade release. That way you get a complete Q/A and customer beta-test cycle to increase the confidence level of the fix.
You know something, I was planning on rebutting, but you're entirely correct. But you make it sound like a bad thing. Some of our clients are educated and experienced in the technical world. Many of them choose to see the forest from above -- and so they demand a taller tree first.
When given the option between a week of effort to fix a security hole (for free), or a week of effort to build a new feature (for cost), not all but most of our clients prefer the latter. They would rather grow their business (even on an unstable balancing act) than safe-guard their business. And all the advice in the world doesn't seem to be able to change that.
I guess it's the typical/traditional backup advice. I have yet to meet a client that is willing to spend the time and effort to develop a reliable backup routine (not even a restoration routine). They'd rather run the risk of losing all of their data, and dealing with the headaches and fall-out, than to spend money to prevent something that may never occur.
I can respect that -- hell I make the same decisions every day. And sometimes I even make those decisions with other people's data. But that's the very same risk/reward decision that every business entrepeneur makes. So in the grand scheme of things, I guess it's insignificant compared to most of the other decisions in running a business.
Just off the top of my head, I'd say that firing an employee is a bigger risk than worrying about hard drive failures.
Wow, what's it like to live in a world where everything's black and white? ;)
But seriously, anyone, and I mean anyone, who's worked with a large program (depending on the language you use, over 10,000 lines) knows that you can't change code without the risk of unintended consequences. If a small security hole requires a large rewrite, it's nearly 100% guaranteed to introduce new bugs and security holes.
And if you rtfa you'd realize that the person has real numbers to back up the fact that most exploits are used after the patch is released and the details are public.
Finally, the argument that the company should do what's best for the consumer is bullshit. Profits drive business, everything dealing with the software should boost the bottom line somehow. If you want people to program something for the pure joy of programming, use open source, otherwise you're buying from a business and you have to deal with the downsides of businesses.
> it begins crashing unpredictably, but very rarely. I know that there is a bug, but I have no idea what the trigger is, I have no idea which part of the code contains the bug
./a.out
./a.out core
// line 5
You sound like a person who needs a good debugger. Take gdb for example. You can ask your customer to send in the core dump file, which the program produces during the crash, then you load this core dump into gdb and not only will you get the exact location of the crash, you can also check where it was called and what values each variable had.
Allow me to demonstrade how easy this is:
# Enable core dumbs (if not enabled by default)
$ ulimit -c unlimited
# Execute program and notice it crash ( this could be done by the customer also, with proper instructions or automation )
$
Floating point exception (core dumped)
# We now have the core dump and a debug version of the program. Start the debugger
$ gdb
(snip)
Program terminated with signal 8, Arithmetic exception.
#0 0x080483eb in main () at b.cpp:5
5 int c = a/b;
(gdb) info locals
a = 4
b = 0
c = -1209610252
d = -1208343920
(gdb)
And there we have it. The programs crashes in b.cpp, line 5 when assigning a/b into c. info locals reveals that b is zero, which is the cause of the crash.
Oh, and the source code for the b.cpp:
int main()
{
int a = 4;
int b = 0;
int c = a/b;
int d = c;
return 0;
}
I'm shocked by how many people answer this with an unqualified "Yes." That's not realistic at all.
Here's an example. An app asks for your password. That password gets written to memory for a period of time. During that time, the page containing the password may get swapped to disk. The system may then crash or lose power, leaving the password on disk. I could then boot into another OS, read the swap file, and recover your password. Unlikely, but possible.
There, I "found" a security hole. Want to patch it? You could try to make every app that uses a password store it in wired (not swappable) memory - but performance will suffer (and good luck doing that in every app). You could also dynamically encrypt all data that gets written to the swap file, and decrypt it when read, at the cost of performance again.
Are you willing to trade performance in every app to defend against this mostly theoretical vulnerability? Probably not. Security is about tradeoffs. Welcome to the realities of software development.
Your example is way to simplistic. I've seen core dump cases in which it was perfectly clear why it was crashing: the data structure got into a logically inconsistent state that it should never be in. The question is how and when. In case of big data structures (in some of the cases I've had to deal with: hundreds of thousands of objects, built gradually and modified heavily during several hours) finding the exact sequence that causes the inconsistency can be a nightmare. Plus, it might have been sitting around in this state for a long time before the program actually enters a path that leads to a crash.
Also, the worst type of bug is the Heisenbug: those that go away as soon as you enable debugging, or add even just a single line of extra code to monitor something while the stuff is running. I've seen my share of those as well. Sometimes persistance pays off and you find the root cause within hours or days, but sometimes reality forces you to give up. It's no use spending five weeks fruitlessly looking for a rare intermittent bug triggered by a convoluted internal unit test case if at the same time easily tracable bugs are being reported by paying users that need a solution within a week.
Linux user since early January 1992.