Slashdot Mirror
Is Finding Security Holes a Good Idea?
ekr writes "A lot of effort goes into finding vulnerabilities in
software, but there's no real evidence that it actually improves security. I've been trying to study this problem and the results (pdf) aren't very encouraging. It doesn't look like we're making much of a dent in the overall number of vulnerabilities in the software we use. The paper was presented at the Workshop on Economics and Information Security 2004 and the slides can be found here (pdf)."
Is finding security holes a good idea?
Writing Security Considerations Sections
Hmmm.
In order to fix vulnerabilities, you have to find them. However, as soon as they are found and publicized, some script kiddie exploits them. So yes finding them is a good idea, patches just need to be released and INSTALLED before script kiddies expliot them.
While we still have a long way to go regarding security, I believe we're still learning how to design security into systems. People are creative. Computers are not. I believe that we're infants at this stage of computer development. Look at how far we've come in 30 years? Where will we be after 30 more?
It's still a brand new world to explore. We have alot of work ahead of us.
Has Comcast disconnected your Internet account? Same here. You can read about it at http://comcastissue.blogspot.com
Sounds alot like something Microsoft has been saying...
I cannot believe that sticking your head in the sand is any better. I would think that there are many examples of security holes being found and patched before they could be exploited.
If anything, the data seems to point to the fact that software companies and users need to act on security holes and patches more quickly. This may require better education of the user, and it also would help to have better patching mechanisms.
Great ideas often receive violent opposition from mediocre minds. - Albert Einstein
'Cause trusting the manufacturer to make their product secure has shown to be such a good solution in the past.
The alternative is to not look and leave that to the people who will fix it or the people that will exploit it. Are you really comfortable with that?
I guess I'm a little unclear on what the research stated is supposed to actually accomplish.
Roving Web-Teleoperated Robot
As a sysadmin, I can tell you for certain that reading bugtraq and other vulnerability lists helps me. I can study trends in software, trends in company response and protect myself against problems. If I know a new worm or vulnerability has a prerequisite configuration then I can make sure to configure software in a way where I won't be vulnerable until a patch is release or until I can apply it.
Anyone who is subscribed to bugtraq can see the bad situation some software is in. Lately there was a lot of posts about Linksys that raised my eyebrow. Do I really want to deal with a company that doesn't properly address vulnerabilities it's made aware of? Good thing bugtraq posters had a workaround for the Linksys remote administration problem.
The global economy is a great thing until you feel it locally.
The real problem in software security lies in the design of the software itself. No amount of patches and service packs can secure unsecure software. Instead to be secure it has to be biult that way from the ground up. These findings seem to make sense in this context beacause patching software doesen't change the fundamental way it works.
411 Y0UR 8453 4R3 8310NG 70 U5!! -NSA
The goal of searching out vulnerabilities is to find them before the people with black-hats do. This is why most clearinghouses of such reports don't release the information until there is a fix (or until such time passes that vendors have demonstrated a lack of interest in producing a fix): the people who would exploit the bugs need to mount their OWN efforts to discover them.
Ignoring actual bugs, there are many other kinds of security vulnerability. We know that software will always have side-effects that we don't intend. In fact, we desire this (e.g. providing a user with the flexibility to use the product in ways not envisioned by the creator). Sometimes those side-effects have security implications (e.g giving someone an environement variable to control a program's behavior lets a good user do something you might not have thought of, but it turns out a malicious user can abuse this in order to raise their security status).
This means that, as long as software is not static, security bugs will continue to be introduced. Discovering them as fast as possible is the only correct course of action... you KNOW the black-hats will be.
one of the reasons I now use Firefox as my primary browser is because so many exploits were found in IE. So even if Microsoft doesn't respond when exploits are found, these exploits do cause some people to look for more secure alternatives.
To answer the question, yes. Finding security holes is a good idea.
To the unasked question, "Is finding individual security holes the best possible use of a security researcher's time?", the answer is No. The best use of security research is to classify different types of security holes and use that information to create a development framework where those security holes are extremely difficult to recreate. For example, you're not going to find buffer overruns in Java code, since the memory is dynamically handled for you. Eventually, having security levels, encrypted buffers, etc. will all be part of a standard developer's library.
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
Evidence wouldn't show us that searching for security holes improves security... Rather, such a judgement requires reasoning and evalution of the evidence. Common sense stuff, here.
Do smashing cars head-on into brick walls improve car safety? No, of course not. Evalution of the results of the crash, and using those findings to build better cars, that is what improves car safety, and the situation is entirely analogous in the security world. The assumption is that there is always a weakest link in security, that link is the most likely one to be exploited, and the trick is finding that link and strengthening it against attacks in the future, hopefully to the point where it is more likely that other links are weaker.
If you find a security hole then the mistake has already been made. Fix the hole, but also make sure the same bug doesn't exist in any other program. Finding the same exploits again and again (buffer overruns, format string vulnerabilities, tempfile symlink vulnerabilities) reflects very badly on free software and programmers' ability to learn from bugs found in other software. (Not that proprietary software is necessarily any better - I am just not discussing it here.)
The OpenBSD people have built up a good track record on security by finding holes and fixing them everywhere possible. I am sure they would disagree with your assertion that finding holes does not help to improve security. Finding the bugs is an important first step towards not putting them back in next time you write code.
-- Ed Avis ed@membled.com
Many posters have already taken to jumping to bad conclusions having not latched on to one of the report author's best conclusions. If patches are not applied then the time and money spent on discovery are worthless. The only ways to make discovery worthwhile is if the patches are applied, otherwise discovery does not resolve the vulnerabilities.
Automatic/Forced patching is the only way to make discovery worthwhile because otherwise the number of vulnerable systems is unpredicatable over time and constitutes a large risk. Security issues must be resolved as quickly as possible in order to mitigate risks, and unless patch application is automated and enforced then discovery becomes meaningless.
Even if I knew that tomorrow the world would go to pieces, I would still plant my apple tree. -Martin Luther
The value of finding security holes is totally dependant upon the company whose product has the hole.
I work at Barnes & Noble as a bookseller; within a few months of working there, I found huge security holes in their in-store search system and network. I reported these to my manager; as soon as she grasped the scope of the hole, she called tech support in New York. I spent 30 minutes explaining all the different problems to them, and that was that.
I didn't think they'd do anything about it, and that's the problem--since it costs time and money, most companies can't or won't fix holes. To my surprise, however, my store began beta testing the next version of the in-store software. What was even more surprising was that every hole I had reported was gone (so I went and found more, of course).
There's never a silver-bullet; a new vulnerability will always surface. It's really hard to stay ahead of the curve, but it's something that must be attempted.
I'm thinking its more along the lines of - if we do not help find security holes, then we are giving less amunition to hackers. The only problem with the hypothesis is it assumes hackers only gain this "ammunition" through legitimate coders who are trying to find vulnerabilities. In fact, as all of us know, hackers do find security holes on their own, without help from other people.
I mod down so you can mod up. Your welcome.
Really. I didn't make that up, check the link! Who is this guy, and why is he giving me software security advice?!
In principle, I agree that automatically installing patches is a good thing in principle. But Microsoft has a habbit of changing their licenses and installing DRM when people "upgrade" and/or "install patches."
Also, imagine I have 2 programs. Both automatically install patches. Unpatched they both work fine. But when program #1 is patched, program #2 cannot run at all. Now this will probably be fixed eventually, but in the mean-time, I cannot run program #2 at all. If I need both programs, I'm fucked with the system of auto-patches.
However when I have a choice, how likely am I to install a patch? Not as likely (due to laziness). So the effectiveness decreases significantly.
Just read the article and have to point out a number of flaws in the methodology. First- it assumes that if the vulnerability is only known to a few then the number of intrustions will be low. Given the number of zombie computers out there, I do not think that is a safe assumption. Look at how the last few big viruses went around. I know those were exploiting known and patched vulnerabilities, but there is nothing to say that the same thing couldn't be done with a "day-zero" exploit.
Second- it doesn't address the level of the vulnerability. If it is an exploit that lets someone take over a machine, or format a drive, the cost of even those first, possibly limited, intrusions will be astronomical.
Someone finds a buffer overflow problem. Someone finds another one. Someone finds another one. Someone finds another one.
Someone realizes: "what if I centralized all my buffer routines and just got one little section of code working perfectly?" Then you get projects like qmail or vsftp, which simply are more secure. Then people start using these programs, and their systems really are less vulnerable.
This paper looks keeps using the phrase "given piece of software." It's talking about (lack of) improvements at a micro-scale, but ignores improvements in the big picture that can happen due to people getting fed up or scared.
If vulnerabilities were not discovered, would anyone bother to develop secure software?
I think this paper has as an underlying assumption, the popular view that it just isn't possible to write secure software, and that every large software system is just a big leaky sieve that requires perpetual patching. I don't accept that. I understand why the belief is widely held: most popular software was not originally written with security issues in mind. But this is something that will be overcome with time, as legacies die off.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
you can plug all the individual holes you want, it is still a crappy designed dam.
/me goes back to condeming the world to doom
if it designed differently, the number of cracks is smaller...
i wish reporters understood that. flame MS for not bringing lonhorn out sooner. XP is not good enough. everyone knows this, nobody in the popular press is saying it in the right way.
*sigh*
The study sheds light into a little studied phenomenon, and therefor shows interesting facts. It shows that the difference between black-hat-discovery and white-hat-discovery basically reduces to the number of exploits between discovery and public disclosure of a bug, which is negible compared to the total number of exploits during the lifecycle of a bug.
That may hold true and make sense if you study the total number of exploitable systems on the net, but totally ignores the fact that there are a very large number of systems with little priority for security while only a few depend on 100% system security.
Those few high security sites have the need and pay for resources to fix known flaws or patch their software asap. It are those who gain from the knowledge of a security flaw before the black-hat guys do. They cannot live with even the shortest vulnerability timeframes and usually patch exploits as soon as they get published on public security channels.
It may hold true that the work put into security auditing does not pay out on whole, taking all software users into account, but for those few who really care about security it surely does...
I think the story raises a good point. The best analogy I could pint out would be a dam where new leaks keep popping up and you quickly rush to patch them. You spend so much time patching over the leaks that the fundamental design problems in the dam are never fixed. :p
There are multiple strategies that will actually improve security far more than just trying to ferret out a new vulnerability. I personally recommend using Java or another type-safe language for programming if at all possible since the most common memory management errors are eliminated. Hoevwer, the best way to stop major security breaches is to have a security layer that will assume software programs will be compromised somehow. Then, the security layer is more interested in enforcing access to the system that program ought to have instead of just trusting the effective user ID of the program to hopefully do the right thing.
A bit of karma-whoring here for my thesis project which is based on earlier work in Mandatory Access Controls in Linux, as well as the much more well-known SELinux
kernel modules.
I personally did my thesis in Domain & Type Enforcment which simply puts running processes into various different domains that have certain access rights to Types. A type is just a name tag assigned to files, and in my case you can also type system calls, network sockets, and eventually even Linux capabilities. It is similar to part of SELinux but also designed to be much simpler to understand & implement as well.
Anyway, these systems all are designed with the assumption that vital processes will be compromised and the onus is on the policy writers to enforce least-privilege on the processes. This may sound difficult to do, but it is actually trivial compared to the approach we are using now which is to try and figure out every possible attack and write perfect software (the point of the article). It is much easier to define what a program is supposed to do than every nasty malicious thing someone on the Internet can dream up that it should not do.
I've ranted long enough, but I think that there are good solutions to stopping about 90% of the crap that we see going on today, and that the other 10% will be fun to keep us security professionals employed
AntiFA: An abbreviation for Anti First Amendment.
The parent is exactly right. Having read through this paper now, I realize what it misses: the economic impact of the information.
Much work has been done in economics regarding the affect that inadequate information flow has on a market; a Nobel Prize was wone in it lately. The paper assumes that there are a constant number of vulnerable machines, as you can see on page 2, for any given vulnerability. First of all, it ignores the fact that someone has to choose to use these vulnerable products. Second, it ignores the choice that comes to sysadmins when they learn that a particular company's products are more likely to have bugs, as the parent describes.
The moral of the story is, the paper tries to be more broad than it can - by assuming that software acquisition decisions never happen, it fails to see the effect of vulnerability disclosure on these decisions. And these decisions, made well, do in fact make us more secure. The "software defect rate" in total might not decrease, but the defect rate in *used* software may well decrease.
|/usr/games/fortune
if the patch breaks an application and the machine goes unpatched there is a loss in security because of potential intrusion. If the patch is applied there is a potential loss of productivity.
Not all patches are security patches. Many patches fix problems, such as the spell check function doesn't work correctly. Or some other function doesn't work correctly. These won't compromise security, but they may interfere with other programs.
Take a sinking boat. If you are bailing water out, and the boat isn't sinking any more, it does not follow that bailing water isn't a good idea. If you stop bailing water, you're sunk. If good guys stop finding and fixing vulnerabilities, you're sunk, too.
I know I'd feel much better if only the blackhats were looking for and exploiting security vulnerabilities. If the whitehats don't look for them, publish them and give the vendors/developers incentive to do something about it, then any response to an attack is purely reactive. Welcome to the world where every system is a zombie. In fact, it would soon be duelling zombies. Wouldn't that be great!
Making the source code available to anyone makes it easier for people to find holes.
This is a proven, incontrovertible fact.
It makes it easier to find, but that doesn't mean open source automatically more secure - you still have to have the right people actually looking, and a defect has to be what they're looking for. I'll explain.
If no-one qualified to spot the hole bothers to look, open source doesn't buy you anything (this is why bugs in things like OpenSSL can linger quite awhile before being discovered - not enough of the right people bothering to look, even though *anyone* can and many do).
A bigger problem is the disconnect between design limitations not meeting end-user expectations. The recent shining example of this is the latest set of CVS vulnerabilities: The CVS team does not claim CVS is secure enough to be publicly-accessible over the internet; yet it frequently IS placed in this position, and that makes it an ongoing security disaster waiting to happen. (Linkage: "We have always said that CVS is not secure")
Bug? No; design limitation. But if the end users aren't aware of that (or, worse, choose to disregard the danger), it's still a vulnerability waiting to be exploited, and open source does NOTHING to prevent that.
So, "easier to find holes"? I'll go with the stock CompSci answer, "It depends". It's certainly not a simple or complete answer.
Xentax
You shouldn't verb words.
Anticipating the shitstorm of people lining up to say 'how stupid' without reading the FA, here's a nice little summary.
The paper is not quite as stupid as it sounded by the description, but it misses/ignores a critical flaw in the argument.
His basic premise is that patching is expensive and people don't do it anyway - probably true for the majority of systems. Therefore, he argues, black-hats are alerted to the security holes by the disclosure. He shows that it doesn't really matter whether holes are discovered by black-hats and are fixes are released after the exploit, or discovered by white-hats and exploited after the fix has been released but not applied.
However, his arguments are based on averages. Where he's wrong is that if you have some systems that are simply so valuable that they cannot be comprimised, proactive bug fixing coupled with a manic obsession for patching your system the moment a patch is released is still the best way to stay safe
foo mane padme hum
exceedingly so, in fact. It boils down to a single sentence: /them/ that it's not worth looking for it.)
It's better to find the security hole yourself and fix it than for someone with malicious intentions to do so and exploit it.
(And good luck convincing
Work is punishment for failing to procrastinate effectively.
His basic theory is that he believes, given the current rates of discovery, poor patching rates and the large number of bugs that:
1) Due to massive information exchange and slow patching/fixing, post announcement explotitations are not significantly less than explotiation caused by un-announced bugs, so announcing does not help.
2) There are so many bugs out there that finding a bug and announcing it does not produce a singificant reduction in the number of "bugs unknown to white Hats" so it does not increase security significantly.
His major errors are ignoring the following: 1) Exploitation post announcement is almost entirely done against the "lesser" computers, i.e. either machines with un-important data/programs (home pc's) or important machines with incompetent sysadmin. As such the real cost of these explotations is either A) practically null, or b) high, but likely to get the incompetent sysadmin fired, resulting in i) better employment prospects for good sysadmin and ii)an over-all improvment in quality of security for that company. So while the # of explots may be higher for Post-announcement bugs, soceity has a REAL social and economic gain that is very significant.
2)A)The announcement of bugs allows people to judge how well written a program is and therefore make an informed decision to NOT buy inferior producs (say Windows).
B)That perhaps our problem is not that we are announcing the bugs but that we are not doing sufficient bug hunting. He seems to be implying that because bug hunts don't find enough bugs, the solution is not to bug hunt. But anyone with a brain should be able to see that if we are not depleating the unknown bugs fast enough than one possible solution is to tremoundously increase our bug hunting, not to stop the bug disclosures.
excitingthingstodo.blogspot.com
Crackers will dissect your patches to create exploits, but you'll at least have protection available when the exploits go wild. If they don't find vulnerabilities from the patches, they'll just spend more time trying to find them manually, and the more you leave unpatched, the better the odds they have of finding one. Your customers who care about security the most will install the patches on time, and get pissed if a cracker exploits something before you've patched it.
But it's even better to find them before the product ships, and design early on to avoid the common ones. I believe the author of qmail is still offering thousands of dollars to the first person who finds even a single vulnerability.
One of the key factors that has kept Mainframes secure for so long is the fact they were designed as a secure environment in the first place:
The society for a thought-free internet welcomes you.
new ones just keep coming along. What is the point. We cured polio and smallpox, but now we have HIV. We should have left well enough alone.
Maybe it is just me, but I think linking bubonic plague to flea infested rats was a beneficial advancement in progressing the human situation, for multiple reasons.
Finding problems which can be disclosed at the same time as a patch is very good.
All the major Linux distributors will release updates in a timely manner, and enable people to install them with minimum effort - much like Microsoft does. The only difference with Microsoft's patches is they can, rarely, break things. I've never seen this happen with a Linux update.
Personally I've never heard anybody say anything bad about the pro-active way which the OpenBSD team audit their codebase and this is one of the reasons why I started the Debian Security Audit.
Having a dedicated team of people auditing code, combined with the ability to release updates in a timely manner is definately a good thing.
(The results of my work show that even with only a small amount of effort security can be increased)
Did I mention that I'm available for hiring? ;)
"A lot of effort goes into funding law enforcement in society, but there's no real evidence that it actually reduces crime. I've been trying to study this problem and the results aren't very encouraging. It doesn't look like we're making much of a dent in the overall number of crimes in our society."
---
If you think security is bad now, just stop fixing security vulnerabilities and see how much worse things get. It's like a sump pump - it may not fix the leak, but it'll keep you from drowning.
Most roads have some holes in them. Some would say it is a natural part of the road building process. Other argues roads can be made hole free. Generally roads are thought to be without holes when initially developed, but over time holes are found. While identifying and patching holes in roads is thought to be a good thing, numerical analysis shows otherwise.
Patching roads requires people to stop the flow of traffic, and puts workers at risk of being injured or killed by users of the roads. A road that is fully patched encourages users to drive faster, burning fossil fuels at a lower efficiency compared to the slow speed drivers use when they see holes. Driving slower will also save lives in the event of an accident and cause drivers to pay more attention to the road since they never know when a hole could be in their local path.
Many times the problem is not with the road, but the surface that it is built on. Patching the road can only be assumed to be a stop gap measure at best and will likely have to be patched again. Holes in the supporting structure will almost always show up in the things built on it.
Fixing pot holes slows innovation. Once it becomes accepted that roads have holes in them, consumers will demand vehicles able to deal with them. If hole patching was stopped right now, studies show we would all be flying to work in our personal jetson mobiles within 8 years. Space previously set aside for roads will be converted to trails for bikes, bladers and walkers. Butterflies will land on your outstretched hand while you stop to observe the wild flowers.
Fixing holes only maintains the status quo and dominance of local government and the corrupt DOT branches of the states. If you reduce their budget by even 1% they will go on strike and roads will quickly deteriorate. Some communities out there are leading the charge in not fixing pot holes to bring you a world of glass houses on stilts and talking dogs with jet packs.
In conclusion our findings indicate the DOT should be abolished. They have served their purpose but have no place in todays innovative world.
Black Hats are dynamic actors -- if the world changes so that Figure 2 in Eric's paper is the norm rather than Figure 1, the Black Hat community will evolve to live in the new world. Their new goal will be to maximize area under the "Private Exploitation" part of Figure 2. We may be better off with the current state of affairs.
Good grief!
You really think the problem is C and C++?? I know that these arn't the holy grail of programming languages but to put some of that blame on them is very nieve. You can write buggy, unsecured code with asm! It's got very little to do with the language involved (the compiler and libraries used may have an effect) since it all gets put into machine code anyway. Blame the design and implementation, not the tool.
Silly rabbit
The author is naive enough to not have considered the Grey Hat Discovery (GHD) of vulnerabilities where the hackers (not being a black hat) do not go on a private exploitation spree, because most people don't want to violate laws and end up in prison, but definitely want to make a big name for themselves, so publish the vulnerability without giving the chance to the Vendor to fix the problem before public disclosure. The percentage of grey hats is enormous compared to both white and black hats, so most of the arguments in this paper are based on a WEAK Foundation of practical reality. This changes many mathematical equations in the article, and lead to significantly different final conclusions. Contact me if you need more specific details.