Slashdot Mirror
Is Finding Security Holes a Good Idea?
ekr writes "A lot of effort goes into finding vulnerabilities in
software, but there's no real evidence that it actually improves security. I've been trying to study this problem and the results (pdf) aren't very encouraging. It doesn't look like we're making much of a dent in the overall number of vulnerabilities in the software we use. The paper was presented at the Workshop on Economics and Information Security 2004 and the slides can be found here (pdf)."
In order to fix vulnerabilities, you have to find them. However, as soon as they are found and publicized, some script kiddie exploits them. So yes finding them is a good idea, patches just need to be released and INSTALLED before script kiddies expliot them.
While we still have a long way to go regarding security, I believe we're still learning how to design security into systems. People are creative. Computers are not. I believe that we're infants at this stage of computer development. Look at how far we've come in 30 years? Where will we be after 30 more?
It's still a brand new world to explore. We have alot of work ahead of us.
Has Comcast disconnected your Internet account? Same here. You can read about it at http://comcastissue.blogspot.com
I cannot believe that sticking your head in the sand is any better. I would think that there are many examples of security holes being found and patched before they could be exploited.
If anything, the data seems to point to the fact that software companies and users need to act on security holes and patches more quickly. This may require better education of the user, and it also would help to have better patching mechanisms.
Great ideas often receive violent opposition from mediocre minds. - Albert Einstein
'Cause trusting the manufacturer to make their product secure has shown to be such a good solution in the past.
The alternative is to not look and leave that to the people who will fix it or the people that will exploit it. Are you really comfortable with that?
As a sysadmin, I can tell you for certain that reading bugtraq and other vulnerability lists helps me. I can study trends in software, trends in company response and protect myself against problems. If I know a new worm or vulnerability has a prerequisite configuration then I can make sure to configure software in a way where I won't be vulnerable until a patch is release or until I can apply it.
Anyone who is subscribed to bugtraq can see the bad situation some software is in. Lately there was a lot of posts about Linksys that raised my eyebrow. Do I really want to deal with a company that doesn't properly address vulnerabilities it's made aware of? Good thing bugtraq posters had a workaround for the Linksys remote administration problem.
The global economy is a great thing until you feel it locally.
The real problem in software security lies in the design of the software itself. No amount of patches and service packs can secure unsecure software. Instead to be secure it has to be biult that way from the ground up. These findings seem to make sense in this context beacause patching software doesen't change the fundamental way it works.
411 Y0UR 8453 4R3 8310NG 70 U5!! -NSA
The goal of searching out vulnerabilities is to find them before the people with black-hats do. This is why most clearinghouses of such reports don't release the information until there is a fix (or until such time passes that vendors have demonstrated a lack of interest in producing a fix): the people who would exploit the bugs need to mount their OWN efforts to discover them.
Ignoring actual bugs, there are many other kinds of security vulnerability. We know that software will always have side-effects that we don't intend. In fact, we desire this (e.g. providing a user with the flexibility to use the product in ways not envisioned by the creator). Sometimes those side-effects have security implications (e.g giving someone an environement variable to control a program's behavior lets a good user do something you might not have thought of, but it turns out a malicious user can abuse this in order to raise their security status).
This means that, as long as software is not static, security bugs will continue to be introduced. Discovering them as fast as possible is the only correct course of action... you KNOW the black-hats will be.
one of the reasons I now use Firefox as my primary browser is because so many exploits were found in IE. So even if Microsoft doesn't respond when exploits are found, these exploits do cause some people to look for more secure alternatives.
To answer the question, yes. Finding security holes is a good idea.
To the unasked question, "Is finding individual security holes the best possible use of a security researcher's time?", the answer is No. The best use of security research is to classify different types of security holes and use that information to create a development framework where those security holes are extremely difficult to recreate. For example, you're not going to find buffer overruns in Java code, since the memory is dynamically handled for you. Eventually, having security levels, encrypted buffers, etc. will all be part of a standard developer's library.
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
Evidence wouldn't show us that searching for security holes improves security... Rather, such a judgement requires reasoning and evalution of the evidence. Common sense stuff, here.
Do smashing cars head-on into brick walls improve car safety? No, of course not. Evalution of the results of the crash, and using those findings to build better cars, that is what improves car safety, and the situation is entirely analogous in the security world. The assumption is that there is always a weakest link in security, that link is the most likely one to be exploited, and the trick is finding that link and strengthening it against attacks in the future, hopefully to the point where it is more likely that other links are weaker.
"but there's no real evidence that it actually improves security"
OK, didn't RTFA, but is there 'real evidence' to the contrary?
You can't fix what you don't know is broken. Is ignorance a better security solution?
then someone who wants to do the real hacking will find them. If a malicious hackers finds the security hole, then he/she might utilize it, and they won't be nice enough to give us a patch to protect against it. So since the holes are there, lets find them and patch them BEFORE some malicious programmer does. Finding security holes is a good choice, making patches for security holes is a better choice, actually UTILIZING these patches for security holes is the BEST choice...unless you want to be on Citibank Identity theft commercials :)
I mod down so you can mod up. Your welcome.
Many posters have already taken to jumping to bad conclusions having not latched on to one of the report author's best conclusions. If patches are not applied then the time and money spent on discovery are worthless. The only ways to make discovery worthwhile is if the patches are applied, otherwise discovery does not resolve the vulnerabilities.
Automatic/Forced patching is the only way to make discovery worthwhile because otherwise the number of vulnerable systems is unpredicatable over time and constitutes a large risk. Security issues must be resolved as quickly as possible in order to mitigate risks, and unless patch application is automated and enforced then discovery becomes meaningless.
Even if I knew that tomorrow the world would go to pieces, I would still plant my apple tree. -Martin Luther
Security holes are bound to be discovered eventually, either by unscrupulous users or professionals with honest intentions.
By hunting for flaws in software and making them public, these flaws can be fixed... Not making a vulnerability public doesn't help anything. It just gives Joe hacker his own personal backdoor that he's free to use indefinitely.
-yeah
I'm thinking its more along the lines of - if we do not help find security holes, then we are giving less amunition to hackers. The only problem with the hypothesis is it assumes hackers only gain this "ammunition" through legitimate coders who are trying to find vulnerabilities. In fact, as all of us know, hackers do find security holes on their own, without help from other people.
I mod down so you can mod up. Your welcome.
... then someone else will. Hard to say if finding and fixing is helping, because noone knows how bad it would be if we didn't do it.
Then again, MS doesn't seem to be trying to find vulnerabilities in their own code; often it's found by others. Sometimes it's the bad guys.
Point being, it's hard to say what effect something is having when you can't contrast it against "not doing it."
Just read the article and have to point out a number of flaws in the methodology. First- it assumes that if the vulnerability is only known to a few then the number of intrustions will be low. Given the number of zombie computers out there, I do not think that is a safe assumption. Look at how the last few big viruses went around. I know those were exploiting known and patched vulnerabilities, but there is nothing to say that the same thing couldn't be done with a "day-zero" exploit.
Second- it doesn't address the level of the vulnerability. If it is an exploit that lets someone take over a machine, or format a drive, the cost of even those first, possibly limited, intrusions will be astronomical.
In the long term, one might hope that the vulnerability finding would feed back into software engineering, and eventually decrease the rate of introduction, but we're clearly not there today, and I'm not holding my breath for tomorrow.
So we've got 18 pages of math measuring an irrelevancy.
I think the story raises a good point. The best analogy I could pint out would be a dam where new leaks keep popping up and you quickly rush to patch them. You spend so much time patching over the leaks that the fundamental design problems in the dam are never fixed. :p
There are multiple strategies that will actually improve security far more than just trying to ferret out a new vulnerability. I personally recommend using Java or another type-safe language for programming if at all possible since the most common memory management errors are eliminated. Hoevwer, the best way to stop major security breaches is to have a security layer that will assume software programs will be compromised somehow. Then, the security layer is more interested in enforcing access to the system that program ought to have instead of just trusting the effective user ID of the program to hopefully do the right thing.
A bit of karma-whoring here for my thesis project which is based on earlier work in Mandatory Access Controls in Linux, as well as the much more well-known SELinux
kernel modules.
I personally did my thesis in Domain & Type Enforcment which simply puts running processes into various different domains that have certain access rights to Types. A type is just a name tag assigned to files, and in my case you can also type system calls, network sockets, and eventually even Linux capabilities. It is similar to part of SELinux but also designed to be much simpler to understand & implement as well.
Anyway, these systems all are designed with the assumption that vital processes will be compromised and the onus is on the policy writers to enforce least-privilege on the processes. This may sound difficult to do, but it is actually trivial compared to the approach we are using now which is to try and figure out every possible attack and write perfect software (the point of the article). It is much easier to define what a program is supposed to do than every nasty malicious thing someone on the Internet can dream up that it should not do.
I've ranted long enough, but I think that there are good solutions to stopping about 90% of the crap that we see going on today, and that the other 10% will be fun to keep us security professionals employed
AntiFA: An abbreviation for Anti First Amendment.
"To err is human, to forgive divine, to persist devilish" at least I think that's the quote...
But what we need to keep in mind is that no matter how hard we try our code is never going to be completely perfect. It's in our nature. I think finding security holes in software is necessary to build on our understanding of security flaws.
Agreed; the first rule of security (let alone *computer* security* is that you can't stop human stupidity.
Vintage computer games and RPG books available. Email me if you're interested.
Take a sinking boat. If you are bailing water out, and the boat isn't sinking any more, it does not follow that bailing water isn't a good idea. If you stop bailing water, you're sunk. If good guys stop finding and fixing vulnerabilities, you're sunk, too.
I know I'd feel much better if only the blackhats were looking for and exploiting security vulnerabilities. If the whitehats don't look for them, publish them and give the vendors/developers incentive to do something about it, then any response to an attack is purely reactive. Welcome to the world where every system is a zombie. In fact, it would soon be duelling zombies. Wouldn't that be great!
Comment removed based on user account deletion
Making the source code available to anyone makes it easier for people to find holes.
This is a proven, incontrovertible fact.
It makes it easier to find, but that doesn't mean open source automatically more secure - you still have to have the right people actually looking, and a defect has to be what they're looking for. I'll explain.
If no-one qualified to spot the hole bothers to look, open source doesn't buy you anything (this is why bugs in things like OpenSSL can linger quite awhile before being discovered - not enough of the right people bothering to look, even though *anyone* can and many do).
A bigger problem is the disconnect between design limitations not meeting end-user expectations. The recent shining example of this is the latest set of CVS vulnerabilities: The CVS team does not claim CVS is secure enough to be publicly-accessible over the internet; yet it frequently IS placed in this position, and that makes it an ongoing security disaster waiting to happen. (Linkage: "We have always said that CVS is not secure")
Bug? No; design limitation. But if the end users aren't aware of that (or, worse, choose to disregard the danger), it's still a vulnerability waiting to be exploited, and open source does NOTHING to prevent that.
So, "easier to find holes"? I'll go with the stock CompSci answer, "It depends". It's certainly not a simple or complete answer.
Xentax
You shouldn't verb words.
Anticipating the shitstorm of people lining up to say 'how stupid' without reading the FA, here's a nice little summary.
The paper is not quite as stupid as it sounded by the description, but it misses/ignores a critical flaw in the argument.
His basic premise is that patching is expensive and people don't do it anyway - probably true for the majority of systems. Therefore, he argues, black-hats are alerted to the security holes by the disclosure. He shows that it doesn't really matter whether holes are discovered by black-hats and are fixes are released after the exploit, or discovered by white-hats and exploited after the fix has been released but not applied.
However, his arguments are based on averages. Where he's wrong is that if you have some systems that are simply so valuable that they cannot be comprimised, proactive bug fixing coupled with a manic obsession for patching your system the moment a patch is released is still the best way to stay safe
foo mane padme hum
If reports aren't and patches aren't made the holes will be found and will be reported on in the black-hatter community.
Report it to the developer, not the whole world.
If reports aren't made but patches are, some (the smart) people will not install patches without knowing exactly what they are installing (especiallly important for Windows users).
You're still installing binary code that you know little about. Whether you have a code sample for the exploit, or you just know that there was an exploit in XXX service through which an attacker could get administrative access, I don't see any difference to the admin.
Those who decide to look after their security shouldn't be hindered.
Well, then the rest of us will just continue to suffer with a constant flow of new Code Reds and other such drivel because you security conscious people think it is better to have a public list of ways to exploit a system.
Actually, I think that reinforces my point. I spend most of my day working with security systems (see here) and so I absolutely know better than to send mail without checking the response line and yet I made that sort of boneheaded mistake anyway. This is exactly why software is riddled with bugs and why it seems unlikely we'll be able to patch them out of existence--people make mistakes.
Enter a patching service, run by say a Linux distributor. SuSE's Yast Online Update (YOU) does this very well. Patches are often zero-day, and are guaranteed by SuSE not to cause trouble with other installed software, or that dependent software is also patched. I'm sure other commercial distributors have similar services, and debian's stable branch has worked well for me as well.
Yes, it involves a certain amount of trust, but if you didn't trust anyone, you'd have to write everything yourself. Also, the company's business model depends on the reliability of said patching service, so they do their best to run it well.
Of course, license changes are evil, but they're unlikely to happen with FOSS. Yet another reason to move away from Microsoft.
Secondly, vulnerabilities will get reported anyway -- perhaps just not so openly. Script kiddies will likely still have access as well as other nasty types -- such as organized spam gangs and other groups with an interest in using compromised machines. At least with the current system there's parity of knowlege -- white hats have access to the same information as black hats, and get it in a timely manner. Supressing open reporting only tips the balance of power the wrong way.
Roving Web-Teleoperated Robot
exceedingly so, in fact. It boils down to a single sentence: /them/ that it's not worth looking for it.)
It's better to find the security hole yourself and fix it than for someone with malicious intentions to do so and exploit it.
(And good luck convincing
Work is punishment for failing to procrastinate effectively.
His basic theory is that he believes, given the current rates of discovery, poor patching rates and the large number of bugs that:
1) Due to massive information exchange and slow patching/fixing, post announcement explotitations are not significantly less than explotiation caused by un-announced bugs, so announcing does not help.
2) There are so many bugs out there that finding a bug and announcing it does not produce a singificant reduction in the number of "bugs unknown to white Hats" so it does not increase security significantly.
His major errors are ignoring the following: 1) Exploitation post announcement is almost entirely done against the "lesser" computers, i.e. either machines with un-important data/programs (home pc's) or important machines with incompetent sysadmin. As such the real cost of these explotations is either A) practically null, or b) high, but likely to get the incompetent sysadmin fired, resulting in i) better employment prospects for good sysadmin and ii)an over-all improvment in quality of security for that company. So while the # of explots may be higher for Post-announcement bugs, soceity has a REAL social and economic gain that is very significant.
2)A)The announcement of bugs allows people to judge how well written a program is and therefore make an informed decision to NOT buy inferior producs (say Windows).
B)That perhaps our problem is not that we are announcing the bugs but that we are not doing sufficient bug hunting. He seems to be implying that because bug hunts don't find enough bugs, the solution is not to bug hunt. But anyone with a brain should be able to see that if we are not depleating the unknown bugs fast enough than one possible solution is to tremoundously increase our bug hunting, not to stop the bug disclosures.
excitingthingstodo.blogspot.com
Even if only a small percentage of computer users apply security patches, there is still the benefit of building up a knowledge base about security vulnerabilities. The blackhats are building up such knowledge anyway, we can't prevent that. But the "good side" needs to also build up this knowledge, otherwise the blackhats would soon be so much more knowledgable and skilled than the whitehats that it becomes impossible to set up any machine in a reasonably secure manner.
Under construction: swpat politics overview article
One of the key factors that has kept Mainframes secure for so long is the fact they were designed as a secure environment in the first place:
The society for a thought-free internet welcomes you.
new ones just keep coming along. What is the point. We cured polio and smallpox, but now we have HIV. We should have left well enough alone.
Maybe it is just me, but I think linking bubonic plague to flea infested rats was a beneficial advancement in progressing the human situation, for multiple reasons.
This is why documents such as The Secure Programming for Linux and Unix should be compulsory reading for developers.
Time after time we see the same flaws being found, sometimes by me, sometimes by more focussed groups.
I seriously believe half the problem is the number of young developers who read manuals/textbooks/online guides which have a paragraph at the introduction saying something like "To keep the code concise we've ommitted all error checking in our examples". With nary a mention of security throughout the rest of the piece.
Half joking - half serious.
Most roads have some holes in them. Some would say it is a natural part of the road building process. Other argues roads can be made hole free. Generally roads are thought to be without holes when initially developed, but over time holes are found. While identifying and patching holes in roads is thought to be a good thing, numerical analysis shows otherwise.
Patching roads requires people to stop the flow of traffic, and puts workers at risk of being injured or killed by users of the roads. A road that is fully patched encourages users to drive faster, burning fossil fuels at a lower efficiency compared to the slow speed drivers use when they see holes. Driving slower will also save lives in the event of an accident and cause drivers to pay more attention to the road since they never know when a hole could be in their local path.
Many times the problem is not with the road, but the surface that it is built on. Patching the road can only be assumed to be a stop gap measure at best and will likely have to be patched again. Holes in the supporting structure will almost always show up in the things built on it.
Fixing pot holes slows innovation. Once it becomes accepted that roads have holes in them, consumers will demand vehicles able to deal with them. If hole patching was stopped right now, studies show we would all be flying to work in our personal jetson mobiles within 8 years. Space previously set aside for roads will be converted to trails for bikes, bladers and walkers. Butterflies will land on your outstretched hand while you stop to observe the wild flowers.
Fixing holes only maintains the status quo and dominance of local government and the corrupt DOT branches of the states. If you reduce their budget by even 1% they will go on strike and roads will quickly deteriorate. Some communities out there are leading the charge in not fixing pot holes to bring you a world of glass houses on stilts and talking dogs with jet packs.
In conclusion our findings indicate the DOT should be abolished. They have served their purpose but have no place in todays innovative world.
Good grief!
You really think the problem is C and C++?? I know that these arn't the holy grail of programming languages but to put some of that blame on them is very nieve. You can write buggy, unsecured code with asm! It's got very little to do with the language involved (the compiler and libraries used may have an effect) since it all gets put into machine code anyway. Blame the design and implementation, not the tool.
Silly rabbit
It's quite hard to compare a status quo to an invisible alternative state -- this is a huge problem in business, politics, and especially economics. But at least I've determined that simply using vulnerability metrics -- i.e. "Finding bugs does not lead to less bugs being found" -- is ultimately not a representative metric for the actual risk mitigated.
To use a straightforward analogy -- possessing an immune system does not by a significant means reduce the pathogenic population, yet lacking one is death. The case is quite similar with vulnerabilities and virii -- it would be very simple for us to completely lack the infrastructure to manage an Internet-wide vulnerability. The low grade malware floating around -- though infuriating -- forces us to create a management infrastructure, on pain of losing connectivity. What the consistent stream of discovered vulnerabilities creates is not fewer vulnerabilities -- software simply isn't mature enough, nor would we really want it to be -- but more managable failures. Put simply, it doesn't matter what this way comes, because we've already been forced to deploy backups, create procedures, and implement recovery strategies.
The alternative state is far more terrifying: Bugs are not talked about, and the strategy is not to fix them but to silence their open researchers. A black market opens up -- it will always be in the benefit of some to have the means to exploit others. These means always work, because nobody defends. Are there fewer with these means? Yes, but one person can write an attack, and the motive to blackmail the entire Internet population (pay me, and I'll "protect" you from the next wave) is quite strong.
Bottom line -- and it's something that took me some time to realize myself, being an active member of the security community who doesn't deal in exploits heavily -- is that whatever the headaches are of full disclosure, the alternative is much worse.
--Dan
Actually, I think that reinforces my point.
I get what you're saying, and you're correct as far as that goes, but I was not concerned that you failed to wipe the CC list.
I was refering to sending your credit card number to someone via electronic mail. Even if I was sure that TLS would occur between my MTA and theirs (ignoring the chance that a third-party secondary MX would get involved) and that they and I were using SSL-enabled IMAP to fetch our mail... I would still hesitate long enough to make it worth my while to just find their phone number and phone in the CC#.
That you claim to be a security expert and yet seemingly advocate against looking for exploits AND send your credit card number out via email... well, I worry is all.
There is an assumption that appears to be raison d'etre for this article, and I think that it is a flawed one. Patching security holes in software is not done for the welfare of its users -- it merely keeps the software alive so that we can continue to use it (and it is a losing battle). Most security problems are remedied with quick patches rather than the rearchitecture necessary to make certain kinds of flaws impossible instead of improbable. This statement should not be interpreted as an attack on the maintainers of software, rather, I would prefer that it be viewed as a lament over the software creator-user dynamic that exists today. Users are riding leaky boats at sea -- they can't very well leap into the water while we make proper repairs, so we merely patch the holes. The sad fact is that, eventually, the boat sinks into the waves. Until user expectations change, software creators will use the practices that encourage these sorts of defects. Assurance of quality begins with the management of expectations.
Can you prove a negative? That's whay this guy is asking in a way. The real question is "What is the cost of NOT finding security holes?" Lots of evidence for that!