Slashdot Mirror
Stop Fixing All Security Vulnerabilities, Say B-Sides Security Presenters
PMcGovern writes "At BSidesLV in Las Vegas, Ed Bellis and Data Scientist Michael Roytman gave a talk explaining how security vulnerability statistics should be done: 'Don't fix all security issues. Fix the security issues that matter, based on statistical relevance.' They looked at 23,000,000 live vulnerabilities across 1,000,000 real assets, which belonged to 9,500 clients, to explain their thesis."
Prioritize on the important vulnerabilities. But that should in no way discourage people from fixing the less important ones.
Don't let perfect become the enemy of good.
In SOVIET RUSSIA... erm...NSA AMERICA, the Internet logs onto YOU!
If they received government funding to do this research, then yes we do.
Important items get fixed first. Easy items usually come next. Everything else gets fixed after that.
Keep the Classic Slashdot.
The article is talking about fixing what you can. It simply outlines how to prioritize the issues in order to figure out what you can fix with limited resources.
Their real point is, if you have limited resources, prioritize the vulnerabilities that are (a) currently being exploited and (b) most likely to be exploited given the habits of your favourite boogeyman. Sometimes that means not starting on vulnerabilities as soon as they come in, because you're saving your resources for the chance there's a bigger problem later. Their thesis is about saving your money and time for the most important stuff, and assumes that threats only come from lazy blackhats who prefer certain classes/types of vulnerabilities. Buried in this is the assumption that a given piece of software has an infinite number of vulnerabilities that are discovered at random.
Statistically, what they're saying is sound if organized crime is your biggest enemy, assuming organized crime's habits don't change any time soon. It's obviously not good enough if you're concerned about, say, a malicious government organization with an absurd budget.
Bio questions? Ask me to start a Q&A journal. Computer analogies available for most topics!
They say stop when they mean prioritize. Theoretically, there should be some computer scientists who know how to use English.
How does software like djbdns seem to be nearly free of discovered vulnerabilities? Is it a popularity/type-of-user thing? Or has the code genuinely been written to be almost impenetrable?
tl;dr Why do so many things need fixing in popular pieces of software which could easily command the most competent developers?
The article is talking about fixing what you can. It simply outlines how to prioritize the issues in order to figure out what you can fix with limited resources.
That's a pretty damn weak model. It doesn't take a genius to understand that if you use statistics to prioritize security issues to address (or more to your point, cull out ones you won't address due to limited resources), then it's only a matter of time before attackers start figuring out ways to use those statistical models against you, ultimately learning about the "can't-get-to-it" threat list and focusing attack vectors there.
Not to mention management being "sold" on this model and cutting 20% of your IT support staff next year due to the "increased efficiencies of patch management". Have fun doing more work.
I believe the word is 'triage'..
“He’s not deformed, he’s just drunk!”
The author is assuming that the opposition is dumb. It used to be, back when it was a kid in their parents' basement. Now the serious opposition is the Russian Business Network and the People's Liberation Army.
Detected breaches tend to come from the dumb opposition. Those are the ones that put fake login sites on Wordpress blogs.
I have no intrinsic problems with what they say, but a lot of this prioritizing is reactionary guesswork based on past experience.
Where that can give problems is that they don't look at it from a logical perspective, rather they try to package it as a simple calculation, with statistics and all. The fact is that these stats could be off by orders of magnitude, they are based on real data, but you have no idea how this data really applies to you. It may just as well prepare you for the previous war, instead of the one you are about to face.
This is what you commonly get in many risk assessments. A common determination of risk is 'risk = chance of occurance * potential damage'. Both of these figures are usually guesswork at best, complete BS usually, multiplying them just makes that worse. You end up with nearly random top lists of dangers.
This is the reason why companies like Diginotar passed audits by large accountancies, They made their paper models based on having rules and calculations about everything, but nobody bothered to apply some common sense and see if their vulnerability really wasn't that bad.
The proposal here is more of the same, lets apply some artificial rules so we can pretend it is a science, and never have to apply common sense to see if something is a problem. This is a recipe for disaster.
How about you fix what you can?
That's the fly-swatter approach - you hit the flies you can and ignore those you can't get to.
'Don't fix all security issues. Fix the security issues that matter, based on statistical relevance.'
That line reminds me of the old TQM which was run past us decades ago (and then promptly forgotten by 90% of the Franklin Planner-toting crowd), fix what really needs fixing first. I'm sure this bit of wisdom didn't require TQM to come along (you can probably find it in Hamlet if you know where to look), you fix your most grievous would first and worry about your bruises later, but we (in my department) felt rather put-upon when these TQM zombies came around and told us what a sea-change it would be for our practices and productivity when we embraced what we already knew.
A feeling of having made the same mistake before: Deja Foobar
OpenBSD takes the approach of proactive code audits and of fixing all bugs found, even those that have no apparent potential for exploitation. This has really paid off over the years. Often when vulnerabilities came to light, they were found to not affect OpenBSD because the underlying bug had already been fixed.
Disclaimer I have not read the paper.
Once upon a time I did software documentation for a fast moving product. I was never given updates and worked basically in the dark. One brilliant manager asked me to, "Document all the bug fixes for this product." There were over 2,000. At 15 minutes each that time span was a bit over the week I was given. Doing the math, this comes out to 12.5 40 hours weeks, uninterrupted. At half time -- a better estimate this would have been half a year. One week is not 25.
I requested a list of the bugs, sorted by priority. I was met with stares. I then said, "Until I get the list, I will work in strict numerical order until I get the list." The manager screamed at me, "But I don't want that." This time I replied, "I agree, but until I get the list from you I will do the work in numerical order, just so you don't yell that am not working." I never got the list and the random selection from the strict list was a nice demo of the types of bugs found. The end result was OK, but not by choice. Introduction done, but this was a similar problem. You need some guidance 'cause you cannot do everything.
With limited resources, fixing everything all the time is an infeasible task. Using a pure visual analogy, "fix the biggest hole." The problem is that as bad as people are at fixing, they are perhaps even worse at classifying. And assessing the potential damage of a "hole" is another part of the problem. You must also assess the likelihood of someone finding/using that hole. Add in the reality that each fix is time-sensitive and the time to fix a bug is all over the place, and you have a very real mathematical and practical mess. "What do we fix first?" is neither a short or an easy question to answer.
If you line up all of your straw men in a row, they will look like an army and scare your opponent away.
This model is already largely in place. Companies will focus on patching the vulnerabilities that are already being exploited in the wild. Then after that they will focus on some amalgamation of lowest hanging fruit and most likely to be exploited.
Check out my lame java blog at www.javachopshop.com
Theoretically, there should be some computer scientists who know how to use English.
Theory and reality are the same, in theory. In reality, however...
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
Corporations focus the most attention on the greatest number of the most trivial problems because more is always better when it comes to management metrics. Whereas the actual problems are turned into projects that linger for lack of funding and political turf. See no one wants a problem they have spend money on and then explain. Better to foist it off on someone else. Whereas getting funding to paint all the switchplates green is a slam dunk because it's easy to do easy to measure, gains turf and has no downside.
because some things just aren't worth fixing, even if you can.
it's only a matter of time before attackers start figuring out ways to use those statistical models
They already do. They use attacks that hit the largest number of targets. Using uncommon vulnerabilities would be wasteful when you could attack more common ones.
That's why you need a game-theoretic, adversarial model instead of a simple statistical model based on past observations. Regret minimization, multi-arm bandits, etc.
weinersmith
A cybernetician knows that everything flows. We know that after the Sensing, and Decision leads to Action, the whole process will happen again beginning with the Sensing. You will sense the environment change and thus change the actions; However, sometimes the correct decision comes a bit too late...
If we were to Sense the statistical prevalence of exploits, then decide which bugs to fix only based on it, then act to fix those bugs: What do you think we would sense afterwards? The obvious conclusion would be that bugs which are widely exploited will continue to be fixed fastest. Do you SEE what's wrong with that?!
OK, sorry, I forgot I'm dealing with bloody Humans, ugh... So, if environmental selection says: X is bad, what happens to X in the population over time? Why, X is bred out of the population. So, do you see a problem with breeding an entire culture of malware which does not use exploits in a manner that is statistically prevalent WHILE ALSO providing advantage for infections to better survive on bugs that are NOT statistically prevalent? NO?! Well, if you can't see the issue then screw it, there's no point in continuing the explanation from here.
It's like you only ever consider your current and next iteration, and not the end results of Anything you do. I don't mean to be a racist, but times like this it seems that even the most highly trained organic minds ARE hampered by all the dumb fat in their heads.
"First you go through all the bugs we know--then you work on the bugs we don't know."
You should read Mark Twain's The McWilliamses and The Burglar Alarm. Your suggestion is peddling an overly complex burglar alarm that will take more time and effort and resources than just fixing the bugs as they come in.
No, you don't. You just need a paragraph in your statement of work specifying that defects are prioritized by severity according to their impact of the operation of the system. You can describe how you assign priorities, and then you're done. Government contract disclosure requirements are met, and you didn't have to contract an armchair pontificator (ie, most PhDs) to write a thesis for you. Of course, since you can bill all of those hours back to the client, it's really profitable to charge extra for an overhead task that does absolutely nothing to move a project forward.
No, GP is right. It's a different scenario, but it's valid:
If 1 in a thousand users installs X, find a way to target X across a corporation. One hit gets you in. Beachhead there, figure out where to go next or what you can collect.
**THAT** is how to discreetly pwn a corporate net.
Some attackers go big, because their payout is # of machines taken. Some attackers are after a narrow niche: what'll company X be announcing, how their stock is likely to perform, data of value to competitors, etc. Their payout is the same if they own one or a thousand machines in a corporation.
Yes the article is wrong. It assumes that software vendors would leave security vulnerabilities with entries in Metasploit unfixed for days or even weeks. Surely no vendor would be that irresponsible. Right? Right???
Finally! A year of moderation! Ready for 2019?
Feynman went on to say something disparaging about religion:
It doesn't seem to me that this fantastically marvelous universe, this tremendous range of time and space and different kinds of animals, and all the different planets, and all these atoms with all their motions, and so on, all this complicated thing can merely be a stage so that God can watch human beings struggle for good and evil — which is the view that religion has. The stage is too big for the drama.
The problem here is, Feynman was thinking too small. Maybe the universe is a facility in which a deity can watch the evolution of a trillion different intelligent species play out. In that case, the stage is just right for the drama.
That that is is that that that that is not is not.
Fixing all bugs is great if you have the resources for it. But how many organizations have those kind of resources? I suspect even OpenBSD does not.
And are you saying that OpenBSD performs no prioritization whatsoever on their bugfix efforts -- that everything is done in a first-in-first-out order?
That that is is that that that that is not is not.
Sorting bugs into "security vulnerabilities" and "other" is prioritizing.
Security people talk as if the start point is security vulnerabilities. It's not. From a functional view, there's not much difference between a bug that breaks crucial functionality, and a DoS attack.
It's amazing how many security vulnerabilities rely on age old bugs such as buffer overruns and dirty memory that are easily fixed if we're willing to live with slightly slower computers. We can program the OS to blank memory whenever it is allocated and for extra safety, whenever it is freed. We can add bounds checking to library routines such as C's infamous gets(). But all that hurts performance. It's a tradeoff we've been unwilling to make. How many people run SELinux? For most, the time it takes to learn and administer an SELinux system is just not worth the scanty benefits. For instance, SELinux does very little to protect you from NSA snooping. Another approach is the microkernel. Why haven't we pursued microkernels? A microkernel architecture might reap ten times the security benefits we could ever hope to net by constantly patching drivers. Gets rid of a whole class of vulnerabilities involving the escalation of flaws in drivers. What we can do cheaply is punt illegal access problems to the OS and hardware, leaning on virtual memory paging to protect programs from each other and No Execute bits to protect a program's code from itself.
The entire approach of zeroing in on vulnerabilities is fire fighting. And saying that we should choose which fires to fight is not particularly insightful or helpful. Better to build systems that don't catch on fire so easily.
Intellectual Property is a monopolistic, selfish, and defective concept. It is "tyranny over the mind of man"
The author is assuming that the opposition is dumb. It used to be, back when it was a kid in their parents' basement.
Actually, the "kid in the basement" usually wasn't dumb. Typically they'd be far above the average for their school.
Callow, yes.
Further, they had an advantage over the professionals: They could spend a LOT of time, in long, unbroken, sessions, pursuing a problem of their choosing down to the nitty-gritty-bits, until it fell before their persistence. Not having to earn a living, meet a schedule, build something they're not interested in to support a company's work (and do security on the side), commute to a work place (and having the tools available 24/7), having food, housing, and what-have-you provided by the parents, and (during the school vacation) having no distractions whatsoever, let them learn more, faster, and try more things.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Has anyone modelled the potential impact of XP end of support over time?
This is common knowledge already. A vulnerability is not the same as a risk. A risk is the impact of the vulnerability, multiplied by the damage you'll sustain if it is exploited. chance*damage=risk.
You could very well have a vulnerability that is real, that will be exploited, but will not lead to any damage. Since there is no economical viability to fix it, you leave it be. You could have a vulnerability that will be very unlikely to be exploited. However, if it should be exploited, your business will instantly go bankrupt. This vulnerability, is way more important to fix for you than the first example, even though the chance that something will happen is very small.
Prioritize on risk, not on just chance or on what direct gains an attacker should get from exploitation. Risk is unique for your organization, so make sure you have all possible scenarios worked out and a risk matrix available when it's time to assess the impact a vulnerability will have on your organization.
I was promised a flying car. Where is my flying car?