Slashdot Mirror
Security Metrics
Ben Rothke writes "The goal of security metrics is to replace fear, uncertainty, and doubt
(FUD) with a more formalized and meaningful system of measurement. The FUD factor is the very foundation upon
which much of information security is built, and the outcome is decades of
meaningless statistics and racks of snake oil products. Let's hope that Andrew Jaquith succeeds, but
in doing so, he is getting in the way of many security hardware and software
vendors whose revenue streams are built on FUD." Read below for the rest of Ben's review.
Security Metrics: Replacing Fear, Uncertainty, and Doubt
author
Andrew Jaquith
pages
336
publisher
Addison-Wesley
rating
10
reviewer
Ben Rothke
ISBN
0321349989
summary
Authoritative text on information security metrics
One could write a book on how FUD sells security products. One of the most memorable incidents was in 1992 when John McAfee created widespread panic about the impending Michelangelo virus. The media was all over him as he was selling solutions for the five million PCs worldwide he said would be affected. The end result is that the Michelangelo virus was a non-event. Nonetheless, it was far from the last time that FUD was used to sell security.
The allure of FUD is that companies can spend huge amounts of money fighting nebulous digital adversaries and feel good about it. They can then put all of that fancy hardware in dedicated racks in their data center, impressing the auditors with the flashing lights giving off an aroma of security and compliance.
And that is the chaos that security metrics comes to solve. Security metrics, if done right, can help transform a company from a nebulous perspective on security to an effective one based on formal security risk metrics.
Security Metrics is a fabulous book that should be in the hands of every security professional. The book demonstrates that companies must establish metrics based on their unique requirements, as opposed to simply basing their requirements on imprecise industry polls, best-practices and other ill-defined methods.
So why don't companies do that in the first place? If security metrics can provide even a quarter of the benefits that Jaquith states, companies should run to implement them. Real security metrics require an organization to open up their security hood and dig deep into the engine that runs their security infrastructure. It necessitates understanding the internal requirements, unique organizational risks, myriad strengths and weaknesses, and much more. Very few companies are willing to dedicate the time and resources for that, and would rather build their security infrastructure on thick layers of FUD. History has shown that the security appliance of the month almost always beats a formal risk and needs assessment.
Chapter 1 lays out the problem with approaches that most companies take to risk management. The main problem is that traditional risk management is far too dependant on identification and fixing, as opposed to quantification and triage based on value. Quantifying and valuing risk is much more difficult than simply identifying, since the software tools used do not have an organization context or knowledge of the specific business domain.
Chapter 2 sets out the foundation of security metrics. The goal of these metrics are to provide a framework in which organizations can quantify the likelihood of danger, estimate the extent of possible damage, understand the performance of their security organizations and weigh the costs of security safeguards against their expected effectiveness.
The time has come for security metrics since information security is one of the few management disciplines that have yet to submit itself to serious analytical scrutiny. The various chapters provide many different metrics that can be immediately used in most organizations to address that.
The author defines various criteria for what makes a good metric. One of his pet peeves is the use of the traffic light as a metaphor for compliance. Jaquith feels that traffic lights are not metrics at all, since they don't contain a unit of measure or are a numerical scale. He suggests using traffic lights colors sparingly, and only to supplement numerical data or draw attention to outliers. He astutely notes that if your data contains more precision than three simple gradations, why dilute their value by obscuring them with a traffic light.
The chapter concludes on what makes a bad metric, defined as any metric that relies too much on the judgment of a person. These metrics can't be relied on since the results can't be guaranteed to be the same from person to person. Also, security frameworks such as ISO-17799 should not be used for metrics. The book also tackles the sacred cow of risk management, namely ALE (annualized loss expectancy), and how it is significantly misused and misunderstood in the industry.
The book states that in developing metrics, there must be formal collaboration between the business units and the security staff. This collaboration serves to increase awareness and acceptance of security. In addition, it ensures that security requirements are incorporated into the lifecycle early on. This is needed as business units generally have no clue as to what the needed security requirements are.
Chapter 5 is a short course on analysis techniques and statistics. The author quotes George Colony who stated that "any idiot can tell you what something is. It is much harder to say what that thing means". With that, the book details a number of techniques for analyzing security data (average, median, time series, etc.) and how each one should be used.
Chapter 6 is about visualization and notes that most information security professionals have no real idea how to show security, both literally and figuratively. Part of the problem is that security is proliferated with esoteric terminology and concepts, and the lack of understanding risk management amongst the masses. Part of the reason for this difficulty in sharing the security message with management is that many security practitioners lack simple metaphors for communicating priorities. This is compounded by the fact that the message is often focused exclusively on technical security issues, as opposed to the underlying business issues, which is was management is concerned with. The chapter is invaluable as it weans one off the malevolent pie chart and traffic light PowerPoint presentation.
Marcus Ranum notes that people seem to want to treat computer security like its rocket science or black magic. In fact, computer security is nothing but attention to detail and good design. FUD is all about emphasizing the black magic aspect of hackers and other rogue threats. Metrics are all about the attention to detail that FUD lives to obfuscate.
Security Metrics: Replacing Fear, Uncertainty, and Doubt is one of the more important security books of the last few years. Jaquith turns much of the common security wisdom on its head, and the world will be a better place for it. Security metrics are a necessity whose time has come and this invaluable book shows how it can be done.
Ben Rothke, CISSP is a security consultant with BT INS and the author of Computer Security: 20 Things Every Employee Should Know.
You can purchase Security Metrics: Replacing Fear, Uncertainty, and Doubt from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
One could write a book on how FUD sells security products. One of the most memorable incidents was in 1992 when John McAfee created widespread panic about the impending Michelangelo virus. The media was all over him as he was selling solutions for the five million PCs worldwide he said would be affected. The end result is that the Michelangelo virus was a non-event. Nonetheless, it was far from the last time that FUD was used to sell security.
The allure of FUD is that companies can spend huge amounts of money fighting nebulous digital adversaries and feel good about it. They can then put all of that fancy hardware in dedicated racks in their data center, impressing the auditors with the flashing lights giving off an aroma of security and compliance.
And that is the chaos that security metrics comes to solve. Security metrics, if done right, can help transform a company from a nebulous perspective on security to an effective one based on formal security risk metrics.
Security Metrics is a fabulous book that should be in the hands of every security professional. The book demonstrates that companies must establish metrics based on their unique requirements, as opposed to simply basing their requirements on imprecise industry polls, best-practices and other ill-defined methods.
So why don't companies do that in the first place? If security metrics can provide even a quarter of the benefits that Jaquith states, companies should run to implement them. Real security metrics require an organization to open up their security hood and dig deep into the engine that runs their security infrastructure. It necessitates understanding the internal requirements, unique organizational risks, myriad strengths and weaknesses, and much more. Very few companies are willing to dedicate the time and resources for that, and would rather build their security infrastructure on thick layers of FUD. History has shown that the security appliance of the month almost always beats a formal risk and needs assessment.
Chapter 1 lays out the problem with approaches that most companies take to risk management. The main problem is that traditional risk management is far too dependant on identification and fixing, as opposed to quantification and triage based on value. Quantifying and valuing risk is much more difficult than simply identifying, since the software tools used do not have an organization context or knowledge of the specific business domain.
Chapter 2 sets out the foundation of security metrics. The goal of these metrics are to provide a framework in which organizations can quantify the likelihood of danger, estimate the extent of possible damage, understand the performance of their security organizations and weigh the costs of security safeguards against their expected effectiveness.
The time has come for security metrics since information security is one of the few management disciplines that have yet to submit itself to serious analytical scrutiny. The various chapters provide many different metrics that can be immediately used in most organizations to address that.
The author defines various criteria for what makes a good metric. One of his pet peeves is the use of the traffic light as a metaphor for compliance. Jaquith feels that traffic lights are not metrics at all, since they don't contain a unit of measure or are a numerical scale. He suggests using traffic lights colors sparingly, and only to supplement numerical data or draw attention to outliers. He astutely notes that if your data contains more precision than three simple gradations, why dilute their value by obscuring them with a traffic light.
The chapter concludes on what makes a bad metric, defined as any metric that relies too much on the judgment of a person. These metrics can't be relied on since the results can't be guaranteed to be the same from person to person. Also, security frameworks such as ISO-17799 should not be used for metrics. The book also tackles the sacred cow of risk management, namely ALE (annualized loss expectancy), and how it is significantly misused and misunderstood in the industry.
The book states that in developing metrics, there must be formal collaboration between the business units and the security staff. This collaboration serves to increase awareness and acceptance of security. In addition, it ensures that security requirements are incorporated into the lifecycle early on. This is needed as business units generally have no clue as to what the needed security requirements are.
Chapter 5 is a short course on analysis techniques and statistics. The author quotes George Colony who stated that "any idiot can tell you what something is. It is much harder to say what that thing means". With that, the book details a number of techniques for analyzing security data (average, median, time series, etc.) and how each one should be used.
Chapter 6 is about visualization and notes that most information security professionals have no real idea how to show security, both literally and figuratively. Part of the problem is that security is proliferated with esoteric terminology and concepts, and the lack of understanding risk management amongst the masses. Part of the reason for this difficulty in sharing the security message with management is that many security practitioners lack simple metaphors for communicating priorities. This is compounded by the fact that the message is often focused exclusively on technical security issues, as opposed to the underlying business issues, which is was management is concerned with. The chapter is invaluable as it weans one off the malevolent pie chart and traffic light PowerPoint presentation.
Marcus Ranum notes that people seem to want to treat computer security like its rocket science or black magic. In fact, computer security is nothing but attention to detail and good design. FUD is all about emphasizing the black magic aspect of hackers and other rogue threats. Metrics are all about the attention to detail that FUD lives to obfuscate.
Security Metrics: Replacing Fear, Uncertainty, and Doubt is one of the more important security books of the last few years. Jaquith turns much of the common security wisdom on its head, and the world will be a better place for it. Security metrics are a necessity whose time has come and this invaluable book shows how it can be done.
Ben Rothke, CISSP is a security consultant with BT INS and the author of Computer Security: 20 Things Every Employee Should Know.
You can purchase Security Metrics: Replacing Fear, Uncertainty, and Doubt from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Guaranteed to protect your investments from intruders no ifs ands or butts
Infiltrated dot Net
Marge, change the channel.
ISO 27004 is supposed to deal with security metrics/management measurement when it is published. It belongs to the same familiy of security standards as ISO 17799 and 27001.
l
/ sp800-55.pdf
About ISO 27004: "The scope is to "provide guidance on the specification and use of measurement techniques for providing assurance as regards the effectiveness of information security management systems. It is intended to be applicable to a wide range of organisations with a correspondingly wide range of information security management systems. [It] provides guidance for measurement procedures and techniques to determine the effectiveness of information security controls and information security processes applied in an ISMS. The purpose of the Information security management measurements development and implementation process, defined in this Standard is to create a base for each organization to collect, analyse, and communicate data related to ISMS processes. This data is ultimately to be used to base ISMS-related decisions and to improve implementation of an ISMS."
Also, NIST has also something to add to the issue of security metrics in SP800-55. A few links:
http://www.iso27001security.com/html/iso27004.htm
http://csrc.nist.gov/publications/nistpubs/800-55
'Defined by Gene Amdahl after he left IBM to found his own company: "FUD is the fear, uncertainty, and doubt that IBM sales people instill in the minds of potential customers who might be considering [Amdahl] products." The idea, of course, was to persuade them to go with safe IBM gear rather than with competitors' equipment. This implicit coercion was traditionally accomplished by promising that Good Things would happen to people who stuck with IBM, but Dark Shadows loomed over the future of competitors' equipment or software. See IBM. After 1990 the term FUD was associated increasingly frequently with Microsoft, and has become generalized to refer to any kind of disinformation used as a competitive weapon.' From the Jargon file at http://www.catb.org/jargon/html/F/FUD.html
The world has changed and we all have become metal men.
Be vewwwy vewwwy quiet, I'm spweading fud.
Kwisatz Haderach
Sell the spice to CHOAM
This Mahdi took Shaddam's Throne
This is just a process in order to 'quanitify' the FUD, in terms of numbers people can better understand.
It's essentially saying, project X is 10% more secure than project Y.
What are the metrics?, what is the basis used for comparison?
Project X has 10% less bugs, but of what type?
What is the weight of a bug?
Do 'root access' bugs have a higher weight?
What's the scale?
Anyone who has ever done ANY benchmarking, etc, knows all about the ratio games that can be played in this field.
It's the same FUD, repackaged with numbers.
Amdahl, the man who dared to take on IBM. His story is interesting.
He worked for IBM as a hardware designer, but spotted a way of improving the performance of the flagship System/360 CPU, by moving from microcode to hardwired control. When he formed his own business to manufacture S/360 compatible CPUs, which were faster, smaller and lower cost, IBM moved to neutralise the threat by spreading FUD. Specifically, they announced extensions to the S/360 instruction set, which were to be used by all new software, but would not be announced until the release of the next S/360 CPU (the IBM 3033). Amdahl's machines would not support these new instructions, so anyone who chose to buy Amdahl rather than IBM was risking his job. What if critical business software did not work on the Amdahl CPU?
Amdahl's business struggled against this FUD. Amdahl could not support the new instructions until he (a) knew what they did, and (b) could manufacture new CPUs. This would take several years, and drove away customers. This was very unfortunate, since his technology remained superior to S/360 for several years. But IBM was the Microsoft of the 1970s, and mercilessly crushing the competition was just a part of good business. References.
The only thing that confuses me about this story is a simple question: why didn't Amdahl give his CPU an extensible instruction set? Unknown opcodes like the IBM 3033 extensions could trigger the execution of emulation software in EPROM, which would provide support for the new instructions. Amdahl could just send out new firmware ROMs to customers that needed them to enable support. It wouldn't be as fast as native support in microcode, but it would work.
I didn't see anything in the note about Risk Assessment Values from isecom.org, which is the foremost methodology for determining risk based on factual evidence of an infrastructure, as well as the other elements of one's security presence (physical, social, etc.). SecurityNOW! from CIOview (www.cioview.com) integrates this idea into a powerful tool in which one can assess IT investments based expected loss, and measure ROI.
Of course, I'm biased being a volunteer at ISECOM, but I still think these are important to bring up in any Security Metrics conversation.
"One of the most memorable incidents was in 1992 when John McAfee created widespread panic about the impending Michelangelo virus. The media was all over him as he was selling solutions for the five million PCs worldwide he said would be affected. The end result is that the Michelangelo virus was a non-event." That statement is in and of itself FUD. Everyone talks about how events like Y2K, the aforementioned Michelangelo, etc. were all hype. They claim that since nothing happened that the entire issue was a farce. Do these people stop to think that it became a non-issue because of all the "FUD" and measures taken to prevent it? What would have happened if we would have decided not to do anything about Y2K since the whole problem was perceived and not proven? We very well could have ended up with the (some of) disasters that the fear-mongers were spreading. The same thing with the Michelangelo virus. They say that John McAfee just used the virus to sell products. Well, if everyone looked at it that way and decided it wasn't worth doing anything about then there would have been a lot of lost data on Michelangelo's birthday. I'm not justifying the use of FUD for profit, especially that made by snake oil. I'm just saying that claiming something is a non-issue because people took the necessary steps to prevent it doesn't mean that there was not a legitimate concern.
Could Security Metrics Destroy the Internet?!
Posted by FUD on Thursday, May 17, @07:12AM
Please stop stalking me, bro.
The ISECOM's RAV is currently a 3 page PDF file, nicely formatted and impressive looking with complicated looking formulas that would surely snow a typical manager with too many problems and not enough people or time. Unfortunately, the actual information content in the PDF is close to zero. ISECOM's RAV appears to be a formula based on variables that are not well defined, or which could be "quantified" in basically any way required to improve one's chances of securing a bid or justifying any project, and doesn't appear to offer much guidance for how to approach risk assessment in a more objective manner.
This is the same type of formula that some consulting organizations use to "quantify" things that really are not easily quantifiable. Their purpose in doing so is typically to shut down dissenting voices within a customer's organization (they gamble that the dissenting voices are not given the same budget with which to demonstrate their counter claims, and they are usually right.)
My criticism falls into a category known as the "Barry Commoner" criticism of quantitative Risk Assessment at the Wikipedia page.
If you mod me down, I shall become more powerful than you could possibly imagine.
Be far the best entertainment in this book is his explanation of the Hamster Wheels of Pain.a ge=Welcome_blogentry_040505_1a ge=Welcome_blogentry_061005_1
p enetration.htmld itorials/point-counterpoint/pentesting.html
http://www.securitymetrics.org/content/Wiki.jsp?p
http://www.securitymetrics.org/content/Wiki.jsp?p
It fits right into the same problem pointed out by Bruce Schneier and Marcus Ranum
when it comes to Pen-Testing:
http://www.schneier.com/blog/archives/2007/05/is_
http://www.ranum.com/security/computer_security/e
Hi everybody,
First off, Ben -- thanks for the kind words. Glad to hear that you liked the book, and of course I am happy to see it slashdotted. :)
Second -- some of the comments have really focused on what the book ISN'T about -- fear, uncertainty, and doubt. Ben's done a good job talking about what the book IS about, but as the author I thought I'd add a quick comment about the goals of the book. Adapted from chapter one:
Goals of Security Metrics
The primary objective of this book is to quantitatively analyze digital security activities. The book suggest ways for using numbers to illuminate the security activities of an organization:
The need for a book like this seems plain to me. Security is one of the few areas of management that does not possess a well-understood canon of techniques for measurement. In logistics, for example, metrics like "freight cost per mile" and "inventory warehouse turns" help operators understand how efficiently trucking fleets and warehouses run. In finance, "value at risk" techniques calculate the amount of money a firm could lose on a given day based on historical pricing volatilities. By contrast, in security... there is exactly nothing. No consensus on key indicators for security exists.
The lack of consensus on security metrics is, in part, due to the fact that the culture surrounding security is largely one of shame. Firms that get hacked tend not to talk about security incidents in public. Likewise, firms that are doing the right things tend not to talk either, lest giant red bulls'-eyes appear on their firewall's flanks. When they do talk, it is typically under NDA, or at small gatherings of like-minded people. Therefore, the book, as a secondary objective, documents effective practices of firms that have take the responsibility of measuring their security activities seriously.
Non-goals of Security Metrics
The dearth of generally accepted security metrics often means that unscrupulous vendors manufacture blood-curdling statistics in a vacuum, devoid of context and designed to scare. Middle managers with agendas promptly recycle these metrics for their own purposes. Therefore, the book is not about:
Of course, because every good deed goes unpunished, it is entirely likely that this book will be used for those purposes regardless. But that, as a student of security analysis might say, is a risk worth bearing.
Audience
I wrote this book for two distinct audiences: security practitioners, and the bosses they report to. Practitioners need to know how, what, and when to measure. Their bosses need to know what to expect. Not for nothing has the security domain resisted measurement. As the bedraggled security manager of a household-name financial services firm recently told me, "my boss doesn't understand what I do every day. All he understands are numbers." Bridging the yawning gap between practitioners and management i
I am 2 chapters into this book. So far it is fantastic. Most good books point out the obvious in a really good way. This was a long time coming and is very obvious, but the point is that it is told in a good way. Everyone knew this was the way things were headed but didn't know how to get the words from their mouth hold.
The "old" @stakers still gots it.
They are just spreading 'F' for Fear. After all if you run MicroSoft and then live in 'F'ear, then 'U'ndoubtedly you are very 'C'ertain they are out there trying to infect your machine and they will 'S'ucceed 'U'sually (MS-FUCSU?). Well the only thing left from the original 'FUD' is the 'F' for Fear, but after all they are 'the experts' aren't they? I really must lay off the Secure Operating Systems (SOS) at work and Start Listening to those Important Experts! (SLIE)
worth a look Software Download http://www.usdownload.net/
Protection from intruders? Sounds good. Protection from "no","ifs","ands" and "or" I do not need. Protection from butts? Hmmmm...depends on the type of butt, obviously.
All assessments of IT controls today are, and should, be based upon a risk based approach. My understanding is that this is a dialogue where the business objectives are outlined and any threat to their successful acheivement are identified and countered through the implementation of measured controls. This is a continual process and forms the backbone of security management. I read the review hopeful that it would indicate that this book offers something new and valuable in the area of quantitative assessment of security risks. "Trashing" ALE is not exactly hard, most shops I've been in don't use quant risk analysis but use qualitative risk analysis, which for better or worse let's us all formally agree and tackle what are frequently blindingly obvious threats in an efficient manner. This allows for the input of the professionals, managers, etc who have a responsibility for and clear idea of the risks to their systems, while allowing for the oversight of security / controls professionals who can winkle out the risks that the business and it's support functions may be either not willing to admit to, or simply ignorant of. My main issue with the authors post is the following quotation about Risk Management: "...simply a vendor excuse about how products help you "manage" risk by serially enumerating and eliminating defects without helping companies understand whether it makes any difference whatsoever." You can't say that. It isn't true, and the briefest of alternate suggestions would be appropriate if you're going to be 'controversial'?... Risk management should and normally is a well defined form of dialogue between the business and the controls staff, whether IT audit, or security experts. While it is true that this oftentimes uses checklists, standards, best practices and assorted generic frameworks as a means of substantiating an approach, it clearly does not preclude but actually enables meaningful debate on the issues the business faces. Having worked in IT security as a penetration tester, an IT auditer, and governance and compliance manager, I find the comment to be either misguided or purposefully disengenous about the nature of the process at work, and if you cant't get the Risk assessment side of things right (which is not really too hard) then it's time for everyone to pack up and go home. This book seems to be playing the same game the reviewer is so concerned with; the propogation of FUD. The only difference here appears to be that we aren't talking firewalls, but governance processes, and the author appears ill-equipped to comment on them...
You may not agree with what I say, but you should fight to the death to allow me to say it, by modding me up.