Slashdot Mirror
Security Metrics
Ben Rothke writes "The goal of security metrics is to replace fear, uncertainty, and doubt
(FUD) with a more formalized and meaningful system of measurement. The FUD factor is the very foundation upon
which much of information security is built, and the outcome is decades of
meaningless statistics and racks of snake oil products. Let's hope that Andrew Jaquith succeeds, but
in doing so, he is getting in the way of many security hardware and software
vendors whose revenue streams are built on FUD." Read below for the rest of Ben's review.
Security Metrics: Replacing Fear, Uncertainty, and Doubt
author
Andrew Jaquith
pages
336
publisher
Addison-Wesley
rating
10
reviewer
Ben Rothke
ISBN
0321349989
summary
Authoritative text on information security metrics
One could write a book on how FUD sells security products. One of the most memorable incidents was in 1992 when John McAfee created widespread panic about the impending Michelangelo virus. The media was all over him as he was selling solutions for the five million PCs worldwide he said would be affected. The end result is that the Michelangelo virus was a non-event. Nonetheless, it was far from the last time that FUD was used to sell security.
The allure of FUD is that companies can spend huge amounts of money fighting nebulous digital adversaries and feel good about it. They can then put all of that fancy hardware in dedicated racks in their data center, impressing the auditors with the flashing lights giving off an aroma of security and compliance.
And that is the chaos that security metrics comes to solve. Security metrics, if done right, can help transform a company from a nebulous perspective on security to an effective one based on formal security risk metrics.
Security Metrics is a fabulous book that should be in the hands of every security professional. The book demonstrates that companies must establish metrics based on their unique requirements, as opposed to simply basing their requirements on imprecise industry polls, best-practices and other ill-defined methods.
So why don't companies do that in the first place? If security metrics can provide even a quarter of the benefits that Jaquith states, companies should run to implement them. Real security metrics require an organization to open up their security hood and dig deep into the engine that runs their security infrastructure. It necessitates understanding the internal requirements, unique organizational risks, myriad strengths and weaknesses, and much more. Very few companies are willing to dedicate the time and resources for that, and would rather build their security infrastructure on thick layers of FUD. History has shown that the security appliance of the month almost always beats a formal risk and needs assessment.
Chapter 1 lays out the problem with approaches that most companies take to risk management. The main problem is that traditional risk management is far too dependant on identification and fixing, as opposed to quantification and triage based on value. Quantifying and valuing risk is much more difficult than simply identifying, since the software tools used do not have an organization context or knowledge of the specific business domain.
Chapter 2 sets out the foundation of security metrics. The goal of these metrics are to provide a framework in which organizations can quantify the likelihood of danger, estimate the extent of possible damage, understand the performance of their security organizations and weigh the costs of security safeguards against their expected effectiveness.
The time has come for security metrics since information security is one of the few management disciplines that have yet to submit itself to serious analytical scrutiny. The various chapters provide many different metrics that can be immediately used in most organizations to address that.
The author defines various criteria for what makes a good metric. One of his pet peeves is the use of the traffic light as a metaphor for compliance. Jaquith feels that traffic lights are not metrics at all, since they don't contain a unit of measure or are a numerical scale. He suggests using traffic lights colors sparingly, and only to supplement numerical data or draw attention to outliers. He astutely notes that if your data contains more precision than three simple gradations, why dilute their value by obscuring them with a traffic light.
The chapter concludes on what makes a bad metric, defined as any metric that relies too much on the judgment of a person. These metrics can't be relied on since the results can't be guaranteed to be the same from person to person. Also, security frameworks such as ISO-17799 should not be used for metrics. The book also tackles the sacred cow of risk management, namely ALE (annualized loss expectancy), and how it is significantly misused and misunderstood in the industry.
The book states that in developing metrics, there must be formal collaboration between the business units and the security staff. This collaboration serves to increase awareness and acceptance of security. In addition, it ensures that security requirements are incorporated into the lifecycle early on. This is needed as business units generally have no clue as to what the needed security requirements are.
Chapter 5 is a short course on analysis techniques and statistics. The author quotes George Colony who stated that "any idiot can tell you what something is. It is much harder to say what that thing means". With that, the book details a number of techniques for analyzing security data (average, median, time series, etc.) and how each one should be used.
Chapter 6 is about visualization and notes that most information security professionals have no real idea how to show security, both literally and figuratively. Part of the problem is that security is proliferated with esoteric terminology and concepts, and the lack of understanding risk management amongst the masses. Part of the reason for this difficulty in sharing the security message with management is that many security practitioners lack simple metaphors for communicating priorities. This is compounded by the fact that the message is often focused exclusively on technical security issues, as opposed to the underlying business issues, which is was management is concerned with. The chapter is invaluable as it weans one off the malevolent pie chart and traffic light PowerPoint presentation.
Marcus Ranum notes that people seem to want to treat computer security like its rocket science or black magic. In fact, computer security is nothing but attention to detail and good design. FUD is all about emphasizing the black magic aspect of hackers and other rogue threats. Metrics are all about the attention to detail that FUD lives to obfuscate.
Security Metrics: Replacing Fear, Uncertainty, and Doubt is one of the more important security books of the last few years. Jaquith turns much of the common security wisdom on its head, and the world will be a better place for it. Security metrics are a necessity whose time has come and this invaluable book shows how it can be done.
Ben Rothke, CISSP is a security consultant with BT INS and the author of Computer Security: 20 Things Every Employee Should Know.
You can purchase Security Metrics: Replacing Fear, Uncertainty, and Doubt from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
One could write a book on how FUD sells security products. One of the most memorable incidents was in 1992 when John McAfee created widespread panic about the impending Michelangelo virus. The media was all over him as he was selling solutions for the five million PCs worldwide he said would be affected. The end result is that the Michelangelo virus was a non-event. Nonetheless, it was far from the last time that FUD was used to sell security.
The allure of FUD is that companies can spend huge amounts of money fighting nebulous digital adversaries and feel good about it. They can then put all of that fancy hardware in dedicated racks in their data center, impressing the auditors with the flashing lights giving off an aroma of security and compliance.
And that is the chaos that security metrics comes to solve. Security metrics, if done right, can help transform a company from a nebulous perspective on security to an effective one based on formal security risk metrics.
Security Metrics is a fabulous book that should be in the hands of every security professional. The book demonstrates that companies must establish metrics based on their unique requirements, as opposed to simply basing their requirements on imprecise industry polls, best-practices and other ill-defined methods.
So why don't companies do that in the first place? If security metrics can provide even a quarter of the benefits that Jaquith states, companies should run to implement them. Real security metrics require an organization to open up their security hood and dig deep into the engine that runs their security infrastructure. It necessitates understanding the internal requirements, unique organizational risks, myriad strengths and weaknesses, and much more. Very few companies are willing to dedicate the time and resources for that, and would rather build their security infrastructure on thick layers of FUD. History has shown that the security appliance of the month almost always beats a formal risk and needs assessment.
Chapter 1 lays out the problem with approaches that most companies take to risk management. The main problem is that traditional risk management is far too dependant on identification and fixing, as opposed to quantification and triage based on value. Quantifying and valuing risk is much more difficult than simply identifying, since the software tools used do not have an organization context or knowledge of the specific business domain.
Chapter 2 sets out the foundation of security metrics. The goal of these metrics are to provide a framework in which organizations can quantify the likelihood of danger, estimate the extent of possible damage, understand the performance of their security organizations and weigh the costs of security safeguards against their expected effectiveness.
The time has come for security metrics since information security is one of the few management disciplines that have yet to submit itself to serious analytical scrutiny. The various chapters provide many different metrics that can be immediately used in most organizations to address that.
The author defines various criteria for what makes a good metric. One of his pet peeves is the use of the traffic light as a metaphor for compliance. Jaquith feels that traffic lights are not metrics at all, since they don't contain a unit of measure or are a numerical scale. He suggests using traffic lights colors sparingly, and only to supplement numerical data or draw attention to outliers. He astutely notes that if your data contains more precision than three simple gradations, why dilute their value by obscuring them with a traffic light.
The chapter concludes on what makes a bad metric, defined as any metric that relies too much on the judgment of a person. These metrics can't be relied on since the results can't be guaranteed to be the same from person to person. Also, security frameworks such as ISO-17799 should not be used for metrics. The book also tackles the sacred cow of risk management, namely ALE (annualized loss expectancy), and how it is significantly misused and misunderstood in the industry.
The book states that in developing metrics, there must be formal collaboration between the business units and the security staff. This collaboration serves to increase awareness and acceptance of security. In addition, it ensures that security requirements are incorporated into the lifecycle early on. This is needed as business units generally have no clue as to what the needed security requirements are.
Chapter 5 is a short course on analysis techniques and statistics. The author quotes George Colony who stated that "any idiot can tell you what something is. It is much harder to say what that thing means". With that, the book details a number of techniques for analyzing security data (average, median, time series, etc.) and how each one should be used.
Chapter 6 is about visualization and notes that most information security professionals have no real idea how to show security, both literally and figuratively. Part of the problem is that security is proliferated with esoteric terminology and concepts, and the lack of understanding risk management amongst the masses. Part of the reason for this difficulty in sharing the security message with management is that many security practitioners lack simple metaphors for communicating priorities. This is compounded by the fact that the message is often focused exclusively on technical security issues, as opposed to the underlying business issues, which is was management is concerned with. The chapter is invaluable as it weans one off the malevolent pie chart and traffic light PowerPoint presentation.
Marcus Ranum notes that people seem to want to treat computer security like its rocket science or black magic. In fact, computer security is nothing but attention to detail and good design. FUD is all about emphasizing the black magic aspect of hackers and other rogue threats. Metrics are all about the attention to detail that FUD lives to obfuscate.
Security Metrics: Replacing Fear, Uncertainty, and Doubt is one of the more important security books of the last few years. Jaquith turns much of the common security wisdom on its head, and the world will be a better place for it. Security metrics are a necessity whose time has come and this invaluable book shows how it can be done.
Ben Rothke, CISSP is a security consultant with BT INS and the author of Computer Security: 20 Things Every Employee Should Know.
You can purchase Security Metrics: Replacing Fear, Uncertainty, and Doubt from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
ISO 27004 is supposed to deal with security metrics/management measurement when it is published. It belongs to the same familiy of security standards as ISO 17799 and 27001.
l
/ sp800-55.pdf
About ISO 27004: "The scope is to "provide guidance on the specification and use of measurement techniques for providing assurance as regards the effectiveness of information security management systems. It is intended to be applicable to a wide range of organisations with a correspondingly wide range of information security management systems. [It] provides guidance for measurement procedures and techniques to determine the effectiveness of information security controls and information security processes applied in an ISMS. The purpose of the Information security management measurements development and implementation process, defined in this Standard is to create a base for each organization to collect, analyse, and communicate data related to ISMS processes. This data is ultimately to be used to base ISMS-related decisions and to improve implementation of an ISMS."
Also, NIST has also something to add to the issue of security metrics in SP800-55. A few links:
http://www.iso27001security.com/html/iso27004.htm
http://csrc.nist.gov/publications/nistpubs/800-55
'Defined by Gene Amdahl after he left IBM to found his own company: "FUD is the fear, uncertainty, and doubt that IBM sales people instill in the minds of potential customers who might be considering [Amdahl] products." The idea, of course, was to persuade them to go with safe IBM gear rather than with competitors' equipment. This implicit coercion was traditionally accomplished by promising that Good Things would happen to people who stuck with IBM, but Dark Shadows loomed over the future of competitors' equipment or software. See IBM. After 1990 the term FUD was associated increasingly frequently with Microsoft, and has become generalized to refer to any kind of disinformation used as a competitive weapon.' From the Jargon file at http://www.catb.org/jargon/html/F/FUD.html
The world has changed and we all have become metal men.
"One of the most memorable incidents was in 1992 when John McAfee created widespread panic about the impending Michelangelo virus. The media was all over him as he was selling solutions for the five million PCs worldwide he said would be affected. The end result is that the Michelangelo virus was a non-event." That statement is in and of itself FUD. Everyone talks about how events like Y2K, the aforementioned Michelangelo, etc. were all hype. They claim that since nothing happened that the entire issue was a farce. Do these people stop to think that it became a non-issue because of all the "FUD" and measures taken to prevent it? What would have happened if we would have decided not to do anything about Y2K since the whole problem was perceived and not proven? We very well could have ended up with the (some of) disasters that the fear-mongers were spreading. The same thing with the Michelangelo virus. They say that John McAfee just used the virus to sell products. Well, if everyone looked at it that way and decided it wasn't worth doing anything about then there would have been a lot of lost data on Michelangelo's birthday. I'm not justifying the use of FUD for profit, especially that made by snake oil. I'm just saying that claiming something is a non-issue because people took the necessary steps to prevent it doesn't mean that there was not a legitimate concern.
The ISECOM's RAV is currently a 3 page PDF file, nicely formatted and impressive looking with complicated looking formulas that would surely snow a typical manager with too many problems and not enough people or time. Unfortunately, the actual information content in the PDF is close to zero. ISECOM's RAV appears to be a formula based on variables that are not well defined, or which could be "quantified" in basically any way required to improve one's chances of securing a bid or justifying any project, and doesn't appear to offer much guidance for how to approach risk assessment in a more objective manner.
This is the same type of formula that some consulting organizations use to "quantify" things that really are not easily quantifiable. Their purpose in doing so is typically to shut down dissenting voices within a customer's organization (they gamble that the dissenting voices are not given the same budget with which to demonstrate their counter claims, and they are usually right.)
My criticism falls into a category known as the "Barry Commoner" criticism of quantitative Risk Assessment at the Wikipedia page.
If you mod me down, I shall become more powerful than you could possibly imagine.
Hi everybody,
First off, Ben -- thanks for the kind words. Glad to hear that you liked the book, and of course I am happy to see it slashdotted. :)
Second -- some of the comments have really focused on what the book ISN'T about -- fear, uncertainty, and doubt. Ben's done a good job talking about what the book IS about, but as the author I thought I'd add a quick comment about the goals of the book. Adapted from chapter one:
Goals of Security Metrics
The primary objective of this book is to quantitatively analyze digital security activities. The book suggest ways for using numbers to illuminate the security activities of an organization:
The need for a book like this seems plain to me. Security is one of the few areas of management that does not possess a well-understood canon of techniques for measurement. In logistics, for example, metrics like "freight cost per mile" and "inventory warehouse turns" help operators understand how efficiently trucking fleets and warehouses run. In finance, "value at risk" techniques calculate the amount of money a firm could lose on a given day based on historical pricing volatilities. By contrast, in security... there is exactly nothing. No consensus on key indicators for security exists.
The lack of consensus on security metrics is, in part, due to the fact that the culture surrounding security is largely one of shame. Firms that get hacked tend not to talk about security incidents in public. Likewise, firms that are doing the right things tend not to talk either, lest giant red bulls'-eyes appear on their firewall's flanks. When they do talk, it is typically under NDA, or at small gatherings of like-minded people. Therefore, the book, as a secondary objective, documents effective practices of firms that have take the responsibility of measuring their security activities seriously.
Non-goals of Security Metrics
The dearth of generally accepted security metrics often means that unscrupulous vendors manufacture blood-curdling statistics in a vacuum, devoid of context and designed to scare. Middle managers with agendas promptly recycle these metrics for their own purposes. Therefore, the book is not about:
Of course, because every good deed goes unpunished, it is entirely likely that this book will be used for those purposes regardless. But that, as a student of security analysis might say, is a risk worth bearing.
Audience
I wrote this book for two distinct audiences: security practitioners, and the bosses they report to. Practitioners need to know how, what, and when to measure. Their bosses need to know what to expect. Not for nothing has the security domain resisted measurement. As the bedraggled security manager of a household-name financial services firm recently told me, "my boss doesn't understand what I do every day. All he understands are numbers." Bridging the yawning gap between practitioners and management i