Slashdot Mirror
Vulnerability Numerology - Defective by Design?
rdmreader writes "RDM has a point by point disassembly of the security vulnerability story phenomenon. We regularly see these, comparing various vulnerability lists for different operating systems. ZDNet's George Ou, for example, condemns Linux and Mac OS X by tallying up reported flaws and comparing them against Microsoft's. What he doesn't note is that his source, Secunia, only lists what vendors and researchers report. Results selectively include or exclude component software seemingly at random, and backhandedly claims its data is evidence of what it now tells journalists they shouldn't report. Is Secunia presenting slanted information with the expectation it will be misused?"
Exactly. IMHO, he's saying that Secunia vulnerability comparisons aren't any more reliable than numerology predictions.
And then unfortunately, their supporters like to bash Linux and Mac for actually working with security agencies and fixing their bugs as well as reporting them. This will forever be the bane of open source and it's benefit... that everyone gets to see its flaws but at the same time, everyone gets to contribute to fix them.
This is my sig. There are many like it but this one is mine.
We keep hearing this again and again and again.
It's very simple, really.
You can _never_ know the relative security of two systems. There simply isn't any way to measure it fairly.
Count disclosed vulnerabilities? What about the vulnerabilities that weren't disclosed?
Have teams search for vulnerabilities and compare the results? What does that tell you? Was one team equally good at finding vulnerabilities in one system as the other was at finding them in the other system? What if one system had many easy to find vulnerabilities, and the other had a couple of severe but harder to find vulnerabilities?
Count actual break-ins? Well, was that due to the system being vulnerable the way the vendor left it, or because of the administrator? What about break-ins you don't know about?
It's always a matter of what you don't know about. You don't know the vulnerabilities that weren't reported. You don't know the vulnerabilities that weren't found. You don't know the relative skills of the teams you used. You don't know if you tested for all possible classes of vulnerability.
And I haven't even mentioned the severity of vulnerabilities, the availability of exploit code, the way vulnerabilities are dealt with by the vendor, and a host of other issues.
The take home message is that you just _can't_ know. It's a hard pill to swallow, but you will just never know which system is more secure. All you have is flawed metrics and your gut feeling.
Please correct me if I got my facts wrong.
When I read the summary, I thought TFA could actually be interesting. But it's not any better than what it is criticizing.
Long story short:
ZDnet published an article comparing Secunia vulnerability counts in Mac OS X and Windows Vista/XP. They spun it the Microsoft way, so Mac OS X loses big time. A mac fanboy wrote a reply spinning it the Apple way.
TFA starts with a long-winded attack against the author of the ZDnet article without ever getting to the point. Let's just say that it talks about Zunes, XBoxes, train wrecks, ballet dancing and many more things.
Then it explains what Secunia does (in about two pages): they track software vulnerabilities which are - among others - reported by the vendors. So "honest" vendors get higher vulnerability counts. Who would have thought.
On it goes by saying that the "border" of an operating system is nowadays blurry; should the vulnerabilities in bundled applications be counted? Even if they are by another vendor?
Then he babbles about how most of the cited vulnerabilites in Mac OS X are related to what he calls "external software" - things such as python, java, perl, samba, tcpdump etc and that those same programs have the the same (or a similar) amount of vulnerabilities on other platforms. What he fails to point out is that Mac OS X *consists* of such "external software" for a big part, and that they are *part* of Mac OS X and cannot be removed easily.
Conclusion: a pointless (and extremely long-winded) article full of Microsoft bashing, as reply to an equally pointless article full of Apple bashing.