WinInformant Says Windows More Secure Than Linux

nihilist_1137 excerpts from this WinInformant article, which reads in part: "For at least the first 8 months of 2001, open-source poster child Linux was far less secure than Windows, according to the reputable NTBugTraq, which is hosted by SecurityFocus, the leading provider of security information about the Internet. ... A look at the previous 5 years--for which the data is more complete--also shows that each year, Win2K and Windows NT had far fewer security vulnerabilities than Linux, despite the fact that Windows is deployed on a far wider basis than any version of Linux." I wonder how many sysadmins (Windows or Linux) would agree with this conclusion. Update: 02/04 16:54 GMT by T : Looks like the WinInfo site has gone down since the story was submitted, so you may have to content yourself in the meantime with the Bugtraq numbers. Update: 02/04 19:30 GMT by T :Several readers have pointed out that the conclusions WinInformant makes based on the Bugtraq data are not those of SecurityFocus; the headline has been changed accordingly.

Comments

There is No Science Here.

[tqbf]

I like SecurityFocus. The people in charge of SecurityFocus are with-it and honest. I am completely confident that this work was done in good faith.

However, the conclusion being drawn here is invalid. The SecurityFocus vulnerability survey is interesting, but it is not itself a reasonable methodology to generate security metrics between operating systems.

I could pick nits at this ad hoc study for hours, but the biggest problems are also the most obvious:

First: the study associates third-party software with the operating system, and aggregates all the distributions together into a meaningless "Linux" category. This study is literally just pattern matching against advisories.

Second: there is no notion of "severity" or "impact" in the study. This is a shame, because SecurityFocus has actually put some real effort into deriving a taxonomy of vulnerabilities from their (enormous) vulnerability database. There is no way to determine whether the N Linux vulnerabilities were equivalent to the K NT vulnerabilities.

Third: the study compares a kit of open-source software, which has received extensive peer review, to a closed-source product. It should surprise nobody that Linux has more documented problems than Windows: it's actually possible to go find vulnerabilities on Linux. Finding Windows vulnerabilities requires black-box reverse engineering.

Finally, both Linux and Windows do a reasonable job of locking down server configurations out of the box. What IT people need to know is vulnerability breakdown by operating system and by deployed configuration. This study does nothing to inform us of whether a Linux web server is at more risk than a Windows web server, or whether it's safer to expose a Linux print server or a Windows print server. Organizations that deploy homogenous Apache+NFS+ssh server farms don't care about XFree vulnerabilities or Samba problems.

I don't think SecurityFocus is actually trying to make claims about the relative security of Linux and Windows. I think they've been a bit careless with this report though; it's a reasonable thing to try to generate from their database, but more thought should have gone into presentation.

SecurityFocus has the on-staff expertise to publish some real conclusions about the distribution of vulnerabilities between Linux and Windows. Before this database report is misconstrued by the trade press, it would be enormously helpful if they could publish a statement about the conclusions that can be legitimately drawn from it. It'd be good press for them, too.

Re:Simply put,

[joshtimmons]

Actually, there aren't SO MANY MORE windows servers on the internet than *nix boxes.

Please see this fine article http://slashdot.org/article.pl?sid=01/07/13/124025 7&mode=thread which tries to compare the number of windows systems vs unix systems on the internet.

Here are a couple of their conclusions:

1. GNU/Linux is the #2 web serving operating system on the public Internet (counting by IP address), according to a study surveying March and June 2001
2. GNU/Linux is the #1 server operating system on the public Internet (counting by domain name), according to a 1999 survey of primarily European and educational sites.
3. GNU/Linux is the #2 server operating system sold in 1999 and 2000, and is the fastest-growing.

Even taking the statistics most favorable to Microsoft, they had almost twice as many IPs on the public internet than Linux did in 1999. However, during that same period, there were many more than twice as many expoits, viruses, etc. that attacked windows vs unix.

Linux has far too many installations on the public internet to be dismissed as too rare to interest hackers.

Re:This, of course, will be ignored and ridiculed

[ryanr]

Sigh...

I can't read the original article, It's been Slashdotted to death. But I think I can make a pretty good guess as to what happened.

First off, we host Bugtraq, not NTBugtraq, which is Russ Cooper's list. (Any chance we can get that fixed in the story intro? Anyone know if the same mistake is in the original article?)

Secondly, I'm constantly amazed at how people mis-read our stats page. The Linux aggregate stats are the total of all unique bugs across all the various distributions we track. It's supposed to answer the question "How many Linux-related bugs were there that year." It's based on things like which distro ships a particular package, and when that package is found to have a hole, it also gets attached to the distro. This is so you can look up your distro, and see what bugs you might need to patch.

Take a look at the top of the page, our script hasn't been running since August, when we switched from Roxen to Apache. So, we're missing the whole last quarter of 2001 stats.

Regardless of anything else, using these number to declare that one thing is more secure than another is a mistake. Based on our numbers, why didn't they declare that everyone should run MacOS for security? Or that if you want to be more secure, run Debian instead of Win2K?

Re:Unfair comparison, uninformed journalist.

[opkool]

What I read was the original article before it went down by /.

So worry for the thing on Win9x/3.x + WinNT/2000.

So they are talking of Server OSes. So Win9x/3.x do not account as such.

What you say is that, of course, they do not include duplicates of the same vulnerability. But then there's no such program as rsync-2.07-3.i386.rpm on Debian 2.2 . Can you see it?

Also, why it is strangely coincidental de number of bugs for Red Hat Linux 6.2 for Alpha and Sparc? See:

For 2001, we see:
RedHat Linux 6.2 sparc - 18
RedHat Linux 6.2 alpha - 18
Debian Linux 2.2 sparc - 18
Debian Linux 2.2 arm - 18
Debian Linux 2.2 alpha - 18
Debian Linux 2.2 68k - 18

Coincidental? See it yourselves at SecurityFocus WebSite

Maybe is a cross-architechture bug? Will this mean that, in fact, it is the same bug?

Then the numbers for Mandrake, Red Hat and Debian are waaay too similar (2001) to be just a coincidence (Mandrake 7.1, Red Hat 7.0 and Debian 2.2 can be thought as "equal distributions" by means of timeline, packets versions and such):

RedHat Linux 7.0 - 28
MandrakeSoft Linux Mandrake 7.1 - 27
Debian Linux 2.2 - 26

Then, on 2001, we can assume that Red Hat 6.2, Mandrake 6.0 and 6.1 have the same package versions :

RedHat Linux 6.2 i386 - 20
MandrakeSoft Linux Mandrake 6.1 - 20
MandrakeSoft Linux Mandrake 6.0 - 20

And those numbers are also very very close to the ones for Red Hat Linux 6.2 on different architectures.

Maybe, just maybe... they are the same bugs?

Then, on previous years, the trend is the same.

With all the respects, I am no FUDing here. I post my comments to some piece of news that was flawled.

And I tried to explain why it was flawed. And I was vry carefull to not to blame conspiracy theories.

Then, again, I'm human. And I make mistakes. Like the Win0x/3.x and Win2000/NT of my previous post.

But this does not invalidate at all my message.

Re:This, of course, will be ignored and ridiculed

[Mr Z]

Or maybe the Slashdot regulars (not the people who hang out at 0 and -1) will look at the piece calmly and discover other very valid flaws with the study.

You mean, like this? The NTBugTraq site itself says (emphasis mine):

There is a distinct difference in the way that vulnerabilities are counted for Microsoft Windows and other operating systems. For instance, applications for Linux and BSD are often grouped in as subcomponents with the operating systems that they are shipped with. For Windows, applications and subcomponents such as Explorer often have their own packages that are considered vulnerable or not vulnerable outside of Windows and therefore may not be included in the count. This may skew numbers.

So, while there may be a stack of Outlook vulnerabilities, those won't get lumped in with Windows. But sendmail vulnerabilities might get lumped in with RedHat. They go on to say (emphasis theirs):

The numbers presented below should not be considered a metric by which an accurate comparison of the vulnerability of one operating system versus another can be made.

Further, the numbers themselves do not support the conjecture that Windows 2000/NT had fewer reported vulnerabilities reported over the 5-year period. Let's compare RedHat (the Linux distro for which the largest number of vulnerabilities was reported) vs. Windows 2000/NT from their data:

• 1997: RedHat 6, Win2K/NT 10
• 1998: RedHat 10, Win2K/NT 8
• 1999: RedHat 47, Win2K/NT 78
• 2000: RedHat 95, Win2K/NT 97
• 2001: RedHat 54, Win2K/NT 42
• Total RedHat 212, Win2K/NT 235

So even though the numbers are potentially skewed against Linux, the totals still come up less for RedHat than for Win2000/NT.

What the other article must be doing (I haven't read it yet, since I wasn't able load it) is totalling across all distributions, which is wrong. One FTPD vulnerability would get multiplied by all the vendors that ship that FTPD, which isn't quite fair.

--Joe

Re:Exactly (it deserves to be rediculed and ignore

[ryanr]

The incompetence of the author writing this story, and of the Security Focus editorial staff for letting it through, is staggering. With this kind of security "expertise" is it any wonder at all that Nimda worms and the like run rampent across the net?

We didn't write the article in question, nor are we hosting, nor did we have any opportunity to see it ahead of time. (Or now... still can't see it.) Sadly, we have very little editorial control over other people's websites.