WinInformant Says Windows More Secure Than Linux

nihilist_1137 excerpts from this WinInformant article, which reads in part: "For at least the first 8 months of 2001, open-source poster child Linux was far less secure than Windows, according to the reputable NTBugTraq, which is hosted by SecurityFocus, the leading provider of security information about the Internet. ... A look at the previous 5 years--for which the data is more complete--also shows that each year, Win2K and Windows NT had far fewer security vulnerabilities than Linux, despite the fact that Windows is deployed on a far wider basis than any version of Linux." I wonder how many sysadmins (Windows or Linux) would agree with this conclusion. Update: 02/04 16:54 GMT by T : Looks like the WinInfo site has gone down since the story was submitted, so you may have to content yourself in the meantime with the Bugtraq numbers. Update: 02/04 19:30 GMT by T :Several readers have pointed out that the conclusions WinInformant makes based on the Bugtraq data are not those of SecurityFocus; the headline has been changed accordingly.

Comments

Actually, to be fair...

[cperciva]

I can't remember hearing about many *new* security holes in win2K recently.

I can't get to the article right now, so I'm not sure exactly what their argument is, but while I can remember hearing about quite a few major security holes in the unixes (I think everyone was bitten at least once by ptrace race conditions) I can't think of any similar issues in win2k.

XP, on the other hand... but we're not talking about XP here.

There is No Science Here.

[tqbf]

I like SecurityFocus. The people in charge of SecurityFocus are with-it and honest. I am completely confident that this work was done in good faith.

However, the conclusion being drawn here is invalid. The SecurityFocus vulnerability survey is interesting, but it is not itself a reasonable methodology to generate security metrics between operating systems.

I could pick nits at this ad hoc study for hours, but the biggest problems are also the most obvious:

First: the study associates third-party software with the operating system, and aggregates all the distributions together into a meaningless "Linux" category. This study is literally just pattern matching against advisories.

Second: there is no notion of "severity" or "impact" in the study. This is a shame, because SecurityFocus has actually put some real effort into deriving a taxonomy of vulnerabilities from their (enormous) vulnerability database. There is no way to determine whether the N Linux vulnerabilities were equivalent to the K NT vulnerabilities.

Third: the study compares a kit of open-source software, which has received extensive peer review, to a closed-source product. It should surprise nobody that Linux has more documented problems than Windows: it's actually possible to go find vulnerabilities on Linux. Finding Windows vulnerabilities requires black-box reverse engineering.

Finally, both Linux and Windows do a reasonable job of locking down server configurations out of the box. What IT people need to know is vulnerability breakdown by operating system and by deployed configuration. This study does nothing to inform us of whether a Linux web server is at more risk than a Windows web server, or whether it's safer to expose a Linux print server or a Windows print server. Organizations that deploy homogenous Apache+NFS+ssh server farms don't care about XFree vulnerabilities or Samba problems.

I don't think SecurityFocus is actually trying to make claims about the relative security of Linux and Windows. I think they've been a bit careless with this report though; it's a reasonable thing to try to generate from their database, but more thought should have gone into presentation.

SecurityFocus has the on-staff expertise to publish some real conclusions about the distribution of vulnerabilities between Linux and Windows. Before this database report is misconstrued by the trade press, it would be enormously helpful if they could publish a statement about the conclusions that can be legitimately drawn from it. It'd be good press for them, too.

Re:Simply put,

[joshtimmons]

Actually, there aren't SO MANY MORE windows servers on the internet than *nix boxes.

Please see this fine article http://slashdot.org/article.pl?sid=01/07/13/124025 7&mode=thread which tries to compare the number of windows systems vs unix systems on the internet.

Here are a couple of their conclusions:

1. GNU/Linux is the #2 web serving operating system on the public Internet (counting by IP address), according to a study surveying March and June 2001
2. GNU/Linux is the #1 server operating system on the public Internet (counting by domain name), according to a 1999 survey of primarily European and educational sites.
3. GNU/Linux is the #2 server operating system sold in 1999 and 2000, and is the fastest-growing.

Even taking the statistics most favorable to Microsoft, they had almost twice as many IPs on the public internet than Linux did in 1999. However, during that same period, there were many more than twice as many expoits, viruses, etc. that attacked windows vs unix.

Linux has far too many installations on the public internet to be dismissed as too rare to interest hackers.

Re:This, of course, will be ignored and ridiculed

[ryanr]

Sigh...

I can't read the original article, It's been Slashdotted to death. But I think I can make a pretty good guess as to what happened.

First off, we host Bugtraq, not NTBugtraq, which is Russ Cooper's list. (Any chance we can get that fixed in the story intro? Anyone know if the same mistake is in the original article?)

Secondly, I'm constantly amazed at how people mis-read our stats page. The Linux aggregate stats are the total of all unique bugs across all the various distributions we track. It's supposed to answer the question "How many Linux-related bugs were there that year." It's based on things like which distro ships a particular package, and when that package is found to have a hole, it also gets attached to the distro. This is so you can look up your distro, and see what bugs you might need to patch.

Take a look at the top of the page, our script hasn't been running since August, when we switched from Roxen to Apache. So, we're missing the whole last quarter of 2001 stats.

Regardless of anything else, using these number to declare that one thing is more secure than another is a mistake. Based on our numbers, why didn't they declare that everyone should run MacOS for security? Or that if you want to be more secure, run Debian instead of Win2K?

Re:Unfair comparison, uninformed journalist.

[opkool]

What I read was the original article before it went down by /.

So worry for the thing on Win9x/3.x + WinNT/2000.

So they are talking of Server OSes. So Win9x/3.x do not account as such.

What you say is that, of course, they do not include duplicates of the same vulnerability. But then there's no such program as rsync-2.07-3.i386.rpm on Debian 2.2 . Can you see it?

Also, why it is strangely coincidental de number of bugs for Red Hat Linux 6.2 for Alpha and Sparc? See:

For 2001, we see:
RedHat Linux 6.2 sparc - 18
RedHat Linux 6.2 alpha - 18
Debian Linux 2.2 sparc - 18
Debian Linux 2.2 arm - 18
Debian Linux 2.2 alpha - 18
Debian Linux 2.2 68k - 18

Coincidental? See it yourselves at SecurityFocus WebSite

Maybe is a cross-architechture bug? Will this mean that, in fact, it is the same bug?

Then the numbers for Mandrake, Red Hat and Debian are waaay too similar (2001) to be just a coincidence (Mandrake 7.1, Red Hat 7.0 and Debian 2.2 can be thought as "equal distributions" by means of timeline, packets versions and such):

RedHat Linux 7.0 - 28
MandrakeSoft Linux Mandrake 7.1 - 27
Debian Linux 2.2 - 26

Then, on 2001, we can assume that Red Hat 6.2, Mandrake 6.0 and 6.1 have the same package versions :

RedHat Linux 6.2 i386 - 20
MandrakeSoft Linux Mandrake 6.1 - 20
MandrakeSoft Linux Mandrake 6.0 - 20

And those numbers are also very very close to the ones for Red Hat Linux 6.2 on different architectures.

Maybe, just maybe... they are the same bugs?

Then, on previous years, the trend is the same.

With all the respects, I am no FUDing here. I post my comments to some piece of news that was flawled.

And I tried to explain why it was flawed. And I was vry carefull to not to blame conspiracy theories.

Then, again, I'm human. And I make mistakes. Like the Win0x/3.x and Win2000/NT of my previous post.

But this does not invalidate at all my message.

Re:Why is this automatically false?

[jdavidb]

Long gone are the Slackware days where you'd download a minimal kernel/utilities package and then compile only the apps you need, by yourself, and understand everything.

Wrong. I entered those days quite recently, with Linux From Scratch. LFS isn't exactly a "security solution," but it's hard to break into a machine when there's nothing running on any port except ssh.

Re:This, of course, will be ignored and ridiculed

[Mr Z]

Or maybe the Slashdot regulars (not the people who hang out at 0 and -1) will look at the piece calmly and discover other very valid flaws with the study.

You mean, like this? The NTBugTraq site itself says (emphasis mine):

There is a distinct difference in the way that vulnerabilities are counted for Microsoft Windows and other operating systems. For instance, applications for Linux and BSD are often grouped in as subcomponents with the operating systems that they are shipped with. For Windows, applications and subcomponents such as Explorer often have their own packages that are considered vulnerable or not vulnerable outside of Windows and therefore may not be included in the count. This may skew numbers.

So, while there may be a stack of Outlook vulnerabilities, those won't get lumped in with Windows. But sendmail vulnerabilities might get lumped in with RedHat. They go on to say (emphasis theirs):

The numbers presented below should not be considered a metric by which an accurate comparison of the vulnerability of one operating system versus another can be made.

Further, the numbers themselves do not support the conjecture that Windows 2000/NT had fewer reported vulnerabilities reported over the 5-year period. Let's compare RedHat (the Linux distro for which the largest number of vulnerabilities was reported) vs. Windows 2000/NT from their data:

• 1997: RedHat 6, Win2K/NT 10
• 1998: RedHat 10, Win2K/NT 8
• 1999: RedHat 47, Win2K/NT 78
• 2000: RedHat 95, Win2K/NT 97
• 2001: RedHat 54, Win2K/NT 42
• Total RedHat 212, Win2K/NT 235

So even though the numbers are potentially skewed against Linux, the totals still come up less for RedHat than for Win2000/NT.

What the other article must be doing (I haven't read it yet, since I wasn't able load it) is totalling across all distributions, which is wrong. One FTPD vulnerability would get multiplied by all the vendors that ship that FTPD, which isn't quite fair.

--Joe

Re:Exactly (it deserves to be rediculed and ignore

[ryanr]

The incompetence of the author writing this story, and of the Security Focus editorial staff for letting it through, is staggering. With this kind of security "expertise" is it any wonder at all that Nimda worms and the like run rampent across the net?

We didn't write the article in question, nor are we hosting, nor did we have any opportunity to see it ahead of time. (Or now... still can't see it.) Sadly, we have very little editorial control over other people's websites.

More interesting statistics...

[sheldon]

Screw securityfocus, let's look at bulletins released by manufacturers.

Microsoft security bulletins released in 2002:
MS02-001

Redhat security bulletins released in 2002:
2002-018
2002-015
2002-014
2002-012
2002-011
2002-009
2002-007
2002-004
2002-005
2002-003
2002-002
2001-171
2001-168
2001-165

And if you look at 2001 results you'll see a somewhat similar trend, although not near as pronounced. Somethink like 80 versus 60.

Are these statistics meaningful? Of course not. If you have read Paul's columns you would know he reported this tongue and cheek. It was a slow news day, he noticed this, had to make fun of it.

What makes this story interesting, and why Paul reported it is because if the numbers had been reversed you would be assured that would be the headline of the day on slashdot, and if anybody questioned it they would be called Microsoftie apologists.

And look at the responses you see here. They're almost comical. Reminds me of the responses to the Mindcraft benchmark. Fear, Uncertainty and Denial. :)

Apples anyone, or how about some tasty Oranges?

[TheFlu]

How about some different numbers...everyone loves statistics. "The following numbers were obtained by counting web site defacements as listed at Attrition.org from June 2000 through May 2001:" Breakin Stats

The trouble with comparing Linux distros to Windows lies in the fact that Linux distros include so many different applications. I just did a count of installed packages on a RedHat box I am using, and I got 780 installed packages. I'd like to see a comparison of the number of exploits between the RedHat distro and Windows installed with 700 of the most common applications for it. That might be a more useful comparison. Also, I will readily acknowledge the weakness and lack of true usefulness of the numbers below, so no need to flame me for the lack of usability...I'm only posting the info I found, so no need to stone the messenger.

Windows
4336 Windows NT
1070 Windows 2000
2 Windows 95
5408 Windows total

All UNIX and Like
1185 Linux Red Hat
999 Linux unknown distributions
36 Linux Connectiva
23 Linux Debian
17 Linux Cobalt
17 Linux SuSE
13 Linux ALZZA
12 Linux Mandrake
1 Linux Slackware
2304 Linux total

485 Solaris & Sun OS (1)
267 IRIX
163 FreeBSD
121 BSDI
44 SCO
28 Generic UNIX
18 Compaq Tru64 UNIX
9 AIX
7 HPUX HP
4 Digital UNIX DG
3 OpenBSD
2 NetBSD
1 PowerBSD
1 Digital OSF1
1153 UNIX & Like total

3457 UNIXs & Linux

8865 Total Windows and all UNIX

Other
2 Mac OS
1 Netware

63 unidentified