Slashdot Mirror
Ask the Author of the Latest MS-Funded Windows vs. Linux Study
Last week on Slashdot you saw a (Microsoft-funded) research
study on Windows vs. (Novell) Linux reliability by Dr.Herbert
Thompson. Novell disagreed
with the study's conclusions. So did most Slashdot readers.
Thompson's work been mentioned on Slashdot before, especially his
famous five-line
script that could change electronic voting machine results
and his novel, The
Mezonic Agenda: Hacking the Presidency. He's a real,
genuine-article computer security expert (and regular Slashdot reader)
who is happy to put on his flame-resistant
suit and discuss his Microsoft vs. Linux study with you. So
ask whatever you like, one question per post. We'll send him 10 of the highest-moderated questions and publish his
answers next Monday. He'll jump into the discussion then, which ought
to make it rather lively.
Dr. Thompson:
Admittedly, I don't know who you are and I haven't read any of your books. Worse, I didn't read your study itself, only its conclusions as reported second-hand by the press. However my lack of knowledge of your backgound is probably consistant with most Slashdot readers and the IT industry as a whole. I have to give you the benefit of the doubt and assume that you are a capable, respected researcher elsewise MS wouldn't have approached you in the first place.
Could you please explain why you decided to risk drawing your objectivity into question by undertaking this project? Your findings may be 100% valid. And MS may very well have straight-up told you: "Please print whatever you find, even if it casts Windows in a bad light." However, who's going to believe it, even if it were true? If I were in your shoes, I'd be affraid that making a deal like this would ruin my career. If I don't tell MS what they want to hear, word would get out that I don't play ball. If I do report what's in the sponsor's best interest, a lot of people start accusing me of being a shill. Seems like a lose-lose proposition.
Entrepreneur : (noun), French for "unemployed"
The study seemed to only compare comercial applications on the various platforms and not the alternatives. Its very common that comercial apps on Linux have poor support on Linux while the free alternatives blows most out of the water on Windows too. Its not especially hard to select a couple of apps with stellar support on Windows and SAP like support on Linux and blame Linux when the problem really lies in the lack of vendor support. Some vendors even support just one specific linux version without! any patches applied.
What care was taken in selecting applications with similar support offerings to not bias the study heavily to Microsofts advantage?
HTTP/1.1 400
How many Microsoft-funded studies have been buried because the conclusion was "incorrect"?
Microsoft and Linux distros have had a policy for some time of including more and more functionality in the base operating system, the latest example is the inclusion of "Local Workflow" in Windows Vista.
As a security expert do you think that bundling more and more increases or decreases the risks, and should both Windows and Linux distros be doing more to create reduced platforms that just act as good operating systems.
An Eye for an Eye will make the whole world blind - Gandhi
Those who pick the metrics always win the pissing match.
:-)
But my questions are: What do you hope to achieve from the study? To dissuade people from Linux and somehow get it shut down? Would the world be better served by a Windows only market?
And an additional would be: How do you suppose to convince all the people who have switched from Windows to Linux and stayed there BECAUSE it met [or exceeded] their needs that Windows is actually the better technical choice?
And last would be: How does it feel to sell your soul to the devil?
Tom
N.B. Seeing how this will get modded -5, stupid I'll also add who cares what some study says. I'm sitting here RIGHT NOW at my Gentoo desktop, I work with my Gentoo computer farm daily. The study could say anything it wants but at the end of the day I get a lot of work done with my Gentoo Linux powered computers.
Someday, I'll have a real sig.
I only skimmed over the public comments and your survey. My impression was that the sample period you chose was very small. Why so small? It seemed so small that it struck me as deliberate to get a predetermined outcome. I am not saying that was your intention but it does give the appearance that it could have been.
Have you considered increasing the sample period?
Keep the Classic Slashdot.
If the same study was not funded by Microsoft and was funded by a company that supports Open source and the linux platform say google or IBM would your results have been the same?
GL HF!
"As they attempt to increase business capabilities over time, customers are telling us that they are hitting a wall with Linux, experiencing significant reliability issues resulting in higher total cost of ownership," said Martin Taylor, general manager of platform strategy at Microsoft.
If scaling up on windows means significant reliability issues, how has google managed to avoid these despite scaling to the level they have?
Or Amazon, which I beleive also runs on linux. These are true enterprise level e-commerce apps, and despite the tons of studies saying they've picked the WRONG computing platform, places like google, amazon have amanged to create profitable businesses on non MS platforms.
What OS do you run personally - and why?
IE: If you run Windows is it because that is what they run at work? If it is an Open Source OS - is it because you believe in open source? If it is OSX - why wasn't it included in the study?
It seems that your study attempted to simulate the growth of an internet startup firm on Windows or Linux. One thing I did not see in the study was a good description of assumptions you made. What assumptions were made in both the design of the requirements and the analysis of the data? What limitations can we place on the conclusions as a result of these assumptions?
LedgerSMB: Open source Accounting/ERP
[..]Windows systems are good for being domain controllers and file servers for Windows clients [...]
Windows:
Client Access Licenses
Linux:
Samba
Additionally, software such as NIS exists to fill the role of a single-sign-on, although I've only had painful experiences with it, personally (using Solaris in a completely crazy setup).
He was paid to evaluate two possible scenarios given a set of initial conditions. Researchers do it all the time in this place we like to call the "real world" - in engineering for example. You take a few alternative designs, apply the constraints you are given, and pick the right tool for the job.
Dr. Thompson was given a set of conditions and two contendors, he gave his evaluation, done deal. It doesn't imply endorsement. I'm an engineer - I evaluate options regularly. Sometimes I have to pick options I didn't like. But I do it because they are the right option for the given scenario. If the conditions were different the results probably would have been different.
-everphilski-
Altho I can understand that Novell are protecting their interests, the same could be said about microsoft.
Also, did Microsoft give you some procedures or methodology to follow in your study?
How much did this report earn you from Microsoft personally? How much did you company get?
Do you get extra money for spreading the news about the report like you do here on Slashdot, or is this included in your original deal?
)9TSS
How many NDAs did you have to sign before starting the study? Did anyone pull you asside to "set the record streight" before the study began? How were you first asked about doing this study? Was it something like "hey, we need a study to boost our TCO stats, here's some cash..." or was it more altruistic like "hey, we need to see how we stack up agaist the competition .. heres some cash, and dont hold any punches!"
-GenTimJS
Simple one: of course I accept that Windows and Linux are a priori equally vulnerable - C programmers make mistakes. the question is which model is most likely to deliver a fix fastest. Given that the one area where Linux is probably in the lead over Microsoft's software is in the realm of the webserver - why are my server logs filled with artifacts of hacked IIS boxes but apache seems to remain pretty safe?
Everyone on /. likes to complain about microsoft security, and microsoft PR people like to point out their improvements. Here's a chance to give ammunition to both sides. What do you think are the three biggest security improvements microsoft has made in the past two years, and what are the three biggest security-related issues that still remain?
"Weapons should be hardy rather than decorative" - Miyamoto Musashi
I think that goes for OS's too
In addition, Digital Rights Management or other copy protection schemes are becoming increasingly demanding and insidious, whether by uniquely identifying and reporting on user activity, intentionally restricting functionality, and even introducing new security issues (the most recent flap involves copy protection software on Sony CDs that not only hides content from the user but permits viruses to take advantage of this feature.)
I would like to know how you feel about the shift of control over the personal computer from the person to the software manufacturers -- is it right, and do we gain more than we're losing in privacy and security?
Try not. Do or do not, there is no try.
-- Dr. Spock, stardate 2822-3.
You tested six people on two different systems; how is that supposed to yield any substantial insight into the underlying OSes themselves?
[At best, your study seems to show that the GNU/Linux distribution you selected was not particularly good at this task. But why does that show that the ``monolithic" style of Windows is better per se than the ``modular" style of GNU/Linux distributions?]
"Every decent man is ashamed of the government he lives under." - H.L. Mencken
No problem, just apply yourself with some integrity. Sadly this is a rarely seen trade theese days. This kid is an Microsoft MVP (Most Valuable Professional) so its not only the funding that makes it hard to see him as unbiased.
HTTP/1.1 400
The Linux administrators faced some out of the ordinary challenges, not faced by most Linux admins, while the Windows admins faced none.
For example, most of the time difference between Windows and Linux was spent upgrading gLibC, something that you're really not supposed to do. It's comparable to trying to manually upgrade parts of a Windows 98 system to run a program that required XP, rather than actually upgrading to XP.
Then, you had the Linux admins getting updates from 4 different sources, rather than just from SuSE's repositories, which is also out of the ordinary, while the Windows admins only visited Windows Update, which only supplies patches to the base operating system, when in reality they'll have to get updates from many other sources if they wanted to keep their apps up to date.
Do you think this was a fair study?
Another concern I have is that while your study simulates the installation and upgrade of two different systems based upon two OS's, it does not seem to simulate the real-world work needed to keep those systems running on a daily basis. In the real world systems break, worms clog the network, and regular maintenance must be done. Your study seems to completely disregard all that work and focus only on install/upgrade. Why did you not base your study on the behaviors of a real working system with a simulated network attached? It seems like the shortcut method you used to quickly evaluate only certain tasks makes the study wholly academic and loses any value as a predictor for the operation of a real network, over time, with real traffic.
Finally, I've seen it suggested that this study requires that all software be updated to the latest versions, but While Linux based servers constantly release the latest patches to each component as they become available, Windows only releases them en masse, How then can you compare the two? To be perfectly fair one would have to know what development has happened on the various components of Windows and rate all of those components as failing to be updated (since MS has not yet released that version). Barring such inside information, any comparison between a system with an open development process and one with a closed development process is critically flawed. Do you not see this as a problem with your study?
Looking at your research report's appendices, it seems that the requirements for Windows Administrators were somewhat different than the Linux Administrators. For instance, you ask for 4-5 years sys admin experience minimum for Windows, whereas it's 3-4 years sys admin experience minimum for Linux.
Why wasn't it equal for both? And doesn't this sort of slight Windows favoring undermine your credibility?
Kudos to you for braving the inevitable flames to answer people's questions here on Slashdot.
Read the EFF's Fair Use FAQ
Dr. Thompson,
How do you explain the different conclusions from studies funded by Microsoft and studies funded by Unix/Linux vendors? Shouldn't studies that essentially study the same issue inevitably arrive in the same conclusions, if the research for the study was made independently, honestly and with no systemic errors? How do you expect people to take any of these studies, whether pro-Microsoft or anti-Microsoft, seriously?
In Soviet Russia, I ruled you
As always, but while this site claims tech coverage, it's mostly LINUX tech coverage.
/. crowd moans that they didn't set up the boxes right. Well, if it's not obvious how to set them up properly, that makes it pretty hard to use, doesn't it?
:/
What I want to know is what's a fair comparison anyway?
If it's MS-funded, it's probably skewed to Windows.
If it's performed by Linux advocates, it's skewed.
If it's done by a research company that doesn't care either way, they end up ruling that Linux is hard to use, and the
I say just ask around in the IT community, though it would really depend on who you know. Most people I've talked to either marginalize it as a non-option (my old boss, when I was extolling the virtues of Redhat 6.4...), consider it as a plague (most of my peers), or a neccesary evil you'll probably have to know sooner or later (my college profs.) Personally, I think it's a brilliant OS-extended-family-and-then-some for a server, but masochistic on the desktop.
So... what's a fair comparison? Even if they loaded a study with a team of Linux gurus, and Windows reps to set up the systems properly, that'd hardly be a realistic environment for either.
Mr Thompson
I've always wondered exactly how much Linux based knowlege a writer should have in order to write a report on the TCO of Linux based networks and software.
How much Real World/In the Trenches experience do you have implementing and supporting large network and software applications that run Microsoft products compared to *nix based solutions?
Exactly how experienced are you with Linux? What is your favourite distro? How long have you been running Linux?
What is the best thing Windows does better than Linux?
What is the best thing Linux does better than Windows?
Have you ever contributed to an Open Source project or been part of an Open Source community?
Thanks
John the Kiwi
Did Microsoft make you sign a non-disclosure agreement? If so, without saying anything to violate the NDA, were there results or findings that MS prevented from being released?
The Data Mining Software used in M1 required the Linux administrators to use MySQL 4.1, which was not part of the SLES distribution. This appears to be where the majority of the problems with the Linux servers stemmed from. Do you think the choice of Linux distribution and/or Data Mining Software biased the outcome report in any way?
From the study:
What I find lacking is the business case for upgrading the OS. And why on earth would any enterprise with even the tiniest amount of foresight and planning deploy Windows 2000/SuSE 8 knowing they will upgrade to the next gen just one year later? (Not that there aren't plenty of enterprises who fit your model, not to mention IT workers seeking to "power level" their skills...)
Now, certainly there is value in trouble-free installs. But can you say with confidence a better upgrade experience is really a fair test of value? Especially when the entire install/patch/upgrade philosophy between Windows and Linux is so disparate?
In other words: It's no surprise that Windows will perform better on the treadmill, constantly upgrading is at the very core of Microsoft's profitability.
--
If I understand the study correctly, the windows side had to do nothing but set up a server to do a few different tasks over time and run windows update. The linux side had to have have multiple incompatible versions of their database server running simultaneously on a single system and had to run unsupported versions of software to do it.
Why wasn't the windows side required to run multiple versions of IIS or SQL server simultaneously? In real life if you need to run multiple database versions you use virtualization or multiple systems, especially if one requires untested software. You don't run some hokie unstable branch on the same system as everything else. Why was a linux solution picked that required this level of work? My other related question is, did any of the unix administrators question why there were being asked to do such a thing? For example, did they come back and say they need a license for vmware? If they did not they do not seem like very competent administrators in my opinion.
Of course, with this audience, you might want to say FireFox, or possibly Safari. I am curious if you use MS IE. (Though I'd like to hear "Opera, of course.")
.. paranoid crackpot leftover from the days of Amiga.
Windows administrators are forced to wait until Windows releases a patch for known vulnerabilities to upgrade their systems. Why, then, were the Linux administrators told to attempt to upgrade their systems before Novell had released newly packaged versions of MySQL? The entire point of a package management system is that administrators rely on companies like Novell to correct dependencies prior to deployment. Since Windows administrators have the same constraint (i.e., waiting for security updates to be released), it is an unfair and arbitrary difference that caused a lot of troubles.
Why did you compare the number of patches required to apply between the systems? This is not a measure of security. Windows patches are bundled and affect many parts of the operating system while Linux patches affect individual components. The overtone in your paper implied that fewer windows patches was in some way easier or more secure; what justification do you have for this assertation?
What is the rationale behind this? Were the Linux administrators required to restart at this point? This is an incredibly contrived situation; one can simply stop and re-start the process in question after the upgrade has completed.
Furthermore, the upgrade methodology questionable. Real companies use development and production servers and don't upgrade the production server until a reproduceable upgrade trajectory has been tested on the development server. The actions of these administrators imply that they had no such access, and that there was no possibility for backtracking or restarting after a failed step. Normally, one would expect the ability to nuke the development server and start over, rather than following a bad plan to worse conclusions.
You conclude from the study that at the enterprise level it is easier to manage Windows in regard to implementing business requirements than it is in Linux. I believe that Linux can and will be as good as Windows and to this end I ask what can we, the community and Linux vendors do to improve this failing i.e. what would you suggest that Linux could do or needs to do to be on par with Windows or even exceed it in this context?
A quick read of the report shows that the real losers here seem to be the Administrators. Some of the Linux admins "could not meet business requirements", and some were judged as failures by not using vendor-supplied solutions.
Isn't one of the points of running Linux servers the freedom to use solutions NOT supplied by the vendor? Is it even possible for the Microsoft admins to make changes that aren't fed from the vendor?
When the only tool you have is the "Upgrade" button, and the button doesn't work, what then? The advantage of Linux in administration is the flexibility to Make It Happen, even if the vendor sends you something broken.
I know good admins on Microsoft, and good ones on UNIX. They seem to Make It Happen no matter what, because that is their job. Making It Happen sometimes include custom fixes, that are documented, so you can undo them when the vendor comes through (hopefully) later.
So the Final Question is, why was it bad for the Linux admins to stray from vendor-supplied fixes, and why is the lack of flexibility on the Microsoft side a "win"?
Dr. Thompson, the way you selected the administrators seems to suggest a strong bias against Linux. In Appendix 3 (page 41), you recruited Windows administrators with at least 4-5 years of Windows administrator experience, while in Appendix 4 (page 43), you recruited Linux administators with just 2 years of Linux experience.
It seems that either you're a true Linux believer thinking that a Linux administrator can out-smart, out-perform a Windows administor with twice the experience, or that your experiment was setup to pit inexperienced Linux admins against experienced Windows admins.
So which is it?
How do your findings hold up against page 31 of the recent leaked MS Singularity OS research document found at ftp://ftp.research.microsoft.com/pub/tr/TR-2005-13 5.pdf, in which MS compares current versions of Windows XP, Linux and FreeBSD, only to show that Linux and FreeBSD outperform Windows XP?
Why do you suppose that MS would even consider building a new OS from the ground up, as they are doing with Singularity, if their current model already beats the competition?
Dr. Thompson,
I will admit that I know very little about administering NT systems, I am basically a Unix guy. But I do work in a mixed shop, and the NT guys I work with have told me that you can't upgrade directly from 2k to 2k3. What they do is install a clean 2k3 image, then migrate services to it, then fix anything tht might have broken along the way. They assure me that this is how MS says they are supposed to do it. I am extremely curious to know more about the upgrade procedure from 2k to 2k3, could you explain what steps are taken to accomplish this operation that MS allegedly says should be avoided?
Question: Were the "underlying assumptions" and basic methodology (which you very responsibly and sensibly do report in your study) dictated to you by Microsoft or some other external entity, or did you yourself come up with the test scenario?
I ask because the consensus around here seems to be that the conditions and methodology were cherry-picked to favor systems with single-vendor provenance and ease of initial installation, and do not include any real measures of operational stability or reliability.
Dr. Thompson,
Though your study pits Windows versus Linux and claims Linux has a higher TCO, what is the actual marginal cost of implementing a Linux box versus a Windows box? Only three machines seems hardly determinant or significant. Implementing one Linux machine may be (although I don't believe it) more expensive, but several Linux machines may cost less than the same number of Windows boxes.
Also, with respect to updates, did you consider all of the upgrades in Red Hat's "up2date" as "patches" or simply as "upgrades" with a few being security patches.
Did your study favor GUI over command-line interface or vice-versa?
Did your study log each crash/reboot/system error thrown by each machine? Also, were you required to run any "system restores" on the Windows machine?
Did your study consider alternative operating systems with high security (such as OpenBSD)?
If you could "fix" Linux (or at least the distros you reviewed), what would you insert, update, or delete?
Would you consider running the same study with a very powerful package management system, such as APT?
Thank you,
Drew E.
His study showed that there was quite a significant effect on marine life, more than previous studies had shown.
He was thanked then disposed of within a month, his study left with him. Needless to say had the study said what BP wanted to here he would never have had such a rapid removal from BP.
"I may be full of crap about this game, and I may be wrong, and that's fine." -Jack Thompson
The report seems to predominantly discuss days of risk of disclosed vulnerabilities.
But it is comparing Apples and Oranges to an extent as in the free software world it is usual, and expected that vulnerabilities will be publically disclosed early (indeed this is a core aspect of Debians developers pledge to their users, that they won't hide the bad news, because you can always do something, even if it is switch the machine off).
The simple truth is that there is no "zero day fix", except arguably where the programmers find and fix their own bugs (vendors, have been known to quietly omit mentioning the huge security flaws thus removed), there is some process, some discussion, and usually a far longer (never ending?) period of deployment following any security fix. "Zero day fix" is marketing speak, for your system has been vulnerable since you installed it, but we finally recruited someone with enough brains to understand the vulnerability report AND fix it.
So the question is "Why use a metric that will only highlight a philosophical difference?", and one on which the best experts in the field can't agree on the significant of from a security perspective. The report does include some random quotation on this, but I assume the answer is that is what they were paid to do.
If you substract out the "zero day" data, you seem to be left with Microsoft taking a long time to fix very few bugs, suggesting they aren't terribly responsive when it counts.
Your computer is not safe simply because you (or the vendor) don't know what the software bugs in it are.
I'm also wondering why vulnerability count is used as a brute statistic again. It is pretty much as discredited a measure as any can be.
For example the Kernel is seen as the most patched feature of the free software system used, and IE as the most patched feature of the Windows world. Interesting because I've seen many, many, boxes compromised by using IE, but I've only ever suspected one case of privilege escalations through kernel vulnerabilities even though it is usually in use 100% of the time the box is on. Surely then I am missing something, as are all the people out there Netcraft, and Uptime reporting as running 2.2 Kernels still, I suspect what I'm missing is realistic analysis of vulnerabilities. I see very few people using IE versions of a similar vintage despite similar vulnerability discovery rates.
Security reports should also cover more basic aspects of system management, like data security, our Windows 2003 server recently reported an error writing data to a filesystem, saying "data might have been lost", alas this error message failed to mention which filesystem data might have been lost from. These kind of experiences cut directly at the first assumption, that the Microsoft product is "Enterprise class". No point in patching it quickly, if it is so broken as to be of doubtful utility in the first place.
Where is the discussion of database functionality, as there is a huge range from MySQL to Oracle, with MS SQL pretty much at the bottom of that spectrum. Where as Oracle is going to win hands down (all however many gigabytes of it there is now). Some of these functions are relevant to security, such as backups, transactional integrity features etc.
You state in your report that the requirements were developed after interviews with "leading CIO's, CTO's,
Moreover, in appendix 5 of your study you show little overlap between the lists of popular component users. Many of the groups listed for one "popular solution" were not listed on another. Nor did you separate these lists by operating system. This give no indication whether the popular components are ever used in concert. Nor does it indicate how many groups are using each feature set or system. Nor even where these user numbers came from.
I bring these points up because they point to potential holes in your study that I am curious about. In particular:
My question is, do you see these as issues? If not why not?
Dr. Thompson
Selecting the methodology for performing research like this must have been difficult. I believe there is already numerous questions that ask you about the various inputs to your methodology.
So my interest is in a different area. The scenario described is based purely on E-Commerce and your conclusions reflect that a Windows Server solution will cause less "IT pain" than a SUSE Linux Solution. My question is thus:
Are there any scenarios in which you suspect a Windows Server Solution is more likely to cause more IT pain? And consequently have you any more research "in the pipe" to test this?
Regards
Darrell
But the bug reports from Securia, which is not sponsored by Microsoft or Linux, show quite clearly that Windows Server 2003 and SQL Server 2000 have more known vulnerabilities than Redhat and Oracle. How can Windows Server 2003 be more secure when it is clear that it has more vulnerabilities?
I'm unable to find any statistics for 2005, but back in 2000, Linux accounted for 36% of webservers, and Windows only 21%, according to Netcraft. It's likely that this hasn't changed.
Windows is certainly more compatable with hardware and the majority of software binaries about, but more versatile? In what way?