Slashdot Mirror
Ask the Author of the Latest MS-Funded Windows vs. Linux Study
Last week on Slashdot you saw a (Microsoft-funded) research
study on Windows vs. (Novell) Linux reliability by Dr.Herbert
Thompson. Novell disagreed
with the study's conclusions. So did most Slashdot readers.
Thompson's work been mentioned on Slashdot before, especially his
famous five-line
script that could change electronic voting machine results
and his novel, The
Mezonic Agenda: Hacking the Presidency. He's a real,
genuine-article computer security expert (and regular Slashdot reader)
who is happy to put on his flame-resistant
suit and discuss his Microsoft vs. Linux study with you. So
ask whatever you like, one question per post. We'll send him 10 of the highest-moderated questions and publish his
answers next Monday. He'll jump into the discussion then, which ought
to make it rather lively.
Dr. Thompson:
Admittedly, I don't know who you are and I haven't read any of your books. Worse, I didn't read your study itself, only its conclusions as reported second-hand by the press. However my lack of knowledge of your backgound is probably consistant with most Slashdot readers and the IT industry as a whole. I have to give you the benefit of the doubt and assume that you are a capable, respected researcher elsewise MS wouldn't have approached you in the first place.
Could you please explain why you decided to risk drawing your objectivity into question by undertaking this project? Your findings may be 100% valid. And MS may very well have straight-up told you: "Please print whatever you find, even if it casts Windows in a bad light." However, who's going to believe it, even if it were true? If I were in your shoes, I'd be affraid that making a deal like this would ruin my career. If I don't tell MS what they want to hear, word would get out that I don't play ball. If I do report what's in the sponsor's best interest, a lot of people start accusing me of being a shill. Seems like a lose-lose proposition.
Entrepreneur : (noun), French for "unemployed"
The study seemed to only compare comercial applications on the various platforms and not the alternatives. Its very common that comercial apps on Linux have poor support on Linux while the free alternatives blows most out of the water on Windows too. Its not especially hard to select a couple of apps with stellar support on Windows and SAP like support on Linux and blame Linux when the problem really lies in the lack of vendor support. Some vendors even support just one specific linux version without! any patches applied.
What care was taken in selecting applications with similar support offerings to not bias the study heavily to Microsofts advantage?
HTTP/1.1 400
How many Microsoft-funded studies have been buried because the conclusion was "incorrect"?
Microsoft and Linux distros have had a policy for some time of including more and more functionality in the base operating system, the latest example is the inclusion of "Local Workflow" in Windows Vista.
As a security expert do you think that bundling more and more increases or decreases the risks, and should both Windows and Linux distros be doing more to create reduced platforms that just act as good operating systems.
An Eye for an Eye will make the whole world blind - Gandhi
I only skimmed over the public comments and your survey. My impression was that the sample period you chose was very small. Why so small? It seemed so small that it struck me as deliberate to get a predetermined outcome. I am not saying that was your intention but it does give the appearance that it could have been.
Have you considered increasing the sample period?
Keep the Classic Slashdot.
If the same study was not funded by Microsoft and was funded by a company that supports Open source and the linux platform say google or IBM would your results have been the same?
GL HF!
"As they attempt to increase business capabilities over time, customers are telling us that they are hitting a wall with Linux, experiencing significant reliability issues resulting in higher total cost of ownership," said Martin Taylor, general manager of platform strategy at Microsoft.
If scaling up on windows means significant reliability issues, how has google managed to avoid these despite scaling to the level they have?
Or Amazon, which I beleive also runs on linux. These are true enterprise level e-commerce apps, and despite the tons of studies saying they've picked the WRONG computing platform, places like google, amazon have amanged to create profitable businesses on non MS platforms.
What OS do you run personally - and why?
IE: If you run Windows is it because that is what they run at work? If it is an Open Source OS - is it because you believe in open source? If it is OSX - why wasn't it included in the study?
It seems that your study attempted to simulate the growth of an internet startup firm on Windows or Linux. One thing I did not see in the study was a good description of assumptions you made. What assumptions were made in both the design of the requirements and the analysis of the data? What limitations can we place on the conclusions as a result of these assumptions?
LedgerSMB: Open source Accounting/ERP
[..]Windows systems are good for being domain controllers and file servers for Windows clients [...]
Windows:
Client Access Licenses
Linux:
Samba
Additionally, software such as NIS exists to fill the role of a single-sign-on, although I've only had painful experiences with it, personally (using Solaris in a completely crazy setup).
He was paid to evaluate two possible scenarios given a set of initial conditions. Researchers do it all the time in this place we like to call the "real world" - in engineering for example. You take a few alternative designs, apply the constraints you are given, and pick the right tool for the job.
Dr. Thompson was given a set of conditions and two contendors, he gave his evaluation, done deal. It doesn't imply endorsement. I'm an engineer - I evaluate options regularly. Sometimes I have to pick options I didn't like. But I do it because they are the right option for the given scenario. If the conditions were different the results probably would have been different.
-everphilski-
Altho I can understand that Novell are protecting their interests, the same could be said about microsoft.
Also, did Microsoft give you some procedures or methodology to follow in your study?
How many NDAs did you have to sign before starting the study? Did anyone pull you asside to "set the record streight" before the study began? How were you first asked about doing this study? Was it something like "hey, we need a study to boost our TCO stats, here's some cash..." or was it more altruistic like "hey, we need to see how we stack up agaist the competition .. heres some cash, and dont hold any punches!"
-GenTimJS
Simple one: of course I accept that Windows and Linux are a priori equally vulnerable - C programmers make mistakes. the question is which model is most likely to deliver a fix fastest. Given that the one area where Linux is probably in the lead over Microsoft's software is in the realm of the webserver - why are my server logs filled with artifacts of hacked IIS boxes but apache seems to remain pretty safe?
Everyone on /. likes to complain about microsoft security, and microsoft PR people like to point out their improvements. Here's a chance to give ammunition to both sides. What do you think are the three biggest security improvements microsoft has made in the past two years, and what are the three biggest security-related issues that still remain?
"Weapons should be hardy rather than decorative" - Miyamoto Musashi
I think that goes for OS's too
In addition, Digital Rights Management or other copy protection schemes are becoming increasingly demanding and insidious, whether by uniquely identifying and reporting on user activity, intentionally restricting functionality, and even introducing new security issues (the most recent flap involves copy protection software on Sony CDs that not only hides content from the user but permits viruses to take advantage of this feature.)
I would like to know how you feel about the shift of control over the personal computer from the person to the software manufacturers -- is it right, and do we gain more than we're losing in privacy and security?
Try not. Do or do not, there is no try.
-- Dr. Spock, stardate 2822-3.
You tested six people on two different systems; how is that supposed to yield any substantial insight into the underlying OSes themselves?
[At best, your study seems to show that the GNU/Linux distribution you selected was not particularly good at this task. But why does that show that the ``monolithic" style of Windows is better per se than the ``modular" style of GNU/Linux distributions?]
"Every decent man is ashamed of the government he lives under." - H.L. Mencken
No problem, just apply yourself with some integrity. Sadly this is a rarely seen trade theese days. This kid is an Microsoft MVP (Most Valuable Professional) so its not only the funding that makes it hard to see him as unbiased.
HTTP/1.1 400
The Linux administrators faced some out of the ordinary challenges, not faced by most Linux admins, while the Windows admins faced none.
For example, most of the time difference between Windows and Linux was spent upgrading gLibC, something that you're really not supposed to do. It's comparable to trying to manually upgrade parts of a Windows 98 system to run a program that required XP, rather than actually upgrading to XP.
Then, you had the Linux admins getting updates from 4 different sources, rather than just from SuSE's repositories, which is also out of the ordinary, while the Windows admins only visited Windows Update, which only supplies patches to the base operating system, when in reality they'll have to get updates from many other sources if they wanted to keep their apps up to date.
Do you think this was a fair study?
Another concern I have is that while your study simulates the installation and upgrade of two different systems based upon two OS's, it does not seem to simulate the real-world work needed to keep those systems running on a daily basis. In the real world systems break, worms clog the network, and regular maintenance must be done. Your study seems to completely disregard all that work and focus only on install/upgrade. Why did you not base your study on the behaviors of a real working system with a simulated network attached? It seems like the shortcut method you used to quickly evaluate only certain tasks makes the study wholly academic and loses any value as a predictor for the operation of a real network, over time, with real traffic.
Finally, I've seen it suggested that this study requires that all software be updated to the latest versions, but While Linux based servers constantly release the latest patches to each component as they become available, Windows only releases them en masse, How then can you compare the two? To be perfectly fair one would have to know what development has happened on the various components of Windows and rate all of those components as failing to be updated (since MS has not yet released that version). Barring such inside information, any comparison between a system with an open development process and one with a closed development process is critically flawed. Do you not see this as a problem with your study?
Looking at your research report's appendices, it seems that the requirements for Windows Administrators were somewhat different than the Linux Administrators. For instance, you ask for 4-5 years sys admin experience minimum for Windows, whereas it's 3-4 years sys admin experience minimum for Linux.
Why wasn't it equal for both? And doesn't this sort of slight Windows favoring undermine your credibility?
Kudos to you for braving the inevitable flames to answer people's questions here on Slashdot.
Read the EFF's Fair Use FAQ
Dr. Thompson,
How do you explain the different conclusions from studies funded by Microsoft and studies funded by Unix/Linux vendors? Shouldn't studies that essentially study the same issue inevitably arrive in the same conclusions, if the research for the study was made independently, honestly and with no systemic errors? How do you expect people to take any of these studies, whether pro-Microsoft or anti-Microsoft, seriously?
In Soviet Russia, I ruled you
As always, but while this site claims tech coverage, it's mostly LINUX tech coverage.
/. crowd moans that they didn't set up the boxes right. Well, if it's not obvious how to set them up properly, that makes it pretty hard to use, doesn't it?
:/
What I want to know is what's a fair comparison anyway?
If it's MS-funded, it's probably skewed to Windows.
If it's performed by Linux advocates, it's skewed.
If it's done by a research company that doesn't care either way, they end up ruling that Linux is hard to use, and the
I say just ask around in the IT community, though it would really depend on who you know. Most people I've talked to either marginalize it as a non-option (my old boss, when I was extolling the virtues of Redhat 6.4...), consider it as a plague (most of my peers), or a neccesary evil you'll probably have to know sooner or later (my college profs.) Personally, I think it's a brilliant OS-extended-family-and-then-some for a server, but masochistic on the desktop.
So... what's a fair comparison? Even if they loaded a study with a team of Linux gurus, and Windows reps to set up the systems properly, that'd hardly be a realistic environment for either.
Mr Thompson
I've always wondered exactly how much Linux based knowlege a writer should have in order to write a report on the TCO of Linux based networks and software.
How much Real World/In the Trenches experience do you have implementing and supporting large network and software applications that run Microsoft products compared to *nix based solutions?
Exactly how experienced are you with Linux? What is your favourite distro? How long have you been running Linux?
What is the best thing Windows does better than Linux?
What is the best thing Linux does better than Windows?
Have you ever contributed to an Open Source project or been part of an Open Source community?
Thanks
John the Kiwi
The Data Mining Software used in M1 required the Linux administrators to use MySQL 4.1, which was not part of the SLES distribution. This appears to be where the majority of the problems with the Linux servers stemmed from. Do you think the choice of Linux distribution and/or Data Mining Software biased the outcome report in any way?
From the study:
What I find lacking is the business case for upgrading the OS. And why on earth would any enterprise with even the tiniest amount of foresight and planning deploy Windows 2000/SuSE 8 knowing they will upgrade to the next gen just one year later? (Not that there aren't plenty of enterprises who fit your model, not to mention IT workers seeking to "power level" their skills...)
Now, certainly there is value in trouble-free installs. But can you say with confidence a better upgrade experience is really a fair test of value? Especially when the entire install/patch/upgrade philosophy between Windows and Linux is so disparate?
In other words: It's no surprise that Windows will perform better on the treadmill, constantly upgrading is at the very core of Microsoft's profitability.
--
If I understand the study correctly, the windows side had to do nothing but set up a server to do a few different tasks over time and run windows update. The linux side had to have have multiple incompatible versions of their database server running simultaneously on a single system and had to run unsupported versions of software to do it.
Why wasn't the windows side required to run multiple versions of IIS or SQL server simultaneously? In real life if you need to run multiple database versions you use virtualization or multiple systems, especially if one requires untested software. You don't run some hokie unstable branch on the same system as everything else. Why was a linux solution picked that required this level of work? My other related question is, did any of the unix administrators question why there were being asked to do such a thing? For example, did they come back and say they need a license for vmware? If they did not they do not seem like very competent administrators in my opinion.
Of course, with this audience, you might want to say FireFox, or possibly Safari. I am curious if you use MS IE. (Though I'd like to hear "Opera, of course.")
.. paranoid crackpot leftover from the days of Amiga.
Windows administrators are forced to wait until Windows releases a patch for known vulnerabilities to upgrade their systems. Why, then, were the Linux administrators told to attempt to upgrade their systems before Novell had released newly packaged versions of MySQL? The entire point of a package management system is that administrators rely on companies like Novell to correct dependencies prior to deployment. Since Windows administrators have the same constraint (i.e., waiting for security updates to be released), it is an unfair and arbitrary difference that caused a lot of troubles.
Why did you compare the number of patches required to apply between the systems? This is not a measure of security. Windows patches are bundled and affect many parts of the operating system while Linux patches affect individual components. The overtone in your paper implied that fewer windows patches was in some way easier or more secure; what justification do you have for this assertation?
What is the rationale behind this? Were the Linux administrators required to restart at this point? This is an incredibly contrived situation; one can simply stop and re-start the process in question after the upgrade has completed.
Furthermore, the upgrade methodology questionable. Real companies use development and production servers and don't upgrade the production server until a reproduceable upgrade trajectory has been tested on the development server. The actions of these administrators imply that they had no such access, and that there was no possibility for backtracking or restarting after a failed step. Normally, one would expect the ability to nuke the development server and start over, rather than following a bad plan to worse conclusions.
You conclude from the study that at the enterprise level it is easier to manage Windows in regard to implementing business requirements than it is in Linux. I believe that Linux can and will be as good as Windows and to this end I ask what can we, the community and Linux vendors do to improve this failing i.e. what would you suggest that Linux could do or needs to do to be on par with Windows or even exceed it in this context?
A quick read of the report shows that the real losers here seem to be the Administrators. Some of the Linux admins "could not meet business requirements", and some were judged as failures by not using vendor-supplied solutions.
Isn't one of the points of running Linux servers the freedom to use solutions NOT supplied by the vendor? Is it even possible for the Microsoft admins to make changes that aren't fed from the vendor?
When the only tool you have is the "Upgrade" button, and the button doesn't work, what then? The advantage of Linux in administration is the flexibility to Make It Happen, even if the vendor sends you something broken.
I know good admins on Microsoft, and good ones on UNIX. They seem to Make It Happen no matter what, because that is their job. Making It Happen sometimes include custom fixes, that are documented, so you can undo them when the vendor comes through (hopefully) later.
So the Final Question is, why was it bad for the Linux admins to stray from vendor-supplied fixes, and why is the lack of flexibility on the Microsoft side a "win"?
Dr. Thompson, the way you selected the administrators seems to suggest a strong bias against Linux. In Appendix 3 (page 41), you recruited Windows administrators with at least 4-5 years of Windows administrator experience, while in Appendix 4 (page 43), you recruited Linux administators with just 2 years of Linux experience.
It seems that either you're a true Linux believer thinking that a Linux administrator can out-smart, out-perform a Windows administor with twice the experience, or that your experiment was setup to pit inexperienced Linux admins against experienced Windows admins.
So which is it?
How do your findings hold up against page 31 of the recent leaked MS Singularity OS research document found at ftp://ftp.research.microsoft.com/pub/tr/TR-2005-13 5.pdf, in which MS compares current versions of Windows XP, Linux and FreeBSD, only to show that Linux and FreeBSD outperform Windows XP?
Why do you suppose that MS would even consider building a new OS from the ground up, as they are doing with Singularity, if their current model already beats the competition?
Question: Were the "underlying assumptions" and basic methodology (which you very responsibly and sensibly do report in your study) dictated to you by Microsoft or some other external entity, or did you yourself come up with the test scenario?
I ask because the consensus around here seems to be that the conditions and methodology were cherry-picked to favor systems with single-vendor provenance and ease of initial installation, and do not include any real measures of operational stability or reliability.
Dr. Thompson,
Though your study pits Windows versus Linux and claims Linux has a higher TCO, what is the actual marginal cost of implementing a Linux box versus a Windows box? Only three machines seems hardly determinant or significant. Implementing one Linux machine may be (although I don't believe it) more expensive, but several Linux machines may cost less than the same number of Windows boxes.
Also, with respect to updates, did you consider all of the upgrades in Red Hat's "up2date" as "patches" or simply as "upgrades" with a few being security patches.
Did your study favor GUI over command-line interface or vice-versa?
Did your study log each crash/reboot/system error thrown by each machine? Also, were you required to run any "system restores" on the Windows machine?
Did your study consider alternative operating systems with high security (such as OpenBSD)?
If you could "fix" Linux (or at least the distros you reviewed), what would you insert, update, or delete?
Would you consider running the same study with a very powerful package management system, such as APT?
Thank you,
Drew E.
His study showed that there was quite a significant effect on marine life, more than previous studies had shown.
He was thanked then disposed of within a month, his study left with him. Needless to say had the study said what BP wanted to here he would never have had such a rapid removal from BP.
"I may be full of crap about this game, and I may be wrong, and that's fine." -Jack Thompson
You state in your report that the requirements were developed after interviews with "leading CIO's, CTO's,
Moreover, in appendix 5 of your study you show little overlap between the lists of popular component users. Many of the groups listed for one "popular solution" were not listed on another. Nor did you separate these lists by operating system. This give no indication whether the popular components are ever used in concert. Nor does it indicate how many groups are using each feature set or system. Nor even where these user numbers came from.
I bring these points up because they point to potential holes in your study that I am curious about. In particular:
My question is, do you see these as issues? If not why not?
Dr. Thompson
Selecting the methodology for performing research like this must have been difficult. I believe there is already numerous questions that ask you about the various inputs to your methodology.
So my interest is in a different area. The scenario described is based purely on E-Commerce and your conclusions reflect that a Windows Server solution will cause less "IT pain" than a SUSE Linux Solution. My question is thus:
Are there any scenarios in which you suspect a Windows Server Solution is more likely to cause more IT pain? And consequently have you any more research "in the pipe" to test this?
Regards
Darrell
But the bug reports from Securia, which is not sponsored by Microsoft or Linux, show quite clearly that Windows Server 2003 and SQL Server 2000 have more known vulnerabilities than Redhat and Oracle. How can Windows Server 2003 be more secure when it is clear that it has more vulnerabilities?
I'm unable to find any statistics for 2005, but back in 2000, Linux accounted for 36% of webservers, and Windows only 21%, according to Netcraft. It's likely that this hasn't changed.
Windows is certainly more compatable with hardware and the majority of software binaries about, but more versatile? In what way?