Slashdot Mirror
Ask the Author of the Latest MS-Funded Windows vs. Linux Study
Last week on Slashdot you saw a (Microsoft-funded) research
study on Windows vs. (Novell) Linux reliability by Dr.Herbert
Thompson. Novell disagreed
with the study's conclusions. So did most Slashdot readers.
Thompson's work been mentioned on Slashdot before, especially his
famous five-line
script that could change electronic voting machine results
and his novel, The
Mezonic Agenda: Hacking the Presidency. He's a real,
genuine-article computer security expert (and regular Slashdot reader)
who is happy to put on his flame-resistant
suit and discuss his Microsoft vs. Linux study with you. So
ask whatever you like, one question per post. We'll send him 10 of the highest-moderated questions and publish his
answers next Monday. He'll jump into the discussion then, which ought
to make it rather lively.
Dr. Thompson:
Admittedly, I don't know who you are and I haven't read any of your books. Worse, I didn't read your study itself, only its conclusions as reported second-hand by the press. However my lack of knowledge of your backgound is probably consistant with most Slashdot readers and the IT industry as a whole. I have to give you the benefit of the doubt and assume that you are a capable, respected researcher elsewise MS wouldn't have approached you in the first place.
Could you please explain why you decided to risk drawing your objectivity into question by undertaking this project? Your findings may be 100% valid. And MS may very well have straight-up told you: "Please print whatever you find, even if it casts Windows in a bad light." However, who's going to believe it, even if it were true? If I were in your shoes, I'd be affraid that making a deal like this would ruin my career. If I don't tell MS what they want to hear, word would get out that I don't play ball. If I do report what's in the sponsor's best interest, a lot of people start accusing me of being a shill. Seems like a lose-lose proposition.
Entrepreneur : (noun), French for "unemployed"
...Will we see this as a dup on /. in about a month?
How can you stay neutral when one side is funding your research?
The study seemed to only compare comercial applications on the various platforms and not the alternatives. Its very common that comercial apps on Linux have poor support on Linux while the free alternatives blows most out of the water on Windows too. Its not especially hard to select a couple of apps with stellar support on Windows and SAP like support on Linux and blame Linux when the problem really lies in the lack of vendor support. Some vendors even support just one specific linux version without! any patches applied.
What care was taken in selecting applications with similar support offerings to not bias the study heavily to Microsofts advantage?
HTTP/1.1 400
How many Microsoft-funded studies have been buried because the conclusion was "incorrect"?
I find that there are too many variables plus unknowns to preemptively measure a TCO before a system has been installed and maintained and migrated to the next system. The maintenance is sometimes addressed, the end of life is rarely if ever addressed.
My personal bias is that Windows systems are good for being domain controllers and file servers for Windows clients, and the UNIX/Linux is better for your typical "headless" dull day to day server stuff like web servers, email, database servers, HPC machines, etc.
So my questions are: Are these studies worth anything more than pseudo-science advertisements, and if so why? And why is the end of life so rarely discussed?
Microsoft and Linux distros have had a policy for some time of including more and more functionality in the base operating system, the latest example is the inclusion of "Local Workflow" in Windows Vista.
As a security expert do you think that bundling more and more increases or decreases the risks, and should both Windows and Linux distros be doing more to create reduced platforms that just act as good operating systems.
An Eye for an Eye will make the whole world blind - Gandhi
I only skimmed over the public comments and your survey. My impression was that the sample period you chose was very small. Why so small? It seemed so small that it struck me as deliberate to get a predetermined outcome. I am not saying that was your intention but it does give the appearance that it could have been.
Have you considered increasing the sample period?
Keep the Classic Slashdot.
"As they attempt to increase business capabilities over time, customers are telling us that they are hitting a wall with Linux, experiencing significant reliability issues resulting in higher total cost of ownership," said Martin Taylor, general manager of platform strategy at Microsoft.
If scaling up on windows means significant reliability issues, how has google managed to avoid these despite scaling to the level they have?
Or Amazon, which I beleive also runs on linux. These are true enterprise level e-commerce apps, and despite the tons of studies saying they've picked the WRONG computing platform, places like google, amazon have amanged to create profitable businesses on non MS platforms.
What OS do you run personally - and why?
IE: If you run Windows is it because that is what they run at work? If it is an Open Source OS - is it because you believe in open source? If it is OSX - why wasn't it included in the study?
It seems that your study attempted to simulate the growth of an internet startup firm on Windows or Linux. One thing I did not see in the study was a good description of assumptions you made. What assumptions were made in both the design of the requirements and the analysis of the data? What limitations can we place on the conclusions as a result of these assumptions?
LedgerSMB: Open source Accounting/ERP
He was paid to evaluate two possible scenarios given a set of initial conditions. Researchers do it all the time in this place we like to call the "real world" - in engineering for example. You take a few alternative designs, apply the constraints you are given, and pick the right tool for the job.
Dr. Thompson was given a set of conditions and two contendors, he gave his evaluation, done deal. It doesn't imply endorsement. I'm an engineer - I evaluate options regularly. Sometimes I have to pick options I didn't like. But I do it because they are the right option for the given scenario. If the conditions were different the results probably would have been different.
-everphilski-
Altho I can understand that Novell are protecting their interests, the same could be said about microsoft.
Also, did Microsoft give you some procedures or methodology to follow in your study?
How many NDAs did you have to sign before starting the study? Did anyone pull you asside to "set the record streight" before the study began? How were you first asked about doing this study? Was it something like "hey, we need a study to boost our TCO stats, here's some cash..." or was it more altruistic like "hey, we need to see how we stack up agaist the competition .. heres some cash, and dont hold any punches!"
-GenTimJS
To be sarcastic, I'd ask "who the heck actually takes these studies seriously?", but obviously *somebody* does. Who are these people, and why do these people take these inudstry analyst firms/journals/reports seriously? Are they right or wrong to do so? This isn't an attack (or endorsement :) of your research -- I'm talking about the credibility gap in industry research, and my observation that it's an industry-wide problem.
The meta-credibility question is this: Given the amount of shoddy pay-for-play research out there, does being published in an analyst journal tend to cost (a researcher, his consulting company, his financial backers) more credibility than it can gains him/her/them? If not, why not -- and more importantly, if so, is there any way to reverse the trend?
Simple one: of course I accept that Windows and Linux are a priori equally vulnerable - C programmers make mistakes. the question is which model is most likely to deliver a fix fastest. Given that the one area where Linux is probably in the lead over Microsoft's software is in the realm of the webserver - why are my server logs filled with artifacts of hacked IIS boxes but apache seems to remain pretty safe?
Everyone on /. likes to complain about microsoft security, and microsoft PR people like to point out their improvements. Here's a chance to give ammunition to both sides. What do you think are the three biggest security improvements microsoft has made in the past two years, and what are the three biggest security-related issues that still remain?
"Weapons should be hardy rather than decorative" - Miyamoto Musashi
I think that goes for OS's too
In addition, Digital Rights Management or other copy protection schemes are becoming increasingly demanding and insidious, whether by uniquely identifying and reporting on user activity, intentionally restricting functionality, and even introducing new security issues (the most recent flap involves copy protection software on Sony CDs that not only hides content from the user but permits viruses to take advantage of this feature.)
I would like to know how you feel about the shift of control over the personal computer from the person to the software manufacturers -- is it right, and do we gain more than we're losing in privacy and security?
Try not. Do or do not, there is no try.
-- Dr. Spock, stardate 2822-3.
You tested six people on two different systems; how is that supposed to yield any substantial insight into the underlying OSes themselves?
[At best, your study seems to show that the GNU/Linux distribution you selected was not particularly good at this task. But why does that show that the ``monolithic" style of Windows is better per se than the ``modular" style of GNU/Linux distributions?]
"Every decent man is ashamed of the government he lives under." - H.L. Mencken
The Linux administrators faced some out of the ordinary challenges, not faced by most Linux admins, while the Windows admins faced none.
For example, most of the time difference between Windows and Linux was spent upgrading gLibC, something that you're really not supposed to do. It's comparable to trying to manually upgrade parts of a Windows 98 system to run a program that required XP, rather than actually upgrading to XP.
Then, you had the Linux admins getting updates from 4 different sources, rather than just from SuSE's repositories, which is also out of the ordinary, while the Windows admins only visited Windows Update, which only supplies patches to the base operating system, when in reality they'll have to get updates from many other sources if they wanted to keep their apps up to date.
Do you think this was a fair study?
Another concern I have is that while your study simulates the installation and upgrade of two different systems based upon two OS's, it does not seem to simulate the real-world work needed to keep those systems running on a daily basis. In the real world systems break, worms clog the network, and regular maintenance must be done. Your study seems to completely disregard all that work and focus only on install/upgrade. Why did you not base your study on the behaviors of a real working system with a simulated network attached? It seems like the shortcut method you used to quickly evaluate only certain tasks makes the study wholly academic and loses any value as a predictor for the operation of a real network, over time, with real traffic.
Finally, I've seen it suggested that this study requires that all software be updated to the latest versions, but While Linux based servers constantly release the latest patches to each component as they become available, Windows only releases them en masse, How then can you compare the two? To be perfectly fair one would have to know what development has happened on the various components of Windows and rate all of those components as failing to be updated (since MS has not yet released that version). Barring such inside information, any comparison between a system with an open development process and one with a closed development process is critically flawed. Do you not see this as a problem with your study?
Looking at your research report's appendices, it seems that the requirements for Windows Administrators were somewhat different than the Linux Administrators. For instance, you ask for 4-5 years sys admin experience minimum for Windows, whereas it's 3-4 years sys admin experience minimum for Linux.
Why wasn't it equal for both? And doesn't this sort of slight Windows favoring undermine your credibility?
Do you think there is reasonable evidence of vote tampering in the 2004 US Presidential election? Do you think the current batch of Diebold machines in Ohio or other electronic voting machines in use for that election are trustworthy?
How is it that Diebold can make ATM machines that will account for every last penny in a banking system, but they can't make secure electronic voting machines?
Also, does the flame-resistant suit come with its own matching tinfoil hat? (don't answer that one)
He who knows best knows how little he knows. - Thomas Jefferson
How do you sleep at night?
On top of a pile of money, surrounded by many beautiful ladies.
Kudos to you for braving the inevitable flames to answer people's questions here on Slashdot.
Read the EFF's Fair Use FAQ
Mr Thompson
I've always wondered exactly how much Linux based knowlege a writer should have in order to write a report on the TCO of Linux based networks and software.
How much Real World/In the Trenches experience do you have implementing and supporting large network and software applications that run Microsoft products compared to *nix based solutions?
Exactly how experienced are you with Linux? What is your favourite distro? How long have you been running Linux?
What is the best thing Windows does better than Linux?
What is the best thing Linux does better than Windows?
Have you ever contributed to an Open Source project or been part of an Open Source community?
Thanks
John the Kiwi
From the study:
What I find lacking is the business case for upgrading the OS. And why on earth would any enterprise with even the tiniest amount of foresight and planning deploy Windows 2000/SuSE 8 knowing they will upgrade to the next gen just one year later? (Not that there aren't plenty of enterprises who fit your model, not to mention IT workers seeking to "power level" their skills...)
Now, certainly there is value in trouble-free installs. But can you say with confidence a better upgrade experience is really a fair test of value? Especially when the entire install/patch/upgrade philosophy between Windows and Linux is so disparate?
In other words: It's no surprise that Windows will perform better on the treadmill, constantly upgrading is at the very core of Microsoft's profitability.
--
If I understand the study correctly, the windows side had to do nothing but set up a server to do a few different tasks over time and run windows update. The linux side had to have have multiple incompatible versions of their database server running simultaneously on a single system and had to run unsupported versions of software to do it.
Why wasn't the windows side required to run multiple versions of IIS or SQL server simultaneously? In real life if you need to run multiple database versions you use virtualization or multiple systems, especially if one requires untested software. You don't run some hokie unstable branch on the same system as everything else. Why was a linux solution picked that required this level of work? My other related question is, did any of the unix administrators question why there were being asked to do such a thing? For example, did they come back and say they need a license for vmware? If they did not they do not seem like very competent administrators in my opinion.
Of course, with this audience, you might want to say FireFox, or possibly Safari. I am curious if you use MS IE. (Though I'd like to hear "Opera, of course.")
.. paranoid crackpot leftover from the days of Amiga.
Windows administrators are forced to wait until Windows releases a patch for known vulnerabilities to upgrade their systems. Why, then, were the Linux administrators told to attempt to upgrade their systems before Novell had released newly packaged versions of MySQL? The entire point of a package management system is that administrators rely on companies like Novell to correct dependencies prior to deployment. Since Windows administrators have the same constraint (i.e., waiting for security updates to be released), it is an unfair and arbitrary difference that caused a lot of troubles.
Why did you compare the number of patches required to apply between the systems? This is not a measure of security. Windows patches are bundled and affect many parts of the operating system while Linux patches affect individual components. The overtone in your paper implied that fewer windows patches was in some way easier or more secure; what justification do you have for this assertation?
What is the rationale behind this? Were the Linux administrators required to restart at this point? This is an incredibly contrived situation; one can simply stop and re-start the process in question after the upgrade has completed.
Furthermore, the upgrade methodology questionable. Real companies use development and production servers and don't upgrade the production server until a reproduceable upgrade trajectory has been tested on the development server. The actions of these administrators imply that they had no such access, and that there was no possibility for backtracking or restarting after a failed step. Normally, one would expect the ability to nuke the development server and start over, rather than following a bad plan to worse conclusions.
A quick read of the report shows that the real losers here seem to be the Administrators. Some of the Linux admins "could not meet business requirements", and some were judged as failures by not using vendor-supplied solutions.
Isn't one of the points of running Linux servers the freedom to use solutions NOT supplied by the vendor? Is it even possible for the Microsoft admins to make changes that aren't fed from the vendor?
When the only tool you have is the "Upgrade" button, and the button doesn't work, what then? The advantage of Linux in administration is the flexibility to Make It Happen, even if the vendor sends you something broken.
I know good admins on Microsoft, and good ones on UNIX. They seem to Make It Happen no matter what, because that is their job. Making It Happen sometimes include custom fixes, that are documented, so you can undo them when the vendor comes through (hopefully) later.
So the Final Question is, why was it bad for the Linux admins to stray from vendor-supplied fixes, and why is the lack of flexibility on the Microsoft side a "win"?
Dr. Thompson, the way you selected the administrators seems to suggest a strong bias against Linux. In Appendix 3 (page 41), you recruited Windows administrators with at least 4-5 years of Windows administrator experience, while in Appendix 4 (page 43), you recruited Linux administators with just 2 years of Linux experience.
It seems that either you're a true Linux believer thinking that a Linux administrator can out-smart, out-perform a Windows administor with twice the experience, or that your experiment was setup to pit inexperienced Linux admins against experienced Windows admins.
So which is it?
How do your findings hold up against page 31 of the recent leaked MS Singularity OS research document found at ftp://ftp.research.microsoft.com/pub/tr/TR-2005-13 5.pdf, in which MS compares current versions of Windows XP, Linux and FreeBSD, only to show that Linux and FreeBSD outperform Windows XP?
Why do you suppose that MS would even consider building a new OS from the ground up, as they are doing with Singularity, if their current model already beats the competition?
OK, I've found and read the report now, and this is just bollocks. From the report:
So the test involved installing on SuSE 8 two applications that (effectively) required SuSE 9. Rather than upgrade to SuSE 9, the test mechanism required the operators to hack their systems to make this work. Some of them did this by taking the ill-advised step of compiling their own glibc; doing this broke the vendor supplied version of 'rpm', leaving them unable to undo their changes. Others did it by partially upgrading their system to SuSE 9 by installing SuSE 9 rpms over their SuSE 8 equivalents.
The Windows equivalent test worked fine because the equivalent applications that the Windows operators were required to install were intended for use with the version of Windows they had installed.
Basically, the test wasn't fair. If SuSE-9 dependent applications were to be used, then SuSE 9 should have been used as the basis of the test. If SuSE 8 had to be tested, then equivalent applications that functioned on SuSE 8 should have been found (chances are, slightly older versions of the same 2 apps would have functioned fine).
So, no, glibc wasn't "mucked up because SUSE's YAST was broken". The operators broke YAST by trying to install a glibc upgrade in order to use an application that wasn't compatible with the system they were running. The test was unrealistic; they weren't given the option of upgrading the system properly. They were told, "make this application run on this system." It's not surprising that some of them failed.
This is utter bollocks. See my analysis of the report in this comment.
They broke RPM by hand compiling glibc, not the other way around. It says so quite explicitly. They hand compiled glibc because they were asked to install (without upgrading to SuSE 9) an application that wasn't compatible with the version in SuSE 8.
See Appendix 5.
The commercial apps in question, though, had dependencies on (1) a very recent version of MySQL, and (2) a more recent version of glibc than is included in the version of SuSE in use. These two dependencies were the root cause of almost all the problems described in this paper.
Question: Were the "underlying assumptions" and basic methodology (which you very responsibly and sensibly do report in your study) dictated to you by Microsoft or some other external entity, or did you yourself come up with the test scenario?
I ask because the consensus around here seems to be that the conditions and methodology were cherry-picked to favor systems with single-vendor provenance and ease of initial installation, and do not include any real measures of operational stability or reliability.
Dr. Thompson.
You note yourself, in your study that the sample is based upon 6 system administrators/systems. That number is, as you yourself note, too small to be considered definitive. That being the case I would argue that this makes the report viable not as a decisionmaking tool but a marketing tool. Were I a CIO I would feel unwilling to base my conclusions soley on a sample size of 6. What is your opinion on this? Do you expect further, more statistically-significant, work to take place? Or do you feel that this is not a problem?
Your study is interesting, but without knowing the 3rd party tools and applications that were used in the test how can we know the results are valid? Without disclosure the results are irreproducible. My hypothesis is that many of the applications were very poorly supported for linux and well-supported for Windows, but without knowing the applications I can't know if this is true or not.