Ask the Author of the Latest MS-Funded Windows vs. Linux Study

Last week on Slashdot you saw a (Microsoft-funded) research study on Windows vs. (Novell) Linux reliability by Dr.Herbert Thompson. Novell disagreed with the study's conclusions. So did most Slashdot readers. Thompson's work been mentioned on Slashdot before, especially his famous five-line script that could change electronic voting machine results and his novel, The Mezonic Agenda: Hacking the Presidency. He's a real, genuine-article computer security expert (and regular Slashdot reader) who is happy to put on his flame-resistant suit and discuss his Microsoft vs. Linux study with you. So ask whatever you like, one question per post. We'll send him 10 of the highest-moderated questions and publish his answers next Monday. He'll jump into the discussion then, which ought to make it rather lively.

Comments

Why risk your creditibilty?

[XorNand]

Dr. Thompson:

Admittedly, I don't know who you are and I haven't read any of your books. Worse, I didn't read your study itself, only its conclusions as reported second-hand by the press. However my lack of knowledge of your backgound is probably consistant with most Slashdot readers and the IT industry as a whole. I have to give you the benefit of the doubt and assume that you are a capable, respected researcher elsewise MS wouldn't have approached you in the first place.

Could you please explain why you decided to risk drawing your objectivity into question by undertaking this project? Your findings may be 100% valid. And MS may very well have straight-up told you: "Please print whatever you find, even if it casts Windows in a bad light." However, who's going to believe it, even if it were true? If I were in your shoes, I'd be affraid that making a deal like this would ruin my career. If I don't tell MS what they want to hear, word would get out that I don't play ball. If I do report what's in the sponsor's best interest, a lot of people start accusing me of being a shill. Seems like a lose-lose proposition.

Re:Why risk your creditibilty?

[CrimsonSamurai]

Good question. I'd be scared to post anything pro-microsoft on here, as a large number of /. users are pro-linux and anti-microsoft. I myself, am not too biased one way or another. I believe at this time that both linux and windows have their places, and aren't in 100% direct competition.

Re:Why risk your creditibilty?

[Kjella]

Could you please explain why you decided to risk drawing your objectivity into question by undertaking this project? Your findings may be 100% valid. And MS may very well have straight-up told you: "Please print whatever you find, even if it casts Windows in a bad light." However, who's going to believe it, even if it were true? If I were in your shoes, I'd be affraid that making a deal like this would ruin my career. If I don't tell MS what they want to hear, word would get out that I don't play ball. If I do report what's in the sponsor's best interest, a lot of people start accusing me of being a shill. Seems like a lose-lose proposition.

Contrary to popular belief, except in circles like slashdot few people have trouble accepting a report that favors a commercial product from the market leader over a distro based on a free product from a minority player. It is after all the way most reports go (which tend to make little fuzz because preserving status quo isn't very exciting), and there's nothing uncommon about being commercially funded by one side. You make it sound like this is something rare and dangerous. It's common and everyday, even when the reports go against Linux. Is a Linux study funded by GNU/FSF/OSI/OSDL or whatever any more impartial? No. Do you have problems finding people doing it? [sarcasm]What? Wouldn't they be afraid to be considered communist hippies?[/sarcasm]. Most people go out there and try to make a honest living (or at least not further than sensationalizing headlines to draw page hits). Of course there's a small group delivering FUD on demand, but they are far from the majority.

Re:Why risk your creditibilty?

[miffo.swe]

What many of you miss to realize(Microsoft included), is that there are a large group of current Microsoft only customers that are unhappy with their current offerings. Just because someone is against Microsofts decisions doesnt meen they like Linux. Many just see Linux as a catalysator wich will free the market, push standards and make interopability more common between vendors. Its very rare with 100% Microsofts network still Microsoft refuses to support any standard that would make life for their customers easier. The constant steering towards 100% MS networks is pissing people off.

This really isnt about Linux its about making computers and their software be as standard as the internet.

Re:Why risk your creditibilty?

[Burz]

Is a Linux study funded by GNU/FSF/OSI/OSDL or whatever any more impartial? No.

I think many here would disagree. Nonprofits are not driven by motives which could be considered the mirrored opposite of commercial corporations. There is not the tremendous pressure to turn a profit (or some analog to monetary gain), and in your examples they're run by mere handfuls of individuals receiving very little compensation with only their reputations to fall back on. They represent what are largely hobbyists, almost to a maddening degree.

OTOH, in Microsoft we have a callow and selfish for-profit entity with a rather abusive track record right up through their financial, er, daliances with SCO.

Need I say more?

Given their dynamics and history, being so dismissive of FOSS organizations as to just say 'well, eveone's biased anyway' really doesn't seem like an acceptable attitude.

Re:Why risk your creditibilty?

[pdbogen]

You make it sound like this is something rare and dangerous. It's common and everyday, even when the reports go against Linux.

I'd just like to point out the fact that just because something is commona and everyday doesn't mean it's not dangerous or doesn't merit fighting against.

GP has a valid question that doesn't really imply one way or another that his findings were bad, or wrong; merely that many people will view them as both things, and (as the prompt seems to imply) since Dr. Thompson seems to be pretty above-board, why he would consent to expose himself to such an attitude.

If anything, the only assumption GP makes is that any reasonable and well-informed computer "expert" would choose Linux / FOSS over Windows / MS.

Re:Why risk your creditibilty?

[CrimsonSamurai]

That's not exactly what I meant. It just frustrates me that there is a good amount of slashdot users that absolutely hate microsoft and condemn all of their products. This isn't the correct thing to do. I myself am primarily a windows user, but I have dabbled in linux as well. Neither side has perfect products. I do agree that Linux is a good catalyst, but it is no replacement for windows in its current form. My biggest complaint is that it is way to hard to use for the average computer user. I consider myself a power user, and I still have struggles with getting some things to work right in linux. Linux often has the opposite problem where a user has the opportunity to customize many things and control them to their liking but, its just too damn hard sometimes. So basically I guess my point is that no OS is perfect at the current time, or really anywhere near perfect. My point was just that far too many slashdot readers like to condemn microsoft, when perhaps it is not appropriate.

Re:Why risk your creditibilty?

[James_Aguilar]

"Could you please explain why you decided to risk drawing your objectivity into question with insane paranoiac Slashdot readers . . ."

Corrected. I know it may seem like a troll, but I don't think it is. Something that a lot of the readers of this site don't understand is that not everyone thinks that Linux is the shit to the point of denying all evidence to the contrary. Don't get me wrong, I have one Linux-only computer that I use for work, my other is dual boot, and I like it. I love Linux both for its principles and because it allows me to do things that I can't normally do with Windows, BUT that does not mean that I believe its raw performance to be equal to that of a more heavily funded operating system. And you know what? That's OK. I'd still rather use it.

Re:Why risk your creditibilty?

[miffo.swe]

In my work as a network administrator i work more with Windows than i do with linux. Still after i have worked with Windows since Windows 95 and Linux since a couple of years back i still think its easier to manage linux servers. The ones having most difficulties with linux is the ones trying to use it like they was using Windows. You have to grasp the ground rules first and then linux aint hard at all. Finding an error in Linux is much easier than in Windows for eg. This is ofcourse if you dont see a reinstall as a successful error checking style. Personally i want to know why something break and how to prevent it the next time. Thats impossible with Windows from my experience.

Re:Why risk your creditibilty?

[Haeleth]

I think many here would disagree.

Hang on, you're saying you believe that you would trust a FSF or OSDL-funded study to be impartial? You're saying that if the FSF funded a study comparing GNU to Windows, and the study came back saying "Windows saves you money in the long term, and Microsoft's Shared Source is as good as Free Software for 99% of users", that the FSF would then be happy to publish that study?

I don't think so, and I suspect you won't either, if you pause to think about it.

Nonprofits are not driven by motives which could be considered the mirrored opposite of commercial corporations. There is not the tremendous pressure to turn a profit (or some analog to monetary gain), and in your examples they're run by mere handfuls of individuals receiving very little compensation with only their reputations to fall back on.

But that doesn't make them impartial! All it means is that the profit motive is replaced by other motives. And there are plenty.

Think about how much time the major contributors to free software projects put into those projects. Hours, days, months, years of personal time, freely given. Time that could have been spent earning money, or doing charitable work, or even just spending time with their families. Time that was wasted, if it turns out that the software they produced is not actually going to help many people do anything at all.

When you reach middle age, and the end starts to heave into sight on the horizon of your life, you start to get very, very uncomfortable about the idea that you might have devoted your precious time to an unworthy cause.

Being so dismissive of FOSS organizations as to just say 'well, eveone's biased anyway' really doesn't seem like an acceptable attitude.

What's dismissive about that? Microsoft really does think that everyone ought to use Microsoft software, and the FSF really does think that everyone ought to use free software. Everyone is biased. Pretty much everyone does have a pre-existing investment, either of time or money, in one of the options. And human nature does dictate that when you have an investment in something, you are biased towards accepting studies that support it and disregarding studies that don't.

What's wrong with telling the truth?

Re:Why risk your creditibilty?

[matt4077]

Maybe he did it because HE understands that research should not be guided by popularity. Go ask Galileo or Pythagoras.

Re:Why risk your creditibilty?

[rpdillon]

Copy: Ctrl-C
Cut: Ctrl-X
Paste: Ctrl-V

That uses one buffer (i.e. KDE's or Gnome's buffer). Not only is it standard, but it is the exact same shortcuts as those used in Windows!

X's buffer is used by:
Select: Copy
Middle-mouse : Paste

You don't need to know about one to use the other. They work independently of one another. I've been using Linux since 1998 and I've never has a problem cutting and pasting between application under Gnome, KDE and Ion3.

Config files? System wide are in /etc. User config files are in ~, usually as .application or, perhaps, in the case of KDE, in .kde subdirectory. Again, in almost 8 years of use, finding a config file has never been an issue.

I find it odd you would choose these things to complain about. Now, if you want to talk about wireless drivers, it's a whole different ballgame... =)

Re:Why risk your creditibilty?

[Antique Geekmeister]

There's a number of huge distinctions between the open source community, such as the FSF, and Microsoft. One of them is money: the other is that Microsoft has been caught tiime and time again lying in court, under oath, and breaking the clearest laws of intellectual property ownership, trade secret theft, coercion of witnesses, and fraud.

The FSF keeps its nose squeaky clean, because they know they have to to keep any respect from their members and from the world at large.

Re:Why risk your creditibilty?

[shadowbearer]

  When you reach middle age, and the end starts to heave into sight on the horizon of your life, you start to get very, very uncomfortable about the idea that you might have devoted your precious time to an unworthy cause.

  Which is supportive evidence that your argument is too biased in itself to consider.

  How is helping to produce freely given and very secure software for EVERYONE ELSE TO USE FREELY an unworthy cause? Indeed, one could (and I do) consider it "charitable work".

  Oh, and I'm on the approach ramp to middle age; and my feelings about helping only get stronger year after year.

  It's not the fact of bias that you're not considering, it's the *reasons* behind the bias. (Yeah, I've heard a lot of people say that altruism doesn't pay the bills. But as a blanket statement, that's bullshit).

SB

Re:Why risk your creditibilty?

[James_Aguilar]

I didn't play my thoughts out to their conclusion in my post, and for that I apologize. Let me write the rest of what I was thinking here.

What I meant to say was that, hey, this guy has a pretty good reputation for putting out good studies. Insane paranoiac people will say, he is getting paid to deliver a certain conclusion! Normal people will say, this guy has a reputation for honesty, I trust what he is saying. That is what I was trying to convey in my post.

Dear Sir....

[Anonymous Coward]

...Will we see this as a dup on /. in about a month?

My Question

[rolfwind]

How can you stay neutral when one side is funding your research?

Re:My Question

[garcia]

How can you stay neutral when one side is funding your research?

This isn't something that I think can be answered as no matter what he says most of the readership here won't believe him (myself included).

Regardless of any study *I* have interpreted data for, I'm always looking to slant it in *my* favor. There's no way that *any* one person is able to present a set of data, paid for or not, in a neutral manner.

Even if they can, we won't believe them unless it's for our side ;)

Re:My Question

[Decaff]

How can you stay neutral when one side is funding your research?

Because if you don't, no-one will fund your research again. Anyone can find marketing people and spin doctors. Quality researchers are hard to find, and if there is evidence of biased or forged research, their career is at an end.

Re:My Question

[miffo.swe]

No problem, just apply yourself with some integrity. Sadly this is a rarely seen trade theese days. This kid is an Microsoft MVP (Most Valuable Professional) so its not only the funding that makes it hard to see him as unbiased.

Re:My Question

[UserGoogol]

I don't know if that's true. Good biased research is done by actually conducting serious research and then selecting the most skewed way to interpret the data. This requires a bit more skill than just pulling numbers out of your ass.

Re:My Question

[peragrin]

Yankee Group
Garter,
Enderle consulting,

Do any of these names ring a bell? all but the last on claims to be unbaised but their reports can be shown in minutes to us predetermined Data.

So what was that about being neutral again? Which group is making more money than you do every minute?

Re:My Question

[slavemowgli]

Well, if you're not looking for a particular outcome, then it'll at least be easier for you to actually get to grips with unexpected outcomes instead of modifying the study until the results suit you - I think that's something that's a definite risk if you get paid by one side. Even when you still try to be objective, and even when you don't get any pressure whatsoever (explicit or implied) to come to a certain conclusion, I don't think you can truly be objective if one side is paying you.

When you're not being paid, though, it is possible. Take a look at science, for example; if you replace "study" with "experiment", you'll see that the whole thing isn't much different really, and in science, most people manage to be at least reasonably objective. You may already have an idea about how things work (that is, you may already have a scientific theory); but if you do an experiment and the results don't fit into your theory, then you'll most likely modify the theory until it works again. (Or you might even throw it away altogether, but I think that's less common.)

Of course, some scientists don't manage to do that and instead get so attached to a particular theory (for whatever reason) or come to dislike a new theory so much that they simply reject it on emotional grounds, without a real, rational, scientific reason. This is unfortunate, but it's not the norm - most scientists manage to be objective.

A good example might be Einstein and the "spooky action at a distance". Did he dislike it? Certainly. Did he think it couldn't be something that actually occured in reality? You betcha. But he didn't reject quantum mechanics because of it; rather, he tried to find a way to modify QM to get rid of what he believed was an artifact.

Of course, we nowadays pretty much know (to the extent that you can truly positively "know" something in science) that he was incorrect and that SAAAD really exists. But the point is that Einstein, while he did not like the idea of SAAAD, reacted in a scientific fashion instead of resorting to dirtier tactics like spreading FUD, buying studies, and all that, which unfortunately seems to be all too common outside of the scientific community.

Regarding Mr. Thompson, I'm not sure if he's a man of questionable ethics who doesn't mind creating a study that comes to a predetermined result if the money is right, or if he is naive enough to truly believe that he can be objective in a study commissioned by and paid for by Microsoft.

And just to make it clear, I don't want to comment on the findings of the study as such at all. I do have my own opinion, of course (Linux/Unix rox, Windows sux, and yes, I've used both for more than a decade), but I don't know exactly what question the study was supposed to evaluate, so I can't comment on it, naturally. I just think that the methodology in this case is rather fishy - praise for company A paid for by company A always is, even if it should be well-deserved.

Re:My Question

[FinchWorld]

Thats kind of funny, I had a cousin who came out of University with a Bsc in chemistry. One of the first jobs he got was research for BP (Oil company). Due to his prior diving expirience he was choosen to study the effects of deep sea drilling on local marine life.

His study showed that there was quite a significant effect on marine life, more than previous studies had shown.

He was thanked then disposed of within a month, his study left with him. Needless to say had the study said what BP wanted to here he would never have had such a rapid removal from BP.

Selection of applications.

[miffo.swe]

The study seemed to only compare comercial applications on the various platforms and not the alternatives. Its very common that comercial apps on Linux have poor support on Linux while the free alternatives blows most out of the water on Windows too. Its not especially hard to select a couple of apps with stellar support on Windows and SAP like support on Linux and blame Linux when the problem really lies in the lack of vendor support. Some vendors even support just one specific linux version without! any patches applied.

What care was taken in selecting applications with similar support offerings to not bias the study heavily to Microsofts advantage?

What about negative results?

[One Louder]

Do you believe your study would have been allowed to be published had the results turned out against them?

How many Microsoft-funded studies have been buried because the conclusion was "incorrect"?

Re:What about negative results?

[Cee]

How many Microsoft-funded studies have been buried because the conclusion was "incorrect"?

How would Dr. Thompson ever know that? Has he been in charge for a lot of MS-funded studies lately?

Aren't these studies worthless?

[hackstraw]

I find that there are too many variables plus unknowns to preemptively measure a TCO before a system has been installed and maintained and migrated to the next system. The maintenance is sometimes addressed, the end of life is rarely if ever addressed.

My personal bias is that Windows systems are good for being domain controllers and file servers for Windows clients, and the UNIX/Linux is better for your typical "headless" dull day to day server stuff like web servers, email, database servers, HPC machines, etc.

So my questions are: Are these studies worth anything more than pseudo-science advertisements, and if so why? And why is the end of life so rarely discussed?

Do you agree with Windows Local Workflow

[MosesJones]

Microsoft and Linux distros have had a policy for some time of including more and more functionality in the base operating system, the latest example is the inclusion of "Local Workflow" in Windows Vista.

As a security expert do you think that bundling more and more increases or decreases the risks, and should both Windows and Linux distros be doing more to create reduced platforms that just act as good operating systems.

Better question

[einhverfr]

It seems to me that the "study" was a simulation or a model. Since such simulations are inherently simplifications of real-world environments, what conclusions should we draw from this? In other words, what are the limitations of your method regarding the conclusions we can draw?

Sample window size

[Monoman]

I only skimmed over the public comments and your survey. My impression was that the sample period you chose was very small. Why so small? It seemed so small that it struck me as deliberate to get a predetermined outcome. I am not saying that was your intention but it does give the appearance that it could have been.

Have you considered increasing the sample period?

Not funded by Microsoft

[Daveznet]

If the same study was not funded by Microsoft and was funded by a company that supports Open source and the linux platform say google or IBM would your results have been the same?

Curious

[Anonymous Coward]

"As they attempt to increase business capabilities over time, customers are telling us that they are hitting a wall with Linux, experiencing significant reliability issues resulting in higher total cost of ownership," said Martin Taylor, general manager of platform strategy at Microsoft.

If scaling up on windows means significant reliability issues, how has google managed to avoid these despite scaling to the level they have?

Or Amazon, which I beleive also runs on linux. These are true enterprise level e-commerce apps, and despite the tons of studies saying they've picked the WRONG computing platform, places like google, amazon have amanged to create profitable businesses on non MS platforms.

Personal OS

[mchawi]

What OS do you run personally - and why?

IE: If you run Windows is it because that is what they run at work? If it is an Open Source OS - is it because you believe in open source? If it is OSX - why wasn't it included in the study?

A better way of putting it:

[einhverfr]

It seems that your study attempted to simulate the growth of an internet startup firm on Windows or Linux. One thing I did not see in the study was a good description of assumptions you made. What assumptions were made in both the design of the requirements and the analysis of the data? What limitations can we place on the conclusions as a result of these assumptions?

Re:A better way of putting it:

[Jerry Coffin]

One thing I did not see in the study was a good description of assumptions you made.

I guess I must have found a different study -- the one I found has a section titled "Assumptions and Rules" starting on page 11, then an "Additional assumptions on Quantitative Data" running from around the middle of page 12 through about the top third of page 14.

Is this a different one than you were looking at?

--
The universe is a figment of its own imagination.

File servers

[StupidKatz]

[..]Windows systems are good for being domain controllers and file servers for Windows clients [...]

Windows:
Client Access Licenses

Linux:
Samba

Additionally, software such as NIS exists to fill the role of a single-sign-on, although I've only had painful experiences with it, personally (using Solaris in a completely crazy setup).

Results

[everphilski]

He was paid to evaluate two possible scenarios given a set of initial conditions. Researchers do it all the time in this place we like to call the "real world" - in engineering for example. You take a few alternative designs, apply the constraints you are given, and pick the right tool for the job.

Dr. Thompson was given a set of conditions and two contendors, he gave his evaluation, done deal. It doesn't imply endorsement. I'm an engineer - I evaluate options regularly. Sometimes I have to pick options I didn't like. But I do it because they are the right option for the given scenario. If the conditions were different the results probably would have been different.

-everphilski-

Re:Results

[Znork]

The study had admins manually resolving dependency conflicts and borking their systems. I dont think 'right tool for the job' is even on the map if that's where the admins end up.

I mean, by whatever deitys protect sysadmins, _manually_ upgrade _glibc_??? I havent done that since before package systems were invented.

"If the conditions were different"

You mean, if the Windows admins spent most of their time manually copying files in dos shells from floppy disk because they for some inexplicable reason didnt want to use more modern methods for handling such problems?

If the conditions are to benchmark people doing things the wrong way then I rather doubt the value of the conclusions.

Re:Results

[Zathrus]

I can't think of one thing windows does better in the desktop or server market that Linux [and the scores of OSS tools] can't totally do better.

Debugger.

Sorry, ddd/gdb, Eclipse, and so forth are pale shadows to MS Visual Studio still, particularly for C and C++ work (Eclipse is probably the best for Java though).

Our code is cross platform Unix, but we maintain a Windows port for one reason only -- debugging. Using Visual Studio is far, far better than the alternatives. We vastly reduce the time involved in finding and eliminating bugs by doing so. And no, we don't sell the Windows version. It's literally only used for internal debugging.

Is there a technical reason why the Linux debuggers couldn't be better? Of course not. That's completely and utterly irrelevant (as are most of your "explanations" you attempt to give in rebuttal) though. The fact of the matter is they aren't, and are actually several years behind in comparison.

And no, I don't develop in Visual Studio -- I prefer vim. And we use CVS for source control. We use gmake (even on Windows) and other OSS tools too. Best tool for the job.

What do you have to say about Novell's statements?

[Spy der Mann]

Altho I can understand that Novell are protecting their interests, the same could be said about microsoft.

Also, did Microsoft give you some procedures or methodology to follow in your study?

What did MS say to you when they gave you funding?

[gentimjs]

How many NDAs did you have to sign before starting the study? Did anyone pull you asside to "set the record streight" before the study began? How were you first asked about doing this study? Was it something like "hey, we need a study to boost our TCO stats, here's some cash..." or was it more altruistic like "hey, we need to see how we stack up agaist the competition .. heres some cash, and dont hold any punches!" -GenTimJS

Meta-credibility?

[Tackhead]

Where I come from (non-management, grunt-level techie), appearing in any of these analysts' journals *costs* an author more credibility than it gains him or her. For example, if $RAG says that $CORP has the best customer support, I immediately assume that $CORP has such horrid customer support that they had to pay someone to make up some research that proves otherwise.

To be sarcastic, I'd ask "who the heck actually takes these studies seriously?", but obviously *somebody* does. Who are these people, and why do these people take these inudstry analyst firms/journals/reports seriously? Are they right or wrong to do so? This isn't an attack (or endorsement :) of your research -- I'm talking about the credibility gap in industry research, and my observation that it's an industry-wide problem.

The meta-credibility question is this: Given the amount of shoddy pay-for-play research out there, does being published in an analyst journal tend to cost (a researcher, his consulting company, his financial backers) more credibility than it can gains him/her/them? If not, why not -- and more importantly, if so, is there any way to reverse the trend?

Re:Meta-credibility?

[Mad_Rain]

Where I come from (non-management, grunt-level techie), appearing in any of these analysts' journals *costs* an author more credibility than it gains him or her. For example, if $RAG says that $CORP has the best customer support, I immediately assume that $CORP has such horrid customer support that they had to pay someone to make up some research that proves otherwise.

So who do you go to when you have question then? Eventually you have to trust somebody when it comes to a topic that you've reached the limits of your knowledge of.

More importantly, when you make a decision, and something goes wrong, how do you explain your decisions? You could say "I used the recommendations from $RAG about $CORP." Or you could say "I didn't trust the people at $RAG, but I asked my colleagues Jimbo and Cleatus," and have a difficult time explaining problems to the non-techie management.

The meta-credibility question is this: Given the amount of shoddy pay-for-play research out there, does being published in an analyst journal tend to cost (a researcher, his consulting company, his financial backers) more credibility than it can gains him/her/them? If not, why not -- and more importantly, if so, is there any way to reverse the trend?

I think once you figure out who to ask, and that multiple people can agree that the source is reliable, then it's just a matter of getting people to use that.

Apache versus IIS

[00_NOP]

Simple one: of course I accept that Windows and Linux are a priori equally vulnerable - C programmers make mistakes. the question is which model is most likely to deliver a fix fastest. Given that the one area where Linux is probably in the lead over Microsoft's software is in the realm of the webserver - why are my server logs filled with artifacts of hacked IIS boxes but apache seems to remain pretty safe?

what are the biggest issues

[evenprime]

Everyone on /. likes to complain about microsoft security, and microsoft PR people like to point out their improvements. Here's a chance to give ammunition to both sides. What do you think are the three biggest security improvements microsoft has made in the past two years, and what are the three biggest security-related issues that still remain?

Convenience vs. security

[Sheetrock]

Lately, I've felt that Microsoft is emphasizing greater trust in their control over your system as a means of increasing your security. This is suggested by the difficulty of obtaining individual or bulk security patches from their website as opposed to simply loading Internet Explorer and using their Windows Update service, the encouragement in Service Pack 2 of allowing Automatic Update to run in the background, and the introduction of Genuine Advantage requiring the user to authenticate his system before obtaining critical updates such as DirectX.

In addition, Digital Rights Management or other copy protection schemes are becoming increasingly demanding and insidious, whether by uniquely identifying and reporting on user activity, intentionally restricting functionality, and even introducing new security issues (the most recent flap involves copy protection software on Sony CDs that not only hides content from the user but permits viruses to take advantage of this feature.)

I would like to know how you feel about the shift of control over the personal computer from the person to the software manufacturers -- is it right, and do we gain more than we're losing in privacy and security?

Scalability of Results?

[hahiss]

You tested six people on two different systems; how is that supposed to yield any substantial insight into the underlying OSes themselves?

[At best, your study seems to show that the GNU/Linux distribution you selected was not particularly good at this task. But why does that show that the ``monolithic" style of Windows is better per se than the ``modular" style of GNU/Linux distributions?]

Do you think the study was fair?

[dtfinch]

The Linux administrators faced some out of the ordinary challenges, not faced by most Linux admins, while the Windows admins faced none.

For example, most of the time difference between Windows and Linux was spent upgrading gLibC, something that you're really not supposed to do. It's comparable to trying to manually upgrade parts of a Windows 98 system to run a program that required XP, rather than actually upgrading to XP.

Then, you had the Linux admins getting updates from 4 different sources, rather than just from SuSE's repositories, which is also out of the ordinary, while the Windows admins only visited Windows Update, which only supplies patches to the base operating system, when in reality they'll have to get updates from many other sources if they wanted to keep their apps up to date.

Do you think this was a fair study?

Re:Do you think the study was fair?

[miffo.swe]

The glibc upgrade was if i understood it correctly done instead of just compiling MySQL manually. I was boggled, why would you change glibc instead of making a fast compile? Upgrading glibc will make the whole OS and all its applications unstable, its almost as it was intentional. I have a hard time beleiving a seasoned linuxadmin would do such a stupid thing as upgrading glibc.

Its smells funny indeed.

Theroy is not Practice

[99BottlesOfBeerInMyF]

I haven't had time to read through this study in its entirety, but from what I have seen it looks like there are several things that could lead it to improper conclusions. First, some of the procedures used to define the benchmarks seem completely arbitrary. For example, security fixes are applied on a monthly basis, rather than as they become available and can be tested, or based upon their severity. In my experience some security patches are tested and applied immediately and others are applied later. Also, no mention is made of mitigating the effects of security vulnerabilities/exploits before a patch is available which, while uncommon does happen. Coincidentally, Microsoft has moved Windows to a monthly patch release cycle. Don't you think defining the operating procedures to be exactly those used by one OS, and not the other biases the test?

Another concern I have is that while your study simulates the installation and upgrade of two different systems based upon two OS's, it does not seem to simulate the real-world work needed to keep those systems running on a daily basis. In the real world systems break, worms clog the network, and regular maintenance must be done. Your study seems to completely disregard all that work and focus only on install/upgrade. Why did you not base your study on the behaviors of a real working system with a simulated network attached? It seems like the shortcut method you used to quickly evaluate only certain tasks makes the study wholly academic and loses any value as a predictor for the operation of a real network, over time, with real traffic.

Finally, I've seen it suggested that this study requires that all software be updated to the latest versions, but While Linux based servers constantly release the latest patches to each component as they become available, Windows only releases them en masse, How then can you compare the two? To be perfectly fair one would have to know what development has happened on the various components of Windows and rate all of those components as failing to be updated (since MS has not yet released that version). Barring such inside information, any comparison between a system with an open development process and one with a closed development process is critically flawed. Do you not see this as a problem with your study?

Why are the requirements different?

[altoz]

Looking at your research report's appendices, it seems that the requirements for Windows Administrators were somewhat different than the Linux Administrators. For instance, you ask for 4-5 years sys admin experience minimum for Windows, whereas it's 3-4 years sys admin experience minimum for Linux.

Why wasn't it equal for both? And doesn't this sort of slight Windows favoring undermine your credibility?

Somewhat less germaine

[killmenow]

Do you think there is reasonable evidence of vote tampering in the 2004 US Presidential election? Do you think the current batch of Diebold machines in Ohio or other electronic voting machines in use for that election are trustworthy?

ATMs vs. Voting Machines

[digitaldc]

How is it that Diebold can make ATM machines that will account for every last penny in a banking system, but they can't make secure electronic voting machines?

Also, does the flame-resistant suit come with its own matching tinfoil hat? (don't answer that one)

OBSimsons...

[schon]

How do you sleep at night?

On top of a pile of money, surrounded by many beautiful ladies.

Who determined the metrics

[Infonaut]

Did Microsoft come to you with a specific set of metrics, or did you work with them to develop the metrics, or did you determine them completely on your own?

Kudos to you for braving the inevitable flames to answer people's questions here on Slashdot.

How do you explain the different conclusions?

[gorbachev]

Dr. Thompson,

How do you explain the different conclusions from studies funded by Microsoft and studies funded by Unix/Linux vendors? Shouldn't studies that essentially study the same issue inevitably arrive in the same conclusions, if the research for the study was made independently, honestly and with no systemic errors? How do you expect people to take any of these studies, whether pro-Microsoft or anti-Microsoft, seriously?

Re:Just goes to show that...

[^_^x]

As always, but while this site claims tech coverage, it's mostly LINUX tech coverage.

What I want to know is what's a fair comparison anyway?

If it's MS-funded, it's probably skewed to Windows.
If it's performed by Linux advocates, it's skewed.
If it's done by a research company that doesn't care either way, they end up ruling that Linux is hard to use, and the /. crowd moans that they didn't set up the boxes right. Well, if it's not obvious how to set them up properly, that makes it pretty hard to use, doesn't it?

I say just ask around in the IT community, though it would really depend on who you know. Most people I've talked to either marginalize it as a non-option (my old boss, when I was extolling the virtues of Redhat 6.4...), consider it as a plague (most of my peers), or a neccesary evil you'll probably have to know sooner or later (my college profs.) Personally, I think it's a brilliant OS-extended-family-and-then-some for a server, but masochistic on the desktop.

So... what's a fair comparison? Even if they loaded a study with a team of Linux gurus, and Windows reps to set up the systems properly, that'd hardly be a realistic environment for either. :/

Real World Experience

[John the Kiwi]

Mr Thompson

I've always wondered exactly how much Linux based knowlege a writer should have in order to write a report on the TCO of Linux based networks and software.

How much Real World/In the Trenches experience do you have implementing and supporting large network and software applications that run Microsoft products compared to *nix based solutions?

Exactly how experienced are you with Linux? What is your favourite distro? How long have you been running Linux?

What is the best thing Windows does better than Linux?

What is the best thing Linux does better than Windows?

Have you ever contributed to an Open Source project or been part of an Open Source community?

Thanks

John the Kiwi

"Real World" scenario

[Gehenna]

If this is a "real world" scenario why is a default install picked? Part of the job description for a sysadmin is to secure a system. If this install "attempted to simulate a "real-world" enterprise e-commerce environment over the course of a year." then how could it be the default configuration? The bugzilla example you annote is for samba, not port of a reasonable database server install.

Also is there a list of the vulnerabilities quantifued in your study?

What are you talking about?

[NineNine]

What commercial apps on Linux did he use, exactly? I just looked over the report, and I saw Apache, PHP, GLIBC, and MySQL. I'd argue that comparing MySQL to MS SQL Server is like comparing a bicycle to a BMW, but still, MySQL, PHP, GLIBC, and Apache are probably the best supported Linux-based apps on the planet. Did you even read the report?

Re:What are you talking about?

[julesh]

What commercial apps on Linux did he use, exactly?

See Appendix 5.

During the experimental trials, 3rd party best-of-breed components were chose to satisfy the needs of the solution. [...] The specific 3rd party vendors are not disclosed because the focus of the study is the methodology and not a specific component.

The commercial apps in question, though, had dependencies on (1) a very recent version of MySQL, and (2) a more recent version of glibc than is included in the version of SuSE in use. These two dependencies were the root cause of almost all the problems described in this paper.

Integrity

[everphilski]

Its called integrity... I take it you've never done scientific research before (and if you have, shame on you)

-everphilski-

Data Mining Software issue

[arevos]

The Data Mining Software used in M1 required the Linux administrators to use MySQL 4.1, which was not part of the SLES distribution. This appears to be where the majority of the problems with the Linux servers stemmed from. Do you think the choice of Linux distribution and/or Data Mining Software biased the outcome report in any way?

Can TCO be extrapolated from install behavior?

[Qrlx]

I wonder if it's really appropriate to make TCO guesstimates from a study which essentialy asks the question "Which OS has nicer installers?"

From the study:

We conducted an experiment pitting Windows 2000 Server against SuSE Linux
Enterprise Server 8, simulating [a] one year period...At the end
of the period, both systems are then transitioned to the more recent versions of their
respective operating systems, Windows Server 2003 and SuSE Linux Enterprise Server 9.

What I find lacking is the business case for upgrading the OS. And why on earth would any enterprise with even the tiniest amount of foresight and planning deploy Windows 2000/SuSE 8 knowing they will upgrade to the next gen just one year later? (Not that there aren't plenty of enterprises who fit your model, not to mention IT workers seeking to "power level" their skills...)

Now, certainly there is value in trouble-free installs. But can you say with confidence a better upgrade experience is really a fair test of value? Especially when the entire install/patch/upgrade philosophy between Windows and Linux is so disparate?

In other words: It's no surprise that Windows will perform better on the treadmill, constantly upgrading is at the very core of Microsoft's profitability.
--

Weak setup

[0xABADC0DA]

If I understand the study correctly, the windows side had to do nothing but set up a server to do a few different tasks over time and run windows update. The linux side had to have have multiple incompatible versions of their database server running simultaneously on a single system and had to run unsupported versions of software to do it.

Why wasn't the windows side required to run multiple versions of IIS or SQL server simultaneously? In real life if you need to run multiple database versions you use virtualization or multiple systems, especially if one requires untested software. You don't run some hokie unstable branch on the same system as everything else. Why was a linux solution picked that required this level of work? My other related question is, did any of the unix administrators question why there were being asked to do such a thing? For example, did they come back and say they need a license for vmware? If they did not they do not seem like very competent administrators in my opinion.

Security and the web

[whitehatlurker]

Given that you are in the field of security and that there is much concern about security holes in web browsers, I'll ask: "What web browser do you use, and why?"

Of course, with this audience, you might want to say FireFox, or possibly Safari. I am curious if you use MS IE. (Though I'd like to hear "Opera, of course.")

Doesn't change the TCO

[everphilski]

But the thing is why should they port to Linux? Why should I purchase Linux versions of software when I already own the Linux versions? So I can say I'm cool and run Linux? No. The cost of a windows license is next to nothing and the cost of the software will be the same on either platform; and when you are talking TCO of engineering software the engineering software costs run in the thousands to tens of thousands of dollars. When we buy our workstations from Dell/Xi/any bulk vendor the windows license runs about $10-$30. Whats the point of recoding part of the software, in the pov of the engineering vendor, to avoid $10-$30 windows license? That's absurd.

-everphilski-

PHBs who listen

[Mr. Underbridge]

To be sarcastic, I'd ask "who the heck actually takes these studies seriously?", but obviously *somebody* does. Who are these people, and why do these people take these inudstry analyst firms/journals/reports seriously?

First, let's recognize that anyone experienced enough with both operating systems will have their own experiences that will tell them which OS is better in various ways. These people are unlikely to be swayed by studies. Therefore, the first thing that is critical to understand is this: these studies are aimed at people who are NOT experienced with both OS's.

As such, it seems there are two potential groups who are targeted by such studies: 1) CIO or sysadmin types who are experienced with windows systems, and who were thinking of trying linux; and 2) PHBs. For the first type, the MS studies are meant to deter. For the second type, the MS studies are meant to indoctrinate.

For example, let's say MS saturates WSJ, Fortune, and similar newspapers/magazines likely to be read by PHBs. They read it enough times, and given they have no field knowledge of the various TCO variables, they believe what they read from seemingly "objective" sources. What MS then wants is this: when an intelligent CIO or sysadmin goes to the CEO and says "Let's try linux, it's great!" the CEO says no, and considers the CIO incompetent for even considering such a blatantly horrible idea.

So basically these studies are meant to influence decision makers who don't have hands-on knowledge. It's a very good idea, really. It will keep Linux adoption a lot lower than it would be otherwise.

A Few Comments:

[abscondment]

1. Windows administrators are forced to wait until Windows releases a patch for known vulnerabilities to upgrade their systems. Why, then, were the Linux administrators told to attempt to upgrade their systems before Novell had released newly packaged versions of MySQL? The entire point of a package management system is that administrators rely on companies like Novell to correct dependencies prior to deployment. Since Windows administrators have the same constraint (i.e., waiting for security updates to be released), it is an unfair and arbitrary difference that caused a lot of troubles.

2. Why did you compare the number of patches required to apply between the systems? This is not a measure of security. Windows patches are bundled and affect many parts of the operating system while Linux patches affect individual components. The overtone in your paper implied that fewer windows patches was in some way easier or more secure; what justification do you have for this assertation?

3. While kernel patches did not require an immediate reboot during installation, the majority of them need a system restart to immunize the system against a specific vulnerability.

   -Page 25, under "Patching and Milestone Upgrades"

   What is the rationale behind this? Were the Linux administrators required to restart at this point? This is an incredibly contrived situation; one can simply stop and re-start the process in question after the upgrade has completed.

4. Furthermore, the upgrade methodology questionable. Real companies use development and production servers and don't upgrade the production server until a reproduceable upgrade trajectory has been tested on the development server. The actions of these administrators imply that they had no such access, and that there was no possibility for backtracking or restarting after a failed step. Normally, one would expect the ability to nuke the development server and start over, rather than following a bad plan to worse conclusions.

constructive question

[petantik f00l]

You conclude from the study that at the enterprise level it is easier to manage Windows in regard to implementing business requirements than it is in Linux. I believe that Linux can and will be as good as Windows and to this end I ask what can we, the community and Linux vendors do to improve this failing i.e. what would you suggest that Linux could do or needs to do to be on par with Windows or even exceed it in this context?

I agree with your point

[Oldsmobile]

I agree. I am a power user, I suppose, and have had computers set up with Linux. I find certain things on Linux much better than on Windows machines, but taken as a whole and looking at the things I do everyday, Windows comes out on top. It really isn't a case of "operating system X is crap and Z is simply wonderfull" but a case of looking at what your needs are and what system works best for you. I do believe that Linux has the very strong potential of overcoming it's weaknesses and would in that case truly win over Windows. However, we are not there yet so in the meanwhile, Windows will do. Also, Windows will probably work on getting better and perhaps Apple will come closer to the proletariat equipment wise, and make it a three way match.

Re: Your sig

[Krach42]

effect:
tr.v. effected, effecting, effects
1. To bring into existence.
2. To produce as a result.
3. To bring about. (*See Usage Note at affect*).

Usage Note: Affect and effect have no senses in common. As a verb affect is most commonly used in the sense of "to influence" (how smoking affects health). Effect means "to bring about or execute": layoffs designed to effect savings. Thus the sentence These measures may affect savings could imply that the measures may reduce savings that have already been realized, whereas These measures may effect savings implies that the measures will cause new savings to come about.

Either way, it's wrong to say that "effect" is not a verb... in fact, it is.

This is besides the whole point that the sibling post made, that it's Grammar, not Grammer.

Administrator Skill Test

[fdisk3hs]

A quick read of the report shows that the real losers here seem to be the Administrators. Some of the Linux admins "could not meet business requirements", and some were judged as failures by not using vendor-supplied solutions.
Isn't one of the points of running Linux servers the freedom to use solutions NOT supplied by the vendor? Is it even possible for the Microsoft admins to make changes that aren't fed from the vendor?
When the only tool you have is the "Upgrade" button, and the button doesn't work, what then? The advantage of Linux in administration is the flexibility to Make It Happen, even if the vendor sends you something broken.
I know good admins on Microsoft, and good ones on UNIX. They seem to Make It Happen no matter what, because that is their job. Making It Happen sometimes include custom fixes, that are documented, so you can undo them when the vendor comes through (hopefully) later.
So the Final Question is, why was it bad for the Linux admins to stray from vendor-supplied fixes, and why is the lack of flexibility on the Microsoft side a "win"?

Linux admins twice as smart as Windows admins?

[epan47]

Dr. Thompson, the way you selected the administrators seems to suggest a strong bias against Linux. In Appendix 3 (page 41), you recruited Windows administrators with at least 4-5 years of Windows administrator experience, while in Appendix 4 (page 43), you recruited Linux administators with just 2 years of Linux experience.

It seems that either you're a true Linux believer thinking that a Linux administrator can out-smart, out-perform a Windows administor with twice the experience, or that your experiment was setup to pit inexperienced Linux admins against experienced Windows admins.

So which is it?

MS's own internal studies don't agree with you

[ramsejc]

How do your findings hold up against page 31 of the recent leaked MS Singularity OS research document found at ftp://ftp.research.microsoft.com/pub/tr/TR-2005-13 5.pdf, in which MS compares current versions of Windows XP, Linux and FreeBSD, only to show that Linux and FreeBSD outperform Windows XP?

Why do you suppose that MS would even consider building a new OS from the ground up, as they are doing with Singularity, if their current model already beats the competition?

Linux by other names

[Heembo]

Dr Thompson, Thanks for sticking your neck out. My question is: has your research given you enough data to provide feedback on other flavors of Linux? I suspect that Microsoft chose to pick on light-weight Novel since their flavor or Linux is one of the relative weakest. Any thought on that?

Re:If you actually read the report...

[julesh]

If you actually read the report, you'd see that GLIBC was all mucked up because SUSE's YAST was broken. And on top of that, part of the study was to see what the administrators would do. Part of the confusion for the sysadmins was WHERE to get the sources when the standard RPM manager broke. It's not clear where they should have gotten GLIBC, and that was part of the test.

OK, I've found and read the report now, and this is just bollocks. From the report:

In the Linux case, the component required an upgrade of the MySQL database component from version 3.23 to version 4.1. Upgrading MySQL means going outside of the supported OS configuration and obtaining the new version from the package distribution site.
[...]
[T]he search component required a newer version of the GLIBC package than the one shipped with SLES 8. There are many ways to resolve such a dependency including trying to get the two versions to co-exist, upgrading the existing version, etc. Upgrading GLIBC directly (a route two administrators took), quickly leads to a cascading sea of dependency failures as other packages on the system that rely on the older version fail. One such failure came in the RPM package installer which left no direct route to reinstall the old version of the GLIBC library.

So the test involved installing on SuSE 8 two applications that (effectively) required SuSE 9. Rather than upgrade to SuSE 9, the test mechanism required the operators to hack their systems to make this work. Some of them did this by taking the ill-advised step of compiling their own glibc; doing this broke the vendor supplied version of 'rpm', leaving them unable to undo their changes. Others did it by partially upgrading their system to SuSE 9 by installing SuSE 9 rpms over their SuSE 8 equivalents.

The Windows equivalent test worked fine because the equivalent applications that the Windows operators were required to install were intended for use with the version of Windows they had installed.

Basically, the test wasn't fair. If SuSE-9 dependent applications were to be used, then SuSE 9 should have been used as the basis of the test. If SuSE 8 had to be tested, then equivalent applications that functioned on SuSE 8 should have been found (chances are, slightly older versions of the same 2 apps would have functioned fine).

So, no, glibc wasn't "mucked up because SUSE's YAST was broken". The operators broke YAST by trying to install a glibc upgrade in order to use an application that wasn't compatible with the system they were running. The test was unrealistic; they weren't given the option of upgrading the system properly. They were told, "make this application run on this system." It's not surprising that some of them failed.

Re:If you'd read the study...

[julesh]

This is utter bollocks. See my analysis of the report in this comment.

They broke RPM by hand compiling glibc, not the other way around. It says so quite explicitly. They hand compiled glibc because they were asked to install (without upgrading to SuSE 9) an application that wasn't compatible with the version in SuSE 8.

Debugging in Linux

[Peaker]

While the Visual Studio debugger has some nice features that gdb frontends lack:

• Partial recompilation during runtime
  
• Convinient stepping into assembly code
  

The two are largely equivalent.

I use emacs gdbsrc mode to debug my code, and I can set breakpoints, conditional breakpoints, step in, step over, print any expression, or call any function I want in the debugger. If I recall correctly, you cannot really manually call functions in the Visual Studio debugger, but correct me if I'm wrong.

There are also advantages to gdb frontends though:

• They are more scriptable. You can run write code to execute at the debug breakpoint, not only for conditionally breakpointing, but also to modify the behaviour of the program.
  
• The same debugger can debug accross multiple languages (this may be true with VS.Net, I have used the VS6 debugger).
  

Please explain what extra productivity or features you gain from the Windows debugger.

As for your selection of tools:

• vim: I prefer Emacs :-)
  
• gmake: Nice for tiny projects. Does not scale up. There are better alternatives (SCons, Python's distutils, Ant, etc).
  

Missing the point

[Peaker]

The point is not that Linux is inherently less powerful.

Its that for certain kinds of purposes, the current situation in the real world, is that, for no good technical reason, software only exists for Windows.

Due to this unfortunate situation, Windows is superior at achieving certain real world tasks.

People who just accept this and go through the path of ethical lazyness get bitten in the ass by the lockin they are themselves creating.

Re:If you actually read the report...

[Fallingcow]

Seriously, what were they thinking? Using an outdated version of SuSE then forcing them to upgrade individual packages to the latest version? F'ing crazy.

They should have had them on Debian Stable or Slackware. For fuck's sake, Gentoo would have been a better choice for this than SuSE. RPM-based distros always seem to be the hardest to change or upgrade piecemeal, without doing a full upgrade to the latest version of the whole OS. I've used Mandrake and Fedora extensively, and pre-Fedora Red Hat and SuSE quite a bit, too. They all have these sorts of problems. You learn to be very careful with upgrades to individual packages, and you learn to upgrade to the newest version of the OS at the first sign of trouble with a package upgrade, before you've dicked with it so much that the system gets broken. If you can't do that for whatever reason, then you use a different distro. Simple as that.

Eh, this is mostly just a "me too!" post... but damn, that's just so dumb that I had to say something!

Did Microsoft dictate your base assumptions?

[Medievalist]

Question: Were the "underlying assumptions" and basic methodology (which you very responsibly and sensibly do report in your study) dictated to you by Microsoft or some other external entity, or did you yourself come up with the test scenario?

I ask because the consensus around here seems to be that the conditions and methodology were cherry-picked to favor systems with single-vendor provenance and ease of initial installation, and do not include any real measures of operational stability or reliability.

Ponder

[cnerd2025]

Dr. Thompson,

Though your study pits Windows versus Linux and claims Linux has a higher TCO, what is the actual marginal cost of implementing a Linux box versus a Windows box? Only three machines seems hardly determinant or significant. Implementing one Linux machine may be (although I don't believe it) more expensive, but several Linux machines may cost less than the same number of Windows boxes.

Also, with respect to updates, did you consider all of the upgrades in Red Hat's "up2date" as "patches" or simply as "upgrades" with a few being security patches.

Did your study favor GUI over command-line interface or vice-versa?

Did your study log each crash/reboot/system error thrown by each machine? Also, were you required to run any "system restores" on the Windows machine?

Did your study consider alternative operating systems with high security (such as OpenBSD)?

If you could "fix" Linux (or at least the distros you reviewed), what would you insert, update, or delete?

Would you consider running the same study with a very powerful package management system, such as APT?

Thank you,
Drew E.

Statistical Significance.

[Irvu]

Dr. Thompson.

You note yourself, in your study that the sample is based upon 6 system administrators/systems. That number is, as you yourself note, too small to be considered definitive. That being the case I would argue that this makes the report viable not as a decisionmaking tool but a marketing tool. Were I a CIO I would feel unwilling to base my conclusions soley on a sample size of 6. What is your opinion on this? Do you expect further, more statistically-significant, work to take place? Or do you feel that this is not a problem?

Vendor Tools

[YoJ]

Your study is interesting, but without knowing the 3rd party tools and applications that were used in the test how can we know the results are valid? Without disclosure the results are irreproducible. My hypothesis is that many of the applications were very poorly supported for linux and well-supported for Windows, but without knowing the applications I can't know if this is true or not.

Schedule, Requirements, and Basis.

[Irvu]

So far as I can tell, the essence of your study, and your conclusions rests on the following assumptions:

1. The set of requirements listed is a natrual one.
2. The schedule is a normal business schedule.
3. The method by which the components were selected, ordered, and applied was natural.

You state in your report that the requirements were developed after interviews with "leading CIO's, CTO's, ..." Nowhere do you state who conducted those interviews and, crucially, how many of them had overlapping requirements. Similarly, you do not state how you selected the particular schedule of your study both in terms of the product-period that you examined and the feature schedule you considered.

Moreover, in appendix 5 of your study you show little overlap between the lists of popular component users. Many of the groups listed for one "popular solution" were not listed on another. Nor did you separate these lists by operating system. This give no indication whether the popular components are ever used in concert. Nor does it indicate how many groups are using each feature set or system. Nor even where these user numbers came from.

I bring these points up because they point to potential holes in your study that I am curious about. In particular:

1. If either Microsoft or Novell supplied either the requirements list and/or the upgrade schedule then the study is vulnerable to the assertion that they schose a schedule, time, etc that was most favorable to them.
2. If the most popular 'component' solution to any one task is used only by one group but not another then this may point to general incompatibilities between them. When making a purchasing decision I typically consider the current state of my system and potential compatibilities with future upgrades. If the components selected for either os are not typically used in concert then this raises the possiblity that the components have known incompatabilities that would keep them from being used together. In that event the system administrators would be installing packages that are not meant to go together and would not be selected by a real-world selection metric, and as a result faced unrealistic issues.
3. In your selection of "popular components" you focused on 3rd party solutions. Nowhere do you state whether you considered only commercial vendors of such solutions or open-source vendors as well. If you focused soley on commercial products that might mean that the system administrators were actually installing less popular, or less viable products on the linux side given the lower amount of such vendors for the Linux platform.
4. If the schedule of upgrades was not one used by many of the real world companies but again, an artificial one constructed as a superset, then the study is vulnerable to the charge that it used an unrealistic technical schedule for installing and testing components.

My question is, do you see these as issues? If not why not?

Upgrade Glibc?

[oli_freyr]

Hello Dr. Thompson

First of all, thank you for participating in this flamefest ;)
I read the Executive Summary of your report and skimmed the rest, so pardon me if I failed to notice something vital.

It seems to me that the demand that your Linux Admins were asked to upgrade Glibc led them to fail the majority of tasks, creating an artificial bias against Linux.

Any Admin worth his weight in pizza knows that you Just Don't Do That.
If you absolutely, positively need some component, you get the version which works with your Glibc. All hell will break loose as soon as you upgrade Glibc and especially if you don't recompile the rest of the system. For an organization which needs commercial support from the OS vendor, this is unacceptable and your Admin should have refused to comply. If your web programmers need a specific component, they should get the component which works with your system.
I understand that this induces "pain" on your organization, but that pain should be much milder than the one your Admins experienced, and as a result, your organization.

My question is therefore: How can you defend the demand to upgrade Glibc when it is so obviously designed to force the Admin to fail?

Thank you very much for your answer, I look forward to reading your reply.

Three Questions about the Study...

[DieBase99]

Hi Hugh, Dr. Thompson, I really liked reading your study. I thought it was well written and setup a nice framework for studying Business Solution Reliabilty. I would like to as you these three questions: 1) When Novell bought Suse they got pretty late into the linux game; about 1-2 years ago (not sure). I am not familiar with Novell/Suse's offering but i am familiar with Red Hat, which has been in the Linux game for a much longer period. The RHN works very well to update key components smoothly... just as well.. if not better than Windows Update. Red Hat should have been picked, but instead Suse was picked, which i believe is like comparing apple to oranges, because Suse/Novell's offering is just too new for a fair comparison. I believe, the study would have been quite different if Red Hat had been picked. -> Why was Suse picked? 2) Study fails to mention the specific software components that were installed citing them as not being relevant. This is major source of bias, since the software components themselves could have been created by software manufacturers who had a higher priority on focusing compatibility with windows than with linux. Since the software vendors were not mentioned... it is impossible to verify if the software vendors were equally committed to create good software on both platforms. -> Why is it unimportant to include the Software vendors? 3) Study fails to measure # of reboots in reliability study. It is not an opinion but a fact that windows requires a lot more reboots than linux when making changes to the system, such as updating key components. A reboot should also be considered as downtime, but wasn't included in the study at all. For example if, if every reboot takes 2 minutes, and windows required 10 reboots and linux only 2; this should be added to the timeline. -> Why were reboots not considered? I am looking forward to a response from you. warmest regards, Daniel

Alternative Scenarios

[darrell73]

Dr. Thompson

Selecting the methodology for performing research like this must have been difficult. I believe there is already numerous questions that ask you about the various inputs to your methodology.

So my interest is in a different area. The scenario described is based purely on E-Commerce and your conclusions reflect that a Windows Server solution will cause less "IT pain" than a SUSE Linux Solution. My question is thus:

Are there any scenarios in which you suspect a Windows Server Solution is more likely to cause more IT pain? And consequently have you any more research "in the pipe" to test this?

Regards
Darrell

Re:Aw, "penguins" can't take it! Their OS lost aga

[arevos]

It's just fact based on what was found is all. Accept it.

But the bug reports from Securia, which is not sponsored by Microsoft or Linux, show quite clearly that Windows Server 2003 and SQL Server 2000 have more known vulnerabilities than Redhat and Oracle. How can Windows Server 2003 be more secure when it is clear that it has more vulnerabilities?

(After all - the rest of the planet seems to, given that Win32 based Operating Systems (by now, I would wager mostly Windows NT-based OS', such as Windows 2000/XP/Server 2003, & software run on 95-99% of all the personal computers on the planet, & not just restricted to laptops/desktops, but servers as well).

I'm unable to find any statistics for 2005, but back in 2000, Linux accounted for 36% of webservers, and Windows only 21%, according to Netcraft. It's likely that this hasn't changed.

It just isn't quite as versatile as Win32 based OS

Windows is certainly more compatable with hardware and the majority of software binaries about, but more versatile? In what way?

Re:If you actually read the report...

[julesh]

How much time would upgrading SuSE8 to SuSE9 have taken?

My experience is about 6 hours for the upgrade, plus another 3 or 4 to check everything still works afterwards. My experience of compiling my own glibc suggests that this will take about twice as much work.

Downtime can cost a lot of money, this would have been a pressure on the admins.

Any real company employing the kind of solutions described (which included so-called "best of breed" commercial applications) would certainly have a staging server to use, and could then swap the staging server for the live one in order to deploy. This would result in no more than a minute's downtime if done correctly. It's possible to do it with zero downtime.

And isn't that kinda the point of the study?

I think by insisting they stick with SuSE8 and use applications on it that blatantly aren't compatible with it they skewed the results. I know if I'd had all of the other requirements given, I'd have done a complete upgrade.