Slashdot Mirror
Windows vs. Linux Study Author Replies
Last week you submitted
questions for Dr. Herb Thompson, author of the latest Microsoft-sponsored
Windows vs. Linux study. Here are his answers. Please feel
free to ask follow-up questions. Dr. Thompson says he'll respond to as
many as he can. He's registered a new Slashdot username, FFE4,
specifically to participate in this discussion. All others claiming to
be him are imposters. So read, post, ask, and enjoy.
1- A
better way of putting it:
by einhverfr
It seems that your study attempted to simulate the growth of an internet startup firm on Windows or Linux. One thing I did not see in the study was a good description of assumptions you made. What assumptions were made in both the design of the requirements and the analysis of the data? What limitations can we place on the conclusions as a result of these assumptions?
Dr. Thompson
This is a really important question. I think there are two sections of the study: the assessment methodology and then the experiment we undertook to illustrate how to apply that methodology. I'll answer the assumption question for both parts:
Methodology - For the methodology, we wanted to provide a tool that organizations could use and apply their own assumptions. Maintaining a system is all about context; some environments favor Linux, others Windows. The question is, how do you know what's likely to be the most reliable (which includes manageable, secure and supportable) solution for your environment? We proposed a methodology a recipe - that looks at a solution in its entirety instead of just individual components. Policies like configuration control vary from organization to organization and to get something that's truly meaningful in your environment, the methodology needs to be carried out in your context. Enterprise customers can and should do this when they are about to trust their critical business processes to a platform. That said, the basic assumptions of the methodology are that patches are applied at 1 month intervals and that business needs evolve over time. How those business needs evolve depends on the scenario you're looking at (in our experiment we looked at ecommerce for example). The methodology doesn't cover steady state reliability, meaning the uptime of a system that is completely static. While this is important, our conversations with CIOs, CTOs, CSOs and IT folks lead us to believe that this was a smaller contributor to pain in a dynamic environment. In an appliance for example, though, steady state reliability is king, and I think an important limitation of this methodology is that we don't capture that well, and I think it's amazingly difficult quality to measure in a time-lapse way.
The purpose of the experiment was to illustrate how to apply the methodology and to begin to get some insights into some of the key model differences between two platforms. For the experiment we picked the ecommerce scenario, for no other reason than there has been a clear shift in how ecommerce sites have serviced their customers in recent years moving from static sites to personalized content. Some specific assumptions were:
* The transition from a basic purchasing site to a personalized portal based on order/browsing history takes place over a one year period.
* The period we looked at was July 1st, 2004 to June 30th, 2005 (the most recent full year at the time of the study).
* A configuration control policy exists that mandates OS version but not much else meaning administrators had fairly free rein to meet business requirements.
* All patches marked as critical or important supplied by the vendor are applied.
* We assume the system to be functioning if the original ecommerce application is running and meets some basic acceptance tests (same for both platforms see Appendix 1 of the report) and the new installed components are also running.
* To add new capabilities, we use leading 3rd party components as opposed to building custom code in-house.
* The business migrates operating system versions at the end of the one year period to the latest versions of the platform.
* The administrators that participated in the experiment reflect the average Linux (specifically SuSE) and Windows administrators in skill, capability and knowledge. While this was strived for, it's important to recognize the small sample size in drawing any conclusions from the data.
As far as limitations, the experiment looks at one specific case with a total of six administrators. I'd love to have done it with a hundred admins on each side on a wide range of business requirement scenarios and my hope is that others will do that and publish their results. Our experiment, however, shows that for this particular, clearly documented scenario, experienced Linux Admins had conflicts between meeting business needs and a recommended best practice like not introducing out-of-distribution components. If one is aware of potential conflicts and challenges upfront, I think you can put controls in place to make reasonable tradeoffs. In the linux case, a precise and specific configuration control policy may have prohibited the problematic upgrade of one of the components that the 3rd party solutions required. This would have likely reduced the number of failures but would have put some hefty constraints on 3rd party solutions. To understand the implications for your environment you really need to run through the methodology with the assumptions and restrictions of your organization and I hope that this study either prompts or provokes people to do that.
************************
2 - Meta-credibility?
by Tackhead
Where I come from (non-management, grunt-level techie), appearing in any of these analysts' journals *costs* an author more credibility than it gains him or her. For example, if $RAG says that $CORP has the best customer support, I immediately assume that $CORP has such horrid customer support that they had to pay someone to make up some research that proves otherwise.
To be sarcastic, I'd ask "who the heck actually takes these studies seriously?", but obviously *somebody* does. Who are these people, and why do these people take these industry analyst firms/journals/reports seriously? Are they right or wrong to do so? This isn't an attack (or endorsement :) of your research -- I'm talking about the credibility gap in industry research, and my observation that it's an industry-wide problem.
The meta-credibility question is this: Given the amount of shoddy pay-for-play research out there, does being published in an analyst journal tend to cost (a researcher, his consulting company, his financial backers) more credibility than it can gains him/her/them? If not, why not -- and more importantly, if so, is there any way to reverse the trend?
Dr. Thompson
This is a really interesting question because it cuts to the heart of what a real research study should provide to the reader. It should provide a baseline and I think research should always be questioned, scrutinized and debated because one can always find reasons for bias. Particularly, if a subject of the study (vendor for example) is behind its funding, whether directly (as in this study) or indirectly (meaning that they are big clients) I think it's critical that the study not provide just a baked cake for readers but the recipe as well. The recipe has to be inherently fair and simple, meaning that it has to map directly to a the quality or pain one is trying to measure without taking into account how the subjects try and provide that service or mitigate that pain. I think slanted opinion pieces, with no backup for those opinions, seriously hurts credibility, at least in my book. If you're presenting facts though and encouraging others to question them then I think that actually helps credibility, even if the search for those facts was paid for.
I agree though that one is tempted to dismiss research a priori though because of funding or some vendor tie. I think a good way to reverse the trend is to open the process up to public scrutiny; that's probably the main reason I came on Slashdot. To use this specific study as an example, some folks disagreed with several points in the experiment from counting patches, to reasons for upgrading key components, to the ecommerce scenario we used. For me, the study's key value is the methodology. Could different applications/scenarios have been chosen: absolutely!
The value I think that this study gives to the practitioner is arming them with a tool to help measure in their own environment. By applying the methodology, the results should take into account things like administrators skillsets, support policies, configuration control policies and the tradeoffs between customizability, maintainability, visibility, security and usability. It's only by looking at this stuff in context can one make a sound judgment; and a true research paper, especially one where funding is in question, needs to fully disclose the method and the funding source. In our case, the methodology has been vetted by industry analysts, IT organizations and several academics. That doesn't mean much, though, if you don't find the methodology meaningful for the questions you want answered. One reason I've come on Slashdot is to get the thoughts, opinions and assessments of the methodology itself from administrators in the trenches. I'm really pleased with the great questions and comments amidst the inevitable flames and I'm looking forward to this being posted so that others can weigh-in with their feedback and I can jump into the threads to get some discussion going.
If the research helps give real insight, and the methodology makes sense, I think there's real value no matter who paid the bill. At the end of the day, you need to decide whether or not you can extract any value from the information presented to you. In the case of this study, my hope is that it will leave you thinking hmmm.... maybe we should actually run through a process like this and check out how this works for ourselves. My more ambitious hope is that you'll implement it and tell me what challenges you faces on Windows, Linux, OSX, BSD, whatever platform you choose to compare. It may not even venture into the perennial Windows versus Linux battle; maybe you're a linux shop trying to decide between multiple distributions for example. Either way, if it's got people thinking about the topic and asking questions, well, that's all any researcher can really hope for.
************************
3 - Weak setup
by 0xABADC0DA
If I understand the study correctly, the windows side had to do nothing but set up a server to do a few different tasks over time and run windows update. The linux side had to have multiple incompatible versions of their database server running simultaneously on a single system and had to run unsupported versions of software to do it.
Why wasn't the windows side required to run multiple versions of IIS or SQL server simultaneously? In real life if you need to run multiple database versions you use virtualization or multiple systems, especially if one requires untested software. You don't run some hokie unstable branch on the same system as everything else. Why was a linux solution picked that required this level of work? My other related question is, did any of the unix administrators question why there were being asked to do such a thing? For example, did they come back and say they need a license for vmware? If they did not they do not seem like very competent administrators in my opinion.
Dr. Thompson
The Windows Admins and Linux admins were given the exact same set of business requirements which doesn't necessarily translate into the same tasks as they went about fulfilling them. The 3rd party components installed were chosen solely based on their market leadership position and any upgrades of OS were unknown at the time of selection. That said, on the Windows side, it turned out that no upgrades of IIS were needed (except for patches) and SQL Server was upgraded to SP4 as part of patch application. On the Linux side, at a high-level there were two main classes of upgrades: MySQL and GLIBC and they were both prompted by the installed components. After the experiment, the administrators were asked on both sides if this kind of evolution of systems met with their real-world experience. They said yes, with the caveat of if they were asked to install a component that required an upgrade of GLIBC that they would likely upgrade the operating system as long as their configuration control policy allowed it.
You make a great point about installing components on some sort of staging system (which is almost always done) as opposed to live running systems. That still means that the problems that the administrators had equal real IT pain. If something weird had to be done to get the system running but it does run and it's then put into production it's like a fuse that gets set on a bomb. A careful configuration control policy would almost certainly help and thats why I think it's so important to conduct this kind of experiment in your own environment with your own policies.
As far as selection of the Linux administrators go, they all had at least 5 years of enterprise administration experience, and two years of experience on SuSE specifically. With three people there's certainly likely to be a lot of variability and to get some conclusive results, I'd love to get a huge group of administrators across the spectrum in terms of experience. I'd also love to do it across multiple scenarios, beyond the ecommerce study. For this experiment, basically the bottom line is that we Illustrate one clearly documented scenario with six highly qualified admins that we selected based on experience. We cant ensure equal competency levels, but there was nothing in our screening that would lead us to believe there were gaps in knowledge on either side. When it comes down to it though, the really meaningful results are the ones you get when you perform the evaluation in your environment. Hopefully this study provides a starting point for asking the right questions when you do that.
************************
4- Who determined the metrics
by Infonaut
Did Microsoft come to you with a specific set of metrics, or did you work with them to develop the metrics, or did you determine them completely on your own?
Kudos to you for braving the inevitable flames to answer people's questions here on Slashdot.
Dr. Thompson
Great question! The metrics and the methodology were developed completely on our own and independent of Microsoft. They were created with the help and feedback of enterprise CIOs as well as industry analysts. I think that this relates to a couple of other questions on Slashdot with the gist of if Microsoft is funding the study aren't you incentivized for them to come out ahead. Besides the standard we would never do that and that would put our credibility at risk which is our primary commodity which are both very true, let me explain a little more about how our research engagements work.
Company X (in this case Microsoft) comes to us and says can you help us measure quality Y (in this case Reliability) to get some insight into how product Z stacks up. We say, sure, BUT we have complete creation and control of the methodology, it will be reviewed and vetted by the community (end users and independent analysts) and must strictly follow scientific principles. The response will either be: great, we want to know whats really going on or um, heres some things to focus on and I think you should set it up this way. In the first case we proceed, in the second case we inform that company that we don't do that kind of research. We are also not in the opinion business, so we present a methodology to follow and illustrate how that methodology is applied with the hope that people will take the methodology and apply it in their own environment.
All of our studies are written as if they will be released publicly BUT it is up to the sponsor if the study is publicly released. The vendor knows that they're taking a risk. They pay for the research either way but only have control over whether it is published, not over content. So if their intent is to use it as an outward facing piece, they may end up with something they don't like. Either way, I think it's of high value to them. If there are aspects of the results that favor the sponsor's product, in my experience, it goes to the marketing department and gets released publicly; if it favors the competitors product it goes off to the engineering folks as a tool to understand their product, their competitor's product, and the problem more clearly. Either way, we maintain complete editorial control over the study and there is no financial incentive for us if it becomes a public study or is used as an internal market analysis piece. The methodology has to be as objective as possible to be of any real value in either case.
************************
5 - ATMs vs. Voting Machines
by digitaldc
How is it that Diebold can make ATM machines that will account for every last penny in a banking system, but they can't make secure electronic voting machines?
Also, does the flame-resistant suit come with its own matching tinfoil hat? (don't answer that one)
Dr. Thompson
This is a question that has passed through my mind more than once. The voting world is very interesting. I don't have experience with the inner workings of Diebolds ATM machines but I can say that the versions of their tabulation software that Ive seen have some major security challenges (see this Washington post documentary for some of the gory details). I'd say I'm concerned about the e-voting systems Ive seen but that would be a serious understatement.
I question whether the economic incentive is there for them to make their voting systems more secure. Take an ATM for example. Imagine the ATM has a flaw and if you do something to it, you can make it give you more money than is actually deducted from your account. Anything involving money gets audited and sometimes audited multiple times and chances are good that the bank is going to figure out that they're loosing money. On the flip side, if there was a flaw in the ATM in the banks favor, someone balancing their checkbook is going to notice a discrepancy. The point is that there's always traceability and there's always someone keeping score. If you think about voting tabulators though we've got this mysterious box that vote data gets fed into and then, in many states, only a fraction of these votes are audited. That means we don't really know what the bank balance is other than what the machine tells us it is. If the system is highly vulnerable and its vulnerability is known by the manufacturer *but* it's going to be expensive to fix it and shore up defenses, there seems to be no huge incentive to fix the problems. I think the only way to get some decent software that counts votes that people can have confidence in is to allow security experts to actually test the systems, highlight potential vulnerabilities, and put some proper checks and balances in place. That would give the general public some visibility into a critical infrastructure system that we usually aren't in the habit of questioning and will hold voting manufacturers directly accountable to voters.
As for the tin foil hat to go with the flame resistant suit; it hasn't been shipped to me yet - apparently the manufacturing company is still filling backorders from SCO :).
************************
6 - Why are the requirements different?
by altoz
Looking at your research report's appendices, it seems that the requirements for Windows Administrators were somewhat different than the Linux Administrators. For instance, you ask for 4-5 years sys admin experience minimum for Windows, whereas it's 3-4 years sys admin experience minimum for Linux.
Why wasn't it equal for both? And doesn't this sort of slight Windows favoring undermine your credibility?
Dr. Thompson
Short answer: Typo. Long answer: We originally were looking for 4 years of general administration experience for both Linux and Windows which is what is reflected in the desired responses to the General Background questionnaire for Linux. We then raised it to 5 years for both Linux and Windows which is reflected in the General Background of the Windows questionnaire. The difference in the two was just a failure to update the response criteria on that shared section of one of the questionnaires. On page 5 though we've got the actual administrator experience laid out:
Each SuSE Linux administrator had at least 5 years experience administering Linux in an enterprise setting. We also required 2 years minimum experience administering SuSE Linux distributions and at least 1 year administering SuSE Linux Enterprise Server 8 and half a year administering SLES 9 (released in late 2004). Windows administrators all had at least 5 years experience administering Windows servers in an enterprise environment. These administrators also had at least 2 years experience administering Windows Server 2000 and at least 1 year administration experience with Windows Server 2003.
************************
7 - Scalability of Results?
by hahiss
You tested six people on two different systems; how is that supposed to yield any substantial insight into the underlying OSes themselves?
[At best, your study seems to show that the GNU/Linux distribution you selected was not particularly good at this task. But why does that show that the ``monolithic" style of Windows is better per se than the ``modular" style of GNU/Linux distributions?]
Dr. Thompson
First, let's look at what we did. We followed a methodology for evaluating reliability with three Windows admins and three Linux admins. This is small sample set and it looked at one scenario: ecommerce. Is this enough to make sweeping claims about the reliability of Linux/Windows? No way. I do however think the results raise some interesting questions about the modularity vs. integration tradeoffs that come with operating systems. I don't think that either the Windows or Linux models are better in a general sense but they *are* different; the question is which is likely to cause less pain and provide more value for your particular business need in your specific environment. Hopefully these are the questions that people will ask after reading this study, and with any luck it will prompt others to carry out their own analysis within their own IT environment, building on what we started here. I think the methodology in this paper has provided a good starting point to help people answer those questions in context.
************************
8 - Convenience vs. security
by Sheetrock
Lately, I've felt that Microsoft is emphasizing greater trust in their control over your system as a means of increasing your security. This is suggested by the difficulty of obtaining individual or bulk security patches from their website as opposed to simply loading Internet Explorer and using their Windows Update service, the encouragement in Service Pack 2 of allowing Automatic Update to run in the background, and the introduction of Genuine Advantage requiring the user to authenticate his system before obtaining critical updates such as DirectX.
In addition, Digital Rights Management or other copy protection schemes are becoming increasingly demanding and insidious, whether by uniquely identifying and reporting on user activity, intentionally restricting functionality, and even introducing new security issues (the most recent flap involves copy protection software on Sony CDs that not only hides content from the user but permits viruses to take advantage of this feature.)
I would like to know how you feel about the shift of control over the personal computer from the person to the software manufacturers -- is it right, and do we gain more than we're losing in privacy and security?
Dr. Thompson
This is an interesting problem because manufacturers have to deal with a wide range of users. If there was real visibility and education for users on the security implications of doing A, B or C then we'd be ok. It's scary though when that line gets crossed. Sony's DRM rootkit is a good example. But if you think about it, we are essentially passively accepting things like this all the time. Every time we install a new piece of software,especially something that reads untrusted data like a browser plugin,we tacitly accept that this software is likely to contain security flaws and can be an entryway into your system; NOW are you sure you want to install it? The visceral immediate reaction is no but then you balance tradeoffs of the features you get versus potential risks. Increasingly, were not even given that choice, and components that are intended to help us (or help the vendor) are installed with out our knowledge. This also brings up the question of visibility; how do we know what security state were really in with a system? Again, there are tradeoffs, some of this installed software may actually increase usability or maintainability but it's abstracting away what's happening on the metal. So far, it seems as though the market has tended towards the usability, maintainability, integration that favors bundling on both the Linux and Windows sides. It's kind of a disturbing trend though.
As another example, think about how much trustaverage programmers put into their compiler these days. Whenever I teach classes on computer security and then go off into x86 op codes or even assembly, it seems to be a totally foreign concept and skillset. We've created a culture of building applications rapidly in super high-level languages which does get the job done, but at the same time seems to have sacrificed knowledge of (or even the desire to know) what's happening on the metal. This places a heavy burden on platform developers, compiler writers and even IDE manufacturers because we are shifting the cloud of security responsibility over to them in big way. Under the right conditions it can be good because the average programmer knows little about security, but we need to make sure that the components we depend on and trust are written with security in mind, analyzed by folks that have a clue, and are tested and verified with security in mind. This means asking vendors the tough questions about their development processes and making sure they've got pretty good answers. Here's what I think is a good start. If that fails, theres always BSD. :).
************************
9 - Apache versus IIS
by 00_NOP
Simple one: of course I accept that Windows and Linux are a priori equally vulnerable - C programmers make mistakes. The question is which model is most likely to deliver a fix fastest. Given that the one area where Linux is probably in the lead over Microsoft's software is in the realm of the webserver - why are my server logs filled with artifacts of hacked IIS boxes but apache seems to remain pretty safe?
Dr. Thompson
You bring up a couple of interesting points. The first is patch delivery. It's true that on Linux if there's a high profile vulnerability you're likely to be able to find a patch out on the net from somebody in a few hours. Sometimes the fix is simple, a one-liner, and other times it may be more complex. Either way, there could be unintended side effects of the patch which is why there's usually a significant lag between these first responder patches and a blessed patch released from the distribution vendor. Most enterprises I know wait for the distribution patch as a matter of policy, and even then, they go through a fairly rigorous testing and compatibility verification process before the patch gets deployed widely. In the Windows world, one doesn't get the alpha or beta patches, just the blessed finished product. So the question is which solution is likely to provide a patch that fixes the problem and doesn't create any more problems the fastest. That's a tough one to answer. I think theres something to be learned by looking historically and that in general theres a big discrepancy between perception and reality. Here's a (pdf) link to a study we did earlier this year based on 2004 data that I think provides a good starting point for answering that question.
As far as why you've got so many attempts on your Windows/IIS box, I think there are two distinct issues: vulnerability and threat profile. In the past, I would argue that the path of least resistance was through Windows because desktop systems were often left unprotected by the home computer user. Bang-for-the-packet favored creating tools that exploited these problems and some of the attacks actually worked on poorly configured servers as well. Then there's the targeted vs. broad attacks. Theres no question that the high-profile worms and viruses in the last several years have favored Windows as a target. The issue gets even more complicated when you look at targeted attacks. These targeted attacks are much harder to measure, even anecdotally, because either an organization gets compromised and doesn't disclose it (unless they're compelled to by law) or the attack goes undetected because it doesn't leave any of the standard footprints, in which case no pain is felt immediately. That may help to explain it but the truth is that there's a lot of conflicting data out there. I remember reading this on Slashdot last year which claims Apache was more attacked than IIS but I've also read reports to the contrary. The reality is that any target of value is going to get attacked frequently. If there is an indiscriminant mass attack like a worm or virus, that's pretty bad and can be really painful. What's scarier though is the attack that just targets you.
************************
10 - Do you agree with Windows Local Workflow
by MosesJones
Microsoft and Linux distros have had a policy for some time of including more and more functionality in the base operating system, the latest example is the inclusion of "Local Workflow" in Windows Vista.
As a security expert do you think that bundling more and more increases or decreases the risks, and should both Windows and Linux distros be doing more to create reduced platforms that just act as good operating systems?
Dr. Thompson
Three years ago I bought my mother a combination TV, VCR and DVD player. It was great; she didn't have to worry about cables or the notorious multi-remote control problem. She didn't even really need the VCR because she hardly ever watches Video tapes, but I thought, why not. It worked great for two years, mom watched her DVDs, and on a blue moon a video tape from a family vacation would find its way into the VCR. All was well at the Thompson household. This past year, tragedy struck. The VCR devoured a videotape, completely entangling it in the machine. This not only knocked out the VCR but the television too (it thought it was constantly at the end of a tape and needing to rewind it). So here's the issue: mom probably only needed a TV and a separate DVD player. I probably could have gotten better quality components individually too, and with some ebay-savvy shopping, the group may have been cheaper. For my mom though, the integration and ease of operation of the three were key assets. The flipside of that is that the whole is only as strong as the weakest of its constituent parts, and by the manufacturer throwing some questionable VCR components into the mix, it caused the whole thing to fail. The meta-question: did I make the right choice, going for the kitchen-sink approach versus individual components? I think for mom I made the right call. For me, my willingness to program a universal remote and my love of tweaking the system would have lead me down a different route.
In operating systems, it depends what you're looking for and what the risk vs. reward equation is for you, and I would argue that the answer varies from user to user. The ideal would be something that gave you integration, ease of use, visibility, manageability and the ability to truly customize and minimize functionality and maintenance requirements. No operating system I've ever seen strikes that balance optimally and for every user. As far as bundling functionality with the distribution, I think it's a question of market demand. There's no question though that from a simple mathematical perspective, the less code processing untrusted data the better. That means if I need a system to perform one specific function, and that function was constant over time, then from a security perspective I only want the stuff on that box that does what I need to serve that goal. For example, I don't ever want X Windows on my linux file server. I just want the minimal code base there because as long as the code itself is reliable, I'll only have to mess with the box to apply patches (and much fewer patches if I strip the system down). That's true of my home fileserver. If I have an army of systems to manage though, my decision is going to come down to which platform is reliable and extends me the most tools to manage it efficiently and effectively. That's a question that can only be answered in context. I can tell you what I run at home though. File server: Red Hat EL 4 (no X windows). Laptop: Windows XP SP2. Desktop: Windows Server 2003 with virtual machines of everything under the sun from Win 9x to SuSE, Red Hat and Debian.
by einhverfr
It seems that your study attempted to simulate the growth of an internet startup firm on Windows or Linux. One thing I did not see in the study was a good description of assumptions you made. What assumptions were made in both the design of the requirements and the analysis of the data? What limitations can we place on the conclusions as a result of these assumptions?
Dr. Thompson
This is a really important question. I think there are two sections of the study: the assessment methodology and then the experiment we undertook to illustrate how to apply that methodology. I'll answer the assumption question for both parts:
Methodology - For the methodology, we wanted to provide a tool that organizations could use and apply their own assumptions. Maintaining a system is all about context; some environments favor Linux, others Windows. The question is, how do you know what's likely to be the most reliable (which includes manageable, secure and supportable) solution for your environment? We proposed a methodology a recipe - that looks at a solution in its entirety instead of just individual components. Policies like configuration control vary from organization to organization and to get something that's truly meaningful in your environment, the methodology needs to be carried out in your context. Enterprise customers can and should do this when they are about to trust their critical business processes to a platform. That said, the basic assumptions of the methodology are that patches are applied at 1 month intervals and that business needs evolve over time. How those business needs evolve depends on the scenario you're looking at (in our experiment we looked at ecommerce for example). The methodology doesn't cover steady state reliability, meaning the uptime of a system that is completely static. While this is important, our conversations with CIOs, CTOs, CSOs and IT folks lead us to believe that this was a smaller contributor to pain in a dynamic environment. In an appliance for example, though, steady state reliability is king, and I think an important limitation of this methodology is that we don't capture that well, and I think it's amazingly difficult quality to measure in a time-lapse way.
The purpose of the experiment was to illustrate how to apply the methodology and to begin to get some insights into some of the key model differences between two platforms. For the experiment we picked the ecommerce scenario, for no other reason than there has been a clear shift in how ecommerce sites have serviced their customers in recent years moving from static sites to personalized content. Some specific assumptions were:
* The transition from a basic purchasing site to a personalized portal based on order/browsing history takes place over a one year period.
* The period we looked at was July 1st, 2004 to June 30th, 2005 (the most recent full year at the time of the study).
* A configuration control policy exists that mandates OS version but not much else meaning administrators had fairly free rein to meet business requirements.
* All patches marked as critical or important supplied by the vendor are applied.
* We assume the system to be functioning if the original ecommerce application is running and meets some basic acceptance tests (same for both platforms see Appendix 1 of the report) and the new installed components are also running.
* To add new capabilities, we use leading 3rd party components as opposed to building custom code in-house.
* The business migrates operating system versions at the end of the one year period to the latest versions of the platform.
* The administrators that participated in the experiment reflect the average Linux (specifically SuSE) and Windows administrators in skill, capability and knowledge. While this was strived for, it's important to recognize the small sample size in drawing any conclusions from the data.
As far as limitations, the experiment looks at one specific case with a total of six administrators. I'd love to have done it with a hundred admins on each side on a wide range of business requirement scenarios and my hope is that others will do that and publish their results. Our experiment, however, shows that for this particular, clearly documented scenario, experienced Linux Admins had conflicts between meeting business needs and a recommended best practice like not introducing out-of-distribution components. If one is aware of potential conflicts and challenges upfront, I think you can put controls in place to make reasonable tradeoffs. In the linux case, a precise and specific configuration control policy may have prohibited the problematic upgrade of one of the components that the 3rd party solutions required. This would have likely reduced the number of failures but would have put some hefty constraints on 3rd party solutions. To understand the implications for your environment you really need to run through the methodology with the assumptions and restrictions of your organization and I hope that this study either prompts or provokes people to do that.
************************
2 - Meta-credibility?
by Tackhead
Where I come from (non-management, grunt-level techie), appearing in any of these analysts' journals *costs* an author more credibility than it gains him or her. For example, if $RAG says that $CORP has the best customer support, I immediately assume that $CORP has such horrid customer support that they had to pay someone to make up some research that proves otherwise.
To be sarcastic, I'd ask "who the heck actually takes these studies seriously?", but obviously *somebody* does. Who are these people, and why do these people take these industry analyst firms/journals/reports seriously? Are they right or wrong to do so? This isn't an attack (or endorsement :) of your research -- I'm talking about the credibility gap in industry research, and my observation that it's an industry-wide problem.
The meta-credibility question is this: Given the amount of shoddy pay-for-play research out there, does being published in an analyst journal tend to cost (a researcher, his consulting company, his financial backers) more credibility than it can gains him/her/them? If not, why not -- and more importantly, if so, is there any way to reverse the trend?
Dr. Thompson
This is a really interesting question because it cuts to the heart of what a real research study should provide to the reader. It should provide a baseline and I think research should always be questioned, scrutinized and debated because one can always find reasons for bias. Particularly, if a subject of the study (vendor for example) is behind its funding, whether directly (as in this study) or indirectly (meaning that they are big clients) I think it's critical that the study not provide just a baked cake for readers but the recipe as well. The recipe has to be inherently fair and simple, meaning that it has to map directly to a the quality or pain one is trying to measure without taking into account how the subjects try and provide that service or mitigate that pain. I think slanted opinion pieces, with no backup for those opinions, seriously hurts credibility, at least in my book. If you're presenting facts though and encouraging others to question them then I think that actually helps credibility, even if the search for those facts was paid for.
I agree though that one is tempted to dismiss research a priori though because of funding or some vendor tie. I think a good way to reverse the trend is to open the process up to public scrutiny; that's probably the main reason I came on Slashdot. To use this specific study as an example, some folks disagreed with several points in the experiment from counting patches, to reasons for upgrading key components, to the ecommerce scenario we used. For me, the study's key value is the methodology. Could different applications/scenarios have been chosen: absolutely!
The value I think that this study gives to the practitioner is arming them with a tool to help measure in their own environment. By applying the methodology, the results should take into account things like administrators skillsets, support policies, configuration control policies and the tradeoffs between customizability, maintainability, visibility, security and usability. It's only by looking at this stuff in context can one make a sound judgment; and a true research paper, especially one where funding is in question, needs to fully disclose the method and the funding source. In our case, the methodology has been vetted by industry analysts, IT organizations and several academics. That doesn't mean much, though, if you don't find the methodology meaningful for the questions you want answered. One reason I've come on Slashdot is to get the thoughts, opinions and assessments of the methodology itself from administrators in the trenches. I'm really pleased with the great questions and comments amidst the inevitable flames and I'm looking forward to this being posted so that others can weigh-in with their feedback and I can jump into the threads to get some discussion going.
If the research helps give real insight, and the methodology makes sense, I think there's real value no matter who paid the bill. At the end of the day, you need to decide whether or not you can extract any value from the information presented to you. In the case of this study, my hope is that it will leave you thinking hmmm.... maybe we should actually run through a process like this and check out how this works for ourselves. My more ambitious hope is that you'll implement it and tell me what challenges you faces on Windows, Linux, OSX, BSD, whatever platform you choose to compare. It may not even venture into the perennial Windows versus Linux battle; maybe you're a linux shop trying to decide between multiple distributions for example. Either way, if it's got people thinking about the topic and asking questions, well, that's all any researcher can really hope for.
************************
3 - Weak setup
by 0xABADC0DA
If I understand the study correctly, the windows side had to do nothing but set up a server to do a few different tasks over time and run windows update. The linux side had to have multiple incompatible versions of their database server running simultaneously on a single system and had to run unsupported versions of software to do it.
Why wasn't the windows side required to run multiple versions of IIS or SQL server simultaneously? In real life if you need to run multiple database versions you use virtualization or multiple systems, especially if one requires untested software. You don't run some hokie unstable branch on the same system as everything else. Why was a linux solution picked that required this level of work? My other related question is, did any of the unix administrators question why there were being asked to do such a thing? For example, did they come back and say they need a license for vmware? If they did not they do not seem like very competent administrators in my opinion.
Dr. Thompson
The Windows Admins and Linux admins were given the exact same set of business requirements which doesn't necessarily translate into the same tasks as they went about fulfilling them. The 3rd party components installed were chosen solely based on their market leadership position and any upgrades of OS were unknown at the time of selection. That said, on the Windows side, it turned out that no upgrades of IIS were needed (except for patches) and SQL Server was upgraded to SP4 as part of patch application. On the Linux side, at a high-level there were two main classes of upgrades: MySQL and GLIBC and they were both prompted by the installed components. After the experiment, the administrators were asked on both sides if this kind of evolution of systems met with their real-world experience. They said yes, with the caveat of if they were asked to install a component that required an upgrade of GLIBC that they would likely upgrade the operating system as long as their configuration control policy allowed it.
You make a great point about installing components on some sort of staging system (which is almost always done) as opposed to live running systems. That still means that the problems that the administrators had equal real IT pain. If something weird had to be done to get the system running but it does run and it's then put into production it's like a fuse that gets set on a bomb. A careful configuration control policy would almost certainly help and thats why I think it's so important to conduct this kind of experiment in your own environment with your own policies.
As far as selection of the Linux administrators go, they all had at least 5 years of enterprise administration experience, and two years of experience on SuSE specifically. With three people there's certainly likely to be a lot of variability and to get some conclusive results, I'd love to get a huge group of administrators across the spectrum in terms of experience. I'd also love to do it across multiple scenarios, beyond the ecommerce study. For this experiment, basically the bottom line is that we Illustrate one clearly documented scenario with six highly qualified admins that we selected based on experience. We cant ensure equal competency levels, but there was nothing in our screening that would lead us to believe there were gaps in knowledge on either side. When it comes down to it though, the really meaningful results are the ones you get when you perform the evaluation in your environment. Hopefully this study provides a starting point for asking the right questions when you do that.
************************
4- Who determined the metrics
by Infonaut
Did Microsoft come to you with a specific set of metrics, or did you work with them to develop the metrics, or did you determine them completely on your own?
Kudos to you for braving the inevitable flames to answer people's questions here on Slashdot.
Dr. Thompson
Great question! The metrics and the methodology were developed completely on our own and independent of Microsoft. They were created with the help and feedback of enterprise CIOs as well as industry analysts. I think that this relates to a couple of other questions on Slashdot with the gist of if Microsoft is funding the study aren't you incentivized for them to come out ahead. Besides the standard we would never do that and that would put our credibility at risk which is our primary commodity which are both very true, let me explain a little more about how our research engagements work.
Company X (in this case Microsoft) comes to us and says can you help us measure quality Y (in this case Reliability) to get some insight into how product Z stacks up. We say, sure, BUT we have complete creation and control of the methodology, it will be reviewed and vetted by the community (end users and independent analysts) and must strictly follow scientific principles. The response will either be: great, we want to know whats really going on or um, heres some things to focus on and I think you should set it up this way. In the first case we proceed, in the second case we inform that company that we don't do that kind of research. We are also not in the opinion business, so we present a methodology to follow and illustrate how that methodology is applied with the hope that people will take the methodology and apply it in their own environment.
All of our studies are written as if they will be released publicly BUT it is up to the sponsor if the study is publicly released. The vendor knows that they're taking a risk. They pay for the research either way but only have control over whether it is published, not over content. So if their intent is to use it as an outward facing piece, they may end up with something they don't like. Either way, I think it's of high value to them. If there are aspects of the results that favor the sponsor's product, in my experience, it goes to the marketing department and gets released publicly; if it favors the competitors product it goes off to the engineering folks as a tool to understand their product, their competitor's product, and the problem more clearly. Either way, we maintain complete editorial control over the study and there is no financial incentive for us if it becomes a public study or is used as an internal market analysis piece. The methodology has to be as objective as possible to be of any real value in either case.
************************
5 - ATMs vs. Voting Machines
by digitaldc
How is it that Diebold can make ATM machines that will account for every last penny in a banking system, but they can't make secure electronic voting machines?
Also, does the flame-resistant suit come with its own matching tinfoil hat? (don't answer that one)
Dr. Thompson
This is a question that has passed through my mind more than once. The voting world is very interesting. I don't have experience with the inner workings of Diebolds ATM machines but I can say that the versions of their tabulation software that Ive seen have some major security challenges (see this Washington post documentary for some of the gory details). I'd say I'm concerned about the e-voting systems Ive seen but that would be a serious understatement.
I question whether the economic incentive is there for them to make their voting systems more secure. Take an ATM for example. Imagine the ATM has a flaw and if you do something to it, you can make it give you more money than is actually deducted from your account. Anything involving money gets audited and sometimes audited multiple times and chances are good that the bank is going to figure out that they're loosing money. On the flip side, if there was a flaw in the ATM in the banks favor, someone balancing their checkbook is going to notice a discrepancy. The point is that there's always traceability and there's always someone keeping score. If you think about voting tabulators though we've got this mysterious box that vote data gets fed into and then, in many states, only a fraction of these votes are audited. That means we don't really know what the bank balance is other than what the machine tells us it is. If the system is highly vulnerable and its vulnerability is known by the manufacturer *but* it's going to be expensive to fix it and shore up defenses, there seems to be no huge incentive to fix the problems. I think the only way to get some decent software that counts votes that people can have confidence in is to allow security experts to actually test the systems, highlight potential vulnerabilities, and put some proper checks and balances in place. That would give the general public some visibility into a critical infrastructure system that we usually aren't in the habit of questioning and will hold voting manufacturers directly accountable to voters.
As for the tin foil hat to go with the flame resistant suit; it hasn't been shipped to me yet - apparently the manufacturing company is still filling backorders from SCO :).
************************
6 - Why are the requirements different?
by altoz
Looking at your research report's appendices, it seems that the requirements for Windows Administrators were somewhat different than the Linux Administrators. For instance, you ask for 4-5 years sys admin experience minimum for Windows, whereas it's 3-4 years sys admin experience minimum for Linux.
Why wasn't it equal for both? And doesn't this sort of slight Windows favoring undermine your credibility?
Dr. Thompson
Short answer: Typo. Long answer: We originally were looking for 4 years of general administration experience for both Linux and Windows which is what is reflected in the desired responses to the General Background questionnaire for Linux. We then raised it to 5 years for both Linux and Windows which is reflected in the General Background of the Windows questionnaire. The difference in the two was just a failure to update the response criteria on that shared section of one of the questionnaires. On page 5 though we've got the actual administrator experience laid out:
Each SuSE Linux administrator had at least 5 years experience administering Linux in an enterprise setting. We also required 2 years minimum experience administering SuSE Linux distributions and at least 1 year administering SuSE Linux Enterprise Server 8 and half a year administering SLES 9 (released in late 2004). Windows administrators all had at least 5 years experience administering Windows servers in an enterprise environment. These administrators also had at least 2 years experience administering Windows Server 2000 and at least 1 year administration experience with Windows Server 2003.
************************
7 - Scalability of Results?
by hahiss
You tested six people on two different systems; how is that supposed to yield any substantial insight into the underlying OSes themselves?
[At best, your study seems to show that the GNU/Linux distribution you selected was not particularly good at this task. But why does that show that the ``monolithic" style of Windows is better per se than the ``modular" style of GNU/Linux distributions?]
Dr. Thompson
First, let's look at what we did. We followed a methodology for evaluating reliability with three Windows admins and three Linux admins. This is small sample set and it looked at one scenario: ecommerce. Is this enough to make sweeping claims about the reliability of Linux/Windows? No way. I do however think the results raise some interesting questions about the modularity vs. integration tradeoffs that come with operating systems. I don't think that either the Windows or Linux models are better in a general sense but they *are* different; the question is which is likely to cause less pain and provide more value for your particular business need in your specific environment. Hopefully these are the questions that people will ask after reading this study, and with any luck it will prompt others to carry out their own analysis within their own IT environment, building on what we started here. I think the methodology in this paper has provided a good starting point to help people answer those questions in context.
************************
8 - Convenience vs. security
by Sheetrock
Lately, I've felt that Microsoft is emphasizing greater trust in their control over your system as a means of increasing your security. This is suggested by the difficulty of obtaining individual or bulk security patches from their website as opposed to simply loading Internet Explorer and using their Windows Update service, the encouragement in Service Pack 2 of allowing Automatic Update to run in the background, and the introduction of Genuine Advantage requiring the user to authenticate his system before obtaining critical updates such as DirectX.
In addition, Digital Rights Management or other copy protection schemes are becoming increasingly demanding and insidious, whether by uniquely identifying and reporting on user activity, intentionally restricting functionality, and even introducing new security issues (the most recent flap involves copy protection software on Sony CDs that not only hides content from the user but permits viruses to take advantage of this feature.)
I would like to know how you feel about the shift of control over the personal computer from the person to the software manufacturers -- is it right, and do we gain more than we're losing in privacy and security?
Dr. Thompson
This is an interesting problem because manufacturers have to deal with a wide range of users. If there was real visibility and education for users on the security implications of doing A, B or C then we'd be ok. It's scary though when that line gets crossed. Sony's DRM rootkit is a good example. But if you think about it, we are essentially passively accepting things like this all the time. Every time we install a new piece of software,especially something that reads untrusted data like a browser plugin,we tacitly accept that this software is likely to contain security flaws and can be an entryway into your system; NOW are you sure you want to install it? The visceral immediate reaction is no but then you balance tradeoffs of the features you get versus potential risks. Increasingly, were not even given that choice, and components that are intended to help us (or help the vendor) are installed with out our knowledge. This also brings up the question of visibility; how do we know what security state were really in with a system? Again, there are tradeoffs, some of this installed software may actually increase usability or maintainability but it's abstracting away what's happening on the metal. So far, it seems as though the market has tended towards the usability, maintainability, integration that favors bundling on both the Linux and Windows sides. It's kind of a disturbing trend though.
As another example, think about how much trustaverage programmers put into their compiler these days. Whenever I teach classes on computer security and then go off into x86 op codes or even assembly, it seems to be a totally foreign concept and skillset. We've created a culture of building applications rapidly in super high-level languages which does get the job done, but at the same time seems to have sacrificed knowledge of (or even the desire to know) what's happening on the metal. This places a heavy burden on platform developers, compiler writers and even IDE manufacturers because we are shifting the cloud of security responsibility over to them in big way. Under the right conditions it can be good because the average programmer knows little about security, but we need to make sure that the components we depend on and trust are written with security in mind, analyzed by folks that have a clue, and are tested and verified with security in mind. This means asking vendors the tough questions about their development processes and making sure they've got pretty good answers. Here's what I think is a good start. If that fails, theres always BSD. :).
************************
9 - Apache versus IIS
by 00_NOP
Simple one: of course I accept that Windows and Linux are a priori equally vulnerable - C programmers make mistakes. The question is which model is most likely to deliver a fix fastest. Given that the one area where Linux is probably in the lead over Microsoft's software is in the realm of the webserver - why are my server logs filled with artifacts of hacked IIS boxes but apache seems to remain pretty safe?
Dr. Thompson
You bring up a couple of interesting points. The first is patch delivery. It's true that on Linux if there's a high profile vulnerability you're likely to be able to find a patch out on the net from somebody in a few hours. Sometimes the fix is simple, a one-liner, and other times it may be more complex. Either way, there could be unintended side effects of the patch which is why there's usually a significant lag between these first responder patches and a blessed patch released from the distribution vendor. Most enterprises I know wait for the distribution patch as a matter of policy, and even then, they go through a fairly rigorous testing and compatibility verification process before the patch gets deployed widely. In the Windows world, one doesn't get the alpha or beta patches, just the blessed finished product. So the question is which solution is likely to provide a patch that fixes the problem and doesn't create any more problems the fastest. That's a tough one to answer. I think theres something to be learned by looking historically and that in general theres a big discrepancy between perception and reality. Here's a (pdf) link to a study we did earlier this year based on 2004 data that I think provides a good starting point for answering that question.
As far as why you've got so many attempts on your Windows/IIS box, I think there are two distinct issues: vulnerability and threat profile. In the past, I would argue that the path of least resistance was through Windows because desktop systems were often left unprotected by the home computer user. Bang-for-the-packet favored creating tools that exploited these problems and some of the attacks actually worked on poorly configured servers as well. Then there's the targeted vs. broad attacks. Theres no question that the high-profile worms and viruses in the last several years have favored Windows as a target. The issue gets even more complicated when you look at targeted attacks. These targeted attacks are much harder to measure, even anecdotally, because either an organization gets compromised and doesn't disclose it (unless they're compelled to by law) or the attack goes undetected because it doesn't leave any of the standard footprints, in which case no pain is felt immediately. That may help to explain it but the truth is that there's a lot of conflicting data out there. I remember reading this on Slashdot last year which claims Apache was more attacked than IIS but I've also read reports to the contrary. The reality is that any target of value is going to get attacked frequently. If there is an indiscriminant mass attack like a worm or virus, that's pretty bad and can be really painful. What's scarier though is the attack that just targets you.
************************
10 - Do you agree with Windows Local Workflow
by MosesJones
Microsoft and Linux distros have had a policy for some time of including more and more functionality in the base operating system, the latest example is the inclusion of "Local Workflow" in Windows Vista.
As a security expert do you think that bundling more and more increases or decreases the risks, and should both Windows and Linux distros be doing more to create reduced platforms that just act as good operating systems?
Dr. Thompson
Three years ago I bought my mother a combination TV, VCR and DVD player. It was great; she didn't have to worry about cables or the notorious multi-remote control problem. She didn't even really need the VCR because she hardly ever watches Video tapes, but I thought, why not. It worked great for two years, mom watched her DVDs, and on a blue moon a video tape from a family vacation would find its way into the VCR. All was well at the Thompson household. This past year, tragedy struck. The VCR devoured a videotape, completely entangling it in the machine. This not only knocked out the VCR but the television too (it thought it was constantly at the end of a tape and needing to rewind it). So here's the issue: mom probably only needed a TV and a separate DVD player. I probably could have gotten better quality components individually too, and with some ebay-savvy shopping, the group may have been cheaper. For my mom though, the integration and ease of operation of the three were key assets. The flipside of that is that the whole is only as strong as the weakest of its constituent parts, and by the manufacturer throwing some questionable VCR components into the mix, it caused the whole thing to fail. The meta-question: did I make the right choice, going for the kitchen-sink approach versus individual components? I think for mom I made the right call. For me, my willingness to program a universal remote and my love of tweaking the system would have lead me down a different route.
In operating systems, it depends what you're looking for and what the risk vs. reward equation is for you, and I would argue that the answer varies from user to user. The ideal would be something that gave you integration, ease of use, visibility, manageability and the ability to truly customize and minimize functionality and maintenance requirements. No operating system I've ever seen strikes that balance optimally and for every user. As far as bundling functionality with the distribution, I think it's a question of market demand. There's no question though that from a simple mathematical perspective, the less code processing untrusted data the better. That means if I need a system to perform one specific function, and that function was constant over time, then from a security perspective I only want the stuff on that box that does what I need to serve that goal. For example, I don't ever want X Windows on my linux file server. I just want the minimal code base there because as long as the code itself is reliable, I'll only have to mess with the box to apply patches (and much fewer patches if I strip the system down). That's true of my home fileserver. If I have an army of systems to manage though, my decision is going to come down to which platform is reliable and extends me the most tools to manage it efficiently and effectively. That's a question that can only be answered in context. I can tell you what I run at home though. File server: Red Hat EL 4 (no X windows). Laptop: Windows XP SP2. Desktop: Windows Server 2003 with virtual machines of everything under the sun from Win 9x to SuSE, Red Hat and Debian.
Kidding!
An Indian-American Hindu committed to non-violent thought/speech/action alarmed by the global explosion of radical Islam
People on slashdot can get pretty upset about the studies Microsft shows the world, and these mostly say Microsoft is the king on the hill. But don't ever forget they don't show ALL of their studies. It could well be that 60% of them does not favor Microsoft good enough or not at all.
Of course I realise they try to use situations that are more likely to favor for them as for [insert competitor].
No if just once a bunch of other studies leaked we could get a real view over what MS is doing with their researches all the time...
Dependency hell? =>
At least the guy has a sense of humor.
See his comment on the Flameproof suit/Tinfoil hat question.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
Okay, so they needed a certain version of MySQL which required a newer version of Glibc. Still, though, any Unix admin should know that upgrading glibc is risky at best (I've broken many systems due to upgrading glibc).
Here's my question: Why didn't they just rebuild the source RPM and install the resulting binaries? This way the binary would be built with the same glibc as everything else on the system. I've done that on many system with no adverse effects. They didn't have to rebuild in on the server, just any machine running the same distro would do fine.
"It ain't a war against drugs.it's a war against personal freedom" --Bill Hicks
When this study was originally posted, many of you slashbots rushed to dismiss it solely on the basis of funding.
When I brought it to your attention that doing so is fallacious, I was modded down into oblivion.
Inevitably the same people will post again, with the same fallacious arguments, claiming that this guy is a shill for MS.
I'll be interested to hear the excuses that are made this time, and I can guarantee that several people will attack this man personally for no reason other than the results of his study.
So how about, instead of relying on old prejudices, we instad attempt to actually examine the research and gauge it on it's own merits.
How pathetic are you that you follow me from topic to topic and waste all your mod points at once modding me down?
Looks like a bunch of honest and detailed answers with no dodging...
"No if just once a bunch of other studies leaked we could get a real view over what MS is doing with their researches all the time..."
And why doesn't Linux sponser some (factual) studies? Complaining about a competitors studies not proving your product is better is simply stupid.
You obviously see the value of public scrutiny in what you do. So do we, we're obviously paying attention to your studies, and are pleased to see the "inner workings." It certainly helps lend credibility to your points. But it also begs the question: why doesn't Microsoft extend that same logic to operating systems or applications?
John
Not to sound like a troll, but meta-credibility does also work the opposite way;
anti-$ rag says that grassroots anti-$ os/app/whatever is "the best" and you will have an immediate knee-jerk reaction from the community defending it to the death and proudly installing it on thier boxes just to say they did, even if it takes several dozen man-hours to get it to do anything even marginally useful.
Dogma is probably even more dangerous and counterproductive than putting blind trust in some $corps marketing stooges, as hard as that is to comprehend.
Sorry, just watched six guys on laptops code and tweak for two hours failing to get the newest, hippest OS du jour to even recognize basic hardware.
From a purely technical point of view, I was mostly interested in seeing the following question [and thread] addressed:
8 949&cid=14084692
http://interviews.slashdot.org/comments.pl?sid=16
--- d'oh
Mostly, becuase unlike ESR, he doesn't seem to have an agenda... Unlike ESR the Dr. doesn't work for Microsoft or any OSS org...
Because he's not a stark raving lunatic?
John
I really thought the answer Dr Thompsons answer to Tackheads question was sincerely put, and I liked the way Tackhead built that question up with such polite delivery. What makes me sad, having done a bit of 'research consulting' is that the honorable intents of the man to conduct a scientific study will still have to go up through the mighty MS spin machine for some heavy editing. What I'm saying is while I think Doc Thompson is genuinely trying his best as an honest empirical researcher he might not be entirely satisfied with the way the company interprets and publishes his results. I think that was what Tackhead was really alluding to, that good men can be tarnished by no fault of their own by keeping the wrong company, and he didn't really answer that. So either he knows exactly how his work is going to be framed, or MS are paying him so much money he doesn't care what people end up thinking of him after MS have doctored his results to suit them (because we all know they always do)
"And I find this guy to be more credible than, say ESR, why, exactly?"
Because you used that jumbo brain of yours, and years of schooling in deductive, and inductive thinking, plus hours of research to make a factual conclusion on weither the author is indeed credible. Or maybe you took the easy way out and simply read "MS Shill" then proceeded to shut down all higher brain functions.
PLEASE tell me he isn't a gun rights type. He cannot even hold a pistol safely...
Yet the Linux sysadmins were downloading mysql code from the mysql site and attempting to backport patches from SLES 9 to SLES 8.
From TFA today:
In every one of these "studies" there is always something that the "study" requires that no intelligent person would do.
I don't care WHO the "researcher" is. Once they participate in one of those "studies", I have no respect for them anymore.
I like it, I find it very difficult to deal with the multi remote problem at someones house.
Surround sound, Satellite, DVD, VHS, cable, PS2 all plugged in. For many peoples house I just give up trying to watch TV or even change channels/volume.
The OS upgrade was already part of the "evaluation".
Why not allow the sysadmins to upgrade from SLES 8 to SLES 9 instead of REQUIRING them to backport the glibc patches from 9 to 8?
Suse is great distribution, but I'd rather place it on desktop instead servers.
I'd like to dare the author to replicate this experiment using Debian stable as linux side server OS.
There are no atheists when recovering from tape backup.
Only on Slashdot. Not only did you not RTFA, you admit it, then have the gall to ask someone else to read the whole monster and summarize it for you. And you'll probably get a up-mod or two somehow.
:)
/. :)
Just Kidding. I was actually going to ask the same thing, and you beat me there, so now I vent.
I love
You could go out and pick 6 new admins and get totally diffrent results, this study is a joke!. Maybe the windows group had dealt with similiar projects before and the linux people had not?
You are not judging any measurable value, instead your study did nothing but judge the performance of the people you picked to do the study.
"Diffrent strokes for diffrent folks"
You are trying to qualify 2 seperate tasks which can both be completed X number of ways into a single conclusion. You can not do that when Y ( people invovled ) is variable. You can not possibly do that with such a small group of people.
For this study to even be close to valid it would have to be approached from a similar direction drug companys use to test there products.. A BROAD range of people ( not just 6 ) would of have to been brought together and your tests would of had to be run multiple times with seperate groups before any formation of a conclusion could come about!
Even then it does not mean your conclusion is fact! ( Every year how many drugs are found unsafe even after going through this type of testing ).
Personal Website
He told you his process. He told you how Microsoft approached his company. He gave you his methodology. Show us where he f*ed up.
I'm waiting... come on... all talk now? yeah...
-everphilski-
Please note, that the OP is actually ifwm. He got modded down for being more troll than having an intelligent thing to say. If you check through his handiwork, you will find that he really does NOT have anything to say. Now the real question is, how did he get modded up, without checking his statements, unless the mod point came from IFWM himself. Editors/Meta-modders, you hearing this? IFWM/Flyingwhitey should be baned from here.
He's an ardent libertarian, I'll give him that. But like most libertarians, he doesn't understand that it takes all of us to make a society. If it were up to him, we'd all still be living in home-made shacks in the woods, because there wouldn't be enough of a society functioning to have paved roads upon which to deliver us construction materials. Or if there were, they'd be toll roads up to your driveway.
John
http://reading.uoregon.edu/scope/trial_scope_index .php
http://www.worldnetdaily.com/news/article.asp?ARTI CLE_ID=23709
http://www.educationnext.org/20061/23.html
DECEMBER 03, 2003 And that was just the first news story google turned up for atm+diebold+flaws
There is a lot of crap that goes on in the banking industry which is not reported. Mostly because there are no laws requiring it to be reported.
[Fuck Beta]
o0t!
King of the Desktop perhaps but not King of servers. Sure they show more REVENUE but as for deployment, Linux still dominates and has been squeezing Microsoft more and more out of server space. While Linux eats into UNIX market share, they also are eating into Windows market share as well.
Don't believe it? Look at what the most widely used Web server is. Look at what the most widely used DB is. look at the most popular scripting languages. And now keep in mind that they all come installed by default on almost all Linux distros.
They can keep putting money into trying to convince people that Microsoft Clusterfuck Edition can replace Linux clusters. That's cool. Just another money pit for them and a great way to divert resources into a nowhere scheme. And sure they have loads of funds but they still have to answer to shareholders and they are not pleased that the stock has stagnated for so long and they won't be pleased when didvidends stop getting payed and products not being sold or delivered on time do to them focusing on a product that will go nowhere.
The entire open source world and all companies supporting open source (IBM, Google, Sun, Amazon, etc.) are all starting a bait and switch where Microsoft throws mony into duplicating anything that it thinks may be a threat. This is turn causes them to waste funds and resources on red herrings when the actual threat is something else entirely.
These past 5 years have seen Linux and open source go from obscurity to mainstream in the business market. The next five years will see it go from obscurity to mainstream in the consumer market.
This is my sig. There are many like it but this one is mine.
Maintaining a system is all about context; some environments favor Linux, others Windows.
I've built many many systems for many people; servers, desktops, multimedia backends, you name it. I personally use linux/unix, but the OS installed upon each of the machines I build is by no means limited by my personal preference. Dr. Thompson makes a wonderful point here. In computing as in life, different situations merit different approaches.
I really wish all of the microsoft-, bsd-, and linux-zealots would realize this. To each, his own.
From the responses it sounds like he did an honest attempt at this study. I think the conclusion however should be that stupid admins cost a lot, so taking away things they could mess up is the key to lowering costs. If it turned out that the windows admins had to actually do anything, I bet the results would have been just as bad or worse for Windows.
In the Windows world, one doesn't get the alpha or beta patches, just the blessed finished product
...) or outlook's bad record of keeping spam from executing malicious code (mostly through the IE engine).
p hp?story=20020924094345962r sjump_1.htmls pr aised-microsoft-patches
yeah, right!
i won't even mention IE's security holes for the last 8 or so years (active x,
but boldly stating how much due diligence is exacted upon the microsoft patches before final release is ridiculous in face of them frequently backfiring and leaving old or new vulnerabilities in their wake:
http://www.hideaway.net/home/public_html/article.
http://www.infoworld.com/article/03/09/08/HNhacke
http://www.eweek.com/article2/0,1895,1753511,00.a
http://www.vnunet.com/vnunet/news/2120864/doubts-
jethr0
You might wanna check out his home page then
...these were highly experienced Linux admins.
- which chose an ancient linux distribution
- which tried to use bleeding-edge software on an old OS software platform
- which didn't know that glibc updates can break things
- which apparently didn't upgrade the system first if that's what they had in mind
- which took more than an afternoon to set up a linux system
- which were stymied by basic systems administration
- which appeared to be unaware of the tools available such as webmin
Wow. That's why I hire kids fresh out of highschool. They're so much more advanced than "experienced professionals" available to this guy.
...Steve
How long will this argument go on? Apples and Oranges I say. More marketing propaganda to buffer the bottom line. Technology will only move forward when we stop arguing over what is better and start working towards a common goal.
I've taken to recommending the Harmony remotes (now from Logitech) for anyone who has a home theater setup that they have a hard time controlling. Even non-techies can set them up fairly easily. Their only drawback is the remotes literally cost more than the TV/DVD/VCR combo box he mentioned above. (The Harmony 880 is $250 at Best Buy.)
John
He said (if you would have read the whole thing) that he wanted a larger sample size, but he didn't have the budget for it. The other point is that it **isnt** "Diffrent strokes for diffrent folks". The constraint was upgrading MySQL and that required an upgrade of glibc. Theres only so many ways to upgrade glibc...
And the distinct difference between drug testing and computers are humans. Computers do the same thing every time. Its their nature. Each human has a unique response to drugs. For example, I'm on a migraine mediacation that has no known interaction with alchohol. Except for me - if I have even a sip of wine, I will have an instant migraine and be incapacitated for the rest of the day. The uniqueness of the human body really can't be compared the the repeititve nature of a computer.
-everphilski-
He doesn't have to be lying. The fact that Microsoft funded the "study" means that you MUST look at the assumptions and process.
In the "study" in question, the Linux sysadmins were, for some reason, backporting patches from SLES 9 to SLES 8 due to the requirements of this "study".
So, no lies required, but because of the criteria chosen, Linux is far more difficult to maintain than ever in my experience.
don't get me wrong, I'm a gun person myself, but one of the first things I learned to follow to the letter was to keep my DAMN FINGER OFF THE TRIGGER UNTIL READY TO FIRE the weapon.
Like your average anti-virus vendor for example. I find it a little rediculous that virus writers eventually just started targeting buffer overflows, etc. in anti-virus software.
I think what we're seeing is the overall move from reactive (patching) to proactive security... and unfortunately, MS and Co. are taking the opportunity to inject DRM into what could be exclusively security related technological advances.
[Fuck Beta]
o0t!
It's all about the criteria. Why was the criteria such that the Linux sysadmins were backporting patches?
He doesn't have to be lying. The fact that Microsoft funded the "study" means that you MUST look at the assumptions and process.
RTFI: Microsoft funded the study but the good Dr. selected the criteria. (see: question #4)
-everphilski-
"All of our studies are written as if they will be released publicly BUT it is up to the sponsor if the study is publicly released."
My understanding is the sponsor will publish only favorable study. Do they have to choose before or after? Let's order a few studies and publish only the "good" ones.
Million Dollar Screenshot
How is it that Diebold can make ATM machines that will account for every last penny in a banking system, but they can't make secure electronic voting machines?
The reason is that Diebold is not required by any law or regulation to do so. The banking industry and financial networks demand and regulate the security and journalling of transactions. If you don't follow the rules, they don't let you run transactions.
The "voting industry," on the other hand, has yet to regulate or stringently demand minumum standards from e-voting machines. Until the constituency informs their lawmakers that they want the security of a) knowing that their vote went through the way they wanted it to, and b) knowing that no one can rig the election so that Snoopy wins, Diebold has no economic incentive to add these features.
BTW - for what it's worth, Diebold can't build an ATM machine worth a crap. They were one of the original ATM manufacturers, and thus have great brand-name recognition in the industry. What they build is over-engineered, over-priced, and over-proprietary. Think of the old IBM PCs that cost much more that their clone counterparts, used nothing that was off-the-shelf, and did no more than a cheaper computer. That's Diebold.
--- This
Tackhead specifically asked, "Who are these people"? I didn't see an answer to that question in the reply.
And the men who hold high places must be the ones who start
To mold a new reality... closer to the heart
"The fact that Microsoft funded the "study" means that you MUST look at the assumptions and process."
No it doesn't. Examining the study in EXACTLY THE SAME WAY as every other study will reveal its flaws. Nothing else is necessary.
The fact that you think the funder matters means you MUST look up "circumstantial ad hominem", because you used one and don't even know it.
I have no skin in this, but I've always wondered why people like you try so hard to stay ignorant. You're wrong about this, and you're using a common fallacy to suport your opinion.
Instead of insisting you are right, just learn something. It's easier than defending an erroneous position.
How pathetic are you that you follow me from topic to topic and waste all your mod points at once modding me down?
If it turned out that the windows admins had to actually do anything
And that's a completely valid response. If your choice of software allows your admins to do less work, perform less upgrades/migrations/etc. over a given timeframe... that's a good thing.
-everphilski-
But regarding windows patches; You say that windwos patches don't come in alpha or beta but wasn't there a windows patch not so long ago that broke VPN - not so great for corporate users.
His Linux admins had at least 5 years of enterprise Linux administration with at least 2 years administering SUSE.
Instead of saying "the admins were idiots" why don't you trying saying "the study was right"? You seem to be blindly rejecting anything that doesn't fit your world view.
If we look at the history of SuSE then we see Novell's big involvement was in the 9.0 world. Right from the get-go we can see that forcing the administrators to remain on SLES 8 is creating problems that would be considered a show stopper in a regular environment. Especially if you're talking about buying components with their required environments. The fact that you even have the option of applying SLES 9.0 patches to an 8.0 environment is something that you can't do in the Windows world.
What were the "third-party components" installed on the systems? The following dodge "The specific 3rd party vendors are not disclosed
because the focus of the study is the methodology and not a specific component." is complete bull if you're crowing about the repeatability of your experiment. How can the experiment be repeated if we don't know the items? (It would be interesting to know if those components didn't support SLES 8 at the time of their installation.)
Also, why this requirement for the components: "Support on both Windows and Linux" when your environments are obviously not equivalent (IIS/ASP versus LAMP instead of J2EE)?
--- I wish I could hear the soundtrack to my life. That way I'd know when to duck.
[At best, your study seems to show that the GNU/Linux distribution you selected was not particularly good at this task. But why does that show that the ``monolithic" style of Windows is better per se than the ``modular" style of GNU/Linux distributions?]
That pretty much sums up the entire study. This isn't really a test of Windows versus Linux, but a test of "modular" operating systems versus "monolithic" operating systems. And, unfortunately, the study didn't even do a good job of testing that.
Linux happens to include several distributions, some more "monolithic" than "modular". Unsuprisingly, the "monolithic" versions are usually those used by "enterprises", such as RedHat and SuSE. The "modular" operating systems, such as Debian, are almost universally ignored by businesses, though you will find IT personnel swear by them. There are Linux distributions that adhere to the Unix philosophy, and there are those that try to emulate Windows and Apple in the name of "ease of use". Hell, even some of SCO's products are more "modular" than commercial Linux distributions.
By requiring "enterprise" sysadmins and a Linux distro that is geared towards "enterprises", the study preselected a Linux competitor with which Windows can easily compete: admins (probably used to using Windows) using Linux distros that attempt to emulate Microsoft's "monolithic" operating system. By virtue of the fact that Microsoft has been building "monolithic" operating systems for at least a decade longer than any of these Linux companies even existed, that the vast majority of Linux components are designed to be used instead in a "modular" fashion, and that most "enterprises" wouldn't know proper system administration from their own asses, anyone can see that this test is designed to fail.
I've spent the last one and a half years doing this exact same study. Guess what I found? You can't treat "monolithic" operating systems, RedHat, Fedora, SuSE, Windows, as though they were "modular". Though doing so is easier with Linux, it's not recommended, and distro makers such as RedHat explicitly warn against doing so. Any IT guy learns this lesson about six months into his career. You either find a truly "modular" OS, such as Debian, or a good Unix, or you very carefully buy products made only by Microsoft or by companies joined at the hip with Microsoft. That is, if you choose modularity, you choose Unix. If you choose out-of-the-box integration, you choose Apple or try to navigate the Microsoft "ecosystem", and you pay monopoly rents for doing so. The people who choose RedHat and SuSE, and expect it to be Windows at this stage, are kidding themselves.
The real headline should be: "Linux admins tasked with using Linux in the same retarded-ass way as Windows, fail." Which should be no suprise.
But the important thing to take out of this is that it is neither technical necessity nor user requirements that make operating systems less "modular", and thus less flexible, less powerful, and ultimately less valuable. It is the commercial requirements of the operating system manufacturers themselves. It is the fact that the OS is commercial that makes it difficult to upgrade, impossible to integrate, and expensive to maintain. The evolution of commercial Linux distributions towards the "monolithic" model of Microsoft, and the concomitant decline in their quality, has proved this beyond a shadow of a doubt. At most, this study only serves to highlight what any competent Linux admin already knew.
"I assumed blithely that there were no elves out there in the darkness"
And get off that high horse you're sitting on and stop telling "us slashbots" what YOU think we should say.
On any public forum there's going to be noise and if you can't deal with that, and learn to sift through it then perhaps this isn't the place for you, hmm? -we don't want you here-
Dr. Herb Thompson talks a good story but it isn't supported by my first hand experiences - Why is that?
Maybe your first hand experience wasn't in a reasonably controlled environment. Maybe your bias will only allow you to see things one way.
Sorry Herb but your study is nothing more than a carefully crafted FUD attack on a superior product.
"Linux is better because I think so" is hardly a refutation. Why don't you point out the flaws in the study?
FFE4: What kind of credibility do you think you have, being a Microsoft MVP?
So how about, instead of relying on old prejudices, we instad attempt to actually examine the research and gauge it on it's own merits.
Oh hush. Why go against everything Slashdot stands for?
Admit it! You're working for Microsoft!
Now that I've accused you, I await a +5 Insightful mod, and the inevitable pats on the back.
Why are you trying to watch TV in my house? Get yer stinkin' hands off my remotes! :)
A used visor handheld with an omniremote module will be less than $250, and you can use it for other stuff, too. OmniREMOTE lets you create your own buttons and layouts. (ObDisclaimer: Omniremote is the product of a friend of mine; I have used it lots.)
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
The problems the study reported with Linux appear to all due to an incompatable unnamed 3rd party software package. Surely then, all this study can conclude is that the 3rd party software used was incompatable with SLES? And if not, why not?
They upgraded glibc? On SuSE? These "admins" aren't qualified to administer their home computers, let alone anything important.
"I assumed blithely that there were no elves out there in the darkness"
At least the one I saw did.
they'd be toll roads up to your driveway.
Like, uh.. large parts of the east coast? Please don't take this as a flame, because it's not, just an observation.
All due respect, but, like most folks who've taken a few short looks at libertarianism, and spend the rest of the energy they relegate to political thought on the intricacies of the spectacle being put on for us by the Democrats and Republicans, you don't seem to have full understanding of the libertarian perspective.
What you've posted is kind of an over simplification of Libertarian values. Libertarians aren't completely anti-govt. That would be Anarchism.
They just believe the govt. should stick with what the govt. does best. Fight our wars, and deliver our mail. I don't know a single libertarian that has a problem with road developement, however, most, like myself probably think that's a job more suited to the states, than the federal government. As are most issues they tend to stick their noses in. For example, if a few states want to legalize marijuana for medical purposes, what business is it of uncle sams? Why do they feel a need to remind people that it doesn't protect them from federal laws?
I agree the original poster's comments about him looking like a raving lunatic, however (even though he may not be). People with lazy eyes shouldn't pose for pictures with guns.
... what did you expect, something profound?
"All of our studies are written as if they will be released publicly BUT it is up to the sponsor if the study is publicly released. The vendor knows that they're taking a risk. They pay for the research either way but only have control over whether it is published, not over content. So if their intent is to use it as an outward facing piece, they may end up with something they don't like. Either way, I think it's of high value to them. If there are aspects of the results that favor the sponsor's product, in my experience, it goes to the marketing department and gets released publicly; if it favors the competitors product it goes off to the engineering folks as a tool to understand their product, their competitor's product, and the problem more clearly. Either way, we maintain complete editorial control over the study and there is no financial incentive for us if it becomes a public study or is used as an internal market analysis piece. The methodology has to be as objective as possible to be of any real value in either case."
But isn't this part of the problem with vendor-funded studies? (Maybe it's THE problem)
This WOULD be fine if it were just science for the advance of knowledge, but in the case of studies of *products* somebody somewhere is looking to use the information to make a product purchasing decision, or to promote a new product. In other words, someone is looking to either save money or make money using the results of the study. But those two goals conflict. For the purchaser, they would like to know both the pros and the cons of all studies involving that product. For the seller, they want to know both the pros and cons of their product, but only want their consumers to know the pros, and minimize the cons as much as possible. Both of these positions make complete sense... except for the group conducting the study. You have two different types of customers that you are trying to satisfy with these studies, but only one group is paying you to do the study - the seller. Hence, the results ARE skewed in favor of the organization purchasing the study, because they maintain control over whether the study gets released to the purchasers of that seller's products or not.
In this case, Microsoft has a win-win proposition, whereas for the rest of us, the purchasers, it's a win-lose proposition. Only if the study is positive for Microsoft will we be given more information necessary to help us save money. But if it's a study that puts Microsoft in a bad light, we lose because we don't get to see such information to make a purchasing decision, and may therefore make an incorrect decision.
I'm still skeptical that these "industry supported" studies are fully worthwhile to us, the purchasers.
How is a comment on the length of an article considered offtopic? I agree with the parent post.
I bet Microsoft (who is loaded) commissioned multiple studies with somewhat credible people. Eg different time periods, different assumptions, etc. Some will favour Windows and some Linux. They just don't release the Linux-friend ones and voila, a Window-friendly study by a credible guy.
Are loose ATMs like loose slots?
This is specifically what I was looking for too and it isn't too surprising that it goes unanswered. The original thread was the clearest attempt at skewing the results that I could identify.
I wasn't trying to find the proverbial 'brick in the gears' but sore thumbs do make themselves rather apparent.
A geek asking for less information? Please turn in your card at the next stop.
Dr. Thompson
Businesses demanded that Microsoft set a schedule when releasing patches. So now everyone using Windows has to wait until the second Tuesday of each month to get the latest patches. So yeah, I agree Linux et al may be faster, not because many people are hacking at the code, but because businesses made these demands with their wallets to purposely slow Microsoft. If someone discovers a new vulnerability after that Tuesday, we still blame Microsoft even though they were listening to their customers. It should be the customer's (some company) fault.
Blame the user, not the software.
slashdot didnt exactly put his "feet to the fire" with all the nasty questions about conflicts of interest, potential NDAs about the funding/details, oversight, or any of the other real issues where the source of the funding (seemingly the only reason this research even got any attention here) would have been a serious problem...
>Maybe your first hand experience wasn't in a reasonably controlled environment.Maybe your bias will only allow you to see things one way.Why don't you point out the flaws in the study?
The flaws in the study? How can I? I have not heard from the supposed 'experienced Linux Admins'. I don't know what proprietary products were deployed. I have no idea why Suse 8.0 was selected (not my first or second choice, by the way).
The study was funded and conducted for the sole purpose of finding a favorable result for Microsoft and that is exactly what it did. How can I possibley find fault with it when it did exactly what it was supposed to do.
I am getting tired of this game, aren't you?
Dude
...upgrading something like kernel.dll under NT4, 2000, XP, etc. It's not something lightly undertaken on a running machine- especially a production machine. Typically, when something of that magnitude needs an update, it's a full system upgrade- doesn't matter if it's Windows, etc. What makes the author of the report think that this was even remotely a fair comparison in question.
And I'll be honest, I find it fishy to say the least that he seemed to need that specific version of glibc; pretty much all vendors that are in the FOSS world try to track deprecated interfaces, avoid making calls to "broken" apis on the machines in question, etc. Even with a security flaw present, unless the glibc actually is the root cause, they will go out of their way to code around problems in most cases instead of mandating a glibc update for customers- it's that big a deal. Better yet, it seems that the official version updates from SuSE DID address all of this, including dealing with a fix to glibc that changed the revision number. If it's on SuSE's update sets, it's been pretty much vetted unless you change something fundamental, like glibc, at which time, all bets are off- it'd be the same way with Windows if you figured out how to accomplish a swap out of kernel.dll, or similar. Currently, for all distributions in main use except for Slackware, a system of handling all dependency relationships and obtaining all the official updates, etc. online. This is a KNOWN feature of all those distributions, whether you're talking Yast, urpmi, apt-get, yum, up-2-date, etc. Given that this is the case, not a single admin that actually knows what he's doing would have ever done what you describe in the draft 13 version of the paper on page 31, where you list things like admins doing by-hand updates of glibc, etc. That's "where Angels fear to tread" territory and would only be attempted by people that either roll custom distributions for embedded use or similar (Myself, for example...)- which would not be your typical sysadmin and they'd not be doing something like that with a production or pre-production server because they know better. And this is just one of numerous flaws with the whole study. I'll try to get to more later.
While I won't label you as a shill for Microsoft (partly because you're brave enough to face the gauntlet on this site...), I will question your ability to frame in adequate tests that actually test something- because you failed to do anything useful here except give Microsoft precisely what they were looking for. The work you did as presented to the whole world is hopelessly flawed in a manner not unlike what Mindcraft did for Microsoft a while back. I'd not consider your firm a reliable source of input or information at this point- while I was going to use one of your other papers that was provided online for a reference item in one of the white papers I am working on for my company, I must now largely discard this and find other sources for the information as everything you've produced is suspect because of the egregious flaws in the paper we're discussing.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
Only the guilty need to speak up!!!
Those idiots :)
Did Microsoft pony up the money for your study after examining the methodologies and metrics you used?
How much did they know before the study was completed?
How much did they know about the study before you released the results publicly?
Did you receive all your funding in one shot or did Microsoft release funds in tranches based on certain milestones?
If you had to make periodic reports to Microsoft to continue the funding, did those reports contain any Data or preliminary conclusions?
Do you know of any other people researching TCO issues that had their funding pulled or denied by Microsoft?
Sorry about the tough questions, but based on your initial replies, I don't think you will have a problem answering them.
Thanks.
deinesh.
Because you are a sheep-person. ESR tends to be a bit, umm, over-the-top. Dumb animals interpret passionate communication as a threat.
Dr. Thompson, however, communicates in a calm, soothing manner. He exudes compassion and empathy. This communication style places you at ease, and lowers your natural adversion to foreign objects and ideas. By communicating in a manner that lowers your stress level, he enables your higher functions (what little there seem to be) a chance to absorb his ideas.
So, at a basic level, whenever you see something that you don't quite understand, which, for you, could be a lot of things, you try to interpret it based on the manner of communication, rather than what was actually communicated. This post, for instance, tends to cast you in a rather dim light. You will interpret it as threatening. If, on the other hand, I had written something like "I understand your concern," or "I see what you're asking," instead of "you are a sheep-person," you would have taken the time to understand what it is I'm saying.
"I assumed blithely that there were no elves out there in the darkness"
Note: I've attempted to be fair to the original questions and responses, but my opinion may have affected how I've summarized things. If so, I apologize.
1) What were the assumptions?
2) does publishing studies like this help or hurt credibility?
3) Why did you force the Linux side to do so much more work?
4) Did you pick the metrics, or did Microsoft?
5) Why does Diebold make good ATMs but lousy voting machines?
Did your flame resistant suit include a matching tinfoil hat?
6) Why did you require 4-5 years experience for Windows but only 3-4 for Linux?
7) You only tested three administrators on one Linux distro. How does that really mean anything about the situation in general?
8) Is it good that vendors seem to be taking more control over what happens on my machine?
9) Which model has better security, especially fewer attacks and faster patches?
10) Is it good that OS vendors keep bundling more and more into the OS, or would it be better to just keep it a basic OS?
--
The universe is a figment of its own imagination.
The universe is a figment of its own imagination.
Libertarianism is the most misunderstood political party in the world. Mainly by Democrats because all they understand is more taxes, more government. So, when they hear of the founding principals of Libertarianism, they immediately jump to conclusions and start calling people names. What else would you expect from a Democrat? An actual intelligent argument against it? Puhlease...
Microsoft seem to be putting a lot of emphasis into how consistent over time their platform is, yet we are seeing a number of changes in current and upcoming Windows that seemt to discredit this. Dot-Net changes dramatically between version 1 and 2. SQL Server also introduces some breaking changes - such as case sensitivity in table and view names which breaks code that used to work assuming it didn't habe to worry.
Are these costs ever factored in to the Windows equation? Would it be interesting to compare system longevity in both Windows and Linux?
Also, how much does quality matter over quick and dirty? This is entirely subjective but I get the impression a lot of Microsoft stuff is quick and dirty. Visual Studio seems great at what may be Joe Average Programmers' tasks - but get into something serious like threading and it rapidly goes downhill. On the other hand Eclipse maybe doesn't have the pretty user interface builder (which can easily be used to build terrible user interfaces) but it's great at more advanced development and its refactoring tools really work well.
I often see a leaning towards quick and dirty. Hire a cheap developer. Knock it out quickly. Throw it away if it breaks or requirements change and it no longer fits. I also often see proven the advantages of a properly thought out and designed system - where maintenance becomes easy and the model responds well to change in requirements.
We say, sure, BUT we have complete creation and control of the methodology, it will be reviewed and vetted by the community (end users and independent analysts) and must strictly follow scientific principles... All of our studies are written as if they will be released publicly BUT it is up to the sponsor if the study is publicly released.
While I understand the reasoning, I don't think this should be represented as following scientific principles. In one of his most famous speeches, Cargo Cult Science, Richard Feynman specifically called out this type of research as being problematic:
IMHO the open source community is just as bad on average, if not worse. You better believe they have an agenda and they often aren't held under the same level of scrutiny as corporations, who have to face up to investors, competitors, governments, and "lottery ticket" lawsuits (especially Microsoft these days). The solution? We need fewer one-sided publishing of studies. We also need more studies overall, as they naturally conflict and are situationally dependent, but together would paint a better picture of the state of the world.
Of course finding funding for unbiased studies that will be published regardless of outcome is probably hard to come by.
The Linux admins were artificially given much more to do and screw up than the Windows admins, if the verbiage in the paper is to be believed. They were mandated to patch much more than is realistic, etc. in a production shop. If you were to have to patch all the local exploits in everything Windows related, you'd be very busy, moreso than the Linux admins- but they only had to do the Windows critical updates as MS provided them. The Linux admins were off patching everything- even if it wasn't very relevent to the servers (i.e. if it's a properly set up server, they shouldn't be ABLE to exploit local exploit possibilities, etc...). Worse, they had the guys doing manual updates to a lot of stuff, even though it WASN'T needed.
The study's heavily stilted to favor Microsoft and Windows- either through ignorance or malice. It'd be your call on how it got there, but it DID get there all the same.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
I'm not to defending the GP, just mentioning something your last sentence brought up.
/. knows bad grammar when they see it, but I doubt 99% of us could explain in a highly technical fashion why it's wrong.
"Why don't you point out the flaws in the study?"
I think it irritates a lot of smart people when they see/hear/read something and it 'feels wrong' or 'doesn't seem right'
I personally don't have specific knowledge in a lot of areas to say why something strikes me as wrong, but significant mental processing is happening at the subconscious level.
Grammar is a fair comparison. Most anyone on
It isn't directly analogous to the flamebot GP, but I hope you got the point.
[Fuck Beta]
o0t!
he did work for microsoft... they paid his bills for this study.
doesn't necessarily make the study right or wrong...
Thanks for giving those details. The study makes sense now. Basically it was a rigged demo.
Given the requirements the answer was a given. Lets count off the big ones:
1. A required OS upgrade after only one year in service? What? On a production system? No, you upgrade the OS when you add something that requires a newer version of something that is only available on a newer version of the OS or the deployed version is about to hit end of errata. And I'm sure you didn't budget a Windows upgrade, seeing as there wasn't one released in the timeframe specified. Now try again when Shorthorn ships and watch the MCSE kids clock up massive overtime.
2. Carefully cherrypicking 3rd party apps that were a nightmare to install on the selected version of Suse. The correct response would of course be to pick an OS from the vendor's recomendations. Then everything would 'just work.' What was done, on the other hand, would be as daft as trying to install an app designed for Windows Server 2003 on XP. Sure, if you are leet enough you might pull it off, but it would be crazy to put into production.
3. Then there was this beaut: "recommended best practice like not introducing out-of-distribution components." Wrong. You don't REPLACE a component in your distribution but without the third party repos (Dag for RH based distros comes instantly to mind) you are screwed.
4. A general mindset of trying to apply Microsoft/Sun server management theory to an Open Source platform.
Democrat delenda est
The study illustrates some of the weaknesses of the GNU/Linux methodology which were previously GNU/Linux strengths. For instance, much software in the Unix world is distributed as source code, yet problems constantly arise because people have moved from source distribution to binary distribution. As a BSD user who hardly ever uses x86 systems, I find it strange that the trend is heading in this direction, but it seems that this isn't the only way that GNU/Linux distros are becoming more similar to Windows. Binary patches seem to be commonplace, and so are "wizards" which are hardly stateful and therefore not particularly suited to a multiuser server, for instance.
Would it be unreasonable to suggest that a good lesson that GNU/Linux people could learn from a study like this is that moving towards the lowest common denominator is NOT a good thing?
with the largest degree of market saturation belonging to windows, doesn't the software with the largest amount of market share automatically equal an application which is by necessity tuned to run best on windows? i think this study is void without assessing the value of a linux system utilizing enterprise-level open source software. essentially what he's saying just means, "we picked out the best selling windows software and tried using the linux ports in a previously untried configuration."
this study is misleading as usual.
How to rig a Windows vs. Linux study in 7 easy steps!
1. Choose hardware that has known difficulties with Linux.
2. Plan simulated study over a time period in which the number of patches favors Windows.
3. Compare minor version change - Win2000 to Win2003- against a more complex Linux migration. SLES 8.0 (2.4 Kernel) to SLES 9.0 (2.6 kernel)
4. Deny administrators use of test systems, which is a Linux cost advantage. Test system can be run on available hardware with free license.
5. Run Linux with all available services instead on the needed minimum. This reduces system performance and adds difficulty to patches and migration.
6. Deliver external data from third party in a Windows favorable format.
7. Require several feature changes that are pre-built into Windows but requires customization in Linux
This "study" was designed to show Linux in an unflattering light. Requiring the "feature upgrade" with MySQL before doing the migration SLES 8.0 (2.4 Kernel) to SLES 9.0 (2.6 kernel) skewed this results in favor of Windows. That decision alone biased the study to the point that it is simply F.U.D.
A major possible fault of subject-is-buyer studies is the possibility of bias by selective publication. Do ten thousand completely fair studies, publish the favourable results and bury the rest. Or, a similar procedure but preemptive, focus the study's remit upon a known strength which is in fact surrounded and dwarfed by (un-studied) weaknesses.
In this the researcher may not actually be methodologically at fault at all. How did you protect your study from this kind of externally induced bias?
All others claiming to be him are imposters.
Hey, you're not Cmdr. Taco! Try as you might, your spelling and grammar lameness can't begin to match that of the Cmdr. Now go away and don't come back until you've learned to truly embarass yourself.
Who were the CIOs and industry analysts who helped determine the metrics? Were they more experienced with Windows or Linux on average? If there was a clear slant towards Windows-oriented participants, they'd tend to produce realistic scenarios in a way that would be soluble on a Windows box. If you needed a daemon doing task A, and this tended to be accomplished on a Windows system by a work-around that involved program B, you might find that the requirements were closer to "implement program B" than "implement a program performing task A".
For the love of God, please learn to spell "ridiculous"!!!
Many of us have several questions about the level of incompetence displayed by these Linux Admins. From the choice of distros to the botched installation of glibc, they made egregious errors that would have sunk ANY startup that they were intended to help setup. And given your knowledge of Linux from your home use, I think you know this.
Do you see this as a credible challenge to your study?
Can we talk with these supposed "admins" to gain insight into why they behaved so incompetently?
And given that you don't have enough admins to be in adherence to the central limit theorem, how do you feel your study applies in a general way to anything at all?
The research is badly done because it did not allow the participants to choose the best tools available for Linux.
I have used Conectiva Linux at work for several years, maintaining and upgrading a server. The only (almost) command I need to know is "apt-get update; apt-get dist-upgrade" for maintenance, and "apt-get update; apt-get install package name" for installing new packages. I have never needed to worry about patches or versions.
Dr. Thompson's words "the basic assumptions of the methodology are that patches are applied at 1 month intervals and that business needs evolve over time" imply a faulty methodology, which imposed unnecessary constraints on Linux. The apt packaging system makes the whole idea of "patching at n-month intervals" obsolete. Linux can be "patched" every day by an automated script, with no effort at all for the administrator.
I understand your point and I agree with you. In coding, when weighing different approaches to a problem sometimes a solution just feels like the right solution, when from a technical view the best path is ambiguous. In this case, however, the OP has a distinct bias. I suspect his feelings are more a product of the bias than any intuition based on experience. I was trying to draw out any rational thoughts he had on the topic, or maybe even learn about his experiences. It didn't work
Microsoft said Spooler was most likely just a DoS. Immunity Inc. let people know that was not true; the Spooler vuln was reliably exploitable remote root code exec & working exploit code was clearly in existence prior to or at least at the time of patch release.
2 005-q3/0221.html
At the time, a few months ago now Dave Aitel from Immunity Inc. said "Linux vulnerabilities are a thousand times harder to exploit than Windows vulnerabilities", and "'many eyes' have reduced Linux to a fished out pond, whereas things like strncpy() bugs are highly likely to still be around in remotely accessible (Microsoft Windows) components."
The following link seems to suggest that Microsoft (as of q3 2005) did not understand or worse misrepresented the "root source of vulnerablility" for Spooler; a critical security risk. Perhaps one could argue that Linux style patch transparency would have made that vulnerability/exploit far more publicly visible and would have resulted in fewer people being misled into believing it was a less severe risk (only a DoS, hah).
http://archives.neohapsis.com/archives/dailydave/
How much value do you place in the fact that Linux patches are always made available in source code form? Do you think that those "many eyes" Aitel talks about bring greater scrutiny to Linux bugs when they become publicly known? Do you think the nature of Linux patches results in a better or worse understanding of vulnerabilities and true risks?
http://secunia.com/product/22/
Currently, 27 out of 122 Secunia advisories (for Windows XP Professional), is marked as "Unpatched" in the Secunia database.
The study was funded and conducted for the sole purpose of finding a favorable result for Microsoft
How do you know this?
By the way, what is your experience?
and I still haven't read it (and I won't, for various reasons, including lack of time and frankly, lack of interest related to the reasons below.)
This point here http://interviews.slashdot.org/comments.pl?sid=168 949&cid=14084692/,however, makes it fairly clear that there were problems with this study. To what degree these were mandated by Microsoft, or added by someone on the research term with a bias one way or the other, or by someone on the team who just didn't know better, or whatever, is unclear. I won't make any accusations here at all.
One thing I would ask is: if the SUSE system had to be upgraded to the point that the RPM manager broke, why weren't backups done beforehand to be able to restore the system to its original configuration, to be able to back out the changes? Seems to me any competent sys admin - particularly one with enough experience to know that upgrading the compiler and/or libraries is risky - would have made sure he could recover the system if something broke.
This indicates to me either that the Linux sys admins weren't as competent as their years would indicate - and having five copies of one year's experience doesn't make you an expert, as the saying goes - or that there were other constraints on their performance NOT mentioned in the study - which would indicate bias (or incompetence or simple error) in the study design.
I think the real problem with this study is the idea of having a reproducible scenario to follow. In the real world, Linux vs Windows entails major differences in IT policy, administration policy, software, admin technigues, etc., etc. To even try to compare these on the basis of a single scenario is to compare apples and oranges. Also such a study does nothing to analyze the overall issues of vendor lock-in, security, quality of software, and many other issues.
It's easy to compare reliability and stability - how often do you reboot the machine? How often does the system crash? How often do you have security penetrations? It is NOT so easy to compare overall system functioning in a live environment. In that sense, this study HAD to be either biased or unable to come to any definite conclusions almost by definition. I am pleased to see the author acknowledging that the sample was too small to make any definitive conclusions, but I question his suggestion that the methodology has value.
This is essentially the problem with TCO studies in general. As a lot of people have said, TCO is very particular to what your overall policies and procedures are and these are specific to a given company. If you're a "Windows shop" and have no clue about anything else, it's going to cost you more IN THE SHORT TERM to switch to Linux than if you come from a UNIX shop. That's obvious. The REAL question is: what is it going to cost you overall OVER TIME to STAY a Windows shop than switch to Linux? Most TCO studies don't even attempt to touch that question. But the problem here is that a Windows shop is going to be totally different from a Linux shop, even if the "same" administration functions have to be done on the same hardware for the same applications.
There's just too much generality being brought down to too much specificity and too much extrapolation from the results to place any trust in these studies. And this author doesn't seem to show any more understanding of that than other authors - not surprising, since he's in the business of producing these studies.
I think it might be better to rely on more anecdotal studies of mixed Windows-Linux shops, such as we've seen occasionally here from sys admins working in them, that indicate the common experience of sys admins working on both sides, or the results of companies who HAVE mass-converted from Windows to Linux and who have then measured their costs and savings.
And in those studies, Linux beats Windows every time.
Again, remember what I always say in interpret
Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
but not in the way you expect.
There are a lot of people that are not Microsoft employees that have seen and have improved the Windows source code.
However, Microsoft is an Intellectual Property company. For better or worse, Microsoft considers its source code its crown jewels. Sharing that in a restriction-free manner is a risk that has been too great to consider thus far. What has instead happened is that MS has worked to get some of the benefits of peer review, but in a way that manages risk (selected audience, NDA, etc) of intellecetual property loss.
There are paid security consultants that review key portions of MS code; there are many Universities that have Windows source licenses.
My opinions are my own, and do not necessarily represent those of my employer.
I would be much more interested in seeing a whitepaper that compares reliability between Windows and FreeBSD.
FreeBSD is a pure, cohesive operating system. Linux is not. I'm not bashing Linux here, but if you have used FreeBSD you already know that Linux does not have the same cohesiveness, and therefore feels like a bunch of utilities slapped on top of a kernel, and a few added scripts to make it all work.
With FreeBSD you would not run into an issue where, "GLIBC had to be updated first" in order to allow for a MySQL upgrade.
Several companies doing what they can to GNU/Linux to personalize it and make it their own. What works on one Linux distribution may or may not work on another. There is, however, only one FreeBSD.
8. ??? :-)
9. Profit!
I wasn't being serious...though some of the replies I got seem to indicate that wasn't obvious! The detail was interesting. However, that is a really good summary.
You'll notice the mods for pointing out the hypocrisy.
Newsflash to the mods, this was a quote, not my statement.
But why bother reading it, when you can just throw your mods points away.
How pathetic are you that you follow me from topic to topic and waste all your mod points at once modding me down?
In fact, it was part of the requirments.
But they did NOT let them upgrade when any normal person would have. They REQUIRED them to stay on SLES 8 and backport patches from SLES 9
Any intelligent person would have skipped the backport process, done the upgrade when it became necessary and bypassed all the "problems" that were "found" in this "study".
Here's what I get from this guy's comparison: "We have compared apples (Windows) with oranges (Linux). First test metric was the color. The apples were green and nice smelling. The oranges were also green but not as nice as apples. The second test was how good the pie made from this taste like. The apple pie was much better and easyer to make since oranges are more difficult to grate. We also found that we can pick apples from a local orchard but oranges need to be transported from Florida. We have then concluded that apples are much better than oranges." Anyone is still in doubt this is a fair comparison ?
The real irony is I actually read through it first, but thought it would be a funny comment. Apparently, the sarcasm/humor was lost on some people. Then again, that seems to happen a lot around here :)
This is the commentary you'll recieve
"Man, that other AC hit the nail on the head: You are a whiny little bitch."
Funny how every AC on the planet decided to post, and all of them made my point for me.
How pathetic are you that you follow me from topic to topic and waste all your mod points at once modding me down?
I have grave concerns as I'm reading the paper. If the 3rd party component needed an upgrade to a new glibc, you would never have done what these admins allegedly did in the paper. It would have been a red-flag on the component in question and if it was something critical to the application, it would be assumed that the official version of the OS that was supported by the components was SLES 9, not 8 because it didn't have support for that version of glibc. You don't hack something like this in a production system, ever- even if you've got the skills to pull something like it off. I've got the skills, but even I wouldn't do what was done. You'd do a migration to the next version- period. There's far, far too many things that can go wrong and you really need to vet everything once you do it. What your esteemed admins did was analogous to someone haxoring kernel.dll by patching it manually and then putting it into a production Windows machine. I honestly don't know of anyone in their right mind that would do that one- ever.
.so file (Currently libc.so.6 on modern Linux and *BSD distributions...). This interface can be safely used for many years at a time, in spite of varying version numbers and the expected behavior will be the same for an older and a newer version- so long as you're not stepping on a bug within the older version or a new feature offered by a later on version of the runtime.
.Net framework for everything and then proceeded to install pieceparts of the OS to get it there.
Another faintly disturbing thing in this paper is that it's assumed that it's Linux at fault, when in reality, it was the ancillary components' requirements and someone trying to bull their way through the "problem". There's several problems with this, but I can number a few key ones for you:
1) glibc's interface, the ABI, doesn't change all that much over time. Typically, it's linked
to at runtime through a sonamed link to the actual
2) Yes, you CAN get away with minor revision updates of glibc without problems, but typically, you need to vet all your compiled code for regression testing purposes. It really, really is like replacing kernel.dll on Windows. If it isn't provided as an update, you've got a lot of regression work ahead of you to ensure that fixes done to the library don't break other code (Typically not a problem, but you never can tell when someone mis-used something...)- this is NOT something that your rank-and-file sysadmin has any real business doing. It's NOT their job.
3) Either the component stepped on a bug, or they're using some new feature of the glibc layer. In either case, you can't bull your way into using it on something that doesn't have the needed support level. What your admins did was analogous to trying to make this work on NT4, only to find out that you need the
The study's flawed- that plain, that simple. You can defend it all you'd like, but it's got bad problems that everyone, myself included, have been pointing out and you've avoided answering several of the key points we've been making.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
I didn't really figure you were -- but this makes sure if I missed something relevant in reading TFA, I'd hear about what it was (loudly, most likely). Personally, I'd rather get flamed now and then than miss something interesting.
--
The universe is a figment of its own imagination.
The universe is a figment of its own imagination.
I struggled through the first few pages, then scanned the article to find some keywords. The basic structure of the study was to place a new installation into a startup company then administer patches and upgrades throughout a year. What safeguards were there in place to prevent self-fullfilling prophecies (or self-enriching profitry ;) ? No offense, but was the study performed as single-blind or double-blind and what were the codewords?
=========
Microsoft's idea of double-blind is to stick their fingers into both eyes of their competitors.
You mention in one of the replies that the company funding the research (Microsoft) had control over publication.
This implies that they could have funded 10 different studies (and perhaps did, even with your company), and only the one favorable to the company was published.
In the case of this study, if they'd funded doing the same study 10 different times, with 10 different small sets of administrators (posssibly being performed by the same research company, possibly being performed by 10 different independent researchers), if even one shows results favorable to Microsoft, that would be the only published results. That's like paying 10 different doctors to perform a drug study on 2 patients, and only publishing the results where the 2 patients recovered, failing to admit that the other 18 died.
Alternately, the might have funded you to research 5 different tasks, but only one favored Microsoft, and the others were buried.
If you aren't paid to publish the results, regardless of who they favor, your efforts are, in my opinion nearly useless. (Especially with the caveats introduced by small sample sizes of administrators.)
So if my auto is recalled, it is all costs(usually) on the mfg. I'm in a mixed environment, win/mac(lose/win)... I recently had to manually update all M$ computers on the network, that all had been set to run updates at 3am once a week, due to the GAV tool had to be manually ok'd... What a JOKE! Who pays the cost for that, surly(I mean surly) not my boss.... So what would be callled a recall in most industries, is in fact fart of the TCO of M$...and I mean fart... I lost two days of productivity due to this TOOL.... I ended up working the following weekend to make up for wa$ted time... So who should pay me for fixing MS problems.... The fools (My Keepers)who bought into it, or M$ itself???
p.s. we have Mac servers that have never needed anything more than a pat on the ass, they just work.....
Sig Hansen?
Reminder: A lot more rides on your computer than on a tv/vcr/dvd combo. Your identity even.
>>All of our studies are written as if they will be released publicly BUT it is up to the sponsor if the study is publicly released. The vendor knows that they're taking a risk. They pay for the research either way but only have control over whether it is published, not over content.
you make this sound like msft *really* took a stand here. puh-lease. they win either way.
if the news is good, the hype machine goes into overdrive and the author becomes a well paid slash dot celebrity. of course, money and fame mean less than nothing to you, right?
if the results turn out badly, they hide the results from the public (no harm there!) and they learn how to improve their SW.
another win.
to paint this as some super risky behavior is a joke, imho.
in fact, it indicates advocacy on your part. you are advocating for the one who paid you. spinning something that a company did precisely to PROTECT THEMSELVES FROM RISK as very risky is ADVOCACY!
suse 8? i'm not even a month into linux and i wouldn't install suse 8. i will likely be installing suse 10.
that's a gaping problem here... besides your advocate assertions of risk.
you remind me of a trial witness we had in a high profile case in san diego... the issue was the date of fly infestations on a body. the entomologists used different temp data and various studies to arrive at minimum times for infestation. this was CRITICAL to the case.
a defense witness had a matrix of times and studies... all of them showed a time much less than expected. there was one point in his matrix left empty, though.
the prosecutor went over the math with this witness, a doctor. guess what? the result yielded the EXACT DAY ONE WOULD EXPECT if the accused had perped the crime. somehow, his portestations that he "just ran out of time" didn't quite ring true. you see, he was ADVOCATING for the accused. he left out data that would reflect poorly on his client. he was happy only discussing the positive stuff. he was paid by his client.
imho, consciously or not, you were happy with the result, predetermined or not, and the sound bite and headline were a go.
more pay days are coming. more name recognition on the way. you know you don't make money telling your employers what they don't want to hear. therefore, you install suse 8 and then require (or figure out how to just get it done) apps not designed for suse 8.
then, like the doctor claiming he ran out of time, you claim, "not my fault, business requirements." name one business that went through that methodology in the real world - w/o an easier alternative.
the validity and trustworthiness of your conclusions are basically worth nothing beyond an advocate advocating for microsoft's marketing advantage.
just for context - how many studies have you done that were unfavorable to and unpublished by the company that hired you to produce the study?
please don't say NONE!
or do. it would be nice to get and honest answer in context.
As others have pointed out, the root problem was a GLIBC incompatability with a closed-source binary-only application which was one of the requirements. For unknown reasons, upgrading to SLES9 was ruled out. As was running the closed-source application on a separate server. As was choosing a compatable product instead of the incompatable one. Moreover, the selection of the "requirement" applications was made solely on "market share" with no consideration as to the actual compatability with existing IT infrastructure. Basically, a series of poor techincal decisions which no competent IT organization would make. The only valid conclusion you can draw from this study is that choosing applications based on market share alone with no thought as to technical considerations can lead to unfavorable outcomes. Is that enough of a refutation for you?
These studies are nice and peek curiosity into the hypothetical. They are nothing to stake your cheese, the future of your company or the security of your government on. If you really want the straight scoop for massive Internet portals just play the old school yard game "Follow the leader". What stack does Slashdot use to avoid the Slashdot effect all day every day? What stack is Google using NOW? What stack does Yahoo use? You surely must dismiss the giants that are selling stacks, because they will try to eat their own dog food no matter how much they choke on it. So the PHD's keep publishing or die, trolls keep flooding the place with mod points, I will ignore you all the same way I ignore Politicians promising to "work for me". Go away I have work to do.
It is a wonder that this research did not look to existing implementations of their design to see what the "best in class" solutions were using. Hummm....
Keep the sheep comments to yourself unless you have millions to spend on independent research yourself. A few engineers and a lab burn up 1 million dollars real quick!
Come on MS trolls you have lots of points to hand out...I bet this little puppy catches a Singapore caning.
Gizmos Gagets For Ninjas
Great question! Blablabla... kitchen sink. His answers don't seem to answer anything.
If this was going to be a comparison of like systems, they would have used PostgreSQL vs MSSQL Server or MySQL vs MS Access.
One of the most striking things I believe the experiment shows is that not only are the operating systems different, but the administrator's job requirements are different as well.
On a Linux system, an administrator (especially a "successful" one) is expected to know how and perform compilations of software on a regular basis. This not only includes knowing how to work a compiler, but also what components to include when compiling a piece of software.
In contrast, in the Windows world, compiling and finding components is the job of the software developer. A product is not considered "finished" if an administrator needs to do something besides double-clicking on setup.exe.
In comparison, would it not be more fair to compare a Windows admin with 5 years experience to a Linux admin with 8 years of experience?
That is more a comment on the sad state of our IT world than on the study (which I haven't read).
Joachim
People don't write Manifestos any more -- what's going on in this world? [Frank Zappa]
I have to say that I found the answer to question nine to be a dodge, so I will give my thoughts on the subject :-)
There has been a lot writen about apache vs. sendmail in terms of security and those issues seem to apply to apache vs IIS as well.
Apache seems to have a lot going for it that IIS does not. (See the Halloween documents)
One of the biggest things going for apache is httpd.conf.
It is documented and a seasoned *n*x admin can quickly tell what it is doing.
This seems to be one of the big advantages exim, postfix, and qmail have over sendmail.
I am never entirely sure that sendmail works as I think it does. I also had the same problem on IIS (thankfully now on the scrap heap.) I didn't have a problem configuring it, I just couldn't quickly tell how it was configured.
I would also say that patch distribution is so varied (ports, apt-get, home brewed, etc.), and people routinely have uptimes in years on *n*x side so I would expect that IIS actually has a better patch distribution system than apache, just a substandard product with even worse patches.
I also suspect that apache has an advantage in the fact that it lives on many different operating systems, many of them incompatible with each other, limiting the scope of the freebsd apache vulnerablity, for example.
Care to chime in?
Work bio at MMWD
I found Dr. Thompson to be very forthcoming and objective in his responses, and certainly not a "Microsoft shill". From his responses I think he was asked to deliver a specific study by Microsoft, and he followed the MS request as closely and objectively as possible. That said, I think that the scenarios he studied don't exactly match real-world practise. He did, however, seem to welcome and encourage scruitiny of his study and others before making critical decisions rather than taking action based on reading executive summaries.
Anyways, here is my "Coles Notes" version (Cliff Notes for those American readers out there):
1. Q: what assumptions did you make in your study and how did they affect the results?
A: Lot of verbage around "every organisation is different/this depends/that depends/etc"...eCommerce application with constatly changing needs (not a steadty-state system), 3rd party software installed, all critical updates applied, upgrade OS at end of study...etc.
2. Q: I'm an IT grunt...every MS-sponsored study I've read is total crap (doesn't reflect my experiences) and many others feel the same. Doesn't this make doing MS-sponsored studies a risk to your credibility?
A: Regardless of funding our studies are objective, peer-reviewed, etc. but you have to carefully read the study and its methodology to see how it matches your situation. More verbage about "it depends" etc...infers that Windows (or Linux) might be better for you.
3. Q: No fair! Linux admins were made to run multiple versions of db on one machine and other unwise/unsupported stuff and Windows users just had to use one DB and didn't have to do such stupid things! Real sysadmins in ay case would use vritual machines or staging hardware.
A: Study wasn't supposed to cover performing the same tasks--it was to make each platform meet the same end goals. We chose 3rd party packages based solely on market share and in the case of Linux thosed packages needed this sort of work done. In Dr. Thompson's opinion he agrees VM or other hardware would've been best but this still represents "more pain" than Windows.
(question 3 in MY opinion reveals a big reason why this study is flawed--see below)
4. Q: Did MS invent the metrics/methodology or did you come up with them?
A: MS did NOT come up with these--MS asks for a study on certain qualities/characteristics and we insist on independenly developing the study metrics and methodology. MS risks getting results that are not in their favour but it is their choice whether to release the study publically. If they don't they at least get value for their money in the form of suggestions to improve their offerings.
5. Q: Diebold makes crap voting machines but we never hear of serious issues with their ATMs. Why is that?
A: Saying Diebold voting machines "concern" him is a "serious understatement". Can't make an informed assesment on ATMs but they are probably more secure and reliable because they are carefully watched 24/7/365 by banks AND their customers to make sure they work right--functions and their results very visible. Voting machines are not scruitinised as thoroughly and are only looked at during elections. Advocates "checks and balances" and more openness and accountability (letting independent security experts have more access to the systems, etc).
6. Q: Why did you require more experience from Windows admins?
A: Oops..that was a typo as a reult of upping the experience reqs for ALL admins. Requirements were matched as closely as possible.
7. Q: study was so small--hw can you make any statements about the benefits of one over the other? How does your study really show "monolithic" windows is better than "modular" linux?
A: It doesn't--study like this cannot make "sweeping generalisations" about any platform--it studies reliability in a certain scenario. Should
A study commissioned by the Northwest Association of Apple Growers proves conclusively that Apples DO taste better than Oranges.
try { do() || do_not(); } catch (JediException err) { yoda(err); }
The study is obviously skewed, no matter how interesting Dr Thompson found the questions he didn't adequately address the major technical flaws in his study. I'm waiting for him to justify the forthcomming Novell study where Windows admins are forced to install an unsupported 3rd party application that requires they hack windows libc with a hex editor.
Not really, it just highlights the lack of standards.
If there was an open standard for remotes and manufacturers ensured interoperability, then it wouldn't matter whose brand Satellite, DVD, VHS, etc. you bought. They would work the same at your house and at your freind's place.
Back to the topic, what the study's author did wrong is to pick a best-fit Microsoft solution for the MS side. Then he tried to make Linux jump through hoops to match Microsoft's way.
The outcome may have been quite different if he picked a best-fit Linux solution and then tried to make Microsoft conform to the Linux solution.
I didn't know what it was so I ate it.
Ok.... You have two basic issues: 1) questionable methodology in *this* experiment and 2) making more out of this document than it should be.
When reading this paper, it did not come across as a recipe that businesses should use in determining their platform commitment. It came across instead as a paper detailing why WIndows was superior.
The second issue is that the experiment itself was designed in such a way as to make any conclusions we can draw from it to be *very* limited. Indeed, what one can draw is that if you want to run your business entirely based on COTS with no modifications from you, then Windows may be the best choice (if you are running an e-commerce business). If you want a more customizable environment, where *you* can keep up with the latest trends by managing extensions to the software products you make, then this study has no value whatsoever.
In essence, this study is only valid where certain limited assumptions hold up. The only real failure is not to state these up front and indicate that these limit the results of this experiment to be meaningless in most cases.
What would be more appropriate would be to give the admins entirely free reign regarding how they impliment the solution and then compare total administrative labor.
LedgerSMB: Open source Accounting/ERP
Actually it IS a comment on the study - because the study took piss-poor IT decisions which favor the Microsoft way of doing things and imposed them on Linux.
More importantly, the study is attempting to compare two systems while the environments themselves are so different as to render the study almost meaningless.
You're right, though, the main result is to demonstrate how IT is totally screwed up.
Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
A lot of people are trying to poke holes in the study itself though it seems to have been fairly well implemented.
I did however notice two interesting bits that cause me to put a lot less importance on the results
With three people there's certainly likely to be a lot of variability and to get some conclusive results, I'd love to get a huge group of administrators across the spectrum in terms of experience. I'd also love to do it across multiple scenarios, beyond the ecommerce study.
And a little later
it is up to the sponsor if the study is publicly released
Simply fund a lot of small legitimate studies with a high variance, publish only the results that fit your case. In a way it's like one big badly done study where someone throws out all the data points that don't fit their hypothesis, for all we know he, or another researcher, might have done a dozen other studies which came out in favour of Linux and were subsequently ignored. The research itself is all completely legitimate but Microsoft creates a false overall conclusion through selective publication, perhaps companies who fund the studies should be held to the same eithical standard as those who do the research?
I stole this Sig
There's a lot of fancy ducking and dodging, none of which changes the facts that:
Tired of Political Trolls? Opt Out!
> But a second point would be to mention that SUSE is not a server
> distribution.
SLES, SuSE Linux Enterprise Server, is not a server distribution? Now, I'm a RedHat guy myself, but that's _really_ unfair.
I agree with most of the rest of what you said, though. The time it took them just to set things up raises a huge red flag in itself.
>I have no idea why Suse 8.0 was selected (not my first or second choice, by the way).
Did you read the study? Or even his response? I really can't tell from your response.
I can't guess what sort of problems you have with the choice, because you chose to give us no information beyond "It stinks".
Was your problem with Suse at all, or why they chose 8.0 instead of 9.0? Where is your problem?
"The period we looked at was July 1st, 2004 to June 30th, 2005"
Out of curiosity, what were your top choices as of July 1st, 2004 for an enterprise level solution? (ie, something breaks you can yell at the vendor...)
>The study was funded and conducted for the sole purpose of finding a favorable result for Microsoft and that is exactly what it did.
Thank you for the clear description of how you reached that conclusion given the evidence.
With such transparency in thinking you should really get into doing studies for profit as well! I can definitely see how you would be able to wade through several people's complex decisions, describing each one perfectly in all detail for every possible reader in addition to choosing people who would have through processes and environmental experience that exactly matches all the readers as well.
I'm not saying to cut the study any slack at all... I'm saying that just because you happen to believe in the prevailing theories of this group here, doesn't mean you can just say "Me too!" and expect everyone to think you're smart. (Did my AOL and MS references make it through?)
Am I smart? I really try not to be... someone might actually ask me to write something useful, and then I'd really be done for.
You suggest that a larger sample or other similar studies would be more insiteful, but you know that more were likely conducted. Your's was published because the results were favorable to MS - some others were probably not. By agreeing to let MS decide if people see the results, you are skewing the results that the public sees. If you're not concerned about what the public sees, then put it in the contract that the results will not be made public regardless of the outcome.
You're only one sample in the "metastudy" that MS conducted. They show the data selectively because you agreed to let them do so. That's just piss-poor research.
"I didn't have a problem configuring it, I just couldn't quickly tell how it was configured."
Good point. Try figuring out how an entire Windows 2003 Server is configured - especially Active Directory.
Bring supplies for a long stay at the office...
Now break it and try to figure out how it broke.
Bring your wife and kids for an even longer stay at the office.
Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
Roblimo's questionable book "Simply Mepis" was a total waste where can I get A refund. He like most LINUX Nerds doesn't consider that there are ever any problems with LINUX and fails to help newbies to LINUX by not pointing out the weaknesses and glossing over these points claiming IT's All so easy! A responsible writter would use the problems to introduce the newbie to the syntax and proceedures for correcting script and making changes. Like GRUB boot loader... the version on his disk in the book ias faulty for some installs the fix turned out to be adding a space to the script but he totally avoids even suggesting this is a problem. He shoul dgo back to LIMO driving and quit being such a LAMEO Author
See, here's the thing. You have qualitative studies, and you have quantitative studies. Qualitative studies can help you discover flaws in a product, basically by systematically collecting anecdotal information. They can tell you a lot about the differences between a couple different products.
But anybody trying to extract judgements about which product is better, based on such a qualitative study, is taking things way too far.
Quantitative studies measure results. To do a quantitative study, you need a representative sample for your study--the more representative the sample compared to the population, the more confidence you have in your results. Six administrators is nowhere near an adequate sample size to have any confidence in the results of this study.
But taken as a qualitative study, we can just learn from the results. Microsoft is counting on all of their customers being dumb enough to think that the experiences of 6 individual administrators in a study can persuade you to believe that Windows is better than Linux. That's a big stretch, a big leap to make in your judgment, but Microsoft is hoping you're too lazy to actually dig into the qualitative results and learn the lessons of the study.
We should actually thank Microsoft for taking Linux seriously, and helping us find its weak spots! We already know that open source is a much more effective way of solving them--just witness the current unresolved IE script vulnerability...
Open Source Solutions for Small Business Problems
Freelock Computing
Etc. etc.
Mind you: I'm not commenting on the actual value of his arguments (I know next to nothing about the subject matter), but this read eerily straight from the "How To Successfully Communicate In Meetings"-Handbook.
sig? Oh, that sig...
Answer: SLES 8 was the most recent at the beginning of the study time period - July 1, 2004
There's your problem. I have had nightmares upgrading from one SuSE release to another, requiring a re-install every time. SuSE tends to break itself very easily. But never had such problems with a well tested distribution like Debian GNU/Linux 'stable'.
Seems to me that state's rights issues are completely separate from libertarianism - what real difference does it make if the government is an individual state or the federal government?
I've worked as an admin on both Windows and Linux (specifically SuSE) I don't understand why these experienced Linux admins didn't just say this is not acceptable for the stability of the system. Disclaimer: I have never worked worked for someone (in an ecommerce setting) that didn't understand a simple statement like that. Especially when it could be easily translated by even the most moronic boss into "You are demanding that we do something stupid here and if we do it and the system fails then I will explain EXACTLY why. It just seems wrong to me that the admins would do something dumb to their system even if it is for just a study.
The lesser of two evils is still evil...
Anybody who asserts that they can conduct an unbiased study that is paid for by a beneficiary of that study is simply fooling himself; trying to defend that is just making him even less credible.
There are lots of other problems with the study apart from its intrinsic bias. The selection of experimental subjects and the statement of business requirements both reflect a naive view of how these things work in practice in a real organization; selecting them in "the same way" is, in fact, not at all selecting them in the same way.
The only thing this study shows is that these people don't know what they are doing and that they can be bought.
So, he came here to make vendor ties to research more acceptable. Is it really appropriate for Slashdot to give him a platform to do this from? Has Slashdot been paid to provide this PR?
I don't see what's wrong with rejecting corporate-sponsored research, and favoring independent research that isn't sponsored by the company.
... and then they built the supercollider.
I was also astonished that they would use SuSE. Who uses SuSE? This is a server ... you use either Debian or Fedora core.
--simon
home page
If you dont belive him dont run a ms server, if you do belive him run a ms server.
I have found that the quality of a product is inversly proporianate to the amount of advertising needed.
ms = large amounts of advertising dollars
linux = word of mouth
Im sure there are ad dollars going to linux the above was just a generalization.
I am certainly not gonna change my server os cause he said ms is better.
All of our studies are written as if they will be released publicly BUT it is up to the sponsor if the study is publicly released. The vendor knows that they're taking a risk. They pay for the research either way but only have control over whether it is published, not over content. So if their intent is to use it as an outward facing piece, they may end up with something they don't like. Either way, I think it's of high value to them. If there are aspects of the results that favor the sponsor's product, in my experience, it goes to the marketing department and gets released publicly; if it favors the competitors product it goes off to the engineering folks as a tool to understand their product, their competitor's product, and the problem more clearly. Either way, we maintain complete editorial control over the study and there is no financial incentive for us if it becomes a public study or is used as an internal market analysis piece. The methodology has to be as objective as possible to be of any real value in either case.
So they are paid to provide honest assessments of MS software vs. competitor software and then if the results are good it goes to marketing, and if they're bad they go to engineering. MS is huge and has zillions of products, and it probably pays for dozens or hundreds of these tests every year. The interesting question (which would never be answered) is "how many such studies have you done for Microsoft which never get shown to the public?" You can guess your own answer.
This kind of thing is a huge issue in academia where non-results (e.g. failures to obtain a predicted result or failures to replicate a published result) are seldom published. So if one researcher uses dodgy data to buttress a result and gets a publication out of it, it's highly unlikely that failures to replicate the result will ever see the light of day. In this case, Microsoft hires someone to perform a test of product A vs. B, it favors A, so they publish. The numbers are so small there's no statistical significance -- so for all we know MS could have given the same brief to the same outfit five times, gotten one positive result, and sent it off to marketing.
My biggest concern is that the methodology drove a glibc upgrade on a production system. In my experience, any methodology that forces a technical disaster like that to occur must be inherently flawed. I wouldn't manually upgrade glibc on my personal systems and I certainly wouldn't dream of doing it on production systems.
I'd like to know if threat-risk assessment was part of the methodology. My own internal TRA suggests that a glibc upgrade is an extreme risk. A chrooted glibc, a parallel glibc, or recompiling the third party application against the existing glibc are minor risks. Why was an extreme risk chosen when minor risks were available? This reeks to me of methodology running rampant over common sense and industry best practise.
How many were "UnPublishable" in MSFT's eyes?
If this issue is covered by an NDA, We understand but, we'd like to know that.
OSGGFG - Open Source Gamers Guide to Free Games
I've seen several posts where people were criticizing the solution chosen by those 'linux experts'. Whether they are experts or not - is a matter of different subject. I think that we should look at the broader picture. And instead of talking about replacing glibc we should talk about the study's conclusion, not how they reached this conclusion.
/. readers is far greater than knowledge of single linux administrator. And what he does can be criticized by others. He did the best he could when he approached that specific problem for the first time.
Ok, so they have chosen to replace glibc, perhaps in their circumstances they couldn't see any other applicable solution. I'm a true linux zealot, I've thrown windows away 5 years ago, and I do the same with computers of my friends and family. And during that time I learned a lot. For instance I've learned that each problem has dozens of solutions, and when I approach each problem for the first time - I do not choose the best solution possible. But I do after solving that same problem several times - because I learn how to do that.
The bottomline is that collective knowledge of all
Another bottomline is that even skilled linux administrator - when meets a new kind of problem - will not choose the optimal solution, and most of the time he will spend on it - will be learning stuff, reading man and googling.
to summarize, I'll repeat my other post - I can belive that setting up a linux sever can take longer time than setting up a windows server doing the same.
But such conclusion from a study means nothing, since everybody will agree that UPTIME of linux server, and its need for MAINTIENCE will score far better than windows server doing the same.
And that's the only reason for which I think that this study is totally useless.
1. I agree that setting a linux server takes more time
2. But 1. is irrevelant because it's the uptime, and maintience for years, that really counts.
Dear Dr. Herb Thompson - tell me if you plan to study how linux vs. windows score in "uptime and maintince" in real corporate environment. I think that this test should be at least 1 year long.
#
#\ @ ? Colonize Mars
#
(BTW, I don't like the IR port on the side of the Visor, I much prefered the "extended IR" of my original Palm, but that wasn't the fault of O.R. As a matter of fact, the "rotate" thing in O.R. was a really cool trick to compensate for it.)
What I most appreciate about the Harmony is that setup is all automatic. I didn't have to train it with 50 buttons on each of four different remotes. I didn't have to draw hundreds of tiny boxes, meticulously laying them all out on aligned grids, giving each an illegible mnemonic. I didn't have to redesign the concept of a remote.
I spent dozens of hours with Omni Remote, and the ORdesktop software. I tried importing codes and stuff from the Pronto (mixed success there.) I tried finding layouts on-line that other people had done. I went so far as to digitize photographs of my existing remotes to try to figure out a sane layout for the screen. I basically spent forever trying to set up the remote, and never got it perfect.
The Harmony eliminated all of that. I bought the remote. I installed the CD-ROM. I plugged it into the USB port, which automatically took me to their web page. I registered a new account. I typed in the model numbers for each of my devices. The web page then gave me a list of activities: "Watch TV", "Watch a DVD", "Listen to Radio", etc. It asked me "what input do you need to set your audio system to watch TV and hear it through the stereo" and gave me an appropriate set of radio buttons to pick from. It asked "what input do you set the TV to watch cable TV?". After answering those few questions, I clicked "done" and it downloaded the new data straight to my remote, which just worked. It took about half an hour, total. I've played with the web page a few more times just because I could, not because I had to. But the remote just worked the first time I downloaded to it, and it had all the functionality it needed to pass the dreaded "wife test."
All in all I find the Harmony is actually cheaper than the Visor + O.R. when I factor in the time investment. I can't recommend O.R. to someone who isn't a techie, or to someone who doesn't have the time to fiddle with creating dozens of screens.
John
Looks like the MS shills are crawling out from under their rocks again...
That if you consider that the decisions are representative for most large companies, then they'd have had issues with Windows. It's as if the admins deliberately chose the absolute wrong path for each Linux decision. I've made these points earlier, if you did the analogous things under Windows you'd end up with a mess much like the one the Linux admins ended up with- probably worse.
You simply don't replace glibc without vetting code- all of it in the system. An official update or the next version should (and typically does) have the bulk of this work done for you already. If they're saying that the revision control process wouldn't let them upgrade the version of the distribution, they already broke that because an update to glibc of the nature in question IS a version update for all intents and purposes.
The whole thing is flawed because the analogous insane decisions were NOT applied to the Windows side as well.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
That is how you professionally answer a question.
I know that is how I do it when giving a presentation, or anything along those lines.
Being surprised and learning that something new is *true* is specifically what geeks search for. Disovering a new paradigm (MS superiority) wasn't found here. The admins were idiots (5 years of knowing how to spell Unix). What is true is that the study was broken and many of us wish it weren't so. You dont hand MS binaries to one guy and tell the other guy to build linux .so's....unless you want to tell the MS guy to build his Dlls too.
Often wrong but never in doubt.
I am Jack9.
Everyone knows me.
JSR $FFE4 (GET whatever key is being pressed
c64 assembly
...niether is glibc updates for Linux admins... When they are, they're handled, typically, in the manner a Windows update is... What they did, wasn't what would be in either world.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
Yes, it's entirely believable that a glibc upgrade was required, because when you compile a program that binary is usually locked to the version of glibc it was compiled with. Newer versions are OK, older versions aren't.
Why not just use a local copy of the required glibc version and set LD_PRELOAD or LD_LIBRARY_PATH in the binary-only program's ENVironment before running it? That way it won't affect anything else.
We can blame the admins, or the people who set the conditions of the test, or whatever, but the real problem is that Linux is crap at handling binaries.
Yes and no - if you compile statically, it doesn't matter what glibc version you're running, or anything else. If you use dynamic linking, then you need the right versions of system libraries, JUST LIKE with Windows. Do you even remember using Win 95 and Win 98 and DLL hell and DirectX upgrades that make your new game work but break the old one, etc., etc., and new software that requires XP or 2003 only? And not only XP, but a certain service pack level?
If you have a constantly changing system (e.g. rapid glibc development OR DirectX upgrades) then there WILL be these sorts of issues. Linux is not unique in this respect. Even Mac software requires certain versions, like 10.4 but NOT 10.2, etc.
It sounds like the study chose a pathological case on purpose (requiring a system-wide glibc backport instead of a local copy particular to the one binary-only 3rd party program). They also arranged the schedule so e.g. vendor fixes were installed monthly -- in accordance with MS release dates -- thus not penalizing them for being 29 days behind with a critical fix that was fixed on Linux the same day it was publicized, that kind of thing.
I'm going to say one last thing on this, which will shut you and all your ignorant friends up.
If the study is flawed, you wouldn't have to resort to ad hominems. Examining the study using traditional criteria is more than sufficient to gauge its accuracy.
Et tu, Brute?
Do you eat your own dog food?
in your response, you indicated that 3rd party components were chosen purely based on their market position. i'm not sure who you were working with, but as a consultant, i've never seen a competent CTO/CIO make a decision based on market position. what they have done, on the other hand, is pick the right tool for the job - namely, evaluated the quality of the software, the features, the technical support *and*, most importantly, its support for the platform they were going to run it on. in my experience, in the real world, a component requiring such hoops (installing unsupported versions of software, upgrading glibc etc) to be jumped through would never even make it to the evaluation stage. with all of that said, it is highly debatable that the 'no custom code' approach is even worth looking at - it does not reflect real life. and this is coming from someone who's worked with some of the top 100 ecommerce sites in several european countries and the US.
Parent does not add anything new to the discussion, yet it's modded insightful? Did Microsoft tell it's employees to start posting on Slashdot or something?
Dr. Thompson,
In your opinion, would this scenario have actually happened in a large enterprise that was running Linux?
I see none in this study.
e +software+sles8&btnG=Search
I have a SLES8 box here in the lab. I would enjoy reproducing the tests to see for myself.
The 3rd party software used is not disclosed. I see no commercial ecommerce solutions that run on both Windows and SLES8. http://www.google.com/search?hl=en&lr=&q=ecommerc
I updated a SuSE8.2 Workstation (same core as SLES8) to SuSE9.0 with no major issues. I clicked on the Yast -> System -> Software Update after inserting the 9.0 CD1. This method is not addressed at all in the research paper.
In appendix 5, the table of companies consulted is listed, but not the contacts. I would like the names of the people contacted for this research paper that work for NASA, the FAA, the FTC, the USAF, the DoC and the DoJ. The FOIA gives me the right to query what was actually contributed from the US Government towards this research paper.
Enjoy,
It's just the normal noises in here.
I really enjoyed your summary. However, could you please put your .sig in the actual .sig area on Slashdot, and not list it here? Some of us have those little taglines turned off on purpose and do not like to see them. The other way I know of to get rid of them is to change the relationship, but that seems kind of drastic. I do enjoy your posts, just not reading your .sig.
He didn't, or at least, that's not the bad part. The key issue is that MICROSOFT DECIDES WHETHER TO RELEASE THE STUDY. This means that only good (for Microsoft) studies are released. A study like this provides an interesting road map for a real study, as mentioned in several of the answers, but it is far too small to be statistically significant. An easy method of sure success for Microsoft is:
1. Commission many too-small studies with their $$$$$.
2. Only allow the statistically insignificant positive results to be published.
(3. Keep the info from all of the studies so that they end up with statistically significant results.)
4. Profit!
Although the moon is smaller than the earth, it is farther away.
I know.
I was just joking, you know.
sig? Oh, that sig...
Is this enough to make sweeping claims about the reliability of Linux/Windows? No way.
I don't think that'll stop Microsoft from making said sweeping claims, though.
Seacrest out.
There are only 10 kinds of people in this world... those who understand binary and those who don't
In your answers to question 7 you say:
"Is this enough to make sweeping claims about the reliability of Linux/Windows? No way."
"Hopefully these are the questions that people will ask after reading this study, and with any luck it will prompt others to carry out their own analysis within their own IT environment, building on what we started here"
In the orginial Slashdot story, you are quoted as:
"The study claims that Windows is "more consistent, predictable, and easier to manage than Linux.""
Why is it that you hope your study will prompt others to carry out their onw analysis within their own IT environment, but the concolusion of your study says that Windows is "more consistent, predictable, and easier to manage than Linux?"
who | grep -i blond | date cd ~; unzip; touch; strip; finger; mount; gasp; yes; uptime; umount; sleep
We all know any self-respecting Linux admin reads Slashdot.
;)
So why can't these guys answer for themselves? Are they too ashamed?
And I'm afraid static linking doesn't solve the problem either. You can't statically link everything, the NSS doesn't like it and will crash, and if you statically link libraries but not glibc you get the same problem.
That's why you STATICALLY LINK glibc if you need to. What don't you understand about that? I used a certain binary-only program for years across multiple distros which was statically linked, and worked on all of them. Transgaming's utils are statically linked as well. Correct, you don't need to do this for everything, just for the binary-only packages that are otherwise a pain. Linux has historically supported multiple simultaneous libc versions since it did not used to use glibc but its own C library. Distros had an option to install old "compat-libs" so your old binaries would continue to work even if they weren't statically linked.
In fact I'd be amazed if you can even build glibc from the sources yourself,
Amazingly prescient that I never mentioned compiling glibc. Just use the old glibc
I have been consultant for several Medium and BIG Corporation and government. It is my personal experience that "BIG Corporation" will forbid updating the GLIBC and many other Linux components without the express consent of a specific groups or individuals. Usually, when a policy mandate something... There is also procedure in place that are limiting the liberty of the SYSADMIN to do thing it's own way. This generally includes any modification that is not "sanctioned/approved" by the OS Distributor/DATABASE Supplier, etc. GLIBC will logically fall in this category.
This is generally true on UNIX, Linux or Windows.
In most cases, when a piece of software need something that is outside of the corporate "Standard", the Supplier of this software get call and asked the best course of action. For example, does it have something compatible with "SES8"? If not, what will be the best alternative solution, etc...?
It is also my personal experience that great many new OS get put "in limited" production (meaning: for a specific function only) to get support from third party tools. A while back, that was true for MS-Windows 2003 mandated by some ISV when most corporations did operate on W2K.
In most "big company", there is a mix of MS-Windows NT 4 that runs some applications that can't be upgraded to MS-Windows 2000. Either many Windows 2000 or many Windows 2003 but generally some of the others because some software mandate it. In extreme cases, there is "specific software" that can even force a specific Service Pack Level (EX:MS-Windows 2000 SP3). This appends when a business application has been heavily updated by an ISV, is too different from the one currently in production but the older one can't be run under the newer OS. On MS-Windows, little option exists but to keep a server, somewhere operating under this non "compliant" OS.
Microsoft OS use to be the worst offender in this area. I know great many company that use "many harware boxen" for the sole purpose of avoiding incompatibility between ISV applications of different vendors. I also know many companies that refuse to use the same "hardware" for too many MS-Windows application because it might become a nightmare to update them -- if the business mandates it.
Of course, I also know about Linux/Unix boxes were the GLIBC has been upgraded/downgraded to accommodate a specific application. Generally, this will be to have otherwise "Incompatible" software operate on the same "computer". This is not a routine decision.
What I really want to say: In most case, at most company, the servers OS policy are stringent but a lot of WAIVER are issue.
At the desktop level, this is different story. In this setup, several different software have to be executed out of a common "OS"/"SPx" level. When new software needs a specific release of the OS... This is where the "$" take it full meaning. Upgrading a few thousand desktops is always a challenge. This is generally under these conditions that we ask a "Windows SYSADMIN" to tweak the configuration to "MAKE IT RUN" without upgrading if possible at all. Often the proposed solutions are ugly and needs considerable time to develop (weeks or months).
This is under these conditions that your "Study" does not fly with me. Generally, big corporation mandate that the OS must be such and such. If the Business application does not run under these conditions, we look at the best alternative to be supported by all software distributors. (We may download from the developer site some software, but this is not a common procedure at most place)
It's kind of a forced, interview-voice sense of humor. I mean, Herb, I know you're a human and a person and all that, but I just don't believe you. You seem to waffle too easily.
You do. Just say it. Go ahead. Say, "I WAFFLE!" and then I'll respect you, I swear.
Please stop stalking me, bro.
I'm left wondering what non-technical skills the admins in question had ?
IME, most linux admins with significant experiance have mainly worked in environments where *they* are the final arbiters of the chagne controll policies etc. They've tended to work in smaller companies, or been putting linux in the edge or certain well defined services (i.e. DNS), and not at the center of larger companies.
Windows admins on the other hand (again, IME) are used to working with the larger comapnies and the beurocrecy that goes allong with it.
I hope you realise it makes everything you say afterwards sound fake.
TWW
"Encyclopedia" is to "Wikipedia" what "Library" is to "Some people at a bus stop"
'nuff said.
[Pruneau
I noticed how the linux results are projected to all distributions of Linux, where only Suse Linux was included in the study. Each distribution has it's strengths and weaknesses. The be truly objective, The report title should not imply that all linux is Suse linux.
Another point is the integration issue. The TV/DVD/VCR analogy in one response could be consider a bit flawed. A more accurate analogy for some linux distributions would be more like the matched component systems from company's like Zenith and RCA where you have a TV, a VCR, and a DVD player in separate boxes with a universal remote that defaults to the same brand conponents.
Everything works fine unless you have to replace a component or the remote with a different brand or model. However, if the VCR eats a tape, your mother can get you to drop by, and take it to the repair shop, (or fix it yourself) and she won't have to miss her soaps.
Over the years I have used redhat, Mandrake, turbolinux, slackware, and debian distributions of linux. I currently use debian, since it includes a lot of games and education software for the kids. All of these distributions had
strengths and weaknesses for certain application.
In summation, Windows is Windows, Linux is not Suse. Linux is to Suse what kernel.dll is to Windows. And the linux kernel can be easily tailored to a specific application, which is one of it's advantages.
and... the study should have included a larger sample....
Similarly I consider the above item on third party out of distibution software stupid - for a start you have the application software that you want to run and bought the machine for in the first place - in the case of servers of any OS it may not necessarily come bundled with the base install. Considering MS Windows you REQUIRE third party applications just to get it to function properly in the first place - antivirus and functional backup software if nothing else.
As all those studies from/sponsored by m$, one subject is allways not touched. What happens when you have to change things like the harddisk or the motherboard?
Every tried to move a domain server or a ts to another hardware? This is just a pain. Microsofts backup almost never works and you can find such answers like "install new" in their knowledge base. What this means for a painfull (through search and click in the best case and normally with tons of stupid registry changes) configured windows system is missed in almost any positive study about windows.
Why was SuSE chosen over perhaps Microsoft's biggest Linux [server] competitor RH ES? For me that tainted the study...
After the experiment, the administrators were asked on both sides if this kind of evolution of systems met with their real-world experience. They said yes, with the caveat of if they were asked to install a component that required an upgrade of GLIBC that they would likely upgrade the operating system as long as their configuration control policy allowed it.
Oh OK so the Linux admins thought everything about the test was real-world, except, ummm, the operating system version.
Sounds great. Oh and the study included SIX groups? Wow this just gets better and better.
The thing that scares me is how excellent people are getting at hiding their agendas, even when they are RIGHT OUT IN THE OPEN. Micro$oft-funded linux vs. windows study? Nah, no conflict of interest there, seriously! (and people believe it!) A couple ex energy company executives decide to bomb the country w/ the largest oil reserves? Nah, no conflict of interest there, serious! (and people believe it!) DIEBOLD makes some e-voting machines, their CEO publicly claims he's "Devoted to delivering bush Ohio", and Computer Science PhDs across the globe are horrified at what they see when DIEBOLD's source code is leaked? Nah, no conflict of interest there.
Open your eyes. Someone stop this idiot before he eats our children.
Why stick up for big business?
i use windows.. i use dos... i use an open source unix emulator on a windows partition too... and I can say that a study that claims test conditions as a medium for comparison between a unix architecture and a windows platform as a way to bring meaning to MSsofts apparent denial of unix/linux expansion at this current time???? ask any real programmer or technician endowed with real programming skills which operating system will do what... 10 to 1 windows will do the job... and unix\linux will effectively control a real programming environment with a command line interface, system heirarchy and built in support for programming in virtually any language in the real world thats 80% of all programmers with any credible ability to run, compile and administer an operating system without replicating on a basic gui interface thats probably going to overheat where a linux\unix mchine just starts to move... using all 139 patches in the test conditions and the 1001 more required for real life application. i think this demonstration just undercut exactly how many patches would be required by windows (39 or something) by 1 too many to even be credible... sorry dr.thompson sounds like a hustle to me!