This is *really* interesting. It gets to the "philosophy" of research as opposed to this study itself - we talk about this internally all the time and about how we can build an industry infrastructure to support this Feynman-esque research. Here's what I'd love to do: get a group of industry folks together on all sides of the fence (so there's no question of funding); agree to some ground rules, a methodology, and then also agree that the work will be published no matter what. To some degree that's what some of the consumer review groups do but I don't think we have a *real* equivalent in the IT world for the really big stuff. This gets down to the question of how could we set up something truly unbiased (perceived or real) in the Feynman sense of the word that would also work as an economic model. It seems like a consortium of consumers (organizations that use technology as opposed to selling it commercially) who do not have a vested interest in the outcome would be ideal. It would be great to get some responses to this thread with some suggestions. Again, the premise is simple, and funding from a fairly neutral third party like the government is one thing, but how would the IT community do something where multiple participants in the user world would be willing to fund it or multiple vendors, as a group, will be willingly to take that risk?
In response to the question you referred to about fairness, here's why I think the study was fair, and here's what I think the limitations are. If I'm a business that needs to deploy some solution I know what I need in terms of business requirements. There are a lot of ways I could technically implement a solution to those business problems. We tried to come up with a methodology to give people insight into the challenges they might run into before they do an enterprise deployment. In the experiment, you've got the assumptions we started with, and the administrators were given fairly free reign. As far as patches, the Linux guys ended up going to different places at Novell for the majority of components and then to MySQL for updates for newer versions they installed. Similarly, the Windows admins had to go to the Windows Update site for patches but also had to check for patches to SQL Server. At a high level, giving some folks business requirements and seeing how they implement them with a particular technology base is fair. The limitations though are the small sample size, the lack of a detailed configuration control policy and the high potential variability of the small group. I think that it's great to question the paths that the admins followed. I think that there are a million ways that they could have approached things, and I guess the key takeaway for me is that given three experienced linux admins we got three really different results. I do think that if that's recognized as a challenge then we can put procedures in to minimize the risk of some of the problems encountered here. You may be prepared to assume that responsibility and in some situations it might even be in your best interest to do so (possibly highly customized environments, embedded,...). I hope that this study will put Company X be in a better position to do their own evaluation.
Answer: SLES 8 was the most recent at the beginning of the study time period - July 1, 2004
- which tried to use bleeding-edge software on an old OS software platform
Answer: All the components used were available in the time-correct period of the study. For example, if they installed a component in the simulated September 2004 time period then that version was available in September 2004.
- which didn't know that glibc updates can break things
Answer: They did know that GLIBC could break things and tries to minimize the breakages (see study)
- which apparently didn't upgrade the system first if that's what they had in mind
Answer: Good point! The only configuration control issue was that the enterprise wouldn't upgrade the OS version until July 1, 2005. This is mainly based on our experience with companies that don't move to the latest OS version until it has had time to "bake" in the community. At that time, SLES 9 was hot off the compiler.
- which took more than an afternoon to set up a linux system
- which were stymied by basic systems administration
Answer: Not sure there's anything to respond to here...
- which appeared to be unaware of the tools available such as webmin
Answer: Hmmm...not really sure how using webmin would have helped in this situation. They were free to use any tools they wanted though.
It was actually one of the 3rd party components that required the GLIBC upgrade and not MySQL. If it had been MySQL and they had the SRPMs I'd agree with you (although that may lead to some wierd patching problems down the road). Many 3rd party commercial vendors only provide the binary RPMs and that was the case here too. Again, let me say that we chose components based on market share without knowing that these issues would crop up. That's why I think it's critical to apply this methodology in your own environment because you get the added benefit of any configuration control policies you may have in place, and going through the exercise may, in addition to helping you select a platform, help you select the 3rd party components that minimize pain too. Most of this kind of stuff just ain't documented in the install/release notes.
FFE4 = JMP ESP on x86 (one of my favorite instructions for certain contexts - buffer overflows in particular:)). It's one I created just for this interview and thus got a UID heading towards infinity!
Slashdot Mirror
This is *really* interesting. It gets to the "philosophy" of research as opposed to this study itself - we talk about this internally all the time and about how we can build an industry infrastructure to support this Feynman-esque research. Here's what I'd love to do: get a group of industry folks together on all sides of the fence (so there's no question of funding); agree to some ground rules, a methodology, and then also agree that the work will be published no matter what. To some degree that's what some of the consumer review groups do but I don't think we have a *real* equivalent in the IT world for the really big stuff. This gets down to the question of how could we set up something truly unbiased (perceived or real) in the Feynman sense of the word that would also work as an economic model. It seems like a consortium of consumers (organizations that use technology as opposed to selling it commercially) who do not have a vested interest in the outcome would be ideal. It would be great to get some responses to this thread with some suggestions. Again, the premise is simple, and funding from a fairly neutral third party like the government is one thing, but how would the IT community do something where multiple participants in the user world would be willing to fund it or multiple vendors, as a group, will be willingly to take that risk?
In response to the question you referred to about fairness, here's why I think the study was fair, and here's what I think the limitations are. If I'm a business that needs to deploy some solution I know what I need in terms of business requirements. There are a lot of ways I could technically implement a solution to those business problems. We tried to come up with a methodology to give people insight into the challenges they might run into before they do an enterprise deployment. In the experiment, you've got the assumptions we started with, and the administrators were given fairly free reign. As far as patches, the Linux guys ended up going to different places at Novell for the majority of components and then to MySQL for updates for newer versions they installed. Similarly, the Windows admins had to go to the Windows Update site for patches but also had to check for patches to SQL Server. At a high level, giving some folks business requirements and seeing how they implement them with a particular technology base is fair. The limitations though are the small sample size, the lack of a detailed configuration control policy and the high potential variability of the small group. I think that it's great to question the paths that the admins followed. I think that there are a million ways that they could have approached things, and I guess the key takeaway for me is that given three experienced linux admins we got three really different results. I do think that if that's recognized as a challenge then we can put procedures in to minimize the risk of some of the problems encountered here. You may be prepared to assume that responsibility and in some situations it might even be in your best interest to do so (possibly highly customized environments, embedded, ...). I hope that this study will put Company X be in a better position to do their own evaluation.
Responses inline:
...these were highly experienced Linux admins.
- which chose an ancient linux distribution
Answer: SLES 8 was the most recent at the beginning of the study time period - July 1, 2004
- which tried to use bleeding-edge software on an old OS software platform
Answer: All the components used were available in the time-correct period of the study. For example, if they installed a component in the simulated September 2004 time period then that version was available in September 2004.
- which didn't know that glibc updates can break things
Answer: They did know that GLIBC could break things and tries to minimize the breakages (see study)
- which apparently didn't upgrade the system first if that's what they had in mind
Answer: Good point! The only configuration control issue was that the enterprise wouldn't upgrade the OS version until July 1, 2005. This is mainly based on our experience with companies that don't move to the latest OS version until it has had time to "bake" in the community. At that time, SLES 9 was hot off the compiler.
- which took more than an afternoon to set up a linux system
- which were stymied by basic systems administration
Answer: Not sure there's anything to respond to here...
- which appeared to be unaware of the tools available such as webmin
Answer: Hmmm...not really sure how using webmin would have helped in this situation. They were free to use any tools they wanted though.
It was actually one of the 3rd party components that required the GLIBC upgrade and not MySQL. If it had been MySQL and they had the SRPMs I'd agree with you (although that may lead to some wierd patching problems down the road). Many 3rd party commercial vendors only provide the binary RPMs and that was the case here too. Again, let me say that we chose components based on market share without knowing that these issues would crop up. That's why I think it's critical to apply this methodology in your own environment because you get the added benefit of any configuration control policies you may have in place, and going through the exercise may, in addition to helping you select a platform, help you select the 3rd party components that minimize pain too. Most of this kind of stuff just ain't documented in the install/release notes.
FFE4 = JMP ESP on x86 (one of my favorite instructions for certain contexts - buffer overflows in particular :)). It's one I created just for this interview and thus got a UID heading towards infinity!