To satisfy your curiosity: I am a software tester. I spend my day looking for problems and trying to convince developers, designers, and managers that we need to fix the problems before we ship the product. To get more fixed, I try to offer workable solutions to many of the problems that I find.
Would you trust the patches more if the patch system told you how many people had installed the patch, how long it has been installed on a critical mass of systems, and how many users reported problems after installing the patch?
(I don't know if any patch system does this...just asking)
Agreed, there is a gigantic difference between a software feature and a hardware enforced software feature.
However, if a sysadmin wants a moderate confidence that a junior admin hasn't installed software that shouldn't be running, or that a user hasn't installed unauthorized games on a critical system, the Software Restriction Policies are a good additional safety feature that can be used in addition to other security techniques.
You can already do this with Windows XP and Windows Server 2003. There is a security policy that allows you to prevent the system from running any binaries that you didn't sign.
The downside is that you also need to individually sign the patches too, and that can be time consuming.
Check out Windows Server 2003 - Microsoft was really trying to focus on security, and even got bashed by customers because they made it so secure that some of the applications wouldn't work anymore.
You don't "fix" 50 million lines of code overnight, especially not when it has taken 10 years (or more) to write. However, all of the developers really did take a few days to go through a set of classes on how to write secure code, and then spent the next month reviewing their code for security problems. All of the program managers really did go to classes to learn about security vulerabilities and how to find security weaknesses in their designs, and then went back and updated designs where needed. All of the testers really did go to classes to learn how to find security bugs and then created security test plans and spent a month doing nothing but looking for security bugs.
It probably isn't perfect, if Microsoft went for perfect you would be paying ten to twenty times more for the software, but for the first stab at really fixing the server operating system so that it is secure out of the box, I would say that 6 months of effort went into making Windows Server 2003 secure that wasn't in the plan prior to the trustworthy computing initiative.
Oh, sorry, I was thinking just binaries...as a tester, that is what I usually deal with.
Yes, diffs are used on an hourly basis all over the place. The SCM system and other tools that everyone uses constantly have both traditional unix-like diffs, and colored graphical side by side diffs, etc.
Developers are frequently working on a dozen little changes, a few large changes, and working on service pack fixes for two or three different products in a given day...they would go nuts if they couldn't compare the changes that they made, track and visualize previous changes, etc.
Microsoft doesn't really use diffs much, when everyone is on 100 megabit networks and physically grouped together by product and technology, it doesn't make sense to make a diff of a 200kb file so that the guy across the hall can toss is on the test bench and run regression tests.
The Windows team uses a custom in-house SCM. Yes, it is rather tricky, and no, it isn't Perforce.
Umm, that is not entirely correct. You can purchase embedded NT, and get just the kernel and the drivers that you select. It won't run most off the shelf software because they expect the "bloat".
If enough people ask for it, the laws of economics demand that someone will provide it......then you have a different testing problem - instead of testing one complex product, you need to test a hundred thousand products of varying complexity!!!
If you are interested in what Microsoft did for NT and Windows 2000, look at the links in the parent post.
Just a couple of highlights: The complete source code was 50 Gigabytes. Build time was 8 hours. The source code control system tracked over 411,000 files.
There were a lot of challenges trying to keep 5000 people working on the same operating system at once, they learned from problems and improved the process for Windows 2000.
It is high-level data, but it is still quite interesting.
For a small slice of how Microsoft actually tests software:
The lifecycle of a software bug in the Windows Division
1) The bug is found, reported in a bug tracking system, and assigned to the developer. 2) The bug is evaluated by managers for severity and triaged in a daily "war" meeting. At this point, the bug may be postponed until the next cycle, or marked to be addressed in the current cycle. 3) For all open bugs in the current cycle, the developer investigates and creates a fix, frequently running a few tests before marking it as ready for test. 4) Testers make sure the bug is fixed, look for any additional problems, look for related issues, and frequently even run a regression test pass to make sure that the developer didn't accidentally break something else while making the fix. If there are additional problems, the bug goes back to the developer to make a better fix, otherwise the bug is marked as okay to check in. 5) The developer then code reviews the changes with another developer, builds the changes for all platforms to catch any possible compile breaks, and then checks in the changes. 6) The build lab picks up the changes for the day and starts to compile. 7) If a compile break occurs, usually because someone was in a hurry and didn't follow the rules, an on-call developer triages and fixes so that the compile can continue. 8) When the build finishes, it is installed on a set of machines, and a series of build verification tests are run to ensure that the build is at least good enough to run some tests. 9) When the build verification tests finish, then the testers install that build and double check that the bug is still fixed, and mark the bug as such. 10) Finally, the tester adds a regression test to their test plan, and automates that test so that it will at least be run before the end of every major cycle, sometimes every minor cycle, every week, every build, or for some issues even as part of the build verification tests.
Major cycles are for betas, and final releases, minor cycles are for releases to be deployed internally, builds tend to come out daily. At the start of a cycle, and in early cycles, the bar is fairly low, almost any bug can be fixed and added to the build. Near the end of each cycle, and at later cycles, the requirements are increased so that only changes that are absolutely necessary are taken, reducing the risk of introducing new problems that won't be discovered until after the product is released. At some point in every major cycle, the bugs and test plans are reviewed to find areas that need improvement.
Additionally, instrumented code to measure test coverage, quality standards in a number of areas like accessibility, reliability, scalability, globalization, localization, integration, interoperability, are measured for improvement, usability studies are performed, code profiling tools are used, code scanning tools look for code execution paths that could result in problems and automatically file bugs, testers bash on other components, and anything else anyone can think of to find the problems early.
However, the pace is incredible and problems can come from anywhere. Imagine testing an Xwindows application to configure networking while the kernel is changing, the networking core is changing, Xwindows is changing, the shell is changing, the compiler is changing, your application is changing, and the tools you use to test with are changing. It is a challenging job.
If you want to bash Microsoft, that's fine, I used to...hence my handle, but now that I have seen inside the "beast", it's just a business, most of the rumors are very off base, and most of the people there are just normal people who want to do the right thing.
The problem is that there are quite a few people out there that assume that just because it is in binary form, that it can't be figured out. For example, they will use XOR to "encrypt" data stored inside the program, or assume that their secret algorithm is safe because it is compiled.
The barrier to entry is definately raised, but it is always possible to figure out what the compiled code is doing given enough time and effort. In fact, I've even heard of people who patch operating system kernel code without the source...
...I don't know why the RFID tag is fried...the bill was just sitting in my wallet. Oh, by the way, I work around power transformers all day, maybe the magnetic fields from those are causing a problem.
I had a number of things going on that box, initially I wanted a dual-nic router so I could have more than one networked computer in my dorm...the dual-nic issue wasn't resolved by changing IRQs, etc. In fact, the dual nics worked perfectly in windows, but required a bit of code tweaking in linux to make it work. And yes, I read all of the FAQs I could find at least a dozen times.
Then I moved out of the dorm and into an apartment, and it took another couple weeks (a few hours each day) to get dial on demand working correctly....no configuration changes on that system were ever simple with linux.
Good point, I guess the way I always looked at it is: Can I take the in-house staff that normally works as a salesperson or a stock-boy, but who has a bit of computer experience, and let them spend a few hours on it at $8.50/hr rather than call in the local computer company for $50/hr with a 1 hour minimum.
The target users won't necessarily be experienced admins. Think home networks, small business networks, branch office networks, etc.
I'm fairly certain that the usability engineer verified that the configuration was correct for the scenario that was given, although the scenario might have been very simple (I don't know).
Your point about by design or by accident, is well taken. Additionally, if you are large enough to have a "production network" I would agree that I would rather not have a novice admin working on it. At the same time, many organizations do not have good IT processes or controls, and sometimes the novice admin is the only one available to solve the problem (I have been in that situation several times in several different organizations...and once I did make the problem worse).
If you are fortunate enough to have the luxury of hiring experienced admins, good for you. Too many organizations hire the cheapest person with the lowest minumum qualifications, and expect them to make do.
I would agree that Win98 isn't a stable system. I wouldn't use anything from the Win9x line on my machines.
I can see significant improvements in many scenarios when moving up the NT line. Today, I wouldn't use NT4.0, and Windows 2000 is significantly better than NT4.0, and Windows 2003 is significantly better than Windows 2000 (from the parts that I have looked at).
And to be fair, my connection sharing experience with linux was 4 or 5 years ago, and linux may have improved significantly since my horror experiences of needing to customize a network driver to coax two identical NICs to work at the same time under linux, how to get dial on demand working, and how to get routing to work with dial on demand.
I'm sorry, I think you are confusing the posts from two different people.
In your original message, you made it very clear that your admins failed to configure Windows for over a week. Are you saying that your Linux admins take over a week to configure a box to run email in an already existing email infrastructure?
My original post talked about configuring DHCP, and in my example, both the linux and windows admins could configure DHCP on Windows in under an hour, where linux admins in groups mostly failed to configure linux as a DHCP server in a several hour study.
I didn't get a chance to meet these linux "admins" perhaps they didn't know what they were doing, but they were managing several linux boxes as a part of their regular work responsibilities. The same class of people (linux "admins") peformed much better when configuring Windows servers.
Maybe the people in this group were ripping people off...I don't know for sure.
At the same time, I remember spending several weeks configuring a linux box to dial on demand and route local network traffic to the internet so that I could share my dialup network connection with my roommates. Granted, I wouldn't claim that I am an experienced admin even though I have installed, configured, and run my own servers for the last 8 years.
Windows NT 4.0 was better than Win95, but it still wasn't up to snuff as a server. However, compare Linux with Windows Server 2003 and I'd go for Windows.
And for the comment about a text file making configuration easy, I don't think you ever looked at a sendmail config file.
Interesting story: A company I work for runs usability tests on Microsoft software. These usability guys have watched Windows Server Administrators with very little experience set up a DHCP server in less than 15 minutes, and almost nobody fails to complete this task when given an hour to work on it. Even novice unix admins can set up a windows DHCP server in under an hour.
Then, they decides to try the same thing with Linux, putting users in groups, with internet access, and after more than a day's worth of man-hours (and several real hours) most of the groups still hadn't been able to configure a linux box to be a functioning DHCP server.
With the 10 hours I'm paying the linux admin to figure out how to do it in linux, I could have paid my Windows Admin to configure DHCP on a Windows box while saving enough money in time to pay for the copy of Windows too!
Slashdot Mirror
He painted it green.
As far as I know, John Deere paints everything they sell green.
Although I'm not sure about the 5,000 part myself.
To satisfy your curiosity: I am a software tester. I spend my day looking for problems and trying to convince developers, designers, and managers that we need to fix the problems before we ship the product. To get more fixed, I try to offer workable solutions to many of the problems that I find.
Would you trust the patches more if the patch system told you how many people had installed the patch, how long it has been installed on a critical mass of systems, and how many users reported problems after installing the patch?
(I don't know if any patch system does this...just asking)
Agreed, there is a gigantic difference between a software feature and a hardware enforced software feature.
However, if a sysadmin wants a moderate confidence that a junior admin hasn't installed software that shouldn't be running, or that a user hasn't installed unauthorized games on a critical system, the Software Restriction Policies are a good additional safety feature that can be used in addition to other security techniques.
You can already do this with Windows XP and Windows Server 2003. There is a security policy that allows you to prevent the system from running any binaries that you didn't sign.
The downside is that you also need to individually sign the patches too, and that can be time consuming.
Check out Windows Server 2003 - Microsoft was really trying to focus on security, and even got bashed by customers because they made it so secure that some of the applications wouldn't work anymore.
You don't "fix" 50 million lines of code overnight, especially not when it has taken 10 years (or more) to write. However, all of the developers really did take a few days to go through a set of classes on how to write secure code, and then spent the next month reviewing their code for security problems. All of the program managers really did go to classes to learn about security vulerabilities and how to find security weaknesses in their designs, and then went back and updated designs where needed. All of the testers really did go to classes to learn how to find security bugs and then created security test plans and spent a month doing nothing but looking for security bugs.
It probably isn't perfect, if Microsoft went for perfect you would be paying ten to twenty times more for the software, but for the first stab at really fixing the server operating system so that it is secure out of the box, I would say that 6 months of effort went into making Windows Server 2003 secure that wasn't in the plan prior to the trustworthy computing initiative.
Oh, sorry, I was thinking just binaries...as a tester, that is what I usually deal with.
Yes, diffs are used on an hourly basis all over the place. The SCM system and other tools that everyone uses constantly have both traditional unix-like diffs, and colored graphical side by side diffs, etc.
Developers are frequently working on a dozen little changes, a few large changes, and working on service pack fixes for two or three different products in a given day...they would go nuts if they couldn't compare the changes that they made, track and visualize previous changes, etc.
Microsoft doesn't really use diffs much, when everyone is on 100 megabit networks and physically grouped together by product and technology, it doesn't make sense to make a diff of a 200kb file so that the guy across the hall can toss is on the test bench and run regression tests.
The Windows team uses a custom in-house SCM. Yes, it is rather tricky, and no, it isn't Perforce.
Hmm, obviously you don't use OS X to manage more than a couple hundred users... OS X simply doesn't scale, or at least the management UI doesn't.
Apple isn't spending money testing scalability, although their GUIs are certainly pretty.
...so, since OSS has no costs, it could be considered "well-funded"...is giving OSS away for free "dumping"?
Umm, that is not entirely correct. You can purchase embedded NT, and get just the kernel and the drivers that you select. It won't run most off the shelf software because they expect the "bloat".
...then you have a different testing problem - instead of testing one complex product, you need to test a hundred thousand products of varying complexity!!!
If enough people ask for it, the laws of economics demand that someone will provide it...
Or how about if 'we' decided that for public health, you should eat your steak well-done rather than medium rare?
People tend to want options, all of those 'bells and whistles' are just the options that people keep asking for.
If you are interested in what Microsoft did for NT and Windows 2000, look at the links in the parent post.
Just a couple of highlights:
The complete source code was 50 Gigabytes.
Build time was 8 hours.
The source code control system tracked over 411,000 files.
There were a lot of challenges trying to keep 5000 people working on the same operating system at once, they learned from problems and improved the process for Windows 2000.
It is high-level data, but it is still quite interesting.
For a small slice of how Microsoft actually tests software:
The lifecycle of a software bug in the Windows Division
1) The bug is found, reported in a bug tracking system, and assigned to the developer.
2) The bug is evaluated by managers for severity and triaged in a daily "war" meeting. At this point, the bug may be postponed until the next cycle, or marked to be addressed in the current cycle.
3) For all open bugs in the current cycle, the developer investigates and creates a fix, frequently running a few tests before marking it as ready for test.
4) Testers make sure the bug is fixed, look for any additional problems, look for related issues, and frequently even run a regression test pass to make sure that the developer didn't accidentally break something else while making the fix. If there are additional problems, the bug goes back to the developer to make a better fix, otherwise the bug is marked as okay to check in.
5) The developer then code reviews the changes with another developer, builds the changes for all platforms to catch any possible compile breaks, and then checks in the changes.
6) The build lab picks up the changes for the day and starts to compile.
7) If a compile break occurs, usually because someone was in a hurry and didn't follow the rules, an on-call developer triages and fixes so that the compile can continue.
8) When the build finishes, it is installed on a set of machines, and a series of build verification tests are run to ensure that the build is at least good enough to run some tests.
9) When the build verification tests finish, then the testers install that build and double check that the bug is still fixed, and mark the bug as such.
10) Finally, the tester adds a regression test to their test plan, and automates that test so that it will at least be run before the end of every major cycle, sometimes every minor cycle, every week, every build, or for some issues even as part of the build verification tests.
Major cycles are for betas, and final releases, minor cycles are for releases to be deployed internally, builds tend to come out daily. At the start of a cycle, and in early cycles, the bar is fairly low, almost any bug can be fixed and added to the build. Near the end of each cycle, and at later cycles, the requirements are increased so that only changes that are absolutely necessary are taken, reducing the risk of introducing new problems that won't be discovered until after the product is released. At some point in every major cycle, the bugs and test plans are reviewed to find areas that need improvement.
Additionally, instrumented code to measure test coverage, quality standards in a number of areas like accessibility, reliability, scalability, globalization, localization, integration, interoperability, are measured for improvement, usability studies are performed, code profiling tools are used, code scanning tools look for code execution paths that could result in problems and automatically file bugs, testers bash on other components, and anything else anyone can think of to find the problems early.
However, the pace is incredible and problems can come from anywhere. Imagine testing an Xwindows application to configure networking while the kernel is changing, the networking core is changing, Xwindows is changing, the shell is changing, the compiler is changing, your application is changing, and the tools you use to test with are changing. It is a challenging job.
If you want to bash Microsoft, that's fine, I used to...hence my handle, but now that I have seen inside the "beast", it's just a business, most of the rumors are very off base, and most of the people there are just normal people who want to do the right thing.
The problem is that there are quite a few people out there that assume that just because it is in binary form, that it can't be figured out. For example, they will use XOR to "encrypt" data stored inside the program, or assume that their secret algorithm is safe because it is compiled.
The barrier to entry is definately raised, but it is always possible to figure out what the compiled code is doing given enough time and effort. In fact, I've even heard of people who patch operating system kernel code without the source...
...I don't know why the RFID tag is fried...the bill was just sitting in my wallet. Oh, by the way, I work around power transformers all day, maybe the magnetic fields from those are causing a problem.
Nahhhahahahah - they'll never track me now!
I had a number of things going on that box, initially I wanted a dual-nic router so I could have more than one networked computer in my dorm...the dual-nic issue wasn't resolved by changing IRQs, etc. In fact, the dual nics worked perfectly in windows, but required a bit of code tweaking in linux to make it work. And yes, I read all of the FAQs I could find at least a dozen times.
...no configuration changes on that system were ever simple with linux.
Then I moved out of the dorm and into an apartment, and it took another couple weeks (a few hours each day) to get dial on demand working correctly.
Good point, I guess the way I always looked at it is: Can I take the in-house staff that normally works as a salesperson or a stock-boy, but who has a bit of computer experience, and let them spend a few hours on it at $8.50/hr rather than call in the local computer company for $50/hr with a 1 hour minimum.
The target users won't necessarily be experienced admins. Think home networks, small business networks, branch office networks, etc.
I'm fairly certain that the usability engineer verified that the configuration was correct for the scenario that was given, although the scenario might have been very simple (I don't know).
Your point about by design or by accident, is well taken. Additionally, if you are large enough to have a "production network" I would agree that I would rather not have a novice admin working on it. At the same time, many organizations do not have good IT processes or controls, and sometimes the novice admin is the only one available to solve the problem (I have been in that situation several times in several different organizations...and once I did make the problem worse).
If you are fortunate enough to have the luxury of hiring experienced admins, good for you. Too many organizations hire the cheapest person with the lowest minumum qualifications, and expect them to make do.
I would agree that Win98 isn't a stable system. I wouldn't use anything from the Win9x line on my machines.
I can see significant improvements in many scenarios when moving up the NT line. Today, I wouldn't use NT4.0, and Windows 2000 is significantly better than NT4.0, and Windows 2003 is significantly better than Windows 2000 (from the parts that I have looked at).
And to be fair, my connection sharing experience with linux was 4 or 5 years ago, and linux may have improved significantly since my horror experiences of needing to customize a network driver to coax two identical NICs to work at the same time under linux, how to get dial on demand working, and how to get routing to work with dial on demand.
I'm sorry, I think you are confusing the posts from two different people.
In your original message, you made it very clear that your admins failed to configure Windows for over a week. Are you saying that your Linux admins take over a week to configure a box to run email in an already existing email infrastructure?
My original post talked about configuring DHCP, and in my example, both the linux and windows admins could configure DHCP on Windows in under an hour, where linux admins in groups mostly failed to configure linux as a DHCP server in a several hour study.
I didn't get a chance to meet these linux "admins" perhaps they didn't know what they were doing, but they were managing several linux boxes as a part of their regular work responsibilities. The same class of people (linux "admins") peformed much better when configuring Windows servers.
Maybe the people in this group were ripping people off...I don't know for sure.
At the same time, I remember spending several weeks configuring a linux box to dial on demand and route local network traffic to the internet so that I could share my dialup network connection with my roommates. Granted, I wouldn't claim that I am an experienced admin even though I have installed, configured, and run my own servers for the last 8 years.
Windows NT 4.0 was better than Win95, but it still wasn't up to snuff as a server. However, compare Linux with Windows Server 2003 and I'd go for Windows.
And for the comment about a text file making configuration easy, I don't think you ever looked at a sendmail config file.
Sorry if I wasn't clear in the original message, but the linux systems were tested with experienced Linux admins.
The Linux admins could configure Windows more quickly than they could configure Linux!
Interesting story: A company I work for runs usability tests on Microsoft software. These usability guys have watched Windows Server Administrators with very little experience set up a DHCP server in less than 15 minutes, and almost nobody fails to complete this task when given an hour to work on it. Even novice unix admins can set up a windows DHCP server in under an hour.
Then, they decides to try the same thing with Linux, putting users in groups, with internet access, and after more than a day's worth of man-hours (and several real hours) most of the groups still hadn't been able to configure a linux box to be a functioning DHCP server.
With the 10 hours I'm paying the linux admin to figure out how to do it in linux, I could have paid my Windows Admin to configure DHCP on a Windows box while saving enough money in time to pay for the copy of Windows too!
This *really* makes me wonder how they attempted to contact Microsoft.