Slashdot Mirror
Why Vista Had To Be Rebuilt From Scratch
iliketrash writes "The Wall Street Journal has a long front-page article describing how Jim Allchin approached Bill Gates in July, 2004, with the news that then-Longhorn, now-Vista, was 'so complex that its writers would never be able to make it run properly.' Also, the article says, 'Throughout its history, Microsoft had let thousands of programmers each produce their own piece of computer code, then stitched it together into one sprawling program. Now, Mr. Allchin argued, the jig was up. Microsoft needed to start over.' And start over they did. The article is astonishing for its frank comments from the principles, including Allchin and Gates, as well as for its description of Microsoft's cowboy spaghetti code culture."
To quote
Microsoft's greatest enemies now are still two for-profit companies - Google and Apple. I'll rest easier when FOSS replaces them (as was promised in 1999). Instead it's just a new master instead of the old one.Quidquid latine dictum sit, altum videtur
It's interesting to hear how their software development survived in such an anarchistic environment - everyone producing their own code, with ad-hoc integration. It's a good example of how software development methodology can work though, even though the specifics of the specification design weren't discussed in the article - if everyone codes to a documented interface, software development can work on such a grand scale.
I personally would like to hear more about the software development procedures and methodologies used in other large projects - how successful different types of development are.
I work for an automotive parts manufacturer, and to see the lack of consistency within the organisation's software development is disturbing. Safety-critical parts are being produced, and the level of testing between said parts varies quite considerably. Additionally, the level of oversight and adherence to software development procedures is rather bad to say the least. I just hope it's not characteristic of the industry as a whole.
Why is it "astonishing" that the article does a decent job of providing hard-hitting information without spin? That's what we are supposed to expect of journalists. The Wall Street Journal is supposed to be (and often is) an example of real journalism. That makes it distinct from computer magazines that rely on advertising revenue from the computer industry, and from discussion forums whose course is steered by peeves and submission sequencing.
When I took C programming in College, one of the points our professors made was if you like your program, rewrite it...
;)
the first time you write something, it's always hackney'd - and it gets that way till you figure out what you want to do and how to do it - afterwards, it then becomes so much clearer to see ways to clean up the code and fix issues...
so one of the first rules he had was once we were almost done, restart our stuff - it ended up being a lot cleaner/modular the 2nd time around...
of course, that won't help MS, but good for the rest of ya to know
RB
----------
ah honey, we're all resplendent - Bill Mallonee
Just look at this quote:
They are comparing an operating system, which has to be backward compatible with a dozen or so earlier versions of Windows and DOS and support an oodle of devices and subsystems, with a bunch of mostly unrelated web-applications and gimmicks from Google.
All I'm getting from the article is that the "let's rewrite from scratch" crowd got the upper hand within Microsoft. But that doesn't necessarily mean that they are right or that the end result will be better than continuous improvements. At the beginning, it is easy to maintain a nice, clean and simple system. But a complex set of requirements can't always be broken down into simple Legolike blocks, as the article suggests.
Right, but have you ever noticed how many successful Free / Open Source software projects use modular architecture? Take (from my own area) Nessus, or Snort. Both consist of a core engine and frameworks that accept plug-ins and modules. Actually they both also have a lower level that allows ordinary non-programmer users to contribute signatures (rules) to the project.) This applies also to Apache, Mozilla, the Linux kernel, and plenty more.
Of course, you are right: Microsoft is indeed one of the most competently managed companies around. And that is exactly their problem.
Why is that a problem? Because their management, sales, and marketing are so good that their technology doesn't have to be. They can ship software with security holes, bugs, poor usability, and bad design, but the non-technical part of the company will somehow manage to still sell it and make a bundle on it.
I'm not too much of a programmer, but I can concur at least on writing websites. It's really hard to get everything right the first time. Web design is just that, design. And software development has plenty of design aspects to it as well. And good design, by its very nature, tends to be iterative. An architect, or an industrial designer, they'll go through dozens, maybe even hundreds of versions of something. Lots of work, plenty of good ideas, they end up not making the final product, but the result is much better because of all that work. Of course, I'd imagine that even the most complicated of buildings is way easier to understand than the windows source code.
One time I threw a brick at a duck.
Throughout its history, Microsoft had let thousands of programmers each produce their own piece of computer code, then stitched it together into one sprawling program.
Sounds like SOP for any massive program/OS. If you've ever been part of a truly massive product's development, you'd know what this is like. There are dozens, if not hundreds, of small groups that each specialize in a particular piece of functionality. Executives and architects determine the work items for a particular release. Responsibilities filter down the chain of command. Teams develop their work items for the release and everything is thrown together into the pot as it's done. Builds break frequently, and problems are addressed as they're encountered. Eventually testers can get their hands on decent builds, and testing/bug fixing commences during the whole process. Some ways down the road, a release finally occurs.
Really, I don't know what the executive in the article thinks should be happening. There really isn't any other way to develop programs on the scale of Windows without the aforementioned "organized chaos". It's not a text editor, it takes numerous small teams working in a coordinated manner to produce such massive piles of code. Obviously, the more teams there are, the harder perfect coordination is to achieve. Hence, things go wrong fairly frequently. This is to be expected, IMO.
You sound like somebody who hasn't used Linux in a long time. In fact, it's amazing how far Linux has come in the last few years.
You've obviously never heard of Synaptic. I suggest you take a look at some of the screenshots. Most distributions now come with Synaptic. To install software, you just load up Synaptic, select the programmes you want to install from a list and click a big "Install" button. What could be simpler?
You seem to have a hard time grasping this but this is actually simpler and better than Windows. Windows has no dependancy tracking. I can't count the number of times I tried to install game X and the installer has told me that before I install, I need to first manually install the latest version of Internet Explorer / Windows Media Player / DirectX.
With Linux, all my programmes are on something equivalent to Windows Update. Not just the OS but also Office Suites, Games, Media Players... you name it. I can install them easily using a graphical interface and they get upgraded automatically when new versions come out.
As for driver support, Linux beats Windows out of the box, hands down. Drivers for most devices come already included with your distribution. They get loaded at boot time if that piece of hardware is detected. On my desktop, my DVB card, Sound Card, Graphics Card and Display were all detected correctly first time. Windows might have a driver for the Sound Card but a DVB Card?
Installation is so easy too. My distro of choice, Ubuntu, all you have to do is select your keyboard layout and where you want to install to and it does the rest. No intervention necessary. If you can't do that, there's something terribly wrong with you.
Microsofts 'conviction' happened several years ago, 1999 if I remember correctly. Has the world stayed the same since then? No. Things change, Microsoft was called a monopoly 6 years ago and that may not be the case today. Labels dont stay attached forever just because you want them to.
There is, as I'm sure you already know, a difference between a C program you wrote in class and an OS. The reason your C program gets better when you rewrite it is because you now have a clear view of what it should look and work like. When it comes to a behemoth like Windows, no one understands the system fully. So even if we have all these people who understand parts of the system rewriting their parts, plenty of design errors can still persist in the way the system is modularized and put together.
So what should they do then? I have no idea.
(I've got a B.Sc. in Comp. Sci from the 1980s - take this comment with that in mind that I'm and old skool type)
The "Second System Effect" is a term coined by Fred Brooks to indicate that the second system you build is usually a horrible monstrosity of things that you wanted to put in the first system but you didn't have time.
http://en.wikipedia.org/wiki/Second-system_effect
I'll wait for your third re-write, thank you very much.
TDz.
Things you should never do: rewrite.
Naah. Software is math and the first proof of a theorem is generally ugly. So, it can pay to start over. I am not going to say in all cases it's better to do one or the other, but sometimes rewriting is the best option. An example from my own life: I wrote a MUD with some neat AI stuff (quests that actually impact the world in large numbers) in it and now I am working with a small startup to make an MMO and started over rewriting because the way I did it was bad the first time, but it was the best I knew how to do because that was all I understood about the problem. Now I have a more modular system and I understand how quickly certain things need to happen, and what needs to interact with what, which means I can split things up among databases and such . One thing to remember is that if you have a system that does X and you just want it to do X with a little bit more, then you don't rewrite. Even if you repeatedly have to do a little bit more. OTOH, if you have a system that does X and then you realize you need to do X and Y and Z...then maybe you need to rewrite depending on what you need the system to do.
Best. Comment. Ever. Enjoy!
Indeed, Microsoft never re-write anything. And arguably any company that did so would be very foolish and probably heading for self-destruction. The article is couched in so much lay-man's talk and analogies it's hard to know what the real engineering changes have been. But at a guess they are just writing the new parts of Windows in a modular plugin way. And that's why they can delay them till after the launch of Longhorn. So WinFS for example was probably originally implemented as spaghetti (co-mingled as MS lawyers like to say) with lots of the system having knowledge of WinFS. Now they've backed that out and made WinFS properly modular, such that it can be issued later, and also back-ported to XP. ...at a guess.
The lack of automated testing and daily builds is a bit of a shocker though isn't it? Either they've been lying all these years, or things had been really falling apart at Microsoft.
There is sometimes a difference between what a word really means and what a court defines a word as meaning in a specific context. In MS's case, a court convicted the company of having a monopoly within the context of anti-trust law. The Wall Street Journal is using the word as it is actually defined by real people, which means to own ALL of a market. The newspaper is properly labeling reality, not showing evidence of bias one way or another. The fact that I detest MS and Windows doesn't keep me from seeing that the WSJ is just doing its job properly in saying "near monopoly." The moment you don't have ANY choice other than Windows in the market, it will be a monopoly. For now, though, the fact that I'm typing this on a Mac and can go buy as many non-Windows computers as I want says MS does NOT have a monopoly. Period.
That might work for small college projects but the real world is a different place.
;)
Often the rewrite never gets completed as there is too much crap added to it.
If you truly want to make something that works you need to plan for an evolution of your software. That is, write the first version with a modular design that can be modified or rewritten in phases. Doing one big rewrite on a non-trivial software system is damn near impossible. It's better to evolve the software over time, always keeping a working system and slowing moving parts in the desired (presumably better) direction.
I could write more on this but it's too early in the morning and I'm not even sure if what I wrote makes sense.
The ratio of people to cake is too big
yeah, I don't recommend trying to install the M$ Exchange Server.. Dependencies, bleh.
having to install it once upon a time definetely makes me realize the importance of portage in my day-to-day computing.
As much as you won't want to believe it, Vista is very similar to the transition from OS 9 to OS X. The big different, of course, is that Apple took someone elses kernel and tools and built on top of them. Apple also essentially forced developers to start writing their software for the new OS, with kind of shoddy emulation of OS 9 programs. The important thing here though is that OS X was pretty much a completely new platform. Vista is not quite as much of a change, but it's pretty damn close. Vista is introducing a whole new API system (WinFX), graphics api (Avalon/WPF), communications platform (Indigo/WCF), completely new audio stack, completely new network stack, and a few other major changes. All this while maintaining compatibility with 95-99% of current windows applications out there without a shitty emulation layer. Microsoft simply won't make a revolutionary OS anytime soon. There are too many people running Windows that simply won't stand for very little of their software running, if any, on a new version of Windows. So Microsoft is doing what I think is a good decision, they are making giant evolutionary steps towards a whole new platform. A transition.
How could they possibly re-write it from scratch in as little as a year. Impossible. If I were a betting man all chips would be in.
Every release of Windows you will hear Microsoft clamor the most secure and stable version ever!
"The irony is that Bill Gates claims to be making a stable operating system and Linus Torvalds claims to be trying to take over the world."
Your Average Joe
The article is very vague. It sounds like they're writing automated tests and rejecting any code that doesn't pass the tests. But I can't imagine that they didn't have a regression test suite before, so I wonder what changed?
And if you believe this, you'll believe anything.
Same as with Half-life 2. I've seen interviews with members of the valve squad who actually said HL2 was completely redone from scratch, when maybee 60% of the code still has its origins in Quake 1.
My guess for vista is the same, they simply had to 'go back to bascis' because all their new stuff was badly organised. This is not the same as starting from scratch.
Put my two cents in as to how the article's storyline doesn't quite track. If Mr. Allchin, despite massive institutional inertia, gave the pig winglets and put it back on a track to actually being releasable then we're missing the motive for why he'll leave on Vista D-Day and why the company wouldn't fight to keep him. In some sense, the article is about the story Microsoft wishes to tell, which is we were writing bad code, but we've fixed that now (and look at the bruises: no pain, no gain, right?), which is what the parent posts suspects.
Now I suspect that the interviews took place before the Microsofta est omnis divisa in partes tres announcement, and there was no desire from Microsoft to have Mr. Allchin candidly describe his reasons for retirement (and maybe Mr. Allchin has a book up his sleeve), so off to press with this peek into the hallowed halls of Redmond.
One quibble I would have with article is in its suggestion that Mr. Gates, as Chief Software Architect has two paradoxical duties to reconcile: coming up with innovations and putting down unrealistic projects. A lot of the candid reporting I've seen is that there's a third element that he practices with zeal, which is to grind into a fine powder any idea he believes shakes a stick at the cash cows.
One implication of the story is that in Summer 2004 Bill Gates didn't know that one of the cash cows was flatlining. There's a thought to ponder.
We may recall how Gates said security was job #1 a while ago. Obviously they are paying more attention to that now, but a large part is to deflect blame for their daily exploits. And in this case the article says how all the nice new features of Vista have fallen by the wayside, it's years late, but the spin is, as always, "the next version will be better than anything ever made". The classic FUD, and in the WSJ; so the CEOs can tell their geeks not to worry about migrating to Linux, or OSX, because Alchin says Vista will be all that and more.
The point not at all investigated is the deliberate encouragement of spaghetti code over the last years, to hook IE, WMP, and coming DRM inextricably into the OS, the very opposite of the clean modular code advocated. Interesting to see which principle will give way.
Did you know, by the way, that a system can be modular on the source code level and then (based upon a compilation flag) it can either be built such that (A) both regions are in kernel space, or (B) one region is in kernel space and the other is in user space. The former would use a very efficient interface, whereas the latter would use one that was more expensive (for having to cross that boundary).
In both cases, the regions exist in separate modules...it's just a compile-time optimization. Modularity is mostly a "maintainability" concept. The user should never care whether two regions are communicating via a Mach message or a pointer on the function-call stack to a struct in the heap. Using the latter does not make the source less modular.
The "cue the foo posts in 3, 2, 1..." posts will commence with no subsequent foo posts in 3, 2, 1...
I love Google, because they rarely promise something and don't deliver. Actually, they rarely promise something. It just shows up one day and it's elegant, clean, and fast.
Hear, hear. MS holds flashy press conferences to announce products that won't ship for a year (if at all), includes laundy-lists of features that will be radically pared down before release, and ultimately ships products that are, at best, incremental improvements over previous versions, although they are touted as 'revolutionary', eg Win 2k vs Win XP.
Google doesn't talk about products in preparation. They quietly release full-function betas before announcing them, and the betas offer features that really are revolutionary. No Gmail wasn't the first web mailer, but it redefined what a web mail program was capable of. No Google didn't make the first map, but maps.google blows everyone else away.
Yes, there is a big difference between between building something like Google Desktop Search and building a whole new filesystem and all the other changes that requires. But the point is what is promised and what is delivered.
Google promises nothing, and delivers products that become essential. Microsoft promises the sky and moon (I thought Windows was supposed to be voice-controlled by now, and my fridge was supposed to automatically order milk when I need it), and delivers products whose importance to daily life is based primarily on the difficulty in avoiding them.
When Google does drop the next bomb (Google TV?, GoogleFS?, Googlix OS for running a smart terminal?), you won't hear about it in a press release. You'll be an invited Beta tester.
It seems pretty clear from the article that its describing Microsoft implementing unit testing on a large scale, but trying to explain it in laymens terms. So they didn't have to "rewrite" everything, they just wrote unit tests for everything they could, and dropped other parts (WinFS) until they could get those properly tested. The part about "code jails" and all of that read right out of an extreme programming book. I'm suprised no one else picked up on this.
Looking under the hood, the Linux development model is more organised than one might expect. Consider the parts that make up a Linux system.
Libraries and applications are typically managed by smaller teams, and even if people contribute, those contributions are reviewed. Accepting that, we only have to look at the big structure. Some observations about libraries:
The whole is a mixture of bottom-up and top-down hierachical control. To understand the dynamics, consider an individual project. At an early stage, the developers looks around to identify what is already done, and tries to identify reasonably stable, common, and well managed libraries which they can use. This is a very feasible thing to do due to open source licenses. They will then start from there, and do occational changes in dependencies throughout the lifetime of the projests due to new needs and changes in availability and quality of dependent parts. Sometimes, libraries are split out of projects by abstracting out identifiable tasks.
An important observation is that by maintainers of a popular project casts a vote when choosing dependent projects. The more important the project is, the higher weighted is the vote for the dependent parts to survive. When most projects thus migrites to a better library, the rest will have to choose to follow suit or to risk loosing ground due to a more difficult installation process. The distributors are the ultimate judges, though their power is limited by what software is available.
In other words, there is a semi-democratic system that organises a hierachical structure of componets, with no single central authority.
You forget Apple, they reinvented themselves more than once AND always have managed to be the frontrunner of computer innovation...
According to their marketing and PR departments, anyway.
That ultimately gave us osX, the ultimate in plug-in philosophy, from the kernel to the GUI.
Apple didn't give us OS X. The kernel came from CMU (an open source project), and NeXT and Apple spent the last 20 years making it less modular. The GUI software architecture came from NeXT, borrowed heavily from Smalltalk, and is client-server, like X11, only not as well architected or as efficient.
In fact, Apple's own systems programming staff screwed up so badly that Apple had to go out and buy a new operating system; all their attempts to develop a next generation Macintosh OS in-house failed.
system7 with its core os and extensions attaching to it, they invented plug-ins before browsers were even invented...
The MacOS extention mechanism was nothing like "plug-ins" -- There was not a defined "extention API" as with a browser, they were they were system call traps and often relied totally on undocumented behavior.
Someone could write such extensions for any OS, but it's generally considered to be a bad practice. As the unstable, conflicting mess of MacOS extentions proved.
Whenever I hear the word 'Innovation', I reach for my pistol.
The key to how an individual perceives Windows XP is based on expectations. Those with low expectations generally think Windows XP is good. Those with high expectations realise it is not. The way to develop realistic expectations is to have an in-depth knowledge of the *potential* of software vs the current reality of software. Kids today who grew up on Windows 98 have low expectations and so they think XP is actually *good*.
BSD lugs around nearly 30 years worth of baggage, but I can boot and reliably run NetBSD and OpenBSD on my 2004 hp pavilion. Linux 2.6.* sometimes will boot from the installer, but only if I disable ACPI (and quite often other things such as agp and usb -wtf?- as well).
Older 2.4.* releases work ok, and the BSDs work ok (except for FreeBSD 5.0-5.3).
After having similiar experience on other computers with 2.6, I've pretty much come to the conclusion that linux has jumped the shark; at least in terms of stability and reliability (hell, I can't even rely on it to successfully boot FFS).
So, rant aside, it's a matter of arrogance and design; the BSDs have an attitude of "do it the right way" and therefore produce a stable system, linux has the attitude of "bugs are part of the FUN" and as a result, you have the mess which is 2.6.*
IMHO, what really killed them was not the code rewrite per say, but attempting to be an early pioneer in open source development. THAT'S what burned up all the time, and everyone knows pioneers have a high risk of ending up with arrows in their backs.
So if you're after morals to the story, I'd say instead never bet your company on open source... *grin*
Any sect, cult, or religion will legislate its creed into law if it acquires the political power to do so.
Here's one - never having to hear "Ship it!". People working on OSS projects on their own time aren't generally being told, "you have to ship before Dec. 31st so we can get the revenue on this quarter's books", with no regard to whether that date is reasonable.
Actually, the large open source projects I've seen DO have pressure to ship fast. Although it may not be due the the quarter's books, there is a lot of pressure on projects to get things done and announce GA. When Apache 2.0 was originally released I believe many developers didn't think it was ready for prime-time, and they still shipped. As a result it took a long time for people to upgrade. Additionally, many important open source projects ARE backed (and essentially developed) by corporations who DO have books to worry about.
I don't know if you were asleep for the latter half of the 90s or just too young to know anything about the computer industry at the time.
:(
Apple single-handedly made USB successful, period. It doesn't matter who invented what, or what any specs said. Nobody was manufacturing USB peripherals, and few computer manufacturers were making systems with USB built-in. USB was a decent technology with a huge chicken and egg problem that dragged on for YEARS until one day Apple released the iMac, removed every other interface, and said from on high "Computers will now use USB".
Immediately, everyone said Apple was crazy, that nobody was making USB devices (other than mice), that it was the stupidest decision ever made in the industry. PC Pundits were laughing at yet another boneheaded decision by the dumbest computer company ever.
Within months, store shelves were filling up with USB devices now that manufacturers had a market to sell to (and one where they could charge a nice price premium, to boot!). Microsoft had to play catchup and actually start releasing USB support in patches so that Windows OSes could actually use all this cool new stuff.
Apple Made USB what it is today. Without Apple's completely unrestrained technical and marketing support of USB, we'd all be using the vastly superior FireWire today and USB would still be "that square plug for your mouse". Hmm, that wouldn't really be so bad.
Recursive: Adj. See Recursive.
For example, I read about that new network architecture in Windows Vista, and how they have implemented per-session routing tables and user-installable tunnelling devices in order to allow VPN connections on a per-user basis without requiring administrator privileges. It is also typical of the kind of engineering that goes on over at Redmond -- they solve the problem at hand, but only superficially and without remedying the real, underlying problem. They added some runtime flexibility, but it only goes as far as modifying the state of the code that exists, instead of adding the ability of loading new code modules dynamically. Doing that wouldn't only have solved their problem, but it would also potentially solve millions of upcoming problems, and not only for them, but for other users as well (for example, if I wanted to write an IP-over-SSH driver and plug it in on a time-sharing system). In other words, it is my not-so-very humble opinion that true runtime flexibility requires dynamic code loading, and that, in turn, requires modularity in the existing code base. Runtime flexibility which only means changing the state of currently loaded code hardly even counts as runtime flexibility.
It is well worth noting that the GNU Hurd is capable of doing all of the things that I mentioned. It can load file system drivers, memory managers, network stacks and much more during runtime. Ordinary users can do it for themselves without requiring administrative privileges or affecting other users. Now when they're switching from the Mach microkernel to the L4 microkernel, they have even added the missing piece of being able to load device drivers dynamically, in user space, and without special privileges (Mach requires device drivers to run in kernel space, much like monolithical kernels). This is why I am an avid supporter of microkernels.