Rewrites Considered Harmful?
ngunton writes "When is "good enough" enough? I wrote this article to take a philosophical look at the tendency for software developers to rewrite new versions of popular tools and standards from scratch rather than work on the existing codebase. This introduces new bugs and abandons all the small fixes and tweaks that made the original version work so well. It also often introduces incompatibilities that break a sometimes huge existing userbase. Examples include IPv4 vs IPv6, Apache, Perl, Embperl, Netscape/Mozilla, HTML and Windows. "
Was it a "good idea" for Microsoft to rewrite Windows as XP and Server 2003? I don't know, it's their code, they can do whatever they like with it. But I do know that they had a fairly solid, reasonable system with Windows 2000 - quite reliable, combining the better aspects of Windows NT with the multimedia capabilities of Windows 98. Maybe it wasn't perfect, and there were a lot of bugs and vulnerabilities - but was it really a good idea to start from scratch? They billed this as if it was a good thing. It wasn't. It simply introduced a whole slew of new bugs and vulnerabilities, not to mention the instability. It's just another example of where a total rewrite didn't really do anyone any good. I don't think anyone is using Windows for anything so different now than they were when Windows 2000 was around, and yet we're looking at a 100% different codebase. Windows Server 2003 won't even run some older software, which must be fun for those users...
.. as they are rewriting the security layer!
As a coder I can assure you that working on somebody else's code is frustrating because you allways say: "I would have done this differently". Most rewrites I think come from there, having the idea of a better implementation.
This introduces new bugs and abandons all the small fixes and tweaks that made the original version work so well. It also often introduces incompatibilities that break a sometimes huge existing userbase.
Microsoft has created an entire, successful, multibillion-dollar-a-year-profiting business model off of this!!
Sheesh.
do() || do_not();
The trick is to include all the tweaks and fixes that were implemented in the old code. Obviously, if you rewrite and then leave open all the gaps and problems from the earlier version, there's no point in rewriting. However, you could rewrite with those fixes in mind, and come out with a completely new (and problem-free) edition.
I think it may have something to do with programmer ego and something to do with the challenge. I'm guilty of it myself. You find something you're interested in and you want to build it. It doesn't matter if someone else has done it or even done it well before you. The challenge is to do it yourself.
~ "When I'm of that age I'm just going to live up a tree."
Microsoft: Ok, Windows XP and 2003 have a full rewrite of the TCP/IP stack and security system.
Slashdoter: Why did Microsoft rewrite the core OS? They just introduced more bugs and lost the stability and security fixes from older versions of the OS?
"Have you ever thought about just turning off the TV, sitting down with your kids, and hitting them?"
Ok, this dude uses netscape 4.x and thinks its fast. next article please.
The minor tweaks, fixes, and changes that made the old version work so well can only go so far. Such is often the nature of code. Tiny fixes and patches are (sometimes haphazardly) hacked on to the code.
Perhaps if true extensive software engineering and documentation techniques were followed, a full rewrite may not be necessary. However, as long as quick fixes continue to pollute the code and make it more and more difficult to work with, an eventual total rewrite will always be necessary.
I propose we redesign Slashdot in brainf*ck instead.
I'm sympathetic to the idea behind this article, but does it deserve a place on /.? There's absolutely no empirical data, or even a reasonable example given in the document. The author is talking about IPv6 and Perl6 both of which are unknown quantities at this point.
He's right that just throwing away old code means yo u lose a lot of valuable bug fixes, on the other hand if you look at some code and realize there is a better way then the solution is to rewrite it.
Of course you can have it both ways. What you do is write an automated test case for every bug that you fix in your code. When you write the new version it has to pass the old test suite, then you've got new code and all the experience from the old code.
John.
This oughtta be good. (puts on asbestos-lined pants)
Your point is well taken about ego often driving rewrites but in my experience the driving force for rewrites is often maintainability.
As a program ages and drifts from the original intent ugly hacks are often placed on top of the original code to add unforseen functionality. There is also the opposite effect where old code is sitting around that no longer has any function. I remember one drastic case of this when rewriting a program where only about 1/2 the code was even beeing utilized.
By rewriting the code you clean things up and make it easier for future programers to understand what the code is doing.
One should not theorize before one has data. -Sherlock Holmes-
For Windows users, Winamp is probably the best example I can think of. Take a stable, usable, simple and elegant audio player (Winamp2) and fuck it up by writing it from scratch (Winamp3), then ultimately abandon that clusterfuck rewrite in favor of yet another rewrite (Winamp5) that fixes what they fucked up with Winamp3.
I'm mighty happy sticking with Winamp2, thank you very much.
Don't rewrite. Refactoring code is the way to go. Refactoring in small pieces allows the app to maintain compatibility as the process progresses.
The other side of the rewrite issue is, how long can you continue to maintain code from a legacy system? I worked on a project a couple years ago that had been migrated from assembler to COBOL and is now being rewritten (as opposed to being redesigned) for Oracle. Nevermind for a moment the fact that the customers wanted to turn the Oracle RDBMS into just another flat-file system--which included designing a database that had no enabled foreign key constraints and that was completely emptied each day so that the next day's data could be loaded. . .
Some of the fields that are now in the Oracle database are bitmapped fields. This is done because there's no documentation for what those fields originally represented in the assembler code and because the designers are afraid of what they might break if they try to drop the fields or attempt to map the fields out into what they might represent. I had the good fortune to get out of the project last August. . . last I checked, they had settled for implementing a Java UI over the COBOL mainframe UI.
Anyway, my point is this: at some point, you have to decide whether the system you're updating is worth further updates. Can you fix everything that's wrong with the code, or are there some things you'll have to jerry-rig or just shrug your shoulders and give up on? Under circumstances like what I mentioned above, I truly think you're better off taking your licks and designing from scratch, because at least that way you can take advantage of the new features that more recent database software and programming languages have to offer.
!#@%*)anks for hanging up the phone, dear.
As I recall, Torvalds made mention that some of his original code in the Linux base was not very good and he would have written it much differently today. Indeed, most anyone that habitually programs naturally becomes more skilled and if the underlying premisis/framework/model of an application or tool is not as good as could be - or is lacking a certain methodology that time has proven to be beneficial and only rewriting it will solve this - what is wrong with rewriting the code from the ground-up?
This guy is full of shit and has no idea of what he is talking about.
Some of the better parts:
- He claims that The mozilla project and everything Netscape >4 is pointless and that Netscape 4 "just works". We all know that Netscape 4 is an awful, crashy, buggy, standards-breaking piece of crap that set the Internet back years.
- He claims that Windows XP was a complete rewrite. Windows XP is NT 5.1 -- (check with ver if you want) Windows 2000 with the PlaySkool OS look.
Okay, so most of the article consists of, "Here's software X. They re-wrote it, and now it's not as good or as accepted. Why'd they do that? They suck."
Software is re-written for many reasons. Sometimes it's ego, sometimes it's for fun, but usually it's because you take a look at the existing codebase and what you want to do with it in the future, and you decide that it's going to cost a lot less to implement the future features by re-writing and fixing the new bugs than to work around the existing architecture.
I've had to make the re-write or extend decision more than once, and it's rarely a simple decision.
What I would have preferred from this article is some interviews with the people responsible for the decision to re-write, and what their thinking was, as well as whether they still agree with that decision or would have done something differently now.
=Brian
There is nothing so good that someone, somewhere, will not hate it.
Joel on software has covered this point in a good article: http://www.joelonsoftware.com/articles/fog00000000 69.html.
It was too messy and unmaintainable. I'll wait until the rewrite comes out to fix all the grammer and spelling bugs.
Things You Should Never Do, Part I
Rewrites are 'bad' from a management point of view (at least, a manager that isn't familiar with software development), which looks at return on investment (ROI).
However, from a developer's point of view, a partial or complete rewrite is sometimes the only way to FIX certain bugs. While it may introduce new, small ones, usually developers are smart enough to read the old code and learn from it's mistakes before the do a rewrite.
A partial or complete rewrite is ALSO sometimes the only way to fix 'spaghetti code' -- code that's become so tangled from patch upon patch being applied to it that it's now impossible to trace and fix a bug. If spaghetti code isn't pursued and rewritten on a regular basis (this is 'constant improvement' -- a management buzzword from the past few years that actually works), new bugs can be inadvertantly introduced -- and it can sometimes take weeks to hunt down an intermittant bug by tracing spaghetti code. Ladies and gents, WEEKS of programmer time is expensive compared to one programmer spending 8-10 hours per week tracking down bad code in the codebase and rewriting it.
Really, there's a case for doing rewrites on a constant basis. The author should have instead addressed adequate testing in software development environments...
--
Vote for your hopes, not for your fears - Vote Third Party
From the Perl 6 development webpage:
"The internals of the version 5 interpreter are so tangled that they hinder maintenance, thwart some new feature efforts, and scare off potential internals hackers. The language as of version 5 has some misfeatures that are a hassle to ongoing maintenance of the interpreter and of programs written in Perl."
For me, this is a necessary and sufficient condition for rewriting something.
Another one is: When changing the original will take longer than rewriting from scratch.
HCG 50a = 2MASX J11170638+5455016
11h17m06.4s +54d55m02s
Every successful piece of software I've ever worked on was rewritten at least once, by the same team (or by myself on private projects) in the process of development, fully or at least partially.
The fact of the matter is, even if you hire an expensive architect and have him do a good job, he's not a God. When you develop software some parts of it tend to become ugly as heck and you can't help but think on how to do the same thing better and/or with less effort, so that it won't become a PITA to run, maintain, improve and extend. When you reach critical mass, you become "enlightened", throw some shit away and rewrite it to save time later on. In all cases where I've seen it done I think it was worth the extra effort. I also think re-engineering code as you go saves money long-term if it's done reasonably.
All of this, of course, doesn't apply to those who start their separate standalone projects even though there are dozens of other reasonably good projects to contribute to (and maybe rewrite some parts of). Freshmeat.net is full of examples.
The Problem: Rewrite Mania
...
Waaaaaaa!!
Case 1: IPv4 vs IPv6
Waaaaaaa!
Case 2: Apache 1.x vs Apache 2.x
Waaaaaaaaaa!
Case 3: Perl 5.x vs Perl 6
Waaaaaaaaa! Waaaaaaaaaaa!
Case 4: Embperl 1.x vs Embperl 2
Waaaaa!
Case 5: Netscape 4.x vs Mozilla
Waaaaaaaaa!
Case 6: HTML 4 vs XHTML + CSS + XML + XSL + XQuery + XPath + XLink +
XML is hard! My HTML for Dummies book weighs too much! Waaaaaaa!
Case 7: Windows 2000 vs Windows XP vs Server 2003
Waaaaaaaa!
Conclusion: In Defense of "good enough" and simplicity
Waaaaa waaaaaaaaa!
The Gnome desktop environment is a prime example of disasters through re-writes.
As we all know, Gnome's oringal purpose was to provide a free rival to KDE, which was the first easy to use Desktop Environment for Linux, this was back before Qt was GPL
Unfortunaltey for Gnome, its problems started as it kept replacing and rewriting core components. For example, it started out with the Enlightenment window manger, then it switched to sawfish, then it switched to the buggy and slow metacity. Metacity has had many problems, and most people want the old sawfish back, but havoc pennington refused to do it and insists that people use it.
The file manager keeps changing too. First it was GMC, then it was the Slow and buggy Nautilus from the now defunct Eazel corporation, now they are writing a new Windows 95 like file manager for gnome called Spiral Nautilus.
It also rewrote the graphics layer GTK and broke compatibillity with GTK 1.x. There are many legacy GTK apps still in wide use and they look ugly on newer desktops.
There is also the many problems with the file dialog, which is now only emerging in GTK 2.4. This is also incompatible with older GTK versions. This means that if you want to use a new program, YOU HAVE to upgrade to Gnome 2.6, and can't keep your leagcy Gnome 2.0,2,4 desktops.
They keep switching default apps, for example, Galeon was dropped in favour of the buggy and far less featureful Epiphany in 2.0. They also dumped several other applications that were useful.
To make matters worse, it is going away from the old philosphy of simple text files and are using an XML based registry clone to configure stuff. KDE keeps the text file format underneeth and has had a standardized API for it.
It also has a lack of true intergration, Micheal de Incanta has PUBLICLY ADMITTED that Bonobo was a faliure. KDE has had this BUILT in from day one using kpart technology, which is now being used in Apples Mac OS X Panther Edition.
Gnome developers, realising they kan't kompete with KDE technology, has spread various FUD about kde, but the message is getting through. Red Hat has abondaned their Gnome desktops, Fedora developers are working hard to make KDE 3.2 the default desktop for Core 2. Debian, who has traditionally been pro-gnome have announced their full support for KDE and they are working hard to make KDE the defualt desktop for
KDE on the other hand has kept consistent technology and has internally has changed very little since 2.0. Distros like Lycoris are still using 2.x because it is very stable and mature. KDE 3.2 will be a good example of why maturity, and not wheel inventing is a better idea overall. They have took their technology and have optimized it for usabillity
Gnome 2.6 will need more than just propoganda about the HIG if it is going to get the attention it needs, but instead it looks like they are reinventing wheels again.
It's hard to write code that is robust enough to not need rewrites. The ability to do that is what separates the really good programmers from amateurs like myself. It's the difference between being a piker (like myself) and an engineer.
I'm not a great programmer, and don't do it regularly, but when I have written fairly big projects, I find that the need for rewrites came out of poor design choices that I had made.
I typically start out with something small, that can handle the core functionality expected from the project. Then I try to add features and fix bugs.
Eventually, the code becomes very difficult to maintain, and ultimately, you get to the point where the ad-hoc architecture simply won't support a new feature.
To the user, everything looks fine, everything runs reliably, but under the hood, there are real problems.
My worst experience was with a web app. I started out with script based pages in ASP (not my call), and kept writing new pages to do different things. It got to the point where I had a about three hundred script pages and lots of redundant code.
When it would become necessary to change the db table structures for another app hitting the same data, I'd have a lot of trouble keeping up, fixing my code quickly in a reliable way.
The problem was that it just wasn't possible to stand still. I couldn't go to my boss and say, "I need a three month feature freeze, to rewrite this stuff."
Writing a new version in parallel was hard because maintaining the crummy but functional code was taking more and more time. It was a real problem, and caused me a fair amount of pain, and suffering.
After digging myself into that hole, I stepped back and tried to figure out how other people did it. I would have been a lot better off building on top of something like struts.
The lesson I took from this is that it's important to study design patterns, and to use tested frameworks whenever possible. You have to think like an engineer, and not someone who codes by the seat of his pants. I'm not an engineer, so it's not easy for me to do that.
I'm not saying that the people who run the projects mentioned are in the same boat that I was. As programmers, they're in a different league.
But they're often working on problems that aren't well understood. Patterns and frameworks are ways to leverage other people's experiences. But if that experience doesn't exist, you have to guess on certain design decisions, and see how it comes out.
Top notch programmers are obviously going to guess a lot better than someone like me will. But they're still going to make mistakes. When enough of those mistakes pile up, you're going to need to do a rewrite.
You could make a point that's opposite of the one that the article makes by looking at the java libraries.
They made choices with their original AWT gui tools that were just wrong. They weren't dumb people -- they just didn't know, the experience necessary to make the right choice simply didn't exist. Once they tried it, they realized it wasn't working, and they came back with Swing.
Rewrites are always going to be necessary for new sorts of projects, because you can't just sit in your armchair and predict how complex systems will work in the real world. You have to build them and see what happens.
Most of the examples given needed rewrites to remain viable. It's easy to look at a package from afar and declare it "perfectly sufficient". Things look different when you have to work with a system daily. In particular, rewrites often address shortcomings in a system's capacity for extension. Just compare the number of third-party extensions available for Netscape 4.* vs. the number now available at mozdev.org for Mozilla and Firebird.
A bigger problem, to my mind, is when a half-dozen projects with the noble intention of replacing an aging kludged-up tool are started, all of which suck in different ways, and none of which learn from each other. And then they lose momentum and stagnate.
Examples? Most programmers agree that "make" is overdue for replacement, but despite many attampts (cmake, jam, cons, ant) no one has managed to come up with one that is compelling enough to catch on. CVS is a crufty mess, but none of it's potential replacements are mature enough or have the kind of widespread tool support to make much of a dent in CVS installations. And there are dozens of written-from-scratch applications which differ primarily on the GUI toolkit they are based on, which would be better apps if they incorporated the best features from all into a joint effort. My idea of the perfect browser combines features of Konqueror, Galeon, Epiphany, Firebird, and Safari.
--
CPAN rules. - Guido van Rossum
It's really common to build something, step back, examine its warts, and start over again with a new perspective and understanding. It's called prototyping. Some people actually build the first one with the intent of throwing it away. Others release it as v1.0, and introduce issues of the kind this author is referring to.
There are many reasons you might prefer a rewrite. The main one, to me, is that complicated applications contain layers and dependencies, not all of which are obvious to a new programmer. If, after some analysis, your assumptions about these dependencies are wrong, you'll break the original code faster than you can say "global variable". In the end, you could easily spend more time and effort patching and praying than you would rebuilding from the ground up.
Of course, if some of the original architects are still involved in the project, arhictectural knowledge and assumptions can be transferred to new programmers in a fairly fluid way, and I suspect it is in these cases where you can confidently add on to an existing code base.
And it's always helpful if the previous programmers were actually good programmers, and who wrote code and comments that were mindful of those who might follow them later. But that's not within your control.
How is it possible to so completely miss the point of Perl 6? The intent is not necessarily to replace Perl 5 - Perl 5 is fantastic and the Perl 6 developers above all people know this. Perl 6 is perhaps best thought of as a DIFFERENT LANGUAGE which will 'just happen' to be, in many places, very similar/identical to Perl 6.
Once you start thinking of Perl 6 in that manner, you realise what it's for. It's not to replace all of the Perl already out there. It's to provide a new tool, a new language for doing new things in, drawing on the experience gained in years of working with Perl 5 and other languages.
Ponie, of course, is part of the effort to make sure that at least some of the vast amounts of Perl 5 code is usable with Perl 6, should programmers wish it. And even that's not a total rewrite of the existing Perl codebase.
So ultimately, that article has nothing of use in it. Yes, programmers should be careful what they rewrite and when they rewrite it, but many times such things are actually worth it. GTK+ 2, anybody?
Miri it is whil Linux ilast...