Slashdot Mirror
Lessons From Six Software Rewrite Stories (medium.com)
A new take on the age-old question: Should you rewrite your application from scratch, or is that "the single worst strategic mistake that any software company can make"? Turns out there are more than two options for dealing with a mature codebase. Herb Caudill: Almost two decades ago, Joel Spolsky excoriated Netscape for rewriting their codebase in his landmark essay Things You Should Never Do . He concluded that a functioning application should never, ever be rewritten from the ground up. His argument turned on two points: The crufty-looking parts of the application's codebase often embed hard-earned knowledge about corner cases and weird bugs. A rewrite is a lengthy undertaking that keeps you from improving on your existing product, during which time the competition is gaining on you.
For many, Joel's conclusion became an article of faith; I know it had a big effect on my thinking at the time. In the following years, I read a few contrarian takes arguing that, under certain circumstances, it made a lot of sense to rewrite from scratch. For example: Sometimes the legacy codebase really is messed up beyond repair, such that even simple changes require a cascade of changes to other parts of the code. The original technology choices might be preventing you from making necessary improvements. Or, the original technology might be obsolete, making it hard (or expensive) to recruit quality developers.
The correct answer, of course, is that it depends a lot on the circumstances. Yes, sometimes it makes more sense to gradually refactor your legacy code. And yes, sometimes it makes sense to throw it all out and start over. But those aren't the only choices. Let's take a quick look at six stories, and see what lessons we can draw.
For many, Joel's conclusion became an article of faith; I know it had a big effect on my thinking at the time. In the following years, I read a few contrarian takes arguing that, under certain circumstances, it made a lot of sense to rewrite from scratch. For example: Sometimes the legacy codebase really is messed up beyond repair, such that even simple changes require a cascade of changes to other parts of the code. The original technology choices might be preventing you from making necessary improvements. Or, the original technology might be obsolete, making it hard (or expensive) to recruit quality developers.
The correct answer, of course, is that it depends a lot on the circumstances. Yes, sometimes it makes more sense to gradually refactor your legacy code. And yes, sometimes it makes sense to throw it all out and start over. But those aren't the only choices. Let's take a quick look at six stories, and see what lessons we can draw.
Rewriting an application with proper software design stages IS a way to improve your software.
When you have a large code base including too many corner cases snippets and dirty patches that stick out from it, then it's brine for a rewrite. But only if you carefully learn from the past mistakes.
Sent as ripples into the electromagnetic field. No single photon has been harmed in the process.
... is undocumented code. As a corollary, the first step on the path to fixing it is to execute the authors of said code and start again with competent developers who understand the value of documentation.
Sometimes you have to completely rewrite everything from the beginning.
Other times, changing a few words here and there, or a paragraph or two is all that is needed.
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
That's how we've ended up approaching this question. Is this really the same application, or are we creating something new. Often, we've found that a major upgrade isn't an upgrade at all, but a new approach to doing a somewhat similar set of things to what our client's application has done. Joel is right; there is an amazing amount of knowledge embedded in the code base. However, if that business knowledge doesn't apply, then throw it out. Artifacts from motor controllers that were obsolete a decade ago, probably not valuable in the new product. User interface paradigms, though, that will persist in the new software really strongly suggest keeping the extant code base, as the UI paradigm reflects and drives the rest of the software. ... unless the UI is horseshit, in which case, you can assume the rest of the code is horseshit and valuable only for growing mushrooms.
Try the first link in TFA.
I know, I know -- it does not look like it right one, but it is.
Nowhere in this article is any flaw in Joel's pronouncement to never rewrite found. The only of the six stories that was a rewrite of an existing product is Netscape and that was a disaster. The fact that Firefox came out years later doesn't make it any less of a disaster.
In the rest of the cases, no one is re-writing their existing product and letting the competition catch up. No, what they are doing it making a new product and moving users over if possible. That makes sense and that is what I learned from Joel's original article.
Making this seem like some kind of intriguing rethink of rewriting legacy code is false and click-baity.
The articles own facts don't support the article's conclusions.
"Netscape’s slide into irrelevance wasn’t entirely due to the rewrite—a court agreed that Microsoft had deliberately abused their monopoly. But the rewrite was certainly a contributing factor"
The graf accompanying this section shows Netscape's market share dropping from about 80% to 50% BEFORE the rewrite. Now that drop continues from 50% to near 0% during and after the rewrite, so the rewrite certainly did not save Netscape. But the slope of the decline barely changes pre- and post-rewrite. Basically, unless there's other evidence not presented, the best conclusion is the rewrite had no effect.
Also, "what finally ended the IE6 era wasn’t Firefox but Google Chrome."
Except your own graf shows IE market share dropping starting in late 2002 in mirror to the rising Mozilla/Firefox. Chrome doesn't even show as a factor until 2008. The articles own facts don't support the article's conclusions.
What really killed Netscape was releases a lousy product. 4.0 suuuuucked. (Folks on the web in '96/97 remember.) And IE at the time was releasing it's first good version, a better version. Fact is, at that time IE was better than Netscape.
What Netscape needed was a better product, and if it took a ground-up rewrite to get that better product, then a rewrite was necessary.
What we know now is the rewrite did not save Netscape. But we'll never know if there was some other course of action that could have saved the it. What we really should be doing is examining what was going on in '95/'96 to produce such a bad product and lose that market share in the first place, not at what rearrangements of the deck chairs was done as the ship went down.
Precisely why re-writing should be on the table.
WHY do the "edge cases" exist? Did you not really understand the situation originally?
Sometime you need to disconnect your software from your ego.
A lot of full rewrites are primarily due to the fact the developer wants to be the Hero, toss out the old broken code, and make a new superior on in their own image. (Which then over times becomes old and broken)
Also trying to keep bad code on life support, because you had invested so much into it over your time there, is just as bad, as the application barely meets the business needs anymore, and every fix is getting increasingly more complex.
Sometimes what the code really needs, is just some reorganization. (Move your function and methods in more logical locations in your code so you visualize the problems more easier. Beautify your code, make white spaces consistent. ) Create a script to see where each function is called and where, Delete unused functions. Identify duplicate logic and merge them, or make a parametrized function of of them. It is actually amazing how a little cleanup work can turn a mess into a manageable application for decades.
Also if designing software really try to split Logic and Interface/System call into separate portions. The routine that checks the required field, should be straight logic. Not generating HTML or Form boxes, giving the errors. Despite what your bosses say, assume the platform it is running on will be retired in 5 years, and try to plan to port the application over.
Finally, really try to use basic core components as much as possible, avoid 3rd party tools as much as you can (this includes semi official repositories such as with pip or cargo). Don't be a Hero and make a ground breaking new interface, just use normal component.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
I've been a lot of projects, some where we re-wrote from scratch, some where we just kept trying to overhaul stuff...
It's true what he says about re-writing either missing, or spending a long time discovering the special understanding of things embedded deep in your code.
What is ALSO true though, is that trying to fold new things into an ancient code base becomes harder and harder over time, to the point where even the smallest of features is torture to implement.
The approach that has actually seemed to work best on a number of projects I've been on, is to make the project kind of a Phoenix - create an overhaul that rises from the ashes of the old as it were.
In practical terms that means really burning out some deep seating issue that creates problems expanding or maintaining the software. Not the whole thing, just the most painful part. Replace it completely, tear into a hundred files and remove the tendrils of evil that have spread throughout.
Then make it all work again.
After that, suddenly it's easier to work with the codebase - not just because some aspect of it is newer, but also because you understand so much of it so much better.
Also probably a rule of thumb related to that is, no-one works on a re-write who has not lived In the code fore at least two years. That may be a tall ask these days.
I'd advise this even switching to some wholly new technology, just do part and build a bridge that works between the new and old halves.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Two programming articles in a row? How in the world am I to satisfy my deep inner thirst for bleeding-edge news about global warming, basic income, the Model 3, and Ajit Pai?
Keep this up and I'm going to have to go back to HuffPost. Sad!
And rewriting XUL into Web Extensions, making the rewrite mistake twice gave Chrome the web. We could have had a strong Mozilla, now we got a weak one. Imagine a Firefox released in 2000, not 2004. We could have defeated IE earlier and avoid IE6 all together.
I hope in five years time, when Chrome has 90%+ market share Mozillians take a hard look at themselves and see that they were responsible for the chromageddon.
His article hinges upon the assumption that Netscape was screwed over by the rewrite. In reality, they were almost certainly screwed on the business side to the extent no amount of technical effort could overcome their position.
To the extent they had technical woes, it may well have been the case they couldn't sort out how to make the improvements they wanted to make given their current design.
Now there are valid points, that 'old code' may look crusty but there is a good chance it is crusty for a reason and that sort of thinking should be ever present while making such a call, to try to understand *why* it is crusty before throwing it out. However sometimes it is for bad reasons:
-Written against a once-presumed 'winner' of the market that becomes defunct. Your shockwave website has to be rewritten because the supporting technology is toast, and you better be scrapping your flash website soon if not already.
-Maybe the runtime is still around, but *your* ability to find willing developers is difficult, so you have to switch languages/runtimes to align with the labor market
-The people doing it didn't know what they were doing and did it incorrectly. Optimally, this is the same team that recognizes they painted themselves into a corner so they know what to do next time.
-The code is full of workarounds for third party libraries that no longer apply. True this doesn't scream 'rewrite', but one of his points was that the ugliness of code is due to fixes for things long forgotten that still matter, but it's frequently the case they do still matter.
In short, like all opinions be informed and influenced, but no simple answer is ever 100% correct no matter what. Internalize the points and evaluate in your scenario.
XML is like violence. If it doesn't solve the problem, use more.
I've experienced it at multiple companies - there's an established product that's showing its age, so the C level announce there will be a new version, which has the effect of making potential customers want to wait for that rather than going with what's currently being sold. No new sales means going into debt to fund the 2.0 version and risking them going to a competitor.
I once worked in a government department that liked silver bullets. One of the things they tried was moving the development shop to one language, Java. That meant rewriting everything they wrote and even used. Even apps that were running without issues had to be re-written just because it was originally written in C. They even wanted to rewrite the utility formmail.pl that most sites were using at the time just because it used Perl.
Well, if you used duct tape and paper clips it would be much easier.
Fight Spammers!
The insistence on rewriting because "The old code is complete crap" is often just about vanity. That it works and meets the business requirements is irrelevant, they just don't like it. They don't care that rewriting it will take a lot of time and cost a lot of money to get to where they already are functionally.
While that rewrite is going on someone else is busy maintaining and enhancing the existing product. By the time rev 2 is near done it is already out of date. Many times I've seen it get canceled. Other times it just keeps meandering along sucking up resources and making grand promises while everyone else happily stays on rev 1.
Seems sort of appropriate for a discussion on "the single worst strategic mistake that any software company can make".
Maybe add this as a seventh cautionary tale.
I developed an application for reserving lab equipment for one of the largest server manufacturers in the world. It was capable of automated inventory, software and image deployment, and automated permissions handling that worked with an out-of-band IPMI interface and was capable of getting data as detailed as PCI bus info, CPU and memory information of virtually any server that runs IPMI. I wasn't the first to attempt an inventory management system like this within our company, but it was the most ambitious and long-lived solution by far and was used by 4000+ employees for thousands of servers and virtual machines at its height in over a dozen sites world-wide.
Unfortunately, it was never an official job duty. I did it during free time at work and during a painful point in my personal life where I sunk most of my waking hours into the project. It was developed on a LAMP server when I was still a very novice programmer and despite how gross and hacky the code was, the software ran exceptionally well, had a professional-looking UI, didn't require any kind of management beyond ensuring the hardware was functional and any auto-generated tickets by my application were solved and gained it gained a lot of positive attention. I had an instance at each location and a master one collecting data from those for backup purposes.
I'd always intended to rewrite the code, but life priorities and job duty changes made progress very slow on that, so despite the issues with the code, I continued updating it and maintaining it to support new products and features for nearly 8 years. The issues with my earlier novice programming skills started creating more and more problems however as I didn't develop it to be modular initially and every change I made to the UI to support the new features I was adding was piling up the problems I had to fix each time. I decided to begin learning NodeJS as I knew by this time that PHP's limitations in presenting real-time data would eventually be an issue and that it wouldn't scale very well as-is.
I began the slow and arduous process of learning a language I'd dabbled in, but never been a huge fan of. Javascript, especially when it comes to programming non-blocking code for things that must happen in a very specific order, but where some parts must be run in parallel, can be an incredibly obtuse language to program with. Javascript starts great until you start using hundreds of modules and libraries and see how splintered the community has become. It's like modding with Skyrim. I spent more time testing mods than actually playing the game. Here, I spent more time learning and testing how to make all the modules work together smoothly than actually programming the software itself, frequently finding certain combinations didn't work properly as documented and requiring extensive troubleshooting.
I'd made steady, but very slow progress as I didn't have a lot of free time to learn the language and I was basically learning as I was developing. The final nail in the coffin was that the company decided a spreadsheet-based solution without any automation, inventory or user management was the way to go... They felt that the IT labs functionality being tied to a single developer's presence was a bad idea. To be fair, I'd been trying to hand over the project to others and act as more of a project manager for awhile, but no one stuck with it long enough to even submit a single commit. The move to the new solution was a complete and utter disaster. Hardware and pieces parts gets swapped, moved, used, etc and as time goes on, the data just gets more and more stale as people complain they can't find what they need. I was happy to just be done with it as it was mandated that all IT were supposed to move away from my application, but I've found myself in an odd scenario as people are going behind IT's back to run my software and I'm having to continue providing support for it despite my desire to escape it since I have no interest in developing an application that will eventually just get hard-blocked by IT.
Over my 40 year career (thankfully retired now), I participated in 3 total rewrites.
The first two never produced anything and sucked up huge amounts of development resources.
The reason was identical both times:
New management wanted to take over a successful product and make a name for themselves.
They never really understood the original product and nor did the (largely) new development staff thay used.
The third rewrite project was a success.
The original management used most of the original development team to rebuild, from the ground up, the product.
While the rewrite took slightly longer than the estimates, the results were better than predicted.
The company was subsequently sold to a huge bank for hundreds of millions of dollars.
The team that built the original product and also did the rewrite still consider themselves a team, despite the fact
that most of us no longer work together.
IT'S ALL ABOUT THE TEAM
Full credit goes to the management for team building, which is the most important aspect of managing just about anything.
I know lots of working VB6 code that will need to be redone because Microsoft deleted the language. First rule of software development, use no Microsoft specific tech. The replacement language Visual Fred (VB.NET) converted 80% OK and left 20% as to fix. The fix requires you to restructure lots of the converted code as they didn't "port" over many of the original concepts that made VB6 productive.