Slashdot Mirror
Lessons From Six Software Rewrite Stories (medium.com)
A new take on the age-old question: Should you rewrite your application from scratch, or is that "the single worst strategic mistake that any software company can make"? Turns out there are more than two options for dealing with a mature codebase. Herb Caudill: Almost two decades ago, Joel Spolsky excoriated Netscape for rewriting their codebase in his landmark essay Things You Should Never Do . He concluded that a functioning application should never, ever be rewritten from the ground up. His argument turned on two points: The crufty-looking parts of the application's codebase often embed hard-earned knowledge about corner cases and weird bugs. A rewrite is a lengthy undertaking that keeps you from improving on your existing product, during which time the competition is gaining on you.
For many, Joel's conclusion became an article of faith; I know it had a big effect on my thinking at the time. In the following years, I read a few contrarian takes arguing that, under certain circumstances, it made a lot of sense to rewrite from scratch. For example: Sometimes the legacy codebase really is messed up beyond repair, such that even simple changes require a cascade of changes to other parts of the code. The original technology choices might be preventing you from making necessary improvements. Or, the original technology might be obsolete, making it hard (or expensive) to recruit quality developers.
The correct answer, of course, is that it depends a lot on the circumstances. Yes, sometimes it makes more sense to gradually refactor your legacy code. And yes, sometimes it makes sense to throw it all out and start over. But those aren't the only choices. Let's take a quick look at six stories, and see what lessons we can draw.
For many, Joel's conclusion became an article of faith; I know it had a big effect on my thinking at the time. In the following years, I read a few contrarian takes arguing that, under certain circumstances, it made a lot of sense to rewrite from scratch. For example: Sometimes the legacy codebase really is messed up beyond repair, such that even simple changes require a cascade of changes to other parts of the code. The original technology choices might be preventing you from making necessary improvements. Or, the original technology might be obsolete, making it hard (or expensive) to recruit quality developers.
The correct answer, of course, is that it depends a lot on the circumstances. Yes, sometimes it makes more sense to gradually refactor your legacy code. And yes, sometimes it makes sense to throw it all out and start over. But those aren't the only choices. Let's take a quick look at six stories, and see what lessons we can draw.
Well this post is crap. No link to the quoted article.
Well gee, who knew.
Rewriting an application with proper software design stages IS a way to improve your software.
When you have a large code base including too many corner cases snippets and dirty patches that stick out from it, then it's brine for a rewrite. But only if you carefully learn from the past mistakes.
Sent as ripples into the electromagnetic field. No single photon has been harmed in the process.
... is undocumented code. As a corollary, the first step on the path to fixing it is to execute the authors of said code and start again with competent developers who understand the value of documentation.
Sometimes you have to completely rewrite everything from the beginning.
Other times, changing a few words here and there, or a paragraph or two is all that is needed.
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
That's how we've ended up approaching this question. Is this really the same application, or are we creating something new. Often, we've found that a major upgrade isn't an upgrade at all, but a new approach to doing a somewhat similar set of things to what our client's application has done. Joel is right; there is an amazing amount of knowledge embedded in the code base. However, if that business knowledge doesn't apply, then throw it out. Artifacts from motor controllers that were obsolete a decade ago, probably not valuable in the new product. User interface paradigms, though, that will persist in the new software really strongly suggest keeping the extant code base, as the UI paradigm reflects and drives the rest of the software. ... unless the UI is horseshit, in which case, you can assume the rest of the code is horseshit and valuable only for growing mushrooms.
I would say, that every program from V1.x should be rewritten for V2.0.
If I have used the app a lot, I would start all over again from scratch.
Then read a good study about what where the bugs and improvements.
Then compare both.
Finally, get the best of both worlds.
Nowhere in this article is any flaw in Joel's pronouncement to never rewrite found. The only of the six stories that was a rewrite of an existing product is Netscape and that was a disaster. The fact that Firefox came out years later doesn't make it any less of a disaster.
In the rest of the cases, no one is re-writing their existing product and letting the competition catch up. No, what they are doing it making a new product and moving users over if possible. That makes sense and that is what I learned from Joel's original article.
Making this seem like some kind of intriguing rethink of rewriting legacy code is false and click-baity.
No and yes. If you code is written in C and you are trying to move the application from a character-based presentation I sure hope that you wouldn't keep your code.
I actually worked with a company back in the 90's who had a mainframe operation that they wanted to run as a Windows GUI. At the time Microfocus COBOL offered code and APIs that would allow you to take a CUI and turn into to a semi-GUI. I spent five months and it worked but as I was only a contractor I wonder how they survived after left as any changes to the CUI required a hell of a lot of work on the GUI.
If your application is a GUI and you want to go the web, same thing. And I (and MANY others) have done that also. That was a rewrite for a major insurance company. Trying to keep any of the C++ code would have a disaster. Eventually, that system was rewritten in C.NET. Again, total rewrite WAS the proper choice.
So, changing platforms and/or hardware puts constraints on what you can do. If you are simply rewriting code because you don't understand it then you don't have a good understanding of what you are doing in the first place and you will FAIL.
The articles own facts don't support the article's conclusions.
"Netscape’s slide into irrelevance wasn’t entirely due to the rewrite—a court agreed that Microsoft had deliberately abused their monopoly. But the rewrite was certainly a contributing factor"
The graf accompanying this section shows Netscape's market share dropping from about 80% to 50% BEFORE the rewrite. Now that drop continues from 50% to near 0% during and after the rewrite, so the rewrite certainly did not save Netscape. But the slope of the decline barely changes pre- and post-rewrite. Basically, unless there's other evidence not presented, the best conclusion is the rewrite had no effect.
Also, "what finally ended the IE6 era wasn’t Firefox but Google Chrome."
Except your own graf shows IE market share dropping starting in late 2002 in mirror to the rising Mozilla/Firefox. Chrome doesn't even show as a factor until 2008. The articles own facts don't support the article's conclusions.
What really killed Netscape was releases a lousy product. 4.0 suuuuucked. (Folks on the web in '96/97 remember.) And IE at the time was releasing it's first good version, a better version. Fact is, at that time IE was better than Netscape.
What Netscape needed was a better product, and if it took a ground-up rewrite to get that better product, then a rewrite was necessary.
What we know now is the rewrite did not save Netscape. But we'll never know if there was some other course of action that could have saved the it. What we really should be doing is examining what was going on in '95/'96 to produce such a bad product and lose that market share in the first place, not at what rearrangements of the deck chairs was done as the ship went down.
I can understand targeted rewrites for clarity, performance, security, etc but not just for funsies.
At a small startup I spent a brief amount of time, I kept in touch with the founder and like 6 months later he said they were in the midst of a major re-write because his new devs wanted to use a different language. At their stage, I definitely think it was a horrible decision, and since I was the original author I'm now unable to be of much use when he reaches out.
Precisely why re-writing should be on the table.
WHY do the "edge cases" exist? Did you not really understand the situation originally?
Sometime you need to disconnect your software from your ego.
A lot of full rewrites are primarily due to the fact the developer wants to be the Hero, toss out the old broken code, and make a new superior on in their own image. (Which then over times becomes old and broken)
Also trying to keep bad code on life support, because you had invested so much into it over your time there, is just as bad, as the application barely meets the business needs anymore, and every fix is getting increasingly more complex.
Sometimes what the code really needs, is just some reorganization. (Move your function and methods in more logical locations in your code so you visualize the problems more easier. Beautify your code, make white spaces consistent. ) Create a script to see where each function is called and where, Delete unused functions. Identify duplicate logic and merge them, or make a parametrized function of of them. It is actually amazing how a little cleanup work can turn a mess into a manageable application for decades.
Also if designing software really try to split Logic and Interface/System call into separate portions. The routine that checks the required field, should be straight logic. Not generating HTML or Form boxes, giving the errors. Despite what your bosses say, assume the platform it is running on will be retired in 5 years, and try to plan to port the application over.
Finally, really try to use basic core components as much as possible, avoid 3rd party tools as much as you can (this includes semi official repositories such as with pip or cargo). Don't be a Hero and make a ground breaking new interface, just use normal component.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
I've been a lot of projects, some where we re-wrote from scratch, some where we just kept trying to overhaul stuff...
It's true what he says about re-writing either missing, or spending a long time discovering the special understanding of things embedded deep in your code.
What is ALSO true though, is that trying to fold new things into an ancient code base becomes harder and harder over time, to the point where even the smallest of features is torture to implement.
The approach that has actually seemed to work best on a number of projects I've been on, is to make the project kind of a Phoenix - create an overhaul that rises from the ashes of the old as it were.
In practical terms that means really burning out some deep seating issue that creates problems expanding or maintaining the software. Not the whole thing, just the most painful part. Replace it completely, tear into a hundred files and remove the tendrils of evil that have spread throughout.
Then make it all work again.
After that, suddenly it's easier to work with the codebase - not just because some aspect of it is newer, but also because you understand so much of it so much better.
Also probably a rule of thumb related to that is, no-one works on a re-write who has not lived In the code fore at least two years. That may be a tall ask these days.
I'd advise this even switching to some wholly new technology, just do part and build a bridge that works between the new and old halves.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Two programming articles in a row? How in the world am I to satisfy my deep inner thirst for bleeding-edge news about global warming, basic income, the Model 3, and Ajit Pai?
Keep this up and I'm going to have to go back to HuffPost. Sad!
And rewriting XUL into Web Extensions, making the rewrite mistake twice gave Chrome the web. We could have had a strong Mozilla, now we got a weak one. Imagine a Firefox released in 2000, not 2004. We could have defeated IE earlier and avoid IE6 all together.
I hope in five years time, when Chrome has 90%+ market share Mozillians take a hard look at themselves and see that they were responsible for the chromageddon.
His article hinges upon the assumption that Netscape was screwed over by the rewrite. In reality, they were almost certainly screwed on the business side to the extent no amount of technical effort could overcome their position.
To the extent they had technical woes, it may well have been the case they couldn't sort out how to make the improvements they wanted to make given their current design.
Now there are valid points, that 'old code' may look crusty but there is a good chance it is crusty for a reason and that sort of thinking should be ever present while making such a call, to try to understand *why* it is crusty before throwing it out. However sometimes it is for bad reasons:
-Written against a once-presumed 'winner' of the market that becomes defunct. Your shockwave website has to be rewritten because the supporting technology is toast, and you better be scrapping your flash website soon if not already.
-Maybe the runtime is still around, but *your* ability to find willing developers is difficult, so you have to switch languages/runtimes to align with the labor market
-The people doing it didn't know what they were doing and did it incorrectly. Optimally, this is the same team that recognizes they painted themselves into a corner so they know what to do next time.
-The code is full of workarounds for third party libraries that no longer apply. True this doesn't scream 'rewrite', but one of his points was that the ugliness of code is due to fixes for things long forgotten that still matter, but it's frequently the case they do still matter.
In short, like all opinions be informed and influenced, but no simple answer is ever 100% correct no matter what. Internalize the points and evaluate in your scenario.
XML is like violence. If it doesn't solve the problem, use more.
I've experienced it at multiple companies - there's an established product that's showing its age, so the C level announce there will be a new version, which has the effect of making potential customers want to wait for that rather than going with what's currently being sold. No new sales means going into debt to fund the 2.0 version and risking them going to a competitor.
Bosses want new features which were sold to customers before they had any idea if they were even feasible and in what time frame. They don't care about technical debt or maintainability. Features, features, features. Sometimes features that nobody will ever use but which were part of the requirements because who knows why.
Never, ever mention refactoring or rewriting or anything like that. They'll say "sure, if you have time" and then give you ten more things to do.
It's a big ball of mud, barely held together by congealed elbow grease and twine.
Everyone was that new guy, once upon a time.
If life and experience doesn't teach you, then you probably aren't paying attention.
Netscape's decline had nothing to do with Netscape at all, it was the manuf of the operating system on 90some percent of computers at the time giving away their own browser packed in with the OS
I once worked in a government department that liked silver bullets. One of the things they tried was moving the development shop to one language, Java. That meant rewriting everything they wrote and even used. Even apps that were running without issues had to be re-written just because it was originally written in C. They even wanted to rewrite the utility formmail.pl that most sites were using at the time just because it used Perl.
The insistence on rewriting because "The old code is complete crap" is often just about vanity. That it works and meets the business requirements is irrelevant, they just don't like it. They don't care that rewriting it will take a lot of time and cost a lot of money to get to where they already are functionally.
While that rewrite is going on someone else is busy maintaining and enhancing the existing product. By the time rev 2 is near done it is already out of date. Many times I've seen it get canceled. Other times it just keeps meandering along sucking up resources and making grand promises while everyone else happily stays on rev 1.
The guy who writes the code, knows what went wrong and rewrites it often does a better job
Seems sort of appropriate for a discussion on "the single worst strategic mistake that any software company can make".
There's a lot more nuance in the original article. It's a long read (30 mins) but worth it. https://link.medium.com/YeSRm7XJwU
I've always liked that theory (see subject line).
Fred Brooks described it: https://en.wikiquote.org/wiki/Fred_Brooks#Quotes
Maybe add this as a seventh cautionary tale.
I developed an application for reserving lab equipment for one of the largest server manufacturers in the world. It was capable of automated inventory, software and image deployment, and automated permissions handling that worked with an out-of-band IPMI interface and was capable of getting data as detailed as PCI bus info, CPU and memory information of virtually any server that runs IPMI. I wasn't the first to attempt an inventory management system like this within our company, but it was the most ambitious and long-lived solution by far and was used by 4000+ employees for thousands of servers and virtual machines at its height in over a dozen sites world-wide.
Unfortunately, it was never an official job duty. I did it during free time at work and during a painful point in my personal life where I sunk most of my waking hours into the project. It was developed on a LAMP server when I was still a very novice programmer and despite how gross and hacky the code was, the software ran exceptionally well, had a professional-looking UI, didn't require any kind of management beyond ensuring the hardware was functional and any auto-generated tickets by my application were solved and gained it gained a lot of positive attention. I had an instance at each location and a master one collecting data from those for backup purposes.
I'd always intended to rewrite the code, but life priorities and job duty changes made progress very slow on that, so despite the issues with the code, I continued updating it and maintaining it to support new products and features for nearly 8 years. The issues with my earlier novice programming skills started creating more and more problems however as I didn't develop it to be modular initially and every change I made to the UI to support the new features I was adding was piling up the problems I had to fix each time. I decided to begin learning NodeJS as I knew by this time that PHP's limitations in presenting real-time data would eventually be an issue and that it wouldn't scale very well as-is.
I began the slow and arduous process of learning a language I'd dabbled in, but never been a huge fan of. Javascript, especially when it comes to programming non-blocking code for things that must happen in a very specific order, but where some parts must be run in parallel, can be an incredibly obtuse language to program with. Javascript starts great until you start using hundreds of modules and libraries and see how splintered the community has become. It's like modding with Skyrim. I spent more time testing mods than actually playing the game. Here, I spent more time learning and testing how to make all the modules work together smoothly than actually programming the software itself, frequently finding certain combinations didn't work properly as documented and requiring extensive troubleshooting.
I'd made steady, but very slow progress as I didn't have a lot of free time to learn the language and I was basically learning as I was developing. The final nail in the coffin was that the company decided a spreadsheet-based solution without any automation, inventory or user management was the way to go... They felt that the IT labs functionality being tied to a single developer's presence was a bad idea. To be fair, I'd been trying to hand over the project to others and act as more of a project manager for awhile, but no one stuck with it long enough to even submit a single commit. The move to the new solution was a complete and utter disaster. Hardware and pieces parts gets swapped, moved, used, etc and as time goes on, the data just gets more and more stale as people complain they can't find what they need. I was happy to just be done with it as it was mandated that all IT were supposed to move away from my application, but I've found myself in an odd scenario as people are going behind IT's back to run my software and I'm having to continue providing support for it despite my desire to escape it since I have no interest in developing an application that will eventually just get hard-blocked by IT.
Over my 40 year career (thankfully retired now), I participated in 3 total rewrites.
The first two never produced anything and sucked up huge amounts of development resources.
The reason was identical both times:
New management wanted to take over a successful product and make a name for themselves.
They never really understood the original product and nor did the (largely) new development staff thay used.
The third rewrite project was a success.
The original management used most of the original development team to rebuild, from the ground up, the product.
While the rewrite took slightly longer than the estimates, the results were better than predicted.
The company was subsequently sold to a huge bank for hundreds of millions of dollars.
The team that built the original product and also did the rewrite still consider themselves a team, despite the fact
that most of us no longer work together.
IT'S ALL ABOUT THE TEAM
Full credit goes to the management for team building, which is the most important aspect of managing just about anything.
I know lots of working VB6 code that will need to be redone because Microsoft deleted the language. First rule of software development, use no Microsoft specific tech. The replacement language Visual Fred (VB.NET) converted 80% OK and left 20% as to fix. The fix requires you to restructure lots of the converted code as they didn't "port" over many of the original concepts that made VB6 productive.
See subject - I went thru it going multiplatform from Windows to Linux in APK Hosts File Engine 10++ SR-1 32/64-bit for Windows https://hosts-file.net/?s=Down... (DL link @ bottom)
PORTED FROM Windows TO Linux/BSD (latter via binary application layer compatibility w/ Linux in BSD)
APK Hosts File Engine 2.0++ 64-bit for Linux h t t p : / / a p k . i t - m a t e . c o . u k / A P K H o s t s F i l e E n g i n e F o r L i n u x . z i p (remove spaces between chars & download)
* The code is a GOOD 90++% or so EXACTLY the same between Operating System platforms (albeit improved on Linux in FEATURES/EFFICIENCY/performance).
(Only REAL "differences" were in WinSock2 vs. *NIX sockets work & drive letters vs. mounted devices work doing the port between OS (which should make my 'ps' below EASIER since BSD is close enough to Linux vs. Windows relation to it))
APK
P.S.=> Soon for MacOS too (I just got a NEW Mac-Mini to port it there)... apk
Your software is shit. It took you over 14,000 lines of code to write a fucking string sorter.
Your software is just fine - well written, functional... I'm going to continue using the Host File Engine by mmell February 17, 2017
Your premise that hostfiles are a good way to deal with advertising and malvertising is quite valid - by JazzLad April 20, 2016
his hosts program is actually pretty good by xenotransplant August 10 2015
his hosts tool is actually useful for those cases in which one does indeed want to locally block stuff outright while consuming minimum system resources by alexgieg September 25 2015
I like your host file system by Karmashock September 09 2015
that APK guy, I use his host file by rogoshen1 Tuesday March 03, 2015
I personally use a HOSTS file blocker produced from a genius called APK by 110010001000 October 27 2017
* SEE SUBJECT & TELL US: How does EATING YOUR WORDS taste?
APK
P.S.=> You're already VASTLY OUTNUMBERED but many more are coming (you haven't done better)
Apk has the answer for that - really... kill automatic updates by adding a hosts file entry setting updates.steam.com or whatever to 127.0.0.1. You have to find the right hostname for each software you want to block updates on by raymorris (2726007) on Friday July 06, 2018
APK your posts on this and the hosts file posts, and more, have never been in error and/or bad advice by BlueStrat (756137) on Wednesday June 21, 2017
I support APK's stand on the hosts file and can't see why it's not used more than it is. My hosts file is 144247 lines long (4,332 Kb) it & a firewall serves me very well - by Trax3001BBS (2368736)
ABP is insufficient as a solid hosts file does everything APK reminds us about fast turtle September 17 2013
You need APK's hosts file - by Teun (17872) on Wednesday August 06, 2014
APK
P.S.=> You EATING YOUR WORDS != GOOD NUTRITION... apk
APK is totally right on this count. Adblock Plus on Firefox mobile is a dog on older, or lower end, phones. A hostfile based adblocker makes for a much better experience in this context. Of course, your phone has to be rooted, which isn't the case with Firefox + adblock." - by chihowa on Saturday May 16, 2015
APK solution STILL relevant Thud457 June 11 2015
In a footnote, I would like to note that I find your hosts file admirable - by vel-ex-tech (4337079) on Tuesday November 24, 2015
APK's monolithic hosts file is looking pretty good at the moment - by Culture20 on Thursday November 17
you're right about hosts files - by drinkypoo (153816) on Thursday May 26
APK, I know people give you a lot of shit regarding hosts, but please don't ever stop - by nasredin (958927) on Friday June 12, 2015 @03:34PM
APK
P.S.=> Are you ENJOYING the taste of EATING YOUR WORDS yet?... apk
APK is kinda right... I've given up on JS based adblocking and gone to blackholing in /etc/hosts, just like it was back in the 90s. The computational load has gotten intolerable for any ad-blocking using JS. I've tried his hosts file generating software. It works. - by bmo (77928) on Thursday October 15, 2015
get around to 'installing' a hosts file list, not sure which one, likely the one from someonewhocares.org. If it works as well as what I used for a while about ten years ago, I'll be happy. And grateful to APK for the lesson and the reminder. - by kermidge (2221646) on Wednesday March 27
I actually went and downloaded a 16k line hosts file and started using that after seeing that post, you know just for trying it out. some sites load up faster. - by gl4ss (559668) on Thursday November 17
dammit MS, you proved APK right about something by lgw
APK
P.S.=> Your words YOU'RE EATING: You choking on them yet?... apk
(APK) is still right a hosts file really does work. It even blocked a some of the video ads that were inserted into a stream OrangeTide February 10 2016
the Host File Engine performs exactly as promised - by mmell (832646) on Thursday February 16, 2017
I do use APK's host file on all my systems at home by OrangeTide December 01 2017
I've never tried to belittle (APK's work), I've flat out said it's good - by BronsCon (927697) on Thursday February 11, 2016 @06:48PM (#51491263)
* Toss on 100,000++ users worldwide too!
APK
P.S.=> You still haven't said how EATING YOUR WORDS tastes? apk
Dozens of /.ers liking/using/praising MY work MADE YOU EAT YOUR SHIT YOU SPEW punk (work of the likes "your kind" in a do-nothing "ne'er-do-well" shit talking WEEZIL who STALKS me by UNIDENTIFIABLE anon in you can't do) lol!
* YOU do all you KNOW how to do: EAT YOUR WORDS & LOSE, lol!
APK
P.S.=> Loser, lmao @ U... apk
We don't see you do better than APK did. You're hotair and no action. You won't answer how eating your words taste after dozens of users shut you up.
I see that retarded bitch APK's pretend friend is here to try and help defend him against the truth. If you aren't APK then you are an even bigger retard than he is. APK didn't refute anything that was said about the quality of his code because he knows the criticism is valid. Instead he just tries to deflect because he has nothing but out of context quotes that if the full comment was included would paint a very different picture of him and his work.
I see that UNIDENTIFIABLE STALKING PUSSY is here to try harass me. If you aren't UNIDENTIFIABLE STALKING PUSSY then you are an even bigger retard than he is. UNIDENTIFIABLE STALKING PUSSY has no code of his own because he knows HE HAS NO SKILLS TO CREATE IT and the criticism is valid. Instead he just tries to deflect because he has nothing but out of context quotes that if the full comment was included would paint a very different picture of him and his LACK OF USEFUL work.
* LMAO!
APK
P.S.=> Truer words were NEVER SPOKEN on /. ... apk
Emojis are in ASCII now!