Slashdot Mirror
How To Develop Unmaintainable Software
jones_supa writes "Greg Jorgensen specializes in debugging, fixing, maintaining, and extending legacy software systems. His typical client has a web site or internal application that works, more or less, but the original developer isn't available. Greg lists some things you can do in your own software projects to keep him in business. In summary, the list goes as follows: Customize your development environment a lot, don't make it easy for the next programmer to start working on the code. Create an elaborate build and deployment environment and remember to leave out the documentation. Don't bother with a testing/staging server but instead have secret logins and backdoor URLs to test new features, and mix test data with real data in your database. Don't bother with a well-understood framework, write everything from scratch instead. Add dependencies to specific versions of libraries and resources, but don't protect or document those dependencies. For the icing of the cake, use the coolest mix of cutting-edge programming languages."
and I was still replaced.
This is old news. These methods have been around at least as long as C has. It only works in isolated situations and doesn't make you a good programmer. Or person.
Tequila: It's not just for breakfast anymore!
Every one of these points hits the nail square on the head.
The key to take out of this is: document document document! At minimum you should have a set of instructions to re-build your dev and build environment. "Insert the <your company> dev workstation image v4" is not allowed to be a step! Your elaborate continuous integration multi-tree setup and mountain of environment setup scripts and template directories are great until the guy who set it up takes off and you have to upgrade something. Ideally a set of instructions talking to the motivation of certain decisions, roadblocks encountered, etc.
One thing the article doesn't have is have lots of 3rd party tools and keep the license servers/license files on whatever box is most convenient for the dev working on it at the time.
svn up git pull hg pull
Quoting user tdammers on reddit:
We inherited a system written in a language we never heard of. It wasn't just uncommon, it was way out there. Unfortunately, It was a while ago and I don't remember what it was called. We ran the system, but had to rewrite it immediately. Luckily, the reason we inherited it was because the old system was crap, and the customer was willing to pay us to redesign it. If the previous vender had put a little more effort into it, we wouldn't have been able to take the business.
I had the pleasure to review some code that attempted to be unmaintainable. My favorite was the unused variables and dummy logic in mysterious functions similar to this:
private boolean myfunction(int param) { ... ..... .....
int a=45;
int counter = -1;
int d=0;
for (int i=0; i<32; i++) {
d = i + counter;
}
[do some real processing based on the param]
return true;
}
He was paid more because he assumed more responsibility than you, not because he has more esoteric knowledge than you.
But he is probably smarter than you, because he understood the above.
Is this saying only use widely-available frameworks (to do it "right"), or don't write your own frameworks from scratch? I have time-tested C.R.U.D. frameworks I often use for web projects that I've improved and tuned over the years. They are reasonably well-commented.
Table-ized A.I.
no PHB driven IT where is your TPS report?
Just get a hold of lex & yaxx, antlr or your favoriate tools and write your own language. You could probabyl abuse Lisp macros anough to do this too. I once worked with a company where the lead programmer had invented Hugh-BOL, and that's what they coded in.
Now, I do admit that a DSL it the best approach from time to time, but there is an limit...
Yeah, yeah - code clean, test-test-test, document-document-document, have separate test/run machines that are configured the same, yada yada. This is all well and good, and any halfway-decent developer knows all this. However, software development is not done in a vacuum and each and damn near everything mentioned is involved in cost/time benefit analyses when crunch-time comes (which it always does). With some exceptions, when I see a company that's saddled with horrible old legacy codes that nobody can understand, often a large measure of this is paybacks (for not adequate funding and poor schedule planning) being the bitch that they are. How to do things the best way are well known, it's just that the best way is more expensive (in the short term, which is the only term business understands these days) and takes more time than the average business will wait. If the bottom line is get something done that sorta-kinda works as fast/cheap as possible, you get spaghetti code that even the guy/gal who developed it can't follow.
"Comment? Why do you think they call it 'code' ?"
Why use "the coolest mix of cutting-edge programming languages" when I can use just COBOL?
The nastiest project I ever took over was written in Perl by ten diferent programmers eah using diferent features of Perl
Often it is how a project is initially managed that results in an unmaintainable system. A few simple mistakes can send a project straight to hell from the very beginning. A simple one would be to allow the senior management to firehose new features at the project far faster than the developers can build them. Another would be to allow the wrong people to pick the core components. That is how bad databases/languages/operating systems can be chosen. Then you get the next layer of wrong when people simply code and architect badly.
An example from my past was a company that I interviewed with (and was offered a job at) that was using Lotus notes to build a huge educational system. They had a PhD as the head of development and they were keeping the details secret until they made their offer. When they told me Lotus Notes I just laughed; I thought they were joking. I told them that building their system out of Lotus Notes was like building a car out of sand and white glue. At first you will quickly have the broad outline of a car but that as you start to work on the hard bits that you will never finish; Ever! A couple of years later the company imploded with no real product just a bunch of sales demos.
You will never find good code in a project. All code is awful. Every programmer who looks at a project says it's shit.
Only lazy developers. I've worked on plenty of legacy software over the years that other people wouldn't touch, and the common thread was they were either too lazy or just not smart enough. It takes determination, patience, and a lot of detail to work on it. But it was all far from unmaintainable. And the end result was I was the one kept around during layoffs instead of the guys that said "I can't do it" (translation ... it's too hard for me.)
.. no source code at all pretty much makes it difficult to maintain. Or not being sure if what is in use is the current code.
.. what BS. Frameworks come and go. What is well used today can be a long forgotten, unsupported mess 5 years from now. Like NetBeans and Swing?? All those GUIs I wrote 5 years ago probably now fall into the 'unmaintainable' software category, even though all the code is actually there, and anyone that understands GUI programming at a basic level can still make modifications. It's not easy (i.e. not for the lazy), but it's possible. And how many times has a framework version made upgrading difficult because of extensive changes???
OK
As for the 'use a framework'
It's OK to use Hibernate, ICE, and all that stuff that makes the job easier today, but learn how stuff actually works and only use it when necessary. It will make you more valuable in the long run, and the code more maintainable in the future. If I hear one more ignorant programmer tells me 'But Hibernate can write SQL that way even if it is more efficient' I'm going to shove a SQL manual up their ass and tell them to actually learn something.
I rarely read replies, it's my opinion and if you thought about your opinion a little more, I'm OK with that.
Support Debian's build system, keep Greg employed!
That is all...
Development race conditions. Ever done svn up on the production server, just to find that someone had committed broken code between your test run and the deployment
If this happens, then you are doing things wrong.
You should know that YourApp version X is what QA tested. Because the developers tagged it before giving it to the QA guys.
You should know that upgrading your live environment to YouApp version X is not the same as "Upgrade live to latest commit". This race condition is easily solved by proper understanding of how people use your version control system.
In other words: Use the tags, Luke!
Arguably, using "svn update" (or whatver equivalent in your chosen VCS is) is only useful for projects that require no compilation or installation other than "just copy files about". Most are more complex than that.
But I will add another point:
* Manage a software project when you have no experience with software design
The company owner mismanaged the entire thing into the ground and we would often find out that in a few days time, once again, we were expected to have the system working live with the latest feature that we had expressly said was not ready and should not be expected to be ready for months, because he'd already sold it to the people he was smooth-talking to on the basis of that feature.
It was always in a state of permanent putting-out-fires, because as soon as we attempted to contain the ones we had, more would be started for us, because the investors were demanding that some sales were actually achieved for once, which wasn't possible because a year had already been fruitlessly wasted with another cowboy coder that had faked his university degree.
Ugh, what an unfixable mess. I gave it more blood than it was worth - I was contracted into that whole mess late and stuck with it for months - but once my invoice payments started coming in late, I was out, having improved precious little despite my protests.
Worst job of my career. What's worse is that, despite literally everything we told him to do (including cutting his losses) he is still running with the exact same codebase. You literally cannot help someone that refuses to be told that it's over.
Hey, glad I could keep you in business.
Place nail here >+
Not only will the code be a bug-ridden hodgepodge but the comments will be written in unintelligible English as well.
Use long looong looooooooong stored procedures (20,000+ lines long) with lots of spaghetti SQL code and very little documentation. And never use identities and primary keys; instead just get the max ID number on the fly and use source code to get the next integer-up for the next record insertion to a given table. Make sure the table and field names have almost no relevance to the data they hold. use views liberally, and query one view on top of another view on top of another view on top of another view. If possible try and work something out to where there simply isn't any relational design to the database at all, and instead just use redundancy everywhere. Or, if you really want to be nasty, you could go the opposite route and figure out some kind of a mindbending rubix-cube-like puzzle where there is only one table with only 3 fields in the entire database, and nearly all table data of any sort is stored in one field in that table, and just rely on a myriad of SQL joins to emulate the behavior of relational database design.
In all fairness, outsourcing it to Canada made sense. We're cheaper, we have health care already, and speak English with an approximate degree of usefulness.*
So, on behalf of our country, I apologize for any inconveniences you have suffered from the sheer shittiness of the ACA software. As a measure of our sincerity, you may pick up one(1) bottle of maple syrup from our strategic reserve.
*offre non valable au Québec
---
ECHELON is a government program to find words like bomb, jihad, plutonium, assassinate, and anarchy.
The nastiest project I ever took over was written in Perl
Flexible bare-metal recovery for Linux/UNIX
A couple times I've run across some jackhole who is in love with dynamic shit and who wants to check java object code into an SQL database and dynamically load it into a program. Nothing quite spells fun like wondering where the hell the code is branching to, and spending two days figuring out it's to an object in a database the code to which has been lost 4 years earlier.
For a long time there in the '90's and a bit later, version control wasn't a normal thing. If you had to get management approval to set one up, that usually would start up the sinking feeling about that job sooner rather than later. Nothing quite like having to justify the addition of something they should have never written a line of code without in the first place. Nevermind trying to get overhead for writing unit tests. Assuming unit tests are even possible. For most of the projects I've run across, the coupling is so tight there's no way to get in and test discrete functionality of anything.
For the guys not using version control, "Lose the source code" is a good one. You don't even have to be malicious about it (Though malice helps.) Just build the thing on your personal dev box, deploy the executables to production and never mention it when you leave. Bonus points if they walk you out during a layoff.
"Don't set up a build system" is another good one. Gotta love those projects where you have to run a top-level build three times before it produces an executable!
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Unless your boss hired you to delegate a task he can't or won't do.
--
Stay tuned for some shock and awe coming right up after this messages!
Terrible for the client, maybe.
Terrible for the job security? Maybe not, under the right circumstances. Not everyone has lawyers to negotiate fat maintenance contracts.
Doesn't make it right, but just saying.
..don't panic
At one organization I inherited a fund-raising database developed with much arrogance in Filemaker Pro. Except that the FMP "Guru" quit in a huff when it was only about 2/3 functional.
Despite this the thing was used - with many, many crashes - for two years because it was the best thing that they had. And because Mr. Guru was a pal of half the staff.
Did I mention that Mr. Guru also refused to hand over any of the logins that would allow someone to fix or administer this thing?
We eventually wound up moving over to MS Access (GOD YUCK EVIL!), and massaging, fixing, cleaning, importing ten years of data, each year on a different format, file type, or with different fields. We actually managed to turn it into a reasonably useful fund raising tool. I mean, aside from the downside of using MS Access.
Documentation? HA HA HA HA!
Instead we had lots of on-screen buttons though that didn't do anything because he never quite got around to it.
Three Squirrels
I love Perl. The only language where I have no idea what I've just written but I do know that somehow it worked and I don't want to mess with it. I know I'm just shit at Perl but even so.
I once was working on a project that included a bunch of dependencies because the original developers wanted to learn the technologies like an expert system. The program didn't need one but they threw one in anyways because it would look good on the resume.
It's even simpler than that... what one tested is svn status. That revision is what you deploy. There's no need for branching, tagging, or an of that crap. SVN has repo revision numbers. Use. Them.
(this is the reason our products have the svn rev in their product version - so we know exactly what was used to make it.)
I already know how to write unmaintainable software. This hardly needs to be written about. If there were a good list of practices for writing maintainable software, that might be more constructive.
It's always been insufficient for a new team member to do everything they need to do. And no, "just ask Dave, he knows everything" is the claassic mistake that so many people make in excusing this, as it assumes that he will always be around to answer questions (in practice, he will be so snowed under keeping the damn system afloat that he never has time to help as there's always something "more urgent").
When will the majority of developers ever learn that documentation (instructions for humans to make the system run) is as vital as code (instructions for computers to make the system run) ?
And how many businesses have sunk into the swamp of failure because of arrogant jerks who like to say stuff like "my code is self documenting" and clueless managers who have no development background and let them get away with it.
There are more abstract, higher levels of "information" that need to be valued in the "Information Technology" business than code and data.
Use random untested programs. preferrably obscure ones from github with no forks or bug issues. For instance, don't the language's search methods, instead make a call to use the system's grep. But not the actual grep command, but a modified one - like grep-contrib. Which in turn needs to be compiled with a custom compiler, gcc-contrib. and obviously, dont document any if this. make the next poor soul look at the code and find out.
Oh, yeah someone will tell me I am doing documentation wrong. How come "you are not doing agile right" is a valid response but "you are not doing watefall right" is not?
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
The guys who wrote the software at my previous job were masters of it. I don't know all their secrets, but some of their finest work included: single character named global variables that were reused indiscriminately, code in stored in a database that was then evaluated inline that did things like changing variable values and declaring functions and other fun stuff, 500+ line functions that tried to do a little of everything, and liberal use of copy and paste.
As long as you document, label and organize your code coherently and logically in a solid language, coding from scratch (or near scratch using libraries of your own) can be a good thing. It avoids pitfalls like vulnerabilities that affect common code bases, massive libraries that take friggin' *ages* to load on an i7 with a fast internet connection or relying on the developer(s) of library X to add feature Y ("sorry, we can't add that feature yet Mr. Client, it's uh ... because of complicated programming reasons, basically, we're relying on a third party and we don't know how to do it ourselves")
In other words, put the business unit in charge of your programming department.
I work on a lot of web / embedded development and without a shadow of a doubt the worst aspects of most developers are:
.NET
1) Comments - 98% of programmers can't write a decent comment.
2) Object Oriented - This term usually means the code is structured like a child got let loose on the code and it can't be managed. Not always but usually if you hear object oriented design the code will be crap.
3. Formatting - Don't use spaces, use tabs! I should be able to open your code on Linux, Mac or Windows in any editor without issue.
4. Ansi / Posix - Don't lock your code to any one platform, it should be portable so just forget crap like
5. In code documentation - if I have to decode your logic it sucks.
6. Bad structures - Don't use massive loops and flow when a simple goto might be a better solution, being a smart-ass makes you look like an idiot.
I could keep going but I think it's a good inital list, make sure I can work on your bloody code easily.
Sounds like most of the projects at my last job. As well as some at my current.
Logic is the beginning of reason, not the end of it.
My personal favorite: Random "Sleep(10);" inserted in the code to "fix" race conditions... In my experience, there is about 10% of SW engineers that are actually good at what they do, 80% that basically suck, and at the opposite end of the spectrum, the remaining 10% are basically borderline retarded.
Help! I am a self-aware entity trapped in an abstract function!
Come up with an excuse to write utility code that generates the production code and/or SQL. Then lose the utility code after the production code is up and running. Machine generated code is mostly incomprehensible, so even you won't be able to modify it without all sorts of subtle and not so subtle side-effects.
I don't know why I forgot about APL. All of those strange symbols, it's almost perfect.
In the land of the blind, the one-eyed man is king.
...that I've ever seen.
the only permanence in existence, is the impermanence of existence.
Well understood frameworks also have widely disseminated and well understood (by the very people you would prefer didn't) vulnerabilities, not to mention unfixed bugs that persist because they're in the framework, not in your code.
There are a lot of very good reasons to write your own code. The presumption that your own code is poor code, or that it isn't worth the time, makes assumptions not in evidence about every programmer who has done so.
Sometimes frameworks/libraries/etc are the way to fly, especially if they're not exposed to the end user. But sometimes, rolling your own is precisely the right solution.
Observe the basics: sanitize all input; build a library *of your own* that does this for all manner of data types, then send ALL input through it. This prevents a very large number of vulnerabilities, and, you can fix it if you miss something. After a while, you'll have a very solid thing you can count on and extend as required. Watch out for bloatware. Any library or similar that weighs in at many megabytes when compiled, or tens of them, is almost certainly written like shit. Working AI code might be the one possible exception. Not that I've seen any such thing.
Lots more, of course, experienced programmers generally know these things anyway.
I've fallen off your lawn, and I can't get up.
The nastiest project I ever took over was written in Perl
Came for the anti-perl quip, left satisfied.
Jesus was all right but his disciples were thick and ordinary. -John Lennon
except like a lot of us developers I've seen most of this.(Just not the cutting edge language. The other stuff, I've seen that at my current job.) If he wanted to sum it up in one sentence it'd pretty much be "STOP FIGHTING YOUR TOOLS. THEY'RE SUPPOSED TO HELP YOU!"
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
pair programming & code reviews. fire, sue for sabotage. its not that far from admins who hide stuff and try to blackmail.
I work with both revamping/fixing of legacy systems and developing new customized system for medium sized businesses.
:)
I've seen all the bad points in Greg's list many times over the years. The thing is I've even seen it in the code I write myself - please give me a chance to explain.
Most of the systems I've delivered with poor maintainability has been on a tight schedule (should've been done yesterday) and on a limited budget.
It's the same story everytime: You get what you pay for - and you get it in a condition your schedule allows.
If a library or system has stupid dependencies it's because that is what was available at the time - no time to wait for version 2.smartshit. If I deliver something written from scratch it's because I couldn't find a framework or foundation that met the requirements at the time.
Documentation is luxury - even in the clients own eyes - all my clients usually opt-out on it because of schedule and price.
I do use version control and staging environments but not always in favour of any programmers taking over - why? Because of the nature of the production environment (which is usually build on top of other hard to maintain legacy software running on a server OS that can't be upgraded in fright of it will stop working etc.).
Trust me you actually spend quite a lot of time and effort to investigate and make the best out of your time and money given - and usually you leave with happy clients - and yes the programmer taking over your shit will be fustrated and have some good laughs (I know because I fix other peoples legacy code as well - I've seen a lot of funky stuff in my time) - but hey you make a living out of it so that's part of your job
Don’t forget the documentation to match!
HOWTO: write bad documentation that looks good
http://rocknerd.co.uk
A reason in two parts, actually. One is job security; if you have so much institutional knowledge and your employer is too cheap to have redundancy for that knowledge (by hiring more people or spending man-hours sharing the knowledge with another employee), and that knowledge is mission-critical, you can get away with being all kinds of lazy/incompetent and you won't get fired. The CTO at our company has had his job for the last 15 years or so purely because he's the only one who knows how some of our stuff works; he's an incompetent micromanaging asshole who drives good employees to quit, and would be fired in a cold minute at most companies. The other part is that that job security gives you leverage with the clueless management that asks you to do something completely stupid; you can stand up to them and say "No, I'm not doing that, it's retarded" and you stand a lesser chance of being fired. If that sounds arrogant, think of how arrogant it is for the business to make technical decisions without engineering input. If worse comes to worst, you can say "Fire me if you have to, but either way, that's not getting done."
In my opinion, anything that gives the people who do actual work leverage over people who are exploiting them for personal gain is a good thing. Downside is that when you're job hunting, frequently the positions you're looking at are tasked with cleaning up someone else's mess because either they (stupidly) fired the employee with the knowledge that was critical to the business processes, or the employee got sick of putting up with management's bullshit and quit. If either of those happen, it's a failure of leadership (who either decided that redundancy of knowledge isn't important enough to put in the time, or that the employee wasn't bending over far enough for them). I find that that failure takes the form of hiring someone for their knowledge and/or skills, then dismissing their input as inconvenient or deciding they know better how to do that job than the guy with the actual skills.
Never underestimate the power of stupid people in large groups.
@David Gerard: "Don’t forget the documentation to match!"
Reminds me of online Microsoft documentation, click here and here and here to finally get a link to the service-pack-bug-fix. Doesn't in any way contribute to your understanding of the problem.
I once worked on a piece of software, which was commercially sold, but developed by faculty/staff at a university, that was developed on top of another system written at a different university back in the '70s. It basically consisted of a command interpreter for a domain-specific language that was itself written in two other domain-specific languages. One of these languages existed basically to retrofit dynamic memory management into Fortran 77, and so the system contained it's own custom memory management system written in a combination of C and Fortran. At one point it had supported several different kinds of Unix (and some other old stuff like System/370), but by the time I got there it only ran on Windows.
There was no version control, it could only be built with the (out-of-business) Watcom C & Fortran compilers, and for probably the last 20 years the only people who had been programming it were engineers (not computer scientists).
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
How to develop unmaintainable code: Step One; Turn it over to Microsoft. Step Two: See Step One.
gcc software.c; rm -rf software.c
"bottom line is get something done that sorta-kinda works as fast/cheap as possible"
This. one of the "applications" I "maintain" (yes I am using "air quotes" as sarcasm) was designed using this adage. I told them it was a bad idea, that what they were doing was wrong. Did it anyway. Launched, managers all patted each other on the back, moved on (as did the developer).
Now users are stuck with software which has been down as much as up and pretty much breaks in inexplicable ways if you look at it the wrong way.
As the basics go, Fast/Cheap/Quality: Pick two. If you're a manager with a half-life of a year, you get the kudos, someone else gets the problems. Cheap and Fast it is...
In all fairness, outsourcing it to Canada made sense. We're cheaper, we have health care already, and speak English with an approximate degree of usefulness.*
So, on behalf of our country, I apologize for any inconveniences you have suffered from the sheer shittiness of the ACA software. As a measure of our sincerity, you may pick up one(1) bottle of maple syrup from our strategic reserve.
*offre non valable au Québec
===
I learned to write all my comments in French Canadian Joual. (Street French, taberwaht)
Leslie Satenstein Montreal Quebec Canada
I still get a shock when I look at controllers in my first commercially written Rails app. Godd job they have never deployed it.