A Gentle Rant About Software Development and Installers

Nerval's Lobster writes "This is the story of the comparison that just wasn't meant to be. It's a story of everything that can go wrong in the customer end of the software world, and some thoughts on what needs to be done, especially in an area known as Installers. I'm a software engineer with 25 years of experience, and for years I've wanted to point out some of the shortcomings of my own industry to help make it better for everyone involved—not only for the end-users, but also for the IT people who have to support the products; the salespeople who have to sell and later, possibly, apologize for the software; for the executives whose hands are tied because they don't have the technical knowledge to roll up their sleeves and help fix problems in the code; and for the programmers themselves who might get stuck with what some consider the absolute worst position for a programmer: maintenance of crappy code written by programmers who have long since left the organization."

Comments

Put everything in the cloud!

[Anonymous Coward]

Problem solved. I will come by later to pick up my consulting fee.

Re:Put everything in the cloud!

[fyngyrz]

I don't mean to rain on your parade, but that seems a bit precipitate.

You know, there was a time when "vaporware" was a bad thing.

Maintenance Isn't a Bad Job

[Greyfox]

As long as you can incrementally improve the design and code, maintenance isn't a bad position to be in. I've seen far too many programmers who whine and whine about how much the code base sucks, but they never do anything to make it better. They insist that the only way to go forward is to rewrite the whole thing, a project that is almost inevitably doomed to failure. If you actually design new code, implement policies if the company doesn't have any in place, and clean up the old code while you're hunting bugs, it can be every bit as rewarding as new development. ANY programming project can be a joy to work on, or a nightmare to work on, and it's entirely the discipline and ability of the team and its management that makes all the difference.

Re:Maintenance Isn't a Bad Job

[Todd Knarr]

The big problem I have with your statement is that often the problem with the software is the basic design. It's 10 years old, completely insufficient for what's being asked of it today, and has so many kludges and hacks bolted on that you can't do anything outside a tight set of constraints without irreparably breaking something else. Usually it gets to this state because management or the team leadership won't permit changes to the design (the usual justification is that there's too much risk and no business benefit right now). Once code's in that state, new code has to follow the old design to work and it's that old design that's adding complications and keeping new requirements from being cleanly implemented. You can't design new code, because the constraints imposed by the design it has to live inside are too tight. You can't implement new policies because they break existing code. You can't clean up code because every bit you try to clean up requires you to first clean up 8 other bits, and eventually you hit a stop-point in the recursion where cleaning up that level requires that the program design be changed, see above.

Yes, good code should never get to this state. But the problem cases aren't good code, that's why they're problem cases.

Re:Maintenance Isn't a Bad Job

[quietwalker]

There is a problem in what you've stated, and it comes down to the last line, "entirely the discipline and ability of the team and its management that makes all the difference"

I've know a lot of exceptional programmers, and I've known a few absolutely horrible developers. However, I've known of NO developer who wanted to push code out that door that was just awful. Rather, I see a lot of developers end up over-engineering on every level to ensure that their product is the brightest, best thing ever. If they had all the time in the world, I'm sure their products would be simply exceptional.

On the other hand, I haven't worked for a single manager who, when the chips were on the table, said something along the lines of, "We'll give you 5 more moths to refactor this, to ensure that ongoing updates and maintenance will be straightforward, and our internal tools can support automation of common tasks," instead of "Just make it work, we'll worry about the rest in the next release." ...and when the next release comes up, it's, "We promised these new features, we don't have time to refactor: if it ain't broke, don't fix it."

I actually just left a company which has been fighting this problem for so long that the entire dev department is spending 80-90% of their time tracking down reported bugs, and the remaining time cramming in whatever was promised to the customers in the fastest way, damn the maintainability. Each year, the cost of bugs and maintenance has gone up, and the devs are now all on call - the operations team cannot support the product themselves anymore. Think about that; you are a developer, and you are on call. 24-7.

The problem is that you need an exceptional development manager who can fight the good fight for the good of the product and the company, and can make the subjective value of code reuse, good architecture, and so on apparent to those above and around him. The company needs that discipline you point out to set realistic customer expectations, to train their sales people not to overreach just to ensure a sale, to listen to the development managers when they describe cost, manpower required, and so on.

Of course, that's only if you want a perfect product, with ever-decreasing quality returns for your time and money investment. It's probably impossible for a developer or even development manager to say where the pivot point is in the market; where time and money spent on quality causes a potential loss of revenue for a product. Unless we're talking about military, healthcare, or hostile environment systems (subs, satellites, space travel, etc), there IS a point where the cost for quality is too high.

In layman's terms, the cost to make high-quality software exceeds the price people would be willing to pay for it. This means that we're all going to be happy 'enough' with imperfect software.

Why?

[BCW2]

Why does every windows application programmer think their program is so special or important that it needs to run in the background? The first thing to speed up any computer is to kill all the crappola trying run at start up. A program should run when started and when exited from it should completely shut down and even delete any temp files it created. I did a cleanup on a customers machine once and deleted over 10,000 temp files. That is lazy and stupid programming.

Re:Why?

[Chris Mattern]

The windows pc+user is "Territory", applications have the main function of occupying it, and if feasible preventing the competition to do the same as much as possible. Exerting control yields profits.

Exactly. Program A completely exits when the user shuts it down. Program B keeps most of itself still in memory and running in the background when the user "shuts it down." Result: When the user starts up Program B again, he is pleased at how fast it comes up. When he starts up Program A again, it has to load--this is an acceptable price for not turning your PC into a mass of sloth all the time, but the user doesn't see that part. And the loading is even slowed down by Program B hanging eating up memory and cycles! So the user thinks of Program B as an efficient, fast program, and Program A as a slow piece of crap.

Re:Why?

[JaredOfEuropa]

Indeed... and that brings me to an area that needs way more attention than installers, which (in my experience) mostly work reasonably well. Lets talk about the way installed software is kept up to date. You'd think by now we would have solved this problem, but the following list of my issues with this process is still sadly valid in 2012:
- Program installs a little update process to run in the background. Don't. Acceptable ways to check for updates are: checking when the software itself is run, or glomming onto some centralised updater (take the App store as an example).
- Program doesn't really update itself, but initiates the download for a fresh install file that I then have to run and click through. Don't. Instead, make sure the program updates itself with minimal user interaction. (Kaspersky does this well; the Battlefield 3 browser plugin doesn't)
- Program updater makes you specify every install option again, like the installation directory, or even the ^&%$*(&@$ license that has to be accepted again. (Java is one of the many things that does this)
- Program updater asks to install a browser toolbar (the default is "yes", of course). Seriously, Adobe, a browser toolbar, in 2012???
- On top of all of the above: publish an update every other day or so. Don't unless you're patching critical security holes.

In short, the updater should not be implemented as a background process unless there is a very, very, very good reason to do so, and should perform the update with minimum user interaction required. That is: asking me for permission, and nothing else. And it certainly should not install anything other than the software being upgraded: publishing an update is not "an opportunity to re-engage with existing customers".

Re:Okay.

[Giant Electronic Bra]

Because most Enterprise software (and I manage a product line which is definitely in this category) isn't something so simple that you can install it with RPM or Installshield (or whatever the heck MS calls it now). There are multiple interacting services, very specific dependencies for things that are often not packaged at all for the target platform and usually cannot be distributed, etc. Complex databases, usually on other machines, have to be set up, XML files edited, etc to even create a basic working environment for our software. Now, MOST things might not be quite this complicated, but I suspect most software that you have to install largely by hand has many of these kinds of issues. Most of this stuff has to integrate fairly deeply with the client's IT infrastructure and setting up the basic applications is only a small part of it.

In our case we have some installers for some specific tools, but the main product? No, it wouldn't materially assist in setup and would just be yet another task to maintain when some tarballs/zips, thorough instructions, and heavy tech support works fine. You're already typically paying from 10's to 100's of K up front and 1000's a month to even run real Enterprise class software anyway. Surprisingly installer tech that was designed around simple desktop apps is just not that helpful. Our installer is a guy that shows up at you office, lol.

Re:Okay.

[sribe]

Bullshit. There's not one thing in your list that cannot be handled with an installer. You just don't care to automate the process, and are making up excuses to justify that after the fact.

Re:The problem is that we still use installers...

[Tackhead]

Instead of creating single exe.

This is enterprise software; it's a much bigger problem than that. You're not installing "a single app", you're typically installing an entire software stack. LAMP is just the beginning.

From TFA:

Except it didn't really install correctly. One of the servers just wouldn't start up, and gave me no indication whatsoever the problem was. Another serverâ"the Tomcat web server that hosts the user interfaceâ"started up fine, so I was able to get to a login screen through the Web browser.

... Oracle provides a pre-built Linux image with all of the necessary business-intelligence tools already installed, running inside Oracle VirtualBox. "What could be easier?" were my famous last words.

The past 20 years have seen us escaped Windows and DLL hell by moving to Linux. Then, we escaped its own little twisted maze of dependency hell by using apt-get. Then, we used all those open source tools to build ... the infrastructure that was then used by the closed-source community. In the case of the SAP product, it comes with Tomcat as a web server. In the case of the Oracle .OVA installation, an entirely preconfigured Linux install that probably comes with its own separate stack. If you keep them all on separate VMs, you've got a shot at getting them to talk to each other, but is that really the best use of the underlying hardware? One bit of Java talking to some other abstracted piece of Java, using dozens of VMs as intermediate layers of abstraction?

And now we're right back where we started. SAP will support this revision of Tomcat which works with this version of Java, because, well, that's the Java way. And Oracle appears to have solved that problem by throwing down a .OVA for every subcomponent.

Web services are a great way to save developer time - but in return for that, they're yet another layer of abstraction that has to be dealt with. Virtualization's a great way to save administrator time. Rather than having to separately install "the right version" of the entire stack from the OS to Java to the configuration properties of the web server, just grab the .ova and work from there. It also gives you some scalability that you might not have had - fire up the .ova on your PC for a demo, or on bigger iron for production.

But in exchange for the ability to get crap out the door ("it works on my machine / Great! Virtualize it and your machine's now the installer!") faster, we've merely exchanged one dependency tree for another: instead of kilobyte-sized .DLLs or megabyte-sized repositories of source from which we can rebuild our binaries, we're now dealing with gigabyte-sized VMs.

Re:Okay.

[GoatCheez]

I agree. By thinking about installation when creating complex enterprise software, you end up making better quality software. When you're forced to handle the inter-dependencies of complex systems in an installation process, you end up having a better handle on how everything interacts. This has been my experience at least.

Well, Oracle and SAP are THE WORST POSSIBLE

[realmolo]

It's hard for me to think of any software companies that are worse at creating software that actually WORKS.

SAP and Oracle are notorious for pushing out incredibly expensive, complex products that are impossible to install and generally don't work like ANYTHING else.

SAP, especially, seems to be incapable of releasing a product without a half-dozen show-stopping bugs that require obscure workarounds that you'll only find out about by calling support. I won't even talk about the unholy mess that is SAP's support site.

There's a rule about software that people often forget, and it's this:

"Software quality is inversely proportional to cost". In other words, the more expensive a given piece of software is, the crappier it is. Oracle and SAP are the NUMBER ONE offenders in this regard.

Re:Agree

[slim]

All of this assumes that the vendor *wants* it to improve. TFA shows that where there's an incentive for good installers, they get written. MySQL installs in a snap. Mainstream open source software installs on Linux in an apt-get/yum/whatever one-liner.

Now look at Oracle DB -- one of TFA's examples of "bad". The people who specify Oracle are seldom the people who will be installing it; or if they are, they're people who've done Oracle training and are charging by the hour for Oracle consultancy.

Lots of people *benefit* from Oracle being a dog to install. Consultants, as above. Staffers who get to put down a week's timesheets for "Oracle installation and configuration". Oracle themselves, because for certain decision making managers, "serious" software is difficult to install -- if you can install it in 20 minutes it must be a toy; and because they can sell books / training / certification.

There's a lot of people who would lose out on profitable (but wasteful) activity if enterprise software was easier to install.

The authors experience is largely my own.

[Vellmont]

I'm a developer who winds up having to do a lot of backend support and installs, I've been installing various enterprise packages for the last 6 years. The authors experience is VERY familiar to me. It's quite hit or miss, with some of the most expensive ($40,000+) pieces of software giving the most miserable experiences. You spend days trying to fix this or that, and it winds up being some obscure setting somewhere that only a super-expert could ever understand.

What sucks is that we have to put up with this crap. End users wouldn't stand for it.... but yet sometimes I swear IT staff think it's somehow OK, and they either blame themselves, or think they've "learned" something by going through these dumb install problems and jumping through the hoops. I'm tired of it, and it wastes a lot of valuable time. There's some things that can't be avoided, but the majority of the problems I've come across could have provided MUCH better indications of what went wrong, or avoided the problem altogether.

Enterprise apps are supposed to be hard

[quietwalker]

To those who haven't read the article, the author posits that testing two same-purpose pieces of software to see if they generate the same result is a simple proposition. Then we find out that the systems are SAP and Oracle. This is not like installing two mp3 players, these are the poorly-defined field of 'enterprise apps'.

I don't disagree that the installation for anything claiming to be an enterprise app is usually hard. Setting either SAP or Oracle ~properly~ requires expert knowledge, and running either ~properly~ requires expert knowledge. This is expected, because it's an extremely complex product which is meant to be deeply customized to your own business solutions. This is similar to the gulf between installing the next version of windows and installing, say, a slackware distribution. One is more about clicking next, while the other requests explicit knowledge of the system in order to tailor it to your specific needs, and even after install requires effort to make it usable. One requires you know nothing, and allows almost no customization, the other expects you know everything, and allows an almost infinite level of customization.

That's why the install of MySQLwas so easy for him. Nothing against MySQL, but I'm not going to put it in the same category as Oracle or SAP. It's not trying to be either of those - it has it's own niche. It just doesn't have the same level of customizations or capabilities.

The other thing that always irritates me: Why the hell would you have a software developer installing 'enterprise' software anyway, unless they're some sort of expert in that software type anyway? Don't get me wrong, I've installed many a developer version of Oracle locally, but I'd be the first to point out that I'm not the Oracle-certified expert that I expect will be running the show in production. Don't people understand that these are separate skillsets? The author states that he came from a j2ee shop running large products, my guess is, he didn't have to know how to admin the application servers. Ever try to dive into enterprise app server configurations? Clustering across firewalled domains with reverse proxies, remote caching, load balancing, certificate management, and active directory tie ins? In something 'big', like JBoss or worse, WebSphere? You could be an incredible java developer, but it has nothing to do with setting up and configuring the app server.

So, he was the wrong person for a task that was, quite honestly, supposed to be hard. Of course he had problems.

Re:Okay.

[sribe]

I agree. By thinking about installation when creating complex enterprise software, you end up making better quality software. When you're forced to handle the inter-dependencies of complex systems in an installation process, you end up having a better handle on how everything interacts. This has been my experience at least.

Absolutely true. I didn't even bother going there in my response, because I figured it was pointless when someone was actually claiming "enterprizey software is simply too complex for installers". Sigh. As usual, the creators of the problems are blind to the problems...

If we build it

[SmallFurryCreature]

If we build it, it is because the user, may his children be cursed for a thousand generations, told us to do so. In fact, that is pretty much the reason ANY crap building has been up to cheaply in a dangerous area. ONLY because THE LAW and BIG GOVERNMENT stops users from demanding high rises build out of mud on top of mud in a flood plane on a fault line, has this stopped... and then only when THE LAW and BIG GOVERNMENT do their job really really well. Just google engineering disasters for examples when THE LAW and BIG GOVERNMENT were stopped from doing their job.

Remember that Engineers have strict laws governing their actions. Software Engineers (HAHA) don't. You want to have the same quality from Software Engineers as real engineers? PAY FOR IT! And WAIT FOR IT.

There are good reasons for building codes and there MIGHT be a cause for coding codes BUT it won't happen when "cheapest" wins out every time. I have seen it far to often when a project was reduced in quality by the customer insisting it be done cheaper and faster. Well guess what, then you get buildings oops code that fall down when someone sneezes.

Want an installer that works in more then one near perfect situation? Sure that will be X on the total bill. User: oops no and I want to save Y amount as well, so cut some more corners please.

Is there a patent on administering electric shocks every time a user demands a shortcut but expects quality?

Re:Okay.

[aztracker1]

Creating an installer for a complex system... a couple man-years of labor from your highest paid employees for a system that only gets installed a dozen times a year.. vs. a checklist... Yeah, you'll be making millions with your strategy.

Re:Okay.

[Applekid]

Not blind, necessarily. Enterprise software is almost certainly designed to be "consultant ware."

Oh, having trouble getting it installed? Yep, our software is very powerful therefore very complex. We can help you with that, let's talk about a retainer.

A function isn't working? It must be your sprawling IT infrastructure. We can help track it down for you with an engineer dispatched to your data center, what's your closest hotel?

Well, that's great

[Giant Electronic Bra]

Yet another incipient genius who probably hasn't had the slightest experience on the ISV side of this equation with the convenient one-size-fits-all answer to end all answers. Thanks, that was amazingly edifying!

Frankly, you have NO FUCKING CLUE what you're talking about. I would presume you really have no familiarity with actual commercial line-of-business enterprise class software. Any enterprise which is not run by morons and is large enough to qualify as an 'enterprise' has an extensive IT infrastructure which is highly integrated and designed around an enterprise information systems architecture, involves capacity, availability, security, and usability analysis and planning etc. When you 'deploy' software into these environments, and I'm talking about large banks, Fortune 500's, large international corps mainly there's no such thing as "drop the installer on a hard drive and run it". You're dealing with integrated management systems, enterprise-wide credentials and role management, complex licensing situations, other corporate IT policy issues, etc etc etc. The idea that MY software, which handles transactions from generally 1000's of clients and integrates with easily on average 4-5 other complex software stacks is 'lacking an installer' because you don't install it clicky--click is hilarious.

You can think what you want, but at least in my world 'enterprise software' means something very different from what some people seem to be using it for. This is not like in your typical small business environment where you slap up a copy of Apache and install Trac on it and call it a day (and even there I'll note there's no single installer that can handle setting that up for you).