A Gentle Rant About Software Development and Installers

Nerval's Lobster writes "This is the story of the comparison that just wasn't meant to be. It's a story of everything that can go wrong in the customer end of the software world, and some thoughts on what needs to be done, especially in an area known as Installers. I'm a software engineer with 25 years of experience, and for years I've wanted to point out some of the shortcomings of my own industry to help make it better for everyone involved—not only for the end-users, but also for the IT people who have to support the products; the salespeople who have to sell and later, possibly, apologize for the software; for the executives whose hands are tied because they don't have the technical knowledge to roll up their sleeves and help fix problems in the code; and for the programmers themselves who might get stuck with what some consider the absolute worst position for a programmer: maintenance of crappy code written by programmers who have long since left the organization."

Comments

Put everything in the cloud!

[Anonymous Coward]

Problem solved. I will come by later to pick up my consulting fee.

Re:Put everything in the cloud!

[fyngyrz]

I don't mean to rain on your parade, but that seems a bit precipitate.

You know, there was a time when "vaporware" was a bad thing.

The problem is that we still use installers...

[gestalt_n_pepper]

Instead of creating single exe. Memory and disk space are cheap. Portable application creators are easily and cheaply available (e.g. Cameyo.com - both free and excellent at what it does). It's easier to copy a single file than to go through the hell that is installation for most apps.

Re:The problem is that we still use installers...

[Tackhead]

Instead of creating single exe.

This is enterprise software; it's a much bigger problem than that. You're not installing "a single app", you're typically installing an entire software stack. LAMP is just the beginning.

From TFA:

Except it didn't really install correctly. One of the servers just wouldn't start up, and gave me no indication whatsoever the problem was. Another serverâ"the Tomcat web server that hosts the user interfaceâ"started up fine, so I was able to get to a login screen through the Web browser.

... Oracle provides a pre-built Linux image with all of the necessary business-intelligence tools already installed, running inside Oracle VirtualBox. "What could be easier?" were my famous last words.

The past 20 years have seen us escaped Windows and DLL hell by moving to Linux. Then, we escaped its own little twisted maze of dependency hell by using apt-get. Then, we used all those open source tools to build ... the infrastructure that was then used by the closed-source community. In the case of the SAP product, it comes with Tomcat as a web server. In the case of the Oracle .OVA installation, an entirely preconfigured Linux install that probably comes with its own separate stack. If you keep them all on separate VMs, you've got a shot at getting them to talk to each other, but is that really the best use of the underlying hardware? One bit of Java talking to some other abstracted piece of Java, using dozens of VMs as intermediate layers of abstraction?

And now we're right back where we started. SAP will support this revision of Tomcat which works with this version of Java, because, well, that's the Java way. And Oracle appears to have solved that problem by throwing down a .OVA for every subcomponent.

Web services are a great way to save developer time - but in return for that, they're yet another layer of abstraction that has to be dealt with. Virtualization's a great way to save administrator time. Rather than having to separately install "the right version" of the entire stack from the OS to Java to the configuration properties of the web server, just grab the .ova and work from there. It also gives you some scalability that you might not have had - fire up the .ova on your PC for a demo, or on bigger iron for production.

But in exchange for the ability to get crap out the door ("it works on my machine / Great! Virtualize it and your machine's now the installer!") faster, we've merely exchanged one dependency tree for another: instead of kilobyte-sized .DLLs or megabyte-sized repositories of source from which we can rebuild our binaries, we're now dealing with gigabyte-sized VMs.

Maintenance Isn't a Bad Job

[Greyfox]

As long as you can incrementally improve the design and code, maintenance isn't a bad position to be in. I've seen far too many programmers who whine and whine about how much the code base sucks, but they never do anything to make it better. They insist that the only way to go forward is to rewrite the whole thing, a project that is almost inevitably doomed to failure. If you actually design new code, implement policies if the company doesn't have any in place, and clean up the old code while you're hunting bugs, it can be every bit as rewarding as new development. ANY programming project can be a joy to work on, or a nightmare to work on, and it's entirely the discipline and ability of the team and its management that makes all the difference.

Re:Maintenance Isn't a Bad Job

[Todd Knarr]

The big problem I have with your statement is that often the problem with the software is the basic design. It's 10 years old, completely insufficient for what's being asked of it today, and has so many kludges and hacks bolted on that you can't do anything outside a tight set of constraints without irreparably breaking something else. Usually it gets to this state because management or the team leadership won't permit changes to the design (the usual justification is that there's too much risk and no business benefit right now). Once code's in that state, new code has to follow the old design to work and it's that old design that's adding complications and keeping new requirements from being cleanly implemented. You can't design new code, because the constraints imposed by the design it has to live inside are too tight. You can't implement new policies because they break existing code. You can't clean up code because every bit you try to clean up requires you to first clean up 8 other bits, and eventually you hit a stop-point in the recursion where cleaning up that level requires that the program design be changed, see above.

Yes, good code should never get to this state. But the problem cases aren't good code, that's why they're problem cases.

Re:Maintenance Isn't a Bad Job

[quietwalker]

There is a problem in what you've stated, and it comes down to the last line, "entirely the discipline and ability of the team and its management that makes all the difference"

I've know a lot of exceptional programmers, and I've known a few absolutely horrible developers. However, I've known of NO developer who wanted to push code out that door that was just awful. Rather, I see a lot of developers end up over-engineering on every level to ensure that their product is the brightest, best thing ever. If they had all the time in the world, I'm sure their products would be simply exceptional.

On the other hand, I haven't worked for a single manager who, when the chips were on the table, said something along the lines of, "We'll give you 5 more moths to refactor this, to ensure that ongoing updates and maintenance will be straightforward, and our internal tools can support automation of common tasks," instead of "Just make it work, we'll worry about the rest in the next release." ...and when the next release comes up, it's, "We promised these new features, we don't have time to refactor: if it ain't broke, don't fix it."

I actually just left a company which has been fighting this problem for so long that the entire dev department is spending 80-90% of their time tracking down reported bugs, and the remaining time cramming in whatever was promised to the customers in the fastest way, damn the maintainability. Each year, the cost of bugs and maintenance has gone up, and the devs are now all on call - the operations team cannot support the product themselves anymore. Think about that; you are a developer, and you are on call. 24-7.

The problem is that you need an exceptional development manager who can fight the good fight for the good of the product and the company, and can make the subjective value of code reuse, good architecture, and so on apparent to those above and around him. The company needs that discipline you point out to set realistic customer expectations, to train their sales people not to overreach just to ensure a sale, to listen to the development managers when they describe cost, manpower required, and so on.

Of course, that's only if you want a perfect product, with ever-decreasing quality returns for your time and money investment. It's probably impossible for a developer or even development manager to say where the pivot point is in the market; where time and money spent on quality causes a potential loss of revenue for a product. Unless we're talking about military, healthcare, or hostile environment systems (subs, satellites, space travel, etc), there IS a point where the cost for quality is too high.

In layman's terms, the cost to make high-quality software exceeds the price people would be willing to pay for it. This means that we're all going to be happy 'enough' with imperfect software.

Re:Maintenance Isn't a Bad Job

[Anonymous Coward]

Try working in a place that doesn't permit rewrites and frowns on refactoring. I've seen software policies that a 120% guaranteed to result in the shittiest, cruftiest, nastiest pile of the most vile mismanaged crap you'll ever have the misfortune of trying to debug. What you call "whining" is usually the programmers trying to get 15 minutes of refactor time into the schedule. Answer: "No, bitches, where iz my features?"

Trying to fix it is futile and "against company policy".

Why?

[BCW2]

Why does every windows application programmer think their program is so special or important that it needs to run in the background? The first thing to speed up any computer is to kill all the crappola trying run at start up. A program should run when started and when exited from it should completely shut down and even delete any temp files it created. I did a cleanup on a customers machine once and deleted over 10,000 temp files. That is lazy and stupid programming.

Re:Why?

[Chris Mattern]

The windows pc+user is "Territory", applications have the main function of occupying it, and if feasible preventing the competition to do the same as much as possible. Exerting control yields profits.

Exactly. Program A completely exits when the user shuts it down. Program B keeps most of itself still in memory and running in the background when the user "shuts it down." Result: When the user starts up Program B again, he is pleased at how fast it comes up. When he starts up Program A again, it has to load--this is an acceptable price for not turning your PC into a mass of sloth all the time, but the user doesn't see that part. And the loading is even slowed down by Program B hanging eating up memory and cycles! So the user thinks of Program B as an efficient, fast program, and Program A as a slow piece of crap.

Re:Why?

[JaredOfEuropa]

Indeed... and that brings me to an area that needs way more attention than installers, which (in my experience) mostly work reasonably well. Lets talk about the way installed software is kept up to date. You'd think by now we would have solved this problem, but the following list of my issues with this process is still sadly valid in 2012:
- Program installs a little update process to run in the background. Don't. Acceptable ways to check for updates are: checking when the software itself is run, or glomming onto some centralised updater (take the App store as an example).
- Program doesn't really update itself, but initiates the download for a fresh install file that I then have to run and click through. Don't. Instead, make sure the program updates itself with minimal user interaction. (Kaspersky does this well; the Battlefield 3 browser plugin doesn't)
- Program updater makes you specify every install option again, like the installation directory, or even the ^&%$*(&@$ license that has to be accepted again. (Java is one of the many things that does this)
- Program updater asks to install a browser toolbar (the default is "yes", of course). Seriously, Adobe, a browser toolbar, in 2012???
- On top of all of the above: publish an update every other day or so. Don't unless you're patching critical security holes.

In short, the updater should not be implemented as a background process unless there is a very, very, very good reason to do so, and should perform the update with minimum user interaction required. That is: asking me for permission, and nothing else. And it certainly should not install anything other than the software being upgraded: publishing an update is not "an opportunity to re-engage with existing customers".

Re:Okay.

[Giant Electronic Bra]

Because most Enterprise software (and I manage a product line which is definitely in this category) isn't something so simple that you can install it with RPM or Installshield (or whatever the heck MS calls it now). There are multiple interacting services, very specific dependencies for things that are often not packaged at all for the target platform and usually cannot be distributed, etc. Complex databases, usually on other machines, have to be set up, XML files edited, etc to even create a basic working environment for our software. Now, MOST things might not be quite this complicated, but I suspect most software that you have to install largely by hand has many of these kinds of issues. Most of this stuff has to integrate fairly deeply with the client's IT infrastructure and setting up the basic applications is only a small part of it.

In our case we have some installers for some specific tools, but the main product? No, it wouldn't materially assist in setup and would just be yet another task to maintain when some tarballs/zips, thorough instructions, and heavy tech support works fine. You're already typically paying from 10's to 100's of K up front and 1000's a month to even run real Enterprise class software anyway. Surprisingly installer tech that was designed around simple desktop apps is just not that helpful. Our installer is a guy that shows up at you office, lol.

Re:Okay.

[sribe]

Bullshit. There's not one thing in your list that cannot be handled with an installer. You just don't care to automate the process, and are making up excuses to justify that after the fact.

Agree

[pr0nbot]

I frequently remark to my colleagues how bizarre it seems to me that after 50 years or so of software engineering, we're still building awful crap. If we were architects, we'd be unveiling skyscrapers built using favela techniques of plugging any old crap together, in the mud, next to a river.

There are lots of reasons, but the two big ones I'd say are time pressures (which we as programmers will never be able to resolve) and a lack of code reuse (which we can resolve). For example, the constant churn of technologies to me is simply a failure to reuse code.

I would start by trying to engineer whole classes of faults out at the language level, as has been done with buffer overflows and garbage collection.

Make static analysis much more anal, forcing the programmer to express their intent up front - static types, constraints, etc. Make the compiler a totally pedantic Nazi. Sure, it's nice to be able to hack shit up in an afternoon in Python or whatever, but then it ships, and the bugs come in, and you end up adding a pile of asserts and whatnot that should have been caught way before the product shipped.

Make unit/integration testing a mandatory part of the build, i.e. the compiler/linker refuses to link with code that hasn't been marked as tested.

If we learned to put the hard thinking and effort into designing APIs, and then reusing those same APIs across whole new classes of problem (because the language makes defining APIs is such a hassle that we'd rather not dream up new ones left and right), I think things would improve massively. Forcing the APIs to be public, but allowing the internals to be as obscure and proprietary as you like, would allow for reuse, interoperability, and hopefully improvement (by replacing particular implementations of APIs with better ones). Add a sane API mechanism for backwards compatibility, so that when you realise the API is fundamentally bad, you can or are required to implement the old API in terms of the new, and you don't just abandon people to DLL version hell. A language could provide support all of these things.

None of this would stop you from writing shitty code. But at least, to do so, you'd have to knowingly subvert the compiler in a bogus way, ignoring screeds of the compiler telling you that you and your code suck goats' balls.

Is there a patent on administering electric shocks every time there's a build error?

Re:Agree

[slim]

All of this assumes that the vendor *wants* it to improve. TFA shows that where there's an incentive for good installers, they get written. MySQL installs in a snap. Mainstream open source software installs on Linux in an apt-get/yum/whatever one-liner.

Now look at Oracle DB -- one of TFA's examples of "bad". The people who specify Oracle are seldom the people who will be installing it; or if they are, they're people who've done Oracle training and are charging by the hour for Oracle consultancy.

Lots of people *benefit* from Oracle being a dog to install. Consultants, as above. Staffers who get to put down a week's timesheets for "Oracle installation and configuration". Oracle themselves, because for certain decision making managers, "serious" software is difficult to install -- if you can install it in 20 minutes it must be a toy; and because they can sell books / training / certification.

There's a lot of people who would lose out on profitable (but wasteful) activity if enterprise software was easier to install.

Re:Okay.

[GoatCheez]

I agree. By thinking about installation when creating complex enterprise software, you end up making better quality software. When you're forced to handle the inter-dependencies of complex systems in an installation process, you end up having a better handle on how everything interacts. This has been my experience at least.

Well, Oracle and SAP are THE WORST POSSIBLE

[realmolo]

It's hard for me to think of any software companies that are worse at creating software that actually WORKS.

SAP and Oracle are notorious for pushing out incredibly expensive, complex products that are impossible to install and generally don't work like ANYTHING else.

SAP, especially, seems to be incapable of releasing a product without a half-dozen show-stopping bugs that require obscure workarounds that you'll only find out about by calling support. I won't even talk about the unholy mess that is SAP's support site.

There's a rule about software that people often forget, and it's this:

"Software quality is inversely proportional to cost". In other words, the more expensive a given piece of software is, the crappier it is. Oracle and SAP are the NUMBER ONE offenders in this regard.

Re:Okay.

[Tanktalus]

Because RPM/DEB makes assumptions. And some of those assumptions are simply invalid for some use cases.

Ever want to install more than one copy of Apache? Maybe you want your database installed somewhere other than default because that's where your GPFS shared disks are mounted? Ok, the latter is possible with RPM, though sometimes a bit more difficult. The former was also possible, but far more painful.

Complex post-requirements? Sure, RPM handles pre-reqs okay, though not circular ones. Post-reqs? When we were switching away from RPM in 2001/2002, no.

Cross platform? No. Want to get your apps installing on Linux, AIX (ok, AIX has RPM now, but it's severely outdated), Sun, HP, and anything else that comes up? Sure, they have their own installers. With their own unique quirks and issues.

In the end, we used tarballs with a Java/C++ front-end. Far simpler. And, for enterprise software, one of the better installers. Though maybe I'm a bit biased there :-)

The authors experience is largely my own.

[Vellmont]

I'm a developer who winds up having to do a lot of backend support and installs, I've been installing various enterprise packages for the last 6 years. The authors experience is VERY familiar to me. It's quite hit or miss, with some of the most expensive ($40,000+) pieces of software giving the most miserable experiences. You spend days trying to fix this or that, and it winds up being some obscure setting somewhere that only a super-expert could ever understand.

What sucks is that we have to put up with this crap. End users wouldn't stand for it.... but yet sometimes I swear IT staff think it's somehow OK, and they either blame themselves, or think they've "learned" something by going through these dumb install problems and jumping through the hoops. I'm tired of it, and it wastes a lot of valuable time. There's some things that can't be avoided, but the majority of the problems I've come across could have provided MUCH better indications of what went wrong, or avoided the problem altogether.

Re:A Gentle Rant About Fixed Width Web Content

[Marc_Hawke]

I see your, 'fixed width web content' rant, and raise you a 'running browser full-screen' rant. There's no useful purpose (as you've so elegantly pointed out) to running your browser the entire width of your monitor. In fact, the entire point of a wide screen monitor is so you can get more done (ie, more done at once.) So, have your browser as a nice window on the side, that's in a size and shape that's useful for the content, and use the rest of your widescreen monitor for something else. Save 'full-screen' for those times when the content dictates that you use full-screen.

I find it interesting that 'zooming' was one of your proffered solutions, but scrolling isn't. You realize that even if you did zoom, you'd still have to scroll?

Enterprise apps are supposed to be hard

[quietwalker]

To those who haven't read the article, the author posits that testing two same-purpose pieces of software to see if they generate the same result is a simple proposition. Then we find out that the systems are SAP and Oracle. This is not like installing two mp3 players, these are the poorly-defined field of 'enterprise apps'.

I don't disagree that the installation for anything claiming to be an enterprise app is usually hard. Setting either SAP or Oracle ~properly~ requires expert knowledge, and running either ~properly~ requires expert knowledge. This is expected, because it's an extremely complex product which is meant to be deeply customized to your own business solutions. This is similar to the gulf between installing the next version of windows and installing, say, a slackware distribution. One is more about clicking next, while the other requests explicit knowledge of the system in order to tailor it to your specific needs, and even after install requires effort to make it usable. One requires you know nothing, and allows almost no customization, the other expects you know everything, and allows an almost infinite level of customization.

That's why the install of MySQLwas so easy for him. Nothing against MySQL, but I'm not going to put it in the same category as Oracle or SAP. It's not trying to be either of those - it has it's own niche. It just doesn't have the same level of customizations or capabilities.

The other thing that always irritates me: Why the hell would you have a software developer installing 'enterprise' software anyway, unless they're some sort of expert in that software type anyway? Don't get me wrong, I've installed many a developer version of Oracle locally, but I'd be the first to point out that I'm not the Oracle-certified expert that I expect will be running the show in production. Don't people understand that these are separate skillsets? The author states that he came from a j2ee shop running large products, my guess is, he didn't have to know how to admin the application servers. Ever try to dive into enterprise app server configurations? Clustering across firewalled domains with reverse proxies, remote caching, load balancing, certificate management, and active directory tie ins? In something 'big', like JBoss or worse, WebSphere? You could be an incredible java developer, but it has nothing to do with setting up and configuring the app server.

So, he was the wrong person for a task that was, quite honestly, supposed to be hard. Of course he had problems.

Re:Enterprise apps are supposed to be hard

[Maximum Prophet]

Setting either SAP or Oracle ~properly~ requires expert knowledge, and running either ~properly~ requires expert knowledge. This is expected, because it's an extremely complex product which is meant to be deeply customized to your own business solutions.

I hear this, but I think, "A C complier is a extremely complex product meant to be deeply customized to your own business solutions." Yet GCC is relatively easy to install, and once installed I can start customzing right away. I can even use customizations that I've written years before and expect them to run with little or no modification. (C programs)

Installing the product, and running a test case should be a no-brainer, no matter how big the product is. After it's installed and running, then you can start to add business customizations that might be difficult and time consuming.

Re:It's the submitter's fault.

[war4peace]

Why should an automobile last longer than an OS?

That has no longer been valid since post-'95. An OS lasts just as long as an automobile, if you consider all the variables. If the infrastructure doesn't change, an OS would last forever. But infrastructure (read hardware support, network support, etc) does change, and rapidly. Your own automobile would be good for nothing if, for example, current roads would be replaced by magnetic field monorails.

Re:Okay.

[sribe]

I agree. By thinking about installation when creating complex enterprise software, you end up making better quality software. When you're forced to handle the inter-dependencies of complex systems in an installation process, you end up having a better handle on how everything interacts. This has been my experience at least.

Absolutely true. I didn't even bother going there in my response, because I figured it was pointless when someone was actually claiming "enterprizey software is simply too complex for installers". Sigh. As usual, the creators of the problems are blind to the problems...

If we build it

[SmallFurryCreature]

If we build it, it is because the user, may his children be cursed for a thousand generations, told us to do so. In fact, that is pretty much the reason ANY crap building has been up to cheaply in a dangerous area. ONLY because THE LAW and BIG GOVERNMENT stops users from demanding high rises build out of mud on top of mud in a flood plane on a fault line, has this stopped... and then only when THE LAW and BIG GOVERNMENT do their job really really well. Just google engineering disasters for examples when THE LAW and BIG GOVERNMENT were stopped from doing their job.

Remember that Engineers have strict laws governing their actions. Software Engineers (HAHA) don't. You want to have the same quality from Software Engineers as real engineers? PAY FOR IT! And WAIT FOR IT.

There are good reasons for building codes and there MIGHT be a cause for coding codes BUT it won't happen when "cheapest" wins out every time. I have seen it far to often when a project was reduced in quality by the customer insisting it be done cheaper and faster. Well guess what, then you get buildings oops code that fall down when someone sneezes.

Want an installer that works in more then one near perfect situation? Sure that will be X on the total bill. User: oops no and I want to save Y amount as well, so cut some more corners please.

Is there a patent on administering electric shocks every time a user demands a shortcut but expects quality?

Re:Okay.

[gorzek]

Bingo! "My organization uses a bunch of home-grown ad hoc'd garbage that isn't versioned or managed by anyone sane or competent" isn't a problem with installers, it's a problem with having a deeply dysfunctional organization. There is no software that's gonna fix that problem, and it's not because packaging/distribution software is lacking in some functionality.

If you have to install it by hand, why? What are the steps? Why can't those steps be automated? Why can't those dependencies also be automated? What, you can't edit XML files, .ini files, or registry settings on the fly? Why not?

Re:Okay.

[aztracker1]

Creating an installer for a complex system... a couple man-years of labor from your highest paid employees for a system that only gets installed a dozen times a year.. vs. a checklist... Yeah, you'll be making millions with your strategy.

Re:Okay.

[Applekid]

Not blind, necessarily. Enterprise software is almost certainly designed to be "consultant ware."

Oh, having trouble getting it installed? Yep, our software is very powerful therefore very complex. We can help you with that, let's talk about a retainer.

A function isn't working? It must be your sprawling IT infrastructure. We can help track it down for you with an engineer dispatched to your data center, what's your closest hotel?

Well, that's great

[Giant Electronic Bra]

Yet another incipient genius who probably hasn't had the slightest experience on the ISV side of this equation with the convenient one-size-fits-all answer to end all answers. Thanks, that was amazingly edifying!

Frankly, you have NO FUCKING CLUE what you're talking about. I would presume you really have no familiarity with actual commercial line-of-business enterprise class software. Any enterprise which is not run by morons and is large enough to qualify as an 'enterprise' has an extensive IT infrastructure which is highly integrated and designed around an enterprise information systems architecture, involves capacity, availability, security, and usability analysis and planning etc. When you 'deploy' software into these environments, and I'm talking about large banks, Fortune 500's, large international corps mainly there's no such thing as "drop the installer on a hard drive and run it". You're dealing with integrated management systems, enterprise-wide credentials and role management, complex licensing situations, other corporate IT policy issues, etc etc etc. The idea that MY software, which handles transactions from generally 1000's of clients and integrates with easily on average 4-5 other complex software stacks is 'lacking an installer' because you don't install it clicky--click is hilarious.

You can think what you want, but at least in my world 'enterprise software' means something very different from what some people seem to be using it for. This is not like in your typical small business environment where you slap up a copy of Apache and install Trac on it and call it a day (and even there I'll note there's no single installer that can handle setting that up for you).