Bad Software Runs the World

whitroth tips a story at The Atlantic by James Kwak, who bemoans the poor quality of software underpinning so many important industries. He points out that while user-facing software is often well-polished, the code running supply chains, production lines, and financial markets is rarely so refined. From the article: "The underlying problem here is that most software is not very good. Writing good software is hard. There are thousands of opportunities to make mistakes. More importantly, it's difficult if not impossible to anticipate all the situations that a software program will be faced with, especially when — as was the case for both UBS and Knight — it is interacting with other software programs that are not under your control. It's difficult to test software properly if you don't know all the use cases that it's going to have to support. There are solutions to these problems, but they are neither easy nor cheap. You need to start with very good, very motivated developers. You need to have development processes that are oriented toward quality, not some arbitrary measure of output."

Comments

The old adage

[killmenow]

Good, Fast, Cheap...Pick Two.

It's a big world

[MozeeToby]

There aren't enough 'good' coders in the world to implement all the software that needs to be written, let along 'very good' ones. Not to mention good architects, designers, requirements analysts, etc, etc, etc. And even if there were, software that needs to work together isn't always designed to do so. Hacks, cludges, and jerry rigged solutions are what hold the tech world together, no amount of wishful thinking is going to change that.

Fixed

[SuperKendall]

That is because corporate infrastructure software does not obviously generate revenue, and losses of opportunity are invisible.

Fixed it for you.

Basically just supporting the last half of what you said.

Re:Numbers don't lie

[bluefoxlucid]

Raw defects doesn't indicate quality. A defect by which the system occasionally has to stop and replay some data write-out because of some hoakey disk driver is not a gerat problem: the disk driver is buggy, and is using a shitty hack to fix it. By contrast, a much better written driver with a very corner case race condition that 1/100 as often simply destroys a ton of data has a huge problem.

Linux is like that. If a hard disk drive starts to not respond, it'll send it a reset command and continue. It'll mount the filesystem read-only without special options; in some conditions that's important, because the OS view of the FS might be completely different due to undetected write failures. In any case, it's still up and you can get information out of the kernel. I've had the system hose itself so bad I couldn't actually read the logs or run dmesg, but if your boot process copies a few utilities into a ramdisk and sets tty1-5=login tty6='chroot /recovery login' you should be able to switch to that tty and run. Bonus points for statically linking chroot on boot (i.e. the boot process copies everything in from installed fs, then uses ld to statically link chroot to all its dependencies), so in a barely-functional active ssh session you can '/recovery/bin/chroot /recovery /bin/sh'

A high-quality system that fails 1/10000 of the time and destroys everything is worse than a low-quality system that fails 1/100 of the time without cause for concern. Yet the low-quality system is clearly shitty.

Re:I've worked at investment banks since '93

[ebh]

Measuring how often something *doesn't* crash is extremely hard to show to the bean-counters. So, be ready to demo.

True story: When I started a sysadmin job, we had JBODs that routinely ran out of space, causing all sorts of downtime problems, like leaving the whole building dead in the water for days. I convinced TPTB that we needed decent storage (in this case, NetApp), but many of them choked when they saw both the purchase and maintenance costs. After we had it installed, I saw that a volume was close to hitting the wall. Before doing anything, I called in the most skeptical of the PHBs, and said, "Wanna see the NetApp pay for itself?"

I showed him the alarm that indicated a space problem, and told him we were on the verge of going down for two days. I waited for his skin to turn pale, and at the right moment, I said, "But we won't, because the NetApp can do this," and in a few keystrokes and mouse clicks, I added enough space to the volume.

I then told him that things like this happen once or twice a week, and I fix them without anyone knowing anything went wrong, but that I documented them in the trouble ticketing system I had installed, so if anyone wanted to know how many disasters were prevented by having decent storage, that's where they should look.

That didn't completely solve the problem of the thousand successes going unnoticed and the one failure never forgotten, but it came close.

Re:Numbers don't lie

[luis_a_espinal]

Is there a real definition for quality?

When it comes to software, yes. We have had it for decades: quality entails

1. architecture, design and implementation decisions that minimize the cost of change,
2. that does not deteriorate with each change (or at least deteriorates linearly with each change),
3. that exhibits strong cohesion and lose coupling,
4. that permits reasonable maintainability and configurability,
5. with relative small bug count per whatever metric one picks (FP, SLOCs, etc.)
6. that is amenable for testing
7. with architecture and design that are understandable (tied with #1)

Just to mention a few. There might not be a single universal definition of software quality, but there are common desirable attributes that have been known for decades. It's when people code without knowing these attributes (or not giving a shit about them) that we get the crappy stuff we get.

Software doesn't have to be perfect. It simply needs to be economical due to its qualities (again, qualities well known for decades.)

Re:Numbers don't lie

[hairyfeet]

I think a better question would be "Why does it NEED to be better?" and the answer in most cases is "it doesn't". Remember perfect is the enemy of good and often the difference between "good enough" and great can be truly insane levels of expense.

At the last shop I worked at before striking out on my own i had to rush home one day and dig my very first two gamer PCs, a Pentium 60MHz and a P100MHz out of the shed, why? because a guy came in the shop about to have a breakdown because the PC they used to control their custom lathe had gone tits up and they had a $60k order due by the end of the week and no columns? No contract. No when i was cloning the drive I asked him "if the thing will only run on ISA, and the company is OOB, why not replace it?' and found out a unit today that will do what that one does would cost upwards of $150k, that one was paid for, it works, its easy to use.

Now I'm sure that if any of the programmers here saw that software they'd either laugh or bawl like babies. We're talking a primitive GUI running on top of DOS 3 that lets you build the design from templates or build your own from various shapes, VERY rudimentary and frankly made Win 3.x look like Win 7, ancient stuff. But ya know what? It works and works well. it does that one task, day after day after day, very WYSIWYG, you could get someone up to speed on running the unit in maybe an hour.

So while i can see why many programmers might look at something like that and think half ass I can also see why it was made the way it was. i'm sure those designing that software they are bitching about in TFA did it that way because they designed it for a task, weren't thinking about someone plugging into it down the line, and just concentrated on that one task. I'm not saying one should design like that today, but usually when i hear programmers screaming about bad software its because they've found something old and creaky and are bitching because they are gonna have to support it. Welcome to the real world Chuck, where jobs often suck.

And just to prove how fussy programmers can be I'll end this with something that always ends up getting programmers to gnash their teeth...I LIKE VB6, okay? If all you need is to have a GUI to a local DB frankly there is NO tool that can compare to VB6. Its fast, uses practically zip when it comes to resources, easy to whip off prototypes right there if you need to so the customer can see what fields he/she needs, for that one task and one task only, putting a GUI to a local DB even now you just can't beat VB6. That is why to this very day its like the third most popular business language, because businesses need that job often and it does it VERY well.