More Than Coding Errors Behind Bad Software

An anonymous reader writes "SANS' just-released list of the Top 15 most dangerous programming errors obscures the real problem with software development today, argues InfoWeek's Alex Wolfe. In More Than Coding Mistakes At Fault In Bad Software, he lays the blame on PC developers (read: Microsoft) who kicked the time-honored waterfall model to the curb and replaced it not with object-oriented or agile development but with a 'modus operandi of cramming in as many features as possible, and then fixing problems in beta.' He argues that youthful programmers don't know about error-catching and lack a sense of history, suggesting they read Fred Brooks' 'The Mythical Man-Month,' and Gerald Weinberg's 'The Psychology of Computer Programming.'"

Comments

Perfection Has a Price

[alain94040]

The most common errors: SQL injection, command injection, cleartext transmission of Sensitive Information, etc.

People make mistakes. Software needs to ship, preferably yesterday.

How much would it cost to have perfect software? I happen to have worked in an industry that requires perfect coding. So I can imagine what it would look like if Microsoft tried it.

The debugger would cost half a million dollar per seat (gdb is free). There would be an entire industry dedicated to analyzing your source code and doing all kinds of proofs, coverage, what-if analysis and other stuff that require Ph.Ds to understand the results.

The industry I'm referring to is the chip industry. Hardware designers code pretty much like software developers (except the languages they use are massively parallel, but apart from that, they use the same basic constructs). Hardware companies can't afford a single mistake because once the chip goes to fab, that's it. No patches like software, no version 1.0.1.

It's just not practical. Let the NSA order special versions of Office that cost 10 times the price and ship three years after the consumer version.

But for me, "good enough" is indeed good enough.

--
FairSoftware.net -- work where geeks are their own boss

Re:Perfection Has a Price

[Opportunist]

The problem is that software doesn't even ship as "good enough" anymore. It's more like "it compiles, ship it".

Your example of hardware, and how it's impossible to patch it, was true to a point for software, too, in the past. Before it became easy to distribute software patches via the internet, companies actually invested a lot more time into testing. Why? Because yes, you could technically patch software, but it was tied to sometimes horrible costs to do just that.

You can actually see a similar trend with the parts of hardware (i.e. BIOSes) that are patchable. Have you ever seen hardware shipped with all BIOS options fully enabled and working? I haven't in the past 2-3 years. More often than not you get a "new" board or controller with the predecessor's BIOS flashed in, and the promise for an update "really soon now".

The easier it is to patch something, the sloppier the original implementation is. You'd see exactly the same with hardware if it wasn't so terribly hard (read: impossible) to rewire that damn printed circuit. I dread the day when they find some way to actually do it. Then the same will apply to hardware that you have today with some OSs: It's not done until it reads "SP1" on the cover.

Re:Perfection Has a Price

[Timothy Brownawell]

I happen to have worked in an industry that requires perfect coding. [...] The industry I'm referring to is the chip industry. [...] Hardware companies can't afford a single mistake because once the chip goes to fab, that's it. No patches like software, no version 1.0.1.

What does "stepping: 9" in /proc/cpuinfo on my home computer mean? What is a f00f, and what happened with the TLB on the early Phenom processors?

Re:Perfection Has a Price

[Jason1729]

People make mistakes. Software needs to ship, preferably yesterday.

This attitude is the number one problem with software development. When all you care about is getting it out the door, you send garbage out the door.

Software today is so damn buggy. I spend hours a week just doing the work of getting my computer to work. And even then it has random slowdowns and crashes.

I'm old enough to remember when it wasn't like that. You'd run your program and it was ready in a second, you'd exit and it left no trace. Crashes were virtually unheard of. We have people where I work who only do data entry, and they still use wordperfect 4.2 on 386 hardware. I've seen their workflow and how fast it works for them and I can see if they "modernized" it would cripple their productivity.

And for the money at stake, what's so wrong with hiring a few Ph.D's to analyze code. Amortized over a few million copies, a few 6-digit salaries aren't so bad. And the losses the software shops suffer in bad-will when their products fail costs them more.

Re:Perfection Has a Price

[bb5ch39t]

I'm an old timer, still working on IBM zSeries mainframes, mainly. We just got a new system, which runs on a combination of Linux and Windows servers, to replace an application which used to run on the mainframe. Nobody likes it. We are apparently a beta test site (though we were told it was production ready). It has a web administration interface. For some reason, on some users, the only PC which can update those users is my Linux PC running Firefox. Nobody can say why. Until early last week, it would take a person a full 5 minutes to login to the product. They made some change to the software and now it takes about 10 seconds. this is production quality? No, I won't say the product. Sorry. Against policy.

Re:Perfection Has a Price

[Schuthrax]

You'd run your program and it was ready in a second, you'd exit and it left no trace. Crashes were virtually unheard of.

And all without managed memory, automatic garbage collection, etc., imagine that! Seriously, I see so many devs (and M$, who has a product to sell) insisting that all that junk is what will save us. What they're doing is attempting to create a Fisher Price dev environment where you don't have to think anymore because they've done it all for you. What's going to happen to this world when GenC# programmers replace the old guard and they don't have the least clue about what is going on inside the computer that makes the magic happen?

Re:Perfection Has a Price

[networkBoy]

they cost a shit ton of money is what happened.

A project I was on in 2000ish went as follows:
Steppings A0, A1, A2, and A3 were halted in-fab because someone found a critical bug in simulations.
A4-A7 did not work.
B0-B4 did not work B6 did not work
C0-C4 did not work
B5, B7, C5 sorta worked.
The company folded.
That's what a software mentality working on hardware will get you.

Steppings in CPUs are a little different. Often an earlier stepping was functional enough to start the design cycle for Dell HP, et.al. but not ideal. The later steppings start by fixing the deficiencies, then beyond that are likely cost cutting.
-nB

Re:Perfection Has a Price

[Enter the Shoggoth]

The most common errors: SQL injection, command injection, cleartext transmission of Sensitive Information, etc.

People make mistakes. Software needs to ship, preferably yesterday.

How much would it cost to have perfect software? I happen to have worked in an industry that requires perfect coding. So I can imagine what it would look like if Microsoft tried it.

The debugger would cost half a million dollar per seat (gdb is free). There would be an entire industry dedicated to analyzing your source code and doing all kinds of proofs, coverage, what-if analysis and other stuff that require Ph.Ds to understand the results.

The industry I'm referring to is the chip industry. Hardware designers code pretty much like software developers (except the languages they use are massively parallel, but apart from that, they use the same basic constructs). Hardware companies can't afford a single mistake because once the chip goes to fab, that's it. No patches like software, no version 1.0.1.

It's just not practical. Let the NSA order special versions of Office that cost 10 times the price and ship three years after the consumer version.

But for me, "good enough" is indeed good enough.

-- FairSoftware.net -- work where geeks are their own boss

I worked within the same space about 10 years ago - I was a sysadmin for a group of asic design jockies as well as the firmware and device driver guys and I'm gonna call you on this...

The hardware designers were under the same sorts of pressures, if not more so, than the software guys and I saw many bugs that would end up in the shipping silicon. The general attitude was always "oh! a bug: well the software guys will just have to work around it."

And as for "no patching", well that's also BS, you can patch silicon, it's just rather messy having to have the factory do it post-fab by cutting traces on die.

So much for perfection!

Re:Perfection Has a Price

[Anonymous Coward]

The same is true in a way of software development.

Back when I was in high school, I could write a program (on punch cards) and put them in the tray to be submitted. Every week the intra-school courier came around and picked up the tray, returning the tray and output from the previous week. When every typo adds 2 weeks to your development time, you check your code *very* carefully, and examine every error message or warning that comes back from the compiler, to try to fix as many errors as possible in each submission.

With interactive compilers/interpretors, it is not worth spending that much time on verifying the mechanical coding - just fix the first obvious problem and re-submit because it is faster to let the compiler (1) refuse to complain about the parts you managed to type correctly, and (2) remove all of the messages that were cascaded issues from the mistake you just fixed, than it is to waste your time scanning for typos, or reading the subsequent error message in case there are some that are not cascades.

Re:Perfection Has a Price

[samkass]

Even in C# and Java there are experts who do know what's going on. They will replace the old guard. The rest will be people for whom contributing to software development was completely impossible before, but can now do the basics (ie. designers, information visualization experts, etc).

And of the top programming errors, many of them still apply to Java and C#. But some don't, and I see that as a positive step. I do Java in my day job, and iPhone development on the side. And while nothing in the industry beats Interface Builder, Objective-C is pretty horrible to develop in when you're used to a modern language and IDE...

Re:Perfection Has a Price

[Anonymous Coward]

I am 40 years old and have been in this world since I was around 10 or so

You're 30 dude. Deal with it.

Re:Perfection Has a Price

[curunir]

But for me, "good enough" is indeed good enough.

It might be of interest to you that Voltaire came up with your conclusion 245 years ago, and a bit more eloquently as well:

The perfect is the enemy of the good.

Re:Perfection Has a Price

[jandrese]

It doesn't help that most programming languages are lousy at validating input. C is especially bad as it has poor pattern matching capabilities by default and no dynamically sized structures. Worse, it offers absolutely no heap checking capabilities meaning at the end of the day your function is forced to trust the input instead of verifying it for itself. You can use fgets() with a buffer size listed, but if that passed buffer size is wrong for whatever reason there is no way in the language for fgets() or anything else to safely handle or even detect it.

That said, it's not impossible to write safe C code. In fact it's not even all that difficult, but it does mean that even small otherwise unrelated errors in your code can be surprising security problems.

Re:Perfection Has a Price

[jandrese]

Anybody who has worked on driver code can tell you that most hardware vendors ship an errata with their chip. Some hardware has rather significant erratas, like the SiI 3112 SATA controller that got installed in pretty much every SATA compatible motherboard back before SATA support was properly integrated into Southbridges. The CMD 640 is another example of a chip with an extensive hardware errata.

Re:Perfection Has a Price

[jellomizer]

I'm old enough to remember when it wasn't like that. You'd run your program and it was ready in a second, you'd exit and it left no trace. Crashes were virtually unheard of. We have people where I work who only do data entry, and they still use wordperfect 4.2 on 386 hardware. I've seen their workflow and how fast it works for them and I can see if they "modernized" it would cripple their productivity.

You remember wrong.
The old stuff was always failing and had a bunch of problems...
You are probably thinking about your old DOS PC. If the floppy disk wasn't corrupted thinks in general worked and it worked well. Why because the programmer could fairly easily test all the cases, and allow security bugs, as a buffer overflow can be prevented as it would take to long for the guy to physically type in the data... And it was one application with essentially the full PC at its whim.
If you tried to do computing with Multi-tasking such as apps like Desqview your stability was greatly reduced. It would make you want to wish for Windows 95. Or remember Windows 3.1 stabilty...

Downloading data via the Modem was a hit or miss activity. I remember getting disconnected trying to download Slackware as there was a combination for the zmodem that passed the hangup code to the modem, forcing me to regzip the file over again to get it.

Just because you were doing simple things back then it wasn't because they were better.

Re:Perfection Has a Price

[ColdWetDog]

Oh it's obviously Notes. Nothing else needs a zSeries to run and 5 minutes to login.

Gotcha.

Re:Perfection Has a Price

[billcopc]

What they're doing is attempting to create a Fisher Price dev environment where you don't have to think anymore because they've done it all for you.

They don't have much of a choice, since about (rand[100]) 97% of all computer schools are absolute garbage, we end up with a bunch of Fisher Price developers who can type out Java bullshit at a steady 120wpm, but the resultant app makes zero sense as they fall into the trap of "checklist development". The app does "this, and that, and that too" but does them all sloppily and unreliably. Then we have managers who review the checklist line by line then sign off because it "passes spec". They completely miss the true purpose of the app in the first place: to fulfill a human need.

If an app is unusable because of bugs or a nonsensical interface, then I don't care what the checklist says, it is a failure!

Re:Perfection Has a Price

[truthsearch]

This is especially true of web development. To patch a web application you don't even need to transmit a patch to clients; just update the web server. It's so easy to patch that many sites let the public use the code before any testing is done at all.

I spent my first 10 years programming clients and servers for the financial industry. Now, as a web developer, I'm shocked at how hard it is to find programmers who strictly follow best practices.

Re:Perfection Has a Price

[Junior J. Junior III]

The problem is that software doesn't even ship as "good enough" anymore. It's more like "it compiles, ship it".

I don't know, if you ask me, software is about a million times better than it used to be. I've never been a user of a mainframe system, and I understand they were coded to be a lot more reliable than desktop class microcomputers. But having started using computers in the early 80's as a small child, and seeing where we are now, there's just no comparison.

It used to be, computers were slow, crashed all the time, on the slightest problem, and encountered problems very frequently. Today, we have many of these same problems, but much less frequently and much less severe. An application crash no longer brings down the entire system. When it does crash, usually there is no data loss, and the application is able to recover most if not all of your recent progress, even if you didn't save. Crashes happen far less frequently, and the systems run faster than they used to, even with a great many more features and all the bloat we typically complain about.

Re:Perfection Has a Price

[chaim79]

I work at a company that tests aircraft instruments to FAA regs (DO-178b) and I know exactly what you are talking about, but for different reasons. What we are working on basically amounts to full computers (one project even used the PowerPC G5 processor, Alphanumeric keypad, and 3-color LCD Screen), but because of where they are failures and bugs are not an option (for software that qualifies as DO-178b Level A, if it fails there is a very high probability that people will die).

The testing that we do for hardware and software is way beyond any other industry I've heard of (before reading your post) and took me a bit to get used to.

For those of you interested, we do Requirements base testing, all the functionality of a product is covered by requirements detailing exactly what it will and will not do. Once the development is underway we are brought in as testers and we write and execute tests that cover all the possibilities for each requirement. An example is a requirement that states: "Function A will accept a value between 5 and 20". This simple statement results in a ton of testing:

Value In range and Value out-of-range is tested:

• Test #1: value = 11
• Test #2: value = 45

Boundry conditions are tested:

• Test #3: value = 4
• Test #4: value = 5
• Test #5: value = 6
• Test #6: value = 19
• Test #7: value = 20
• Test #8: value = 21

Limits of Value Type is tested (assuming unsigned short):

• Test #9: value = 0
• Test #10: value = 255

So 10 tests from a simple statement. Once all tests are written we do "Structural Coverage" tests, we instrument the code and run all the tests we've written and make sure that every line of code is covered in a test.

Only once all that passes is that instrument considered ready to go, that takes a LOT of man-hours to do, even in situations where we can script the running of the tests (some situations are 'black box' testing where we just get the instrument itself and have to manually push the buttons to run any tests... takes a Long time!)

This level of testing is making me take a second look at some of the software I've written before starting this job and wincing... spending a few quick min playing around with a few different values really doesn't cut it for testing anymore in my eyes...

Re:Perfection Has a Price

[TheLink]

Is there really a thing as "perfect software"?

Often a bug is a matter of opinion/taste, and opinions/tastes change.

For instance some might say a program should stop if it discovers it suddenly can't log anymore. Others might say it should continue running and ignore the logging problem. And others might say it should continue running and try to log elsewhere. What would a perfect program do? I really don't know.

And the other reason why you don't get perfect software:

If shopping malls cost the same as now to design, but cost the equivalent of "make mall" and "coffee break" to build, people would be shopping in "buggy" shopping malls as soon as the blueprint is good enough to "compile".

Because the bosses won't bother waiting for stuff to be "perfect". They'd want to start collecting rent/$$$ ASAP.

And most shoppers and shopkeepers won't care either - as long as nobody gets killed or maimed, just stick a few warning signs over stuff that isn't working properly yet.

Re:Perfection Has a Price

[aztracker1]

I feel your pain. I've come across about 3 major instances where race conditions were a problem under load in a heavy traffic website. I'm now on my third web application refactoring. The first, I cut minutes out of a reporting process, and changed the way data was pulled in from various dbms resources. The second I did some changes to the way data settings were looked up and persisted. At first glance it wasn't so bad, because each setting call was taking only 0.11 seconds. The problem was there were about 50 calls on each page request.. that added up to a major page that took about 8 seconds to load, and was spiking the cpu's on the farm to 90% under our heaviest load. After the changes, the page load is well under 2 seconds, and cpu load under heaviest user load is under 32%.

I don't think a lot of people understand where resources get used, and how they get used, nearly as much as they should. Let alone security risks. I see in enterprise web applications people will use static resources in their classes not understanding how to use them safely. Often even *if* they should use them.

Re:Perfection Has a Price

[Chabil Ha']

The main reason why your DVD never required updates is because Blu-Ray is a trainwreck of a spec that is still in heavy flux, while DVD was finalized and stable by the time players started shipping.

It didn't require updates because in 1995 only 0.4% of the world's population had access to the Internet, and the spec didn't allow for such things as 'upgrades'. It hadn't even entered their minds that people would actively crack the thing. Fast forward ten years later (digitally mind you, we had passed up analog tapes!) and one of the requirements was to be able to update the code used to decrypt the disc in order to combat piracy. The fact that many updates to the specification have already been made is only an ancillary to this motivation.

Re:Perfection Has a Price

[tsm_sf]

Software is bad because it's not designed by MBAs, or even marketing departments. It's designed by engineers and developers with no practical experience.

(( I agree with your statement as well, btw ))

Re:Perfection Has a Price

[TaoPhoenix]

We might be seriously seeing the Ultimate Slashdot Car Analogy. My library informs me that the auto industry struggled with exactly this 30 years ago. Spurred by the Japanese that time, someone noticed that while the cost profile shifts, it really wasn't all that bad making quiet quality improvements across the line.

Yes, we have some fun little beta tech fragments in the works, but the big engines of Office and Browsers are pretty solid, and the OS market is going to hit the comparable maturity in another 5ish years.

With nothing earth shattering available, someone is gonna get 100 OldTimers into a big building for a month and decide to razor down the cruft of existing apps to sell the next iteration on speed improvements.

Re:Perfection Has a Price

[plover]

Yes, on the whole applications have become more stable, while growing an order of magnitude more complex. But TFA is not about stability as much as it is about security -- people leaving inadvertent holes in software that a hacker can exploit. You can have a perfectly functioning program, one that passes every test 100% of the time, but it may still have a SQL injection flaw or fail to validate input.

Re:Perfection Has a Price

[HeronBlademaster]

At the company I work for, I used to be on one of the software teams (now I'm in charge of IT infrastructure). When I was given a task, I would ask for details, and would receive vague comments that would only suffice for a vague implementation. Whenever I would ask for more details (knowing that the manager had them), my manager would say "just go implement what you can, then we'll change it later for what we need." No amount of begging would get more out of him.

One particular piece of the software was reimplemented a grand total of five times (twice by me, three times by someone else); all of the information that would have prevented the rewrites was in the manager's possession before the first implementation was written (as he later admitted).

Granted, the early versions were never released to the public (beyond a few development partners), but you can see how this mentality might cause poor code to end up in the "finished" product.

I finally transferred to IT because that manager refused to allow me, or anyone else, to clean up poor code as I went about implementing new features. Incidentally, he did put some code cleanup tasks on the to-do list, but he assigned them to a guy who didn't want to do them, despite both me and the manager's boss asking that those tasks be assigned to me.

Before anyone asks - no, I have no idea why I wanted to clean up old code. Maybe I'm insane. But I figure, if a guy has an itch to clean up old code, and you know it needs to be done, why on earth would you give it to a guy who doesn't want to do it instead of the guy who wants to do it (assuming equal skill sets)?

Re:Perfection Has a Price

[Opportunist]

Best practices? Following any sensible practice would already be a blessing!

Now, I know I'm jaded and cynical, but having to deal with people you get from temp agencies does that to you. It's the latest fad, and it arrived at "coding jobs". Watch your productivity plummet, code quality leaping off the cliff and your sanity following close behind.

First of all, what do you get from a temp agency? Hell, it's not like programmers have a really, really hard time finding a job around here. If they can code, they have a normal job. So what the hell do you think you get from a TA? Right. The sludge of the trade. The ones that cling to "doing something with computers" despite all odds and the fact that they simply can't.

The few gems that you might get, because they're young and need some sort of work experience, are gone before you got them up to par. And that time is spent mostly trying to find some better (read: permanent) job. You'd be amazed how often people get "sick" just before they jump ship and are miraculously hired by some other company.

The average turnover is 3 months. Now, I'm sure anyone who spent more than a week in programming knows what sort of productive work you get out of someone in his first three months (remember: You get either sludge or fresh meat, nobody with experience would willingly waste his time in a TA employment situation).

After a year or two you end up with code written by about 20-30 different people, every single of them caring more about their next lunch than the project, with 20-30 different coding styles, and near zero documentation.

I left the company, too. I still have a tiny bit of sanity that I treasure.

Re:Perfection Has a Price

[Opportunist]

Now, I don't think that code got less secure. I rather think there are simply a hell lot more ways a system's insecurities can be exploited, and the propagation of such malicious code is magnitudes faster.

Think back to the 80s. Whether you take an Amiga or a PC, both had horrible systems from a security point of view. It was trivial on either system to place code into ram so it could infect every disk inserted, every program executed, much more than it is today, with home users having systems that have at least nominal levels of security. Yet the propagation of malware was way slower. Because most people rebooted between using programs (they sometimes had to because the programs were not kind to the system and simply kicked it out of the ram so they could use more of it... and a program being able to do that just shows how secure the system was in the first place). Even if the malicious program could somehow survive as a TSR program, spreading from one machine to another required more often than not that a disc was carried over to the next machine and put into this other machine...

This is all a matter of the past today. Dozens of programs running at the same time, machines getting rebooted maybe once a day and spreading malware through the internet is a matter of seconds.

The same applies to databases. How many databases were actually "online" in the old days, with people having (potentially) access to them that had no business there? Most databases were inside a company's local network. Maybe accessable by other company outlets somewhere, but it was usually a quite well shielded network, not accessable by "mundane" means. The amount of hackers (or people who would like to pose as one) that could possibly access the database and inject something was way smaller.

So I doubt systems got more secure. They're just exposed to more insecure input these days.

Re:Perfection Has a Price

[ozphx]

I pull a 6-digit salary (even when you convert our shitty AUD to USD ;)) - and its worse.

If anything I am more rushed than I was when I was a junior dev. Managers flat out don't care what you are doing when you are on 50k. You could whack off all day if you like.

As soon as you start earning more than your manager it all goes to shit. Possibly an ego thing - I don't really care about the reasons - but every place I've had managers over my shoulder monitoring my times, trying to cut schedules, etc. Where I'm consulting at the moment is a good case in point - every 3 weeks theres a new "final deadline". I think its to try and motivate us to finish up (theres a good several months of work left) - all they are getting is bandaid fixes.

Re:Perfection Has a Price

[nonsequitor]

The horror you described is for Level C certification. Level B has the requirement of testing each implied else. Meaning for every standalone if statement, you have one test that makes sure it does the right thing when the conditions are true, and a second test which makes sure that thing does not happen when the conditions are not true.

Level A certification requires Modified Condition Decision Coverage (MCDC). The means a test case must exist for every possible logical combination of terms for every conditional expression. These are extensions of the structural coverage requirements for Level C which is only statement coverage, every line has to be exercised at least once by a test case.

This sort of exhaustive testing is done because of a very different operating environment from desktop software. Desktops don't have single bit events. A single bit event occurs when a gamma ray strikes a 0 in RAM and turns it into a 1. As a result some places forbid the use of pointers in their code, since a single bit event could have catastrophic consequences. Every function has to be tested to be able to best withstand random garbage input and fail gracefully since you cannot rely on RAM storing values correctly.

However, the medical industry has similar guidelines for things like X-Ray machines and ventilators, which I believe have a very similar set of best practices codified into a certification process. Depending on the certification level, the cost of testing varies. It is VERY expensive to develop software this way since Test Driven Development doesn't cut it. For these industries the testers need to be independent of the developers meaning developers don't write their own unit tests, which in turn leads to more cumbersome software processes.

I still work in safety critical embedded software, but am eternally thankful I no longer have to worry about certification with a mandated software process. I now write software for boats instead of planes and have started doing model based controls design, test driven development, and in general have tailored the software process to best suit the company. The goal is to produce better software faster, in order to respond to changing business needs, not to fill out checklists to pass audits. The FAA process is the most reliable way to the highest quality software possible; it is not efficient, cheap, or even modern.

Re:Perfection Has a Price

[Grishnakh]

The hardware designers were under the same sorts of pressures, if not more so, than the software guys and I saw many bugs that would end up in the shipping silicon. The general attitude was always "oh! a bug: well the software guys will just have to work around it."

Most of my work involves software workarounds for hardware bugs. The reason is a little different, though. My company, in its infinite wisdom, completed the very first revision of this particular chip, did a preliminary test and the core seemed to work (it's a system-on-chip), so they laid off the hardware design team! Then they started finding the many bugs in the hardware, and ran around looking for another design team to fix the bugs, but it took several years, and they had to ship millions of the buggy chips to customers, which are deployed in the field.

Re:Perfection Has a Price

[jellomizer]

Very true I have worked on maintaining these old systems; your statement is quite true.
The cost to replace legacy apps are massive, like millions of dollars.
Why...
Analysis of all the features of the old app: Before you buy an app you should know what your current one does... A form that fits on a 80x25 display could be doing a bunch of complex calculations.
Analysis on the current usages of the program: Some parts are used all the time others code hasn't been touched in decades and will never be used again, some havent been used in a while but is needed once in a blue moon. You don't want to code unneeded code but you need to make sure when they upgrade they are not missing anything.
Analysis of the data: How is the data layout how can you migrate the data from one system to an other. Are those values in easy to read text numbers. Or in a binary form with one number next to each other, with an endianness that is unknown, and with data types of undocumented size. (1 Bit bool or an 8 bit bool?)
Then after all that Analysis there is the coding of the new app.
Importing all the data from the old app to the new one, and insure data integrity.
Finding a group of people willing to test the new software.
Training people to use the new software.
Getting people to STOP using the old software (and there will be resistance on this, as there be that one stupid feature that was nicer on the old system vs the new one)

This is a HUGE Process and I am not factoring bugs. So cost of upgrading is defiantly a big problem.

Re:Perfection Has a Price

[mollymoo]

First of all, what do you get from a temp agency? Hell, it's not like programmers have a really, really hard time finding a job around here. If they can code, they have a normal job. So what the hell do you think you get from a TA? Right. The sludge of the trade. The ones that cling to "doing something with computers" despite all odds and the fact that they simply can't.

I used to do contract work (if that's what you mean by "temp agency"), not because I couldn't get a permanent job - I was offered a permanent position at every single place I did contract work - but because being paid twice as much as you'd get in a permanent position, getting to work on a variety of projects with a variety of companies and being able to take several months a year off is just a nicer way of working.

Re:Perfection Has a Price

[Eskarel]

The software of yester-year ran largely on single threaded operating systems, didn't have to interact with the internet or defend against attacks originating from it, had to manage miniscule feature and data sets, and was still buggy.

There was no magical era of bug free computing, there was an era when systems were orders of magnitude less complex, where about a tenth of the software was running, and where features which most people depend on didn't exist. The software was still buggy and most of it was so full of security holes that it may as well have been a sieve.

Complex systems have more bugs, modern systems are more complex. The workflow of your data entry people might be fast, and perfectly adequate, but what does it cost to support those 386's, what does it cost to train someone to actually use word perfect 4.2(the old dos verisons of word perfect had a pretty high learning curve). Data entry folks are basically doing monkey work, but training them up on an older system could take days.

a book never written

[jollyreaper]

Fred Brooks's 'The Mythical Man-Month',

I read that as "the Mythical Man-Moth." I bet that would be a great book.

Re:a book never written

[LMacG]

Not to mention cartoon and live-action series.

When I was breaking in

[PingXao]

In the early '80s there were no "older" programmers unless you were talking mainframe data processing. On microprocessor CPU systems the average age was low, as I recall. Back then we didn't blame poor software on "youthful programmers". We blamed it on idiots who didn't know what they were doing. I think it's safe to say that much hasn't changed.

Re:When I was breaking in

[Skapare]

This is true with any group. There are geniuses and idiots in all groups. The problems exist because once the supply of geniuses have been exhausted, businesses tap into the idiots. And this is made worse when employers want to limit pay across the board based on what the idiots were accepting. Now they are going overseas to tap into cheaper geniuses, which are now running out, and in the mean time, lots of local geniuses have moved on to some other career path because they didn't want to live at the economic level of an idiot.

Re:When I was breaking in

[77Punker]

After working 3 months at my first programming job, the other two developers quit which left just me. I felt inadequate until I started interviewing other programmers to fill in the gap. Apparently lead developers with 10 years of experience can't solve the simplest programming problems, explain how databases work, or explain OOP. I'm convinced that most software sucks because most people writing it have no idea what they're doing and shouldn't be allowed to touch a computer. I'm currently in my 5th month at the same job and we've got someone good who will start soon, but it took a long time to find even one competent developer.

Re:When I was breaking in

[Opportunist]

Quite dead on. But the difference is that today, with all the RAD tools around, you have a fair lot of "programmers" who don't even know what they're doing and get away with it. They got some course, maybe at their school (and that's already the better ones of the lot), maybe as some sort of an attempt to get them back onto a job from their unemployment services, and of course tha intarwebz is where da money is, so everyone and their dog learned how to write a few lines in VB (or C#, the difference for this kind of code molesters is insignificant) and now they're let loose on the market.

Then you get hired by some company that signed up those "impressively cheap programmers, see, programmers needn't be horribly expensive" when the project goes south because deadlines lead to dead ends but no product, and you're greeted with code that makes you just yell out WTF? You got conditional branches that do exactly the same in every branch. You get loops that do nothing for 90% of the loop time and when asked you just get a blank stare and a "well, how do you think we could count from 80 to 90 besides counting from 0 to 90 and 'continue;' for 0-70?", because that's how they learned it and they never for a nanosecond pondered just WHAT those numbers in the 'for' block meant. And so on.

Granted, those are the extreme examples of people who learned programming like a poem. By heart. They have a hammer as a tool, so every problem has to be turned into a nail. But you'd be amazed at the blank stares and "what do I need that for?" when you ask some of those "programmers" about hash tables (include snide joke about Lady Mary Jane here...) or Big-O notation. And we're talking people who are supposedly able to write a database application here.

This is the problem here. It's not youthful programmers. It's simply people who know a minimum about programming and managed to trick some HR goon into believing they could actually do it.

Re:When I was breaking in

[fishbowl]

>There are geniuses and idiots in all groups.

Most of both groups are within two standard deviations of a norm. Your idiots are probably smarter than you think and your geniuses are probably not as smart as you'd like to believe.

Re:When I was breaking in

[frosty_tsm]

I'm convinced that most people presenting themselves as lead developers in interviews are far from it. There's a reason why thedailywtf.com has a "Tales from the Interview" section.

Re:When I was breaking in

[77Punker]

"Write a function to sum all the numbers from 0 to 100"

Every code question I ask is about that simple. The solutions I get to EASY questions are almost always really stupid, incorrect, or get answered with "I don't know how to do that"

Re:When I was breaking in

[wgaryhas]

That should have been:

int SumNumbers(int x){
return x * (x + 1) / 2;
}

Re:When I was breaking in

[Surt]

Do you care whether they write a loop or return (n*n+1)/2? (where n=100 in this case?)

(curious whether you are looking for the person who knows the clever solution, or the guy who can write a basic loop).

Re:When I was breaking in

[77Punker]

That's a very good answer, but any good answer would be one that didn't involve declaring an array of all the number between 0 and 100 and then iterating over the array. Yes, that's a typical solution.

Re:When I was breaking in

[77Punker]

I'm just looking for a guy who can solve the problem at all and explain how he solved it. You know, just trace it by hand and tell me why he decided to approach the problem the way he did. I tend to go for "no wrong answer" types of questions. I count this question in that group since the only wrong answers are "I can't do it" or the interviewee writes an obviously wrong solution and can't trace his own code well enough to know that it's wrong even after having it explained to him.

Re:When I was breaking in

[Opportunist]

What you describe is a wannabe-genius. A showoff. Not a genius.

The code that makes me mutter "that's pure genius" is usually not the kind of code you can't understand or is winded, twisted and a worthy entry for the obfuscated C-Code contest. It's usually code that is brilliantly simple yet very functional, fast and easy to understand.

Genius code isn't the code that takes a year to read and a decade to understand. It's the code that makes you wonder why you didn't come up with it because it, well, looks so simple that it's pure genius. A good example of "pure genius" code is the square root function that relies on the quirks of the IEEE754 format. Works only on 32bit little endian, but there it works and is fast.

Pure genius code doesn't mean it has to be complicated, quite the opposite. "Pure genius" is what it mostly attributed to finding new ways to do something in a better way. Not making it overly complicated to look like you did something great.

Re:When I was breaking in

[Chabo]

Well that's unfair...

First off, I started off as self-taught, then moved on to get a B.S. in Computer Science (is there a school that offers a B.A.?). Would you fault me for getting an education?

Second, I started teaching myself at age 18. You'd reject me simply because I started six years later?

Just because it's harder to learn something later in life doesn't mean you can't learn it, whether it be French, Italian, or C.

Re:When I was breaking in

[Gordonjcp]

Sometimes the right answer is "do it the stupid way and buy a faster computer"...

Look at it this way, if I cost £50 an hour, and a new fast-as-blazes CPU costs £100, then will two hours of my work make as much improvement as putting a faster chip in? If not, I can go and do some useful coding and you get new shiny to play with.

Re:When I was breaking in

[Anonymous Coward]

int f() { return 5050; } ?

Re:When I was breaking in

[CorporateSuit]

"Write a function to sum all the numbers from 0 to 100"

easy

dim a a = 1+2+3+4+5+6+7+8+9+10+11+12+13+14+15+16+17+18+19+20+21+22+23+24+25;
a = a+26+27+28+29+30+31+32+33+34+35+36+37+38+39+40+41+42+43+44+45+46+47+48+49+50;
a = a+51+52+53+54+55+56+57+58+59+60+61+62+63+64+65+66+67+68+69+70+71+72+73+74+75;
a = a+76+77+78+79+80+81+82+83+84+85+86+87+88+89+90+91+92+93+94+95+96+97+98+99+100; print a;

Re:When I was breaking in

[againjj]

"Write a function to return the sum of all the numbers from 0 to 100" is even more fun. Because many people, while getting the "return n(n+1)/2" solution, miss "return 5050".

Re:When I was breaking in

[clone53421]

All loops are coded as GOTOs in assembly, you insensitive clod!

Re:When I was breaking in

[77Punker]

I'm beginning to realize that I chose a terrible sample question to post here.

Re:When I was breaking in

[Anonymous Coward]

ah ah you failed. You forgot to add the zero. Idiot.

Re:When I was breaking in

[Surt]

Yes, I mean (n*(n+1))/2.
Tens of AC's are all trying to correct me, and I assume none can see each other's posts.

Re:When I was breaking in

[greg1104]

xor ax,ax
xor bx,bx ;_loop
inc bx
add ax,bx
cmp bx,64 ;64 hex = 100 decimal
jb 0104 ;_loop

You damn wasteful kids, just using a CMP like it doesn't cost anything. For the love God, can't you just load bx with the upper boundary and use dec/jnz in that loop? Been faster since day one, and both instructions execute in one cycle on the Pentium and later. Makes it easier to turn into a subroutine, too--make setting bx the first instruction, then you just can put the number you want to sum up to into bx and jump into the second instruction.

Don't even get me started on how you just throw in a div there like they're giving them away for free at the store or something.

Waterfall

[Brandybuck]

The waterfall method is still the best development model. Uou have to analyze, then plan, then code, then test, then maintain. The steps need to be in order and you can't skip any of them. Unfortunately waterfall doesn't fit into the real world of software development because you can't freeze your requirements for so long a time. But cyclic models are a good second place, because they are essentially iterated waterfall models. When you boil all the trendy stuff out of Agile, you're basically left with a generic iterated waterfall, which is why it works. The trendy crap is just so you can sell the idea to management.

Re:Waterfall

[Timothy Brownawell]

The waterfall method is still the best development model. [...] Unfortunately waterfall doesn't fit into the real world

WTF? Not working in the real world makes it a crap model.

When you boil all the trendy stuff out of Agile, you're basically left with a generic iterated waterfall, which is[...]

...not a waterfall.

Re:Waterfall

[radish]

The waterfall is broken, seriously. I'm paraphrasing from an excellent talk I attended a while back, but here goes.

For a typical waterfall you're doing roughly these steps: Requirements Analysis, Design, Implementation, Testing, Maintenance. So let's start at the beginning...requirements. So off you go notebook in hand to get some requirements. When are you done? How do you know you got them all? Hint: you will never have them all, and they will keep changing. But you have to stop at some point so you can move onto design, so when do we stop? Typically it's when we get to the end of the week/month/year allocated on the project plan. Awesome. Maybe we've got 50% of the reqs done, maybe not. It'll be a long time until we find out for sure...

Next up - Design! Woot, this bit is fun. So we crank up Rose or whatever and get to work. But when do we stop? Well again, that's tough. Because I don't know about you but I can design forever, getting better and better, more and more modular, more and more generic, until the whole thing has flipped itself inside out. So we stop when it's "good enough" - according to who? Or more likely, it's now 1 week to delivery and no-one's written any code yet so we better stop designing!

Implementation time. Well at least this time we know when we're done! We're up against it time wise though, because we did such a good job on Reqs & Design. Let's pull some all nighters and get stuff churned out pronto, who cares how good it is, no time for that now. That lovely, expensive design gets pushed aside.

No time to test...gotta hit the release date.

Sure this isn't the waterfall model as published in the text books, but it's how it works (fails) in real life. And the text books specifically don't say how to fix the problems inherent in the first two stages. What to do instead? Small, incremental feature based development. Gather requirements, assign costs to them, ask the sponsors to pick a few, implement & test the chosen features, repeat until out of time or money.

Re:Waterfall

[Brandybuck]

You can't create quality software without planning before coding. Ditto for not testing after coding. This isn't rocket science, yet too many "professionals" think all they need to do is code.

The waterfall model isn't a management process, it's basic common sense. It's not about attending meetings and getting signatures, it's about knowing what to code before you code it, then verifying that that is what you coded. The classic waterfall took too long because you had to plan Z before you started coding A, but with an iterated waterfall (which is still a waterfall, duh) you only need to plan A before you code A.

Re:Waterfall

[radish]

The phase is over when the stakeholders agree and sign off on the requirements.

Hahahahahahahahahahahahahaha...breathe....Hahahahahahahahahahahahahaha!!

I have no idea where you're coming from (government?) but in _my_ real world stakeholders change requirements all the time. Like daily. And saying "no you can't have this because you signed off on something last week" is a great way to dramatically shorten your career. The real world that my stakeholders live in changes all the time, so their requirements change all the time, so you have to be able to react.

According to whom? According to the stakeholders who mutually agree on the design

I've had stakeholders who understand requirements, and even some who understand specs. Never one who understood a design in any useful way. This ties into my first point though, without knowing the requirements are right how can you validate a design?

Says you who very clearly have not been on a large waterfall project with people who know what they're doing?

As far as I'm concerned, anyone who tries to do anything useful with a rigid waterfall doesn't know what they're doing by definition, so I guess not :)

Re:Waterfall

[jstott]

Hint: you will never have them all, and they will keep changing. But you have to stop at some point so you can move onto design, so when do we stop?

This is a solved problem in engineering. You write the contract so that if the customer [internal or external] changes the requirements it carries a financial penalty and allows you [the developer] to push back the release date. On the other side, if you discover a problem after you've shifted out of the design phase, then you are SOL and your company eats any associated cost. Contracts like these are motivation for both of you to get it right the first time.

-JS

Re:Waterfall

[recharged95]

Obvious you were working in the consumer-commercial side of the waterfall practice.

.

On the DoD Space Systems side, we had stuff like:

• A Requirements Traceability Matrix
• PDR: Preliminary Design Review with the Customer
• CDR: Critical Design Review with the Customer
• IOC: Initial Op capability
• FAT: Preliminary Acceptance Testing
• FOC: Final Op capability
• FAT: Final Acceptance Testing

And never had a launch delay due to writing software last minute. And as long as the h/w flew fine, we had birds that ran in space for years. Years.

.

The Waterfall model "failed" because of a. it got too expensive due to competition (mainly smaller firms that cut corners resulting in more failures); it wasn't "faster, cheaper". And b. all the OO and Agile guys were better salesmen.

.

I bet DeMarco is enjoying the responses.

Re:Waterfall

[turbidostato]

"One of the main commandments in agile development is that the requirements cannot change."

Maybe you read the wrong book. In my book requirements don't change *within a sprint*.

"Also, you are supposed to plan and code for a feature to be shippable when it is done."

And you plan it so it can be done within a sprint. So before the sprint you get the client to accept feature "A" (you will show him a list of features so is he, not you, then one that sorts them). After the sprint you offer your client the promised feature "A" (and ideally, you bill your client for the feature).

"But in the office, requirements keep changing as much as they did with waterfall"

And that's exactly why agile exists: because it recognices requirements *will* change. Since they'll change why expend a lot of time on producing a very long list of intermingled requirements, then a long (and costly) development cycle, then after a long time and expenditure show the whole list of asked for requirements done only to find that 25% of the original requirments were not indeed required because the client didn't properly understood them, 25% of them were wanted no more because on such a long time situation changed and 50% new requirements arosen because, again, the long time awaited?

You better agree on a sketchy *cheap* and *fast* gross list of requirements, short them as for the client's percieved value and start producing functionally complete code ASAP. The client is glad because gets to see the advance on the project; your company is glad, because the client is glad, it can suck money faster from it, and due to the constant exchange of ideas is easy to engage on new projects. When agile is possible is a win-win proposition.

"This messes up the schedule."

That clearly means you are not doing agile at all. If you have a long run schedule, you are not doing agile. If you are doing agile, then there's no long run schedule to mess up with. You can have a starting grosstimate at best and the continous feedback with your client will make him easly understand and accept any delays because it will have strong feedback about being him not you the main responsible for the slippage.

"And features only get to the "done enough for now" stage which means that they are good enough to sort of work and build on, but they are not complete or shippable."

Again, you are clearly not doing "agile". Of course bugs can arise but if you do it the proper way there's no way for a feature to be so long running and complex that they can slip too much (they should fit within a single sprint), and since you are doing tests->autodoc->prototype->implementation there's no chance of shipping "done enough" features as is the case when you develop functionality for the easy cases->corner cases->documentation (this too common way is the one that always gets cut up at step one "easy cases" since then it "seems" to work; the first way only seems to work when it indeed works).

"not everyone is going to say no to the boss"

Again, "agile" works on the assumption that in fact nobody can say "no" to the boss producing a framework where you won't need to say "no" to the boss.

"The seminar example was a web app [...] In my office, the software is much more complex"

This seems to be the only sensical assertion you made to this point. Yes it's true: agile is not the solution for every project. While proper practice and experience can help you to find a way to split more and more problems so it can become "agile-tractable" there will definetly be situations where due to the inherent complexity and internal coupling of the problem "agile" is not the way to go. If the seminar guy explicitly told or make the impression that "agile" is a silver bullet for any software project, then that just were a "snake oil vendor seminar", not to say that there are no little numbers of them.

Damn Lazy Programmers!

[pete-wilko]

Yeah, those good for nothing programmers cramming in features all over the place and not ad-hearing to time honored development practices like waterfall!

And requirement changes? WTF are those? Using waterfall you specify your requirements at the beginning, and these are then set in stone, IN STONE! Nothing will ever change in 6 -12 months.

It's not like they're working 60-80 hour weeks, been forced to implement features, having new requirements added and not being listened to! That would be like marketing driving engineering! Insanity!

As an aside - why is he dragging OO into this? Pretty sure you can use waterfall with OO - you even get pretty diagrams

Re:Modus Operandi

[John Hasler]

Except that what you actually do is promise to paint it red even though you know that you do not have and cannot get any red paint. Then you deliver it green and try to tell the customer he is colorblind and besides the next model really will be red.

Extensible Framework

[should_be_linear]

Most horrible projects I've seen were "extensible frameworks" that can do just about anything with appropriate extensions (plugins or whatever). But currently, without any existing extensions, it is bloated pile of crap. Also, there is nobody in sight willing to make one extension for it (except sample, done by author himself, on how to easily create extension).

Those who fail to learn the lessons of history

[overshoot]

... are destined for greatness, because their bullshit is not burdened by reality.

I've heard from several ex-Softies that the company inculates its recruits with a serious dose of übermensch mentality: "those rumors about history and 'best practices' are for lesser beings who don't have the talent that we require of our programmers." "We don't need no steenking documentation," in witness whereof their network wireline protocols had to be reverse-engineered from code by what Brad Smith called 300 of their best people working for half a year.

However, I'll note that they were right: anyone who wants to say that they did it wrong should prove it by making more money.

Re:Those who fail to learn the lessons of history

[jd]

The Brinks-Matt Robbery, in which thieves brazenly stole truck-loads of gold bullion, and the two gigantic heists at Heathrow Airport, were all 100% correct - from the perspective of the thieves. And given that next to nothing from any of these three gigantic thefts was ever recovered, I think most people would agree that the implementation was damn-near flawless. As economic models go, though, they suck. They are so utterly defective in any context other than the very narrow one for which they were designed, and even then only from the very narrow, short-sighted perspective of the thief, that they have zero reusability.

Just because something makes a person rich doesn't mean that something is worth copying, would even work for someone else, or could ever be to the advantage of anyone but the original person. It is equally flawed logic to assume that "correct" behaviour could be equally profitable to one-off flawed behaviour. Flaws can make some damn good income, short-term, even if they must degenerate in the long-term.

You will find socially-irresponsible companies have the fastest-rising stocks in the stock-market. Well, until they crash and burn. Cults are the most profitable of all societies. Until they self-destruct. Extremists have the ultimate in work ethics, until they die from the strain. In the Dark Ages, you could build far and away more wooden palisades than you could build stone fortifications. Well, until the builders all died horribly from Greek Fire (early napalm).

And nobody will ever make more money than Microsoft by writing responsibly and sensibly. In the long-term, Microsoft's code is unsupportable and its methodology is unscalable. Vista showed that they are approaching (but are not yet) at the limits. They might not reach the limits for another 10-20 years, but they will someday reach them in such a way that they have blocked off all avenues of escape.

A software company producing a good, solid product won't necessarily make much money in any given year, now or ever, but if the design is truly as good as all that, it will sell longer and require less maintenance. Instead of a 50-year lifespan for the company, you might well be looking at a 250-year lifespan or more. (There are companies today that are much older than that.) I'd say that such a company can legitimately say that it is doing things right, because they meet the ultimate requirement: will you adapt or will you die?

It is also possible that a company that lasts that long will actually end up making more money than Microsoft (as a sum of gross revenue, not net), even if they never matched the sorts of sales Microsoft have achieved and even if they never end up as rich because they spent that resource on improving their product over and above improving their bank balance. Failure to invest is not an admirable quality in my book, nor is cutting corners. In terms of social evolution and being able to adapt to new environmental pressures, Microsoft is an evolutionary dead-end. Massively successful in the short-term, but incapable of survival on a meaningful timeframe.

Users are to blame

[Chris_Jefferson]

The sad truth is, given the choice between a well-written, stable and fast application with a tiny set of features and a giant slow buggy program with every feature under the sun, too many users choose the second.

If people refused to use and pay for buggy applications, they would either get fixed or die off.

Re:Users are to blame

[curunir]

The even sadder truth is that when faced with the choice of the two apps you describe and a third buggy application with a tiny set of features, users will choose the most visually appealing app, regardless of lack of features or the app being buggy.

The under-the-covers stuff is important, but finding a good designer to make it pretty is the single most important thing you can do to make people choose your product. If it's pretty, people will put up with a lot of hassle and give you the time necessary to make it work reliably and have all the necessary features.

This is clearly Microsoft's fault!

[sheldon]

Oh great a rant by someone who knows nothing, providing no insight into a problem.

Must be an Op-Ed from a media pundit.

And they wonder why blogs are replacing them?

cheap shot

[Anonymous Coward]

I work at Microsoft. We use agile development and almost everybody I know here has read the Mythical Man Month. Get your facts straight before taking cheap shots in story submissions. Thanks.

Waterfall versus OOD

[BarryNorton]

A waterfall process and object-oriented design and programming are orthogonal issues. The summary, at least, is nonsense.

Complete BS?

[DoofusOfDeath]

For the life I me, I can't figure out what the choice of {waterfall vs. cyclic} has to do with {writing code that checks for error return codes vs. not}.

Waterfall vs. cyclic development is mostly about how you discover requirements, including what features you want to include. It also lets you pipeline the writing of software tests, rather than waiting until the end and doing it big-bang approach. Whether or not you're sloppy about checking return codes, etc., is a completely separate issue.

Despite the author's protests to the contrary, he really is mostly complaining incoherently about the way whipper-snappers approach software development these days.

Microsoft?

[dedazo]

Most of the teams I've had contact with inside the tools group at MS (in the last four years or so) use SCRUM.

I don't know how widespread that is in other divisions (say the MSN/Live folks or the Microsoft.com teams) but that clever comment in the submission is nothing more than an ignorant cheap shot.

Don't be so twitterish and make up crap about Microsoft. Get your facts straight or you just come across as an idiot.

Waterfall was never valid

[wezeldog]

Dr. Royce used it as an example of a methodology that doesn't work, but what he described was easy to understand so it gained traction with management types. It's like the joke where the guy says he's looking for a lost quarter under a streetlamp because the light is better than where he lost it.
http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf
I think his suggestion was to 'build it twice' via prototyping to discover what was missed in the requirements gathering and design phases.

Typical: blame the process

[SparkleMotion88]

It is very common for people to blame all of the problems of software engineering on some particular methodology. We've been shifting blame from one method to another for decades, and the result is that we just get new processes that aren't necessarily better than the old ones.

The fact is that software development is very difficult. I think there are several reasons why it is more difficult to develop robust software now than it was 20 years ago. Some of these reasons are:

• The customer expects more from software: more features, flashier interfaces, bigger datasets -- all of these things make the software much more complicated. The mainframe systems of a few decades ago can't even compare to the level of complexity of today's systems (ok, maybe OS/360).
• There is just more software out there. So any new software that we create is not only supposed to do its job, but also interconnect with various other systems, using the standards and interfaces of various governing bodies. This effect has been growing unchecked for years, but we are starting to counter it with things like SOA, etc.
• The software engineering talent has been diluted. There, I said it. The programmers of 20-30 years ago were, on average, better than the programmers of today. The reason is we have more need for software developers, so more mediocre developers slip in there. Aggravating this issue is the fact that the skilled folks, those who develop successful architectures and programming languages, don't tend to consider the ability level of the people who will be working with the systems they develop. This leads to chronic "cargo cult programming" (i.e. code, test, repeat) in many organizations.
• Software has become a part of business in nearly every industry. This means that people who make high-level decisions about software don't necessarily know anything about software. In the old days, only nerds at IBM, Cray, EDS, etc got involved with software development. These days, the VP of technology at Kleenex (who has an MBA) will decide how the new inventory tracking software will be built.

I'm sure there are more causes and other folks will chime in.

It's economics, too...

[gillbates]

As long as:

• Consumers buy software based on flashy graphics and bullet lists of features, without regard for quality...
• Companies insist on paying the lowest wages possible to programmers...
• Programmers are rewarded for shipping code, rather than its quality...

You will have buggy, insecure software.

Fast. Cheap. Good. Pick any two.

The market has spoken, and said that they would rather have the familiar and flashy than secure and stable. Microsoft fills this niche. There are other niches, such as the Stable and Secure Computer market, and they're owned by the mainframe and UNIX vendors. But these aren't as visible as the PC market, because they need not advertise as much; their reputation precedes them. But they are just as important, if not moreso, than the consumer market.

There (was) is no such thing as Waterfall Method

[hopopee]

I find it more than a little amusing that summary mentions waterfall method being a time honed, excellent example of how all software should be made. Here's the history of the none existent waterfall method has to say about it.. Waterfall Method does not exist!

grumpy old coder

[girlintraining]

It's just another "I have 40 years of experience doing X... Damn kids these days. Get off my lawn." Hey, here's something to chew on -- I bet he screwed up his pointers and data structures just as much when he was at the same experience level. Move along, slashdot, nothing to see here. I will never understand the compulsion to compare people with five years experience to those with twenty and then try to use age as the relevant factor. Age is a number... Unless you're over the age of 65, or under the age of about 14, your experience level is going to mean more in any industry. This isn't about new technology versus old, or people knowing their history, or blah blah blah -- it's all frosting on the poison cake of age discrimination.

P.S. Old man -- reading a book won't make you an expert. Doubly so for programming books. I'd have thought you'd know that by now. Why not get off your high horse and side-saddle with the younger generation and try to impart some of that knowledge with a little face time instead?

Re:grumpy old coder

[benj_e]

Meh, it goes both ways. How many younger coders feel they are god's gift to the industry?

Personally, I welcome anyone who wants to be a programmer. Show me you want to learn, and I will mentor you. I will also listen to your ideas and will likely learn something from your fresh insight.

But show me you are an asshat, and I'll treat you accordingly.

Re:Its all true

[Cornflake917]

I think refusing to hire someone solely because of their age is naive. Is there some magical event at the age of 30 that bestows knowledge of linkers to the aging programmer? Give me a break. You are making bad assumptions. Your first bad assumption is that just because of your anecdotal experience dealing with one individual, that all schools no longer teach anything about linkers. Your second bad assumption is that even if that was true, no programmer would learn that information on their own, as if no one is generally interested in learning comp sci any more outside of the classroom.

Re:Its all true

[yttrstein]

You must be under 30.

It's the fault of the Internet.

[rickb928]

Several posters have alluded to this, but I blame the Internet. Just follow me here:

- Back in the 'Old Days', productivty software at the time was dominated by WordPerfect, VisiCalc and then 1-2-3, and what else? MS-DOS as the operating system. Everything shipped on diskettes. There was no Internet.

- Fixing a major bug in WordPerfect required shipping diskettes to users, usually under 'warranty'. Expensive, time-consuming, and fraught with uncertainty.

- Fixing bugs in MS-DOS really wasn't done. It was a minor release. Again, diskettes everywhere, and more costs.

- Patch systems were important. Holy wars erupted on development teams over conflicting patch methods, etc. Breaking someone else's code was punished. Features that weren't ready either waited for the next release or cost someone their job.

Today, patches can be delivered 'automatically'. It take how long, seconds, to patch something minor? Internet access is assumed. the 'ship-it-now' mentality is aided by this 'ease' of patching.

If it weren't for Internet distribution, we would see real quality control. It would be a matter of financial survival.

No, Internet distribution is not free. But it's both cheaper, I suspect, than shipping any media, and also less frustrating to a user than waiting at least overnight (more likely 5-7 days) for shipment.

And it leads to the second distortion - Bug fixes as superior service. The BIG LIE.

It is not superior service to post a patch overnight. It is not superior service to respond immediately to an exploit. It is a lie. Having to respond to another buffer overflow exploit after years (YEARS) of this is incompetence, either incompetence by design or incompetence of execution. This afflicts operating systems, software, utilities, nothing is innocent of this.

The next time you marvel just a little bit when Windows or Ubuntu tells you that you were automatically updated, or that udpates are awaiting your mere consent to be installed, remember - they just admitted your software was imperfect, and are asking you to take time to let their process, designed with INEVITABLE errors expected, perform its task and fix what should never have been broken to begin with.

ps- I love Ubuntu. I cut it some slack 'cause you get what you pay for, and many who work on Ubuntu are unpaid, and any rapid response to problems is above expectations. Microsoft, Symantec, Adobe, Red Hat, to name a few, are not in that business. They purport to actually *sell* their products, and make claims to make money. When they fail to deliver excellent products, the lie of superior service is still a lie.

Just the voice of one who remembers when it was different.

pps - EULAs tell it all. I wish I had an alternative. Oh, wait, I do... envermind, lemme get that Ubuntu DVD back out here and... Except at work...

Re:It's the fault of the Internet.

[PitaBred]

Don't forget that Ubuntu patches EVERYTHING on your system, versus the monthly Microsoft patches which are just the core OS and possibly Office. And since most of the developers of those other programs that Ubuntu patches are not on Ubuntu's payroll... well, they don't really have much control on whether something is patched or not. They'll patch some egregious things themselves and send them upstream, but... Anyway, you probably know all that. Just wanted to make sure it was clear to everyone else who may not use it ;)

error 26: 15 !=25

[cylcyl]

The article link says top 25 errors....

Question

[coryking]

When you were working on those punch cards, using your green screen console (kids these days with color monitors and mice), what were you doing?

Did you ever transcode video and then upload it to some website called Youtube on "the internet"? Did you then play it back in a "web browser" that reads a document format that your grandma could probably learn? Did your mainframe even have "ethernet"? Or is that some modern fad that us kids use but will probably pass and we'll all go back to "real" computers with punch cards.

Did you ever have to contend with botnets, spyware or any of that? And dont say "if we used The Right Way Like When I Was Your Age, we wouldn't have those things because software would be Designed Properly". because if we used "The Right Way" like you, software would take so long to develop and cost so much that we wouldn't even have the fancy systems that even make malware possible.

Old timers crack me up. Ones that are skeptical of object oriented programming. Ones who think you can waterfall a huge project. I'd like just one of them to run a startup and waterfall Version 1.0 of their web-app (which, they wouldn't because the web is a fad, unlike their punch cards).

Sorry to be harsh, but get with the times. Computing these days is vastly more complex then back in the "good old days". Your 386 couldn't even play an mp3 without pegging the CPU, let alone a flash video containing one.

I've seen their workflow and how fast it works for them and I can see if they "modernized" it would cripple their productivity.

Until they try to bring in new-hires. How long does it take to train somebody who is used to modern office programs to use a DOS program like wordperfect? You think they'll ever get as proficient when what they see isn't what they get (a fad, I bet, right?)

Again, sorry to sound so harsh. You guys crack me up. Dont worry though, soon enough we'll see the errors in our ways and go back to time honored methods like waterfall. We'll abandon "scripting languages" like Ruby or C and use assembler like god intends.

Sheesh.

Re:Question

[ClosedSource]

Not all of us "old timers" think everything was great "back in the day" and shit now.

As you say, what is being built now is much more ambitious than what we used to make. There were challenges then too, they were just different.

Newer technologies can be a two-edged sword. Way back when a serious bug in an embedded system would require a new PROM or EPROM to be made and installed by a technician.

Today you can download an update over the Internet in a few minutes. That convenience weakens a company's motivation to getting it right the first time.

Of course today your product probably relies on software that you didn't write and aren't familiar with. We used to write every byte we delivered and couldn't blame an error on libraries because we didn't use any. But you couldn't compete in the market that way now.

The constant in this business is there will always be those who try to push the limits whatever they are.

Re:Question

[DoofusOfDeath]

Sorry to be harsh, but get with the times. Computing these days is vastly more complex then back in the "good old days".

You. are. fucking. kidding. me., right?

The sources of complexity have changed, but not significantly increased.

When's the last time your code had to:

• Employ overlays to make your code fit into memory?
• Write a large, complex algorithm in assembly, or even (in the 50's) straight machine language?
• Consider writing self-modifying code, just to make the program require less memory?
• Make "clever" use of obscure, corner-case behavior of certain machine instructions, not because you like to screw the people who will inherit the code, but because you have only a time amount memory to work with?
• Intentionally mis-DIMENSION an array in fortran 77 code, knowing that it will work out okay at runtime, just to avoid (by necessity) the runtime cost of using array-slicing functions?

Time-Honored WHAT?!

[mihalis]

he lays the blame on PC developers (read: Microsoft) who kicked the time-honored waterfall model to the curb

The waterfall model is long-discredited and its downfall is nothing to do with Microsoft. Ian Sommerville in his Software Engineering book was discussing this in the early 90s. I recall a diagram of the spiral model which is very much like agile development, although we didn't call it that then.

Went on to fame and fortune

[CarpetShark]

A project I was on in 2000ish went as follows:
Steppings A0, A1, A2, and A3 were halted in-fab because someone found a critical bug in simulations.
A4-A7 did not work.
B0-B4 did not work B6 did not work
C0-C4 did not work
B5, B7, C5 sorta worked.
The company folded.
That's what a software mentality working on hardware will get you.

Luckily Microsoft stepped in and bought the company, and now market the product as X-Box.

Re:Defects have a cost. Who pays? Change that.

[TheLink]

So how does full warranty work for OSS software?

There is no shortage of good coders or IT people

[jeko]

I bought three Mercedes. Two of them got repossessed. Now, the dealers won't finance me when I go to buy another. Clearly, there is a shortage of Mercedes.

Look at your story. You had three programmers. Two quit (Yeah, I know, it wasn't because they were unhappy. Look, no one wants to be known as a malcontent or difficult. They lied to you.) Now, you can't get anyone in to interview who knows what they're doing.

You think maybe it's possible that your company's reputation precedes it? I know of half a dozen places in my town that nobody in their right mind would agree to work for.

Show me a man who says he can't find anyone to hire, and I'll show you a man nobody wants to work for.

Take that same man, triple the wages he's offering and wire a pacifier into his mouth and the ghosts of Ada Lovelace and Alan Turing will fight for the interview.

Re:Its all true

[yttrstein]

You know, once you GET to 30, you do immediately understand why you should never trust anyone over 30.

It's because we know how fulla shit people under 30 are. Don't worry, you'll agree before ya know it, champ.

Re:Modus Operandi

[dacut]

Except that what you actually do is promise to paint it red even though you know that you do not have and cannot get any red paint.

It doesn't matter, anyway, because the customer changes his mind mid-project and wants it blue. After it's delivered, they realize they wanted a bicycle, not software. Nonetheless, they attempt to ride it.

While this makes about as much sense as the Chewbacca Defense, this explains a good 50% of the projects I've worked on. Hm. Right, I'll just sob in my tea now...

The issue is a lack of local coding standards.

[Richard Steiner]

If you perform certain types of validation on a routine basis, write a set of common routines to do the work, and reuse them over and over again. Standardize your code. Define standard buffer names, sizes and buffer attributes, and make sure that anyone working on that code is acutely aware of the standards which are already in place.

Reject code that doesn't follow the standard. Even if it works otherwise.

Modular coding isn't rocket science, and one can be very structured and modular in any language. No OO needed. We had an extensive library of common routines, common buffers, etc. back when I wrote Fortran 66 and 77 code at my last major place of employment, and we have the exact same thing here on both the C and Fortran sides of life.

Feature, not bug ...

[trolltalk.com]

The program he left worked wonderfully and was incredibly complicated until he wasn't there due to poor coding practice with documentation/comments to allow others to manage the tool.

It's called "coding for job security . . ."

It also comes with a companion use case "coding for future consultancy gigs . . ."

HR couldn't find a hooker in Vegas

[jeko]

"I'm beginning to think that our HR person just does a terrible job at finding resumes for me"

Well, there's your problem...

You sound like a great, amiable guy who'd be a great coworker. HR is screwing that up for you.

Why are you letting HR dictate who you're going to get saddled with? HR doesn't bring you resumes, you should be taking resumes to them. Talk to your friends and acquaintences, guys in the users' group. People that you know can do the job.

"Hey, we need a coder for Project X. Ya want it, or know anybody?"

"Whatcha offering?"

"120K, 30 days vacation, free milk and cookies..."

"See you Monday morning."

.
.

"Hey, we need a coder for Project X. Ya want it, or know anybody?"

"Whatcha offering?"

"$4.25 an hour plus all the stress and scapegoating you can handle..."

"Gee, not really looking myself. Let me see if I can find anyone for you...."

Basically, with unemployment penciled in to hit nine percent next month, you WILL find someone competent to hire. You just have to be offering market rates.

Re:Its all true

[Xtifr]

Not necessarily. I'm well over 30, and I agree with him. Programming is a talent that people have or they don't, and age doesn't seem to be a big factor either way. Skill (as opposed to talent) can be acquired with age, but I'd rather have a 22 year old with great talent and instinct than a 40 year old stick-in-the-mud who's been dragging down his teams for the last 18 years. I'd even more rather have a 40 year old with great talent and instinct, but you take what you can get.

Or to put it another way--just because the schools often seem to miss out on teaching certain useful skills, that doesn't mean that someone fresh out of school can't have acquired them by other means. For example, a kid might know what a linker is because she's been hacking her dad's Linux box since she was in grade school. (I have strong hopes that this will end up being a valid description of at least one of my nieces.) The 22 year old just might have more real-world practical experience hacking Debian than the 40 year old who spent years doing nothing but generating reports with RPG, Access and VB "programming".

Re:The customer pays in the end. Every time.

[Opportunist]

You know that managers buy the software with the best box design and the best marketing spinners, right?

Can't have your cake and eat it Mr. Wolfe

[Stormie]

I love the way Alex Wolfe blames shoddy programming on the PC industry, which apparently replaced the waterfall development model with "cramming in as many features as possible before the shipping cut-off date, and then fixing the problems in beta". He then goes on to reference two books, from 1971 and 1975 respectively, which provide wisdom regarding this problem.

They're excellent books - but I wasn't aware that the PC industry was around in 1971. Could it be.. that bad programming practices have always been with us? That the PC isn't to blame?