Software Development's Evolution towards Product Design

An anonymous reader writes: "The Lost Garden site has an excellent post on software development's evolution into product design. He starts with the first attempts at software design (for yourself or a colleague), and brings the conversation forward to modern design settings." From the article: "At the dawn of software history, programmers wrote software for other programmers. This was a golden era. Life was so simple. The programmers understood their own technical needs quite intimately and were able to produce software that served those needs. The act of software development was a closed circuit. A programmer could sit in a corner and write code that he wanted. By default it also happened to apply to other programmers."

Comments

Software design

[nizo]

Randomly throwing elements into software from the ground up rarely works well; everyone knows that early design is important for the software to be easy to use. In fact, intelligent design saves people from thinking about the software creation process at all, since the intelligent designer keeps the underlying processes of the software hidden from the user. Put your faith in a good intelligent designer and you can simply use the software and remain blissfully ignorant of how the software actually works.

Walk a mile in their shoes...

[Quintios]

That's one of the things I've found most interesting in my n00bie programming travels. As a chemical engineer, I find that the programming I do (mostly scripting for automating Office and text manipulation to get stuff into Excel, or Word, etc.) serves me and my buddies quite well, but the solutions developed by central IT are usually complicated, buggy, and just plain awful. They seem to have little idea of what *our* (the engineers) workflow/work process is.

Understanding the actual needs of the enduser, I think, is one of the biggest challenges for programmers. What do they really need? Will they understand it? Will new "gee-whiz" features really be welcomed? And for that matter, do the programmers really undersand my job?

To sum up, it's easier to program for yourself than for others, it seems. You know your job better than anyone else. Otherwise, you have to do a lot of interviewing and discussing before you code a single line. You'll end up with a *much* better product if you listen to your endusers well.

Re:Walk a mile in their shoes...

[computational super]

They seem to have little idea of what *our* (the engineers) workflow/work process is.

This would work well if the programmer just sat down with the user, with no fixed delivery date, and they started working towards a common goal. This is similar to what you're doing with Excel and Word - you have no project manager, no "budgeted hours", no "chief architect", no "technical lead", and no "requirements design meetings". Just a problem to be solved as efficiently as it can be solved.

Unfortunately, for the programmer, you're not allowed to talk to the users (or, at least, I never have been). Talking to the end-users (actually, there are no users - they're "stakeholders". You'll never meet them.) is for the Birkenstock-wearing, ponytailed, hybrid-driving "usability engineers" the author is so slobberingly excited about. Programmers just get 1500-page "requirements documents", all written in Microsoft Word over a six-month period of review meetings. The less actual information, the better. From those specifications, the programmers are expected to fill out an Excel spreadsheet listing the "tasks" they must complete to fulfill "requirements" with such descriptions as "6.G.9.d.z. The freemulator frooble must goblify the cooblestocken whenever the user selects the remulize option". No asking the "stakeholders" what that actually means - they're far too busy to talk to you. The programmer must then randomly compile a list of "tasks" and a completely wild guess as to how long each task might take. You can estimate any amount of time for any given task and add as many tasks as you like, as long as the total time you estimate adds up to the target date that the stakeholders made up out of the air without consulting you. Once the task list is complete, the whole list is handed over to a project manager who "manages" each task. He does so using sophisticated project management techniques he learned in a one-hour training seminar such as: requiring all programmers to attend a weekly two-hour status meeting where he solemnly reads the list of tasks, one by one, and says, "what percent complete are you on this task?", and "you're falling behind on these tasks. Can we offload some of these to somebody else?". Tasks can never be added or deleted once they project manager "finalizes" his Excel spreadsheet.

At the same time, the stakeholders will change their minds daily. They'll randomly remove one of the 20,000 requirements (the one you spent the last three months coding for) from the requirements list and announce that they've "flexed the scope" to meet a new "compressed delivery date" (which is next Friday).

Slowly but surely, in spite of all the obstacles that have been placed in your way in the name of improving "Product Design" efficiency, you (the programmer) will finally start to understand what it is the users might actually want and what this thing actually does. Unfortunately, about the time you see exactly what needs to be done and the best, most flexible way to do it, the "two months past code complete" date will be hit. Then you will enter "crunch time". The weekly two-hour meetings will change to daily three-hour meetings. One of the agenda items on each meeting will be to discuss how to improve programmer productivity. To make up for the lost time spent in these meetings, you'll be required to work on the weekends. Fifty new programmers will be added to the effort to "help". You'll spend 10 hours of your 16 hour day getting them up to speed. At this point, you do whatever the hell it takes to get this monkey off your back.

Then, at the end of the year, after months and months of 80+ hour work weeks, a failing liver from your increased alcohol consumption (it's the only way you can actually manage to fall asleep) and a divorce since your wife and kids can't remember your first name or what you look like, the users complain about the stinking pile of poo that you finally were able to produce in spite of the "efficiency experts" driving the process. You see, you nee

Extreme Programming

[StarvingSE]

This is why I think certain elements of extreme programming should become prominent in the software industry, particularly user stories and incremental releases.

The non-technical customer can provide the programming team with "stories" about how they would like their software to function, and rate these stories based on priority. It is up to the programming team to figure out how to do the technical work in the software to accomplish make the story (or use case) a functional part of the software.

Then, the team can incrementally add these user stories and show the customer a working prototype, so that if the design isn't exactly what they expect, it is easier to change and hopefully to maintain.

The author has no clue...

[xxxJonBoyxxx]

"At the dawn of software history, programmers wrote software for other programmers."

I thought the first programmers wrote software to figure out ballistic missle trajectories, crack ciphers, count census figures and perform other useful work. What exactly is our clueless author whining about?

Can it really be engineering right now?

[MikeRT]

People often treat code like a paper. You want something added, you type it in and flesh out the paper. I think that this is why software often ends up working so badly. If people would treat their programs more like engineered pieces of hardware than written works in progress, things would be better. It just seems to me that people all too often forget that software development requires real planning and that it's not as simple as "I want feature X" in many cases. Maybe the solution is extreme extensibility. Cars can be modded in pretty complex ways. Perhaps the solution for software development is building a consistent core that just works and then plugging new modules into it, sort of like Eclipse.

Huh.

[Z0mb1eman]

I read this blog entry with a growing sense of unease, until I got to this point:

The benefits of a product design process are well documented. New products that deliver superior, unique benefits to the customer have a commercial success rate of 98% compared to 18.4% for undifferentiated products. These products reach an outstanding 53.5% market share.

As much as I wanted to finish reading the article, I just couldn't get past that. It is well documented that 83.8% of Slashdotters who share my interests and read the RTFA reacted the same way.

Cute illustrations, impressive list of references... but I haven't been able to extract any useful information from the article. Yes, writing software that people want to use is hard. Yes, listening to customers is very important, but also a lot trickier than "listen to your customers". Yes, to write successful software you need a mix of many different skills, not just programming, and yes, it is often difficult to even know what those skills are, let alone to find people who have them and to get them to work together productively.

Is any of this theory really that groundbreaking? I like to think that all these concepts are self-obvious to anyone involved in the software industry - the difficult part is actually translating them into reality.

Identifying user pain

[c0d3h4x0r]

I'm a developer on a product team at a huge software company. I've worked on 6 shipped releases of the same product, and I've worked within the same core feature areas for about 3 of those releases. More than half my time each release cycle is spent helping the feature team to identify user scenarios and optimize ways to solve them.

One of these core feature areas had frustratingly low usage and user-satisfaction ratings for years, until we got serious about feeling users' pain. It took lots of usability testing, using the right tests and asking the right questions, to finally expose a number of thematic problems users were having. It took even more usability testing on many iterations of designs to find approaches that really solved those problems well for most users.

The most educational lesson in all of this was that the things the product team suspected were user pain points were often not so, and the things the product team thought were fine were often problematic. In other words, the product team's very educated guesses were frequently wrong. These were people who had worked on the same product, on the same feature areas, for years, often looking into bugs and suggestions sent in by real end-users. If anyone was qualified to make an educated guess, it was these people, and yet they were often wrong.

We didn't make huge technical changes under the hood or introduce loads of new power-user functionality. We didn't just try to pile hacky bug fixes on top of the existing user experience. We didn't just try to optimize the performance or speed of the existing feature. We listened to what real users were telling us, and we squarely addressed their frustrations and confusions.

In the latest round of usability testing, the feature scored more than double the old user-satisfaction numbers, and there will be even more improvements made to address more user feedback gathered from that testing. We anticipate that when the next release ships, this feature area will have dramatically improved user-satisfaction and significantly reduced abandonment.

Now, I think about my Kubuntu installation on my PC at home, and about the variety of open-source applications that I use on it, and I skeptically wonder: is the same kind of feedback loop and concern for non-technical users applied in the open-source world? It seems like most developers of open-source software spend more time developing what they think is cool, or what other geeks might want, than trying to identify and eliminate the pain experienced by non-technical users. Even when some open-source projects (say, GNOME, KDE, or Firefox) are genuinely trying to make things easier for non-technical folk, they are often just flying blind, copying the UI of commercial software or taking wild personal guesses at what they think non-technical users want. Their guesses, although well-intentioned, are often completely wrong.

The moral of my story: you have to approach identifying user needs in a scientific way, or you'll almost never get it right. You have to perform your own research and perform it frequently as the design evolves/iterates. And no matter how crazy the results of that research seem to you, the software designer/developer, you should still trust in them.