Software Development's Evolution towards Product Design

An anonymous reader writes: "The Lost Garden site has an excellent post on software development's evolution into product design. He starts with the first attempts at software design (for yourself or a colleague), and brings the conversation forward to modern design settings." From the article: "At the dawn of software history, programmers wrote software for other programmers. This was a golden era. Life was so simple. The programmers understood their own technical needs quite intimately and were able to produce software that served those needs. The act of software development was a closed circuit. A programmer could sit in a corner and write code that he wanted. By default it also happened to apply to other programmers."

Comments

Software design

[nizo]

Randomly throwing elements into software from the ground up rarely works well; everyone knows that early design is important for the software to be easy to use. In fact, intelligent design saves people from thinking about the software creation process at all, since the intelligent designer keeps the underlying processes of the software hidden from the user. Put your faith in a good intelligent designer and you can simply use the software and remain blissfully ignorant of how the software actually works.

Re:Software design

[imstanny]

Put your faith in a good intelligent designer and you can simply use the software and remain blissfully ignorant of how the software actually works.

I think you copied/pasted this from a religious thread.

Re:Software design

[$RANDOMLUSER]

Actually, I put in some features to decieve you - to test your faith in the software.

Re:Software design

[Heembo]

Something I hear few say, is that a LONG design process can be very dangerous to a projects success.

For example, if you have a one year design process, you capture the ebbs and flows of the business over that period of time, more likely you have captured a fluid structure. Your chance of success drops dramatically.

But, if you drop in, do a rapid design with elite analysts, whip out solid, secure code with an elite team in short period of time, you gain a great often undocumented benefit: You capture the business at a discrete moment in time. You codify what is exactly happening with the company at that moment. Often, the company rallies around that codified process.

I love design - but the moral of the story is, solid, detailed design alone will not give you good software or a sucessful project.

Software Evolution?

[n6kuy]

Momma told me software was created by Flying Spaghetti Monster...

Re:Software Evolution?

[Spy der Mann]

Momma told me software was created by Flying Spaghetti Monster...

Eeeew spaguetti code! >_<

Walk a mile in their shoes...

[Quintios]

That's one of the things I've found most interesting in my n00bie programming travels. As a chemical engineer, I find that the programming I do (mostly scripting for automating Office and text manipulation to get stuff into Excel, or Word, etc.) serves me and my buddies quite well, but the solutions developed by central IT are usually complicated, buggy, and just plain awful. They seem to have little idea of what *our* (the engineers) workflow/work process is.

Understanding the actual needs of the enduser, I think, is one of the biggest challenges for programmers. What do they really need? Will they understand it? Will new "gee-whiz" features really be welcomed? And for that matter, do the programmers really undersand my job?

To sum up, it's easier to program for yourself than for others, it seems. You know your job better than anyone else. Otherwise, you have to do a lot of interviewing and discussing before you code a single line. You'll end up with a *much* better product if you listen to your endusers well.

Re:Walk a mile in their shoes...

[TubeSteak]

You sound like every engineer I've ever talked to.

Engineers always say that if the suits just asked, the engineers would help make everything better. The suits, in the meantime, are busy doing everything but listening to the engineers.

Why is there such a fundamental disconnect between the engineers and *everyone* else in a business environment?

Re:Walk a mile in their shoes...

[Duncan3]

Why is there such a fundamental disconnect between the engineers and *everyone* else in a business environment?

Because it's the only way to get any work done.

Every group usually has one person that does all the work, the rest just have meetings all day and pretend to be useful. If you're smart about it, you can sacrifice a member of the group, and they go do all the meeting stuff, and the other n-1 can actually do something. Those are the companies that win.

Re:Walk a mile in their shoes...

[Quintios]

Why is there such a fundamental disconnect between the engineers and *everyone* else in a business environment?

This is sort of off topic, but I'm answering TubeSteak's question. I would opine that it's because the suits don't want the engineers doing stuff that doesn't directly make themselves money. I've gotten into "trouble" before because of my penchant and ability with computers. Doggone it if I can take 2 hours and develop a tool that will save me 15 minutes a day for the next 5 years, or longer, why shouldn't I?

Cause it's not my job. :(

I think also that perhaps the suits look down on us in some ways for being overly excited about equations, calculations, and being just really really anal and annoying at times when things aren't *exactly* right. In my line of work if you're not careful people get killed. I *have* to be this way, but in the end I enjoy it. :)

In the same respect, there's a lot of engineers out there that don't care about the nifty tools I develop. Probably because they're not really that nifty, but in the end a few of them have gotten distributed. Some people are very "toe-the-line" and if it's not an official company-sponsored work process or tool, or they didn't get training on it, they won't touch it. A lot of engineers have blinders on when it comes to their work, which is bad because what if a better widget comes out? You have to stay abreast of current and future technology.

I could continue to ramble, but in the end I think we're all paid to do different things. The suits run the business deals and make sure the engineers are working towards making those deals successful, and the engineers are always looking to improve things. Anything. Everything. Whatever they can touch. Well, at least, that's *me* doing that. :)

Re:Walk a mile in their shoes...

[cwgmpls]

It's not just engineers. It is the worker bees in any organization.

I happen to work for a school district. Our main (only?) job is supposed to be teaching students. Yet our administration spends almost no time asking teachers what they need to be more productive. They come out with these great initiatives that they've heard or read about somewhere and the teachers are supposed to fall in line.

Then they wonder why teachers aren't more "productive". Its quite simple to figure out. The teachers spend half of the time jumping through hoops that the administration has set up, rather than the administration providing what the teachers need to do their job.

Teachers know how to teach, engineers know how to engineer, what the suits are supposed to do is listen to what those people need and provide it for them.

Instead, and especially with IT, the administration thinks they know it all and can tell the teachers what to do. It should be the other way around!

Re:Walk a mile in their shoes...

[ChilyWily]

In addition to many good points, the reality is that once a company achieves a certain reputation and a product (which is doing reasonably well.. even in appearance), then *everything* becomes a 'business case'. It is for that reason that so-called 'Leadership' comes from getting as much revenue as possible out of the said product... Engineering types are usually kept out of the loop - because typically they see through all this BS. Sad part is that most of them, especially the technically and intellectually capable Engineers ones don't engage themselves to the next level - so the business types implicitely take over control - then begins a long cycle of reaching SEI levels and process where as the real money, innovation and most of all, creativity, which fueled all the initial success is lost.

bleh.

Re:Walk a mile in their shoes...

[Quintios]

We're slow to punish people for several reasons, I think:

1. Lawyers
2. Invested capital
3. Circumstances

I think we try to be an understaing society, forced by lawyers suing companies for wrongful termination. Then, when you bring someone into an organization, there's a certain amount of time that has to pass while you try to make money off the person. If you pay for relocation, training, and a few months of "getting used to the company" time, you've got a lot of money invested in that person. You sometimes have to wait to see if it pays off. And then we also try to take into account the circumstances of the individual, or at least I hope we do. If that person is going through some personal trauma, we try to take that into account if they are going through an unproductive period. Of course, if #2 and #3 happen at the same time there's not much of a future for you at the company. :(

Now, if it's been verified that you are indeed an *idiot* and management knows it, you won't be around long...

Re:Walk a mile in their shoes...

[mattkime]

bah, success ALLOWS for idiocy within a company. success isn't being as streamlined as possible. success is selling a product.

i don't understand why you bring school systems into it. i don't know where you live, but there are extremely excellent public school systems in this country. the school system tends to be as good as the community demands.

Re:Walk a mile in their shoes...

[computational super]

They seem to have little idea of what *our* (the engineers) workflow/work process is.

This would work well if the programmer just sat down with the user, with no fixed delivery date, and they started working towards a common goal. This is similar to what you're doing with Excel and Word - you have no project manager, no "budgeted hours", no "chief architect", no "technical lead", and no "requirements design meetings". Just a problem to be solved as efficiently as it can be solved.

Unfortunately, for the programmer, you're not allowed to talk to the users (or, at least, I never have been). Talking to the end-users (actually, there are no users - they're "stakeholders". You'll never meet them.) is for the Birkenstock-wearing, ponytailed, hybrid-driving "usability engineers" the author is so slobberingly excited about. Programmers just get 1500-page "requirements documents", all written in Microsoft Word over a six-month period of review meetings. The less actual information, the better. From those specifications, the programmers are expected to fill out an Excel spreadsheet listing the "tasks" they must complete to fulfill "requirements" with such descriptions as "6.G.9.d.z. The freemulator frooble must goblify the cooblestocken whenever the user selects the remulize option". No asking the "stakeholders" what that actually means - they're far too busy to talk to you. The programmer must then randomly compile a list of "tasks" and a completely wild guess as to how long each task might take. You can estimate any amount of time for any given task and add as many tasks as you like, as long as the total time you estimate adds up to the target date that the stakeholders made up out of the air without consulting you. Once the task list is complete, the whole list is handed over to a project manager who "manages" each task. He does so using sophisticated project management techniques he learned in a one-hour training seminar such as: requiring all programmers to attend a weekly two-hour status meeting where he solemnly reads the list of tasks, one by one, and says, "what percent complete are you on this task?", and "you're falling behind on these tasks. Can we offload some of these to somebody else?". Tasks can never be added or deleted once they project manager "finalizes" his Excel spreadsheet.

At the same time, the stakeholders will change their minds daily. They'll randomly remove one of the 20,000 requirements (the one you spent the last three months coding for) from the requirements list and announce that they've "flexed the scope" to meet a new "compressed delivery date" (which is next Friday).

Slowly but surely, in spite of all the obstacles that have been placed in your way in the name of improving "Product Design" efficiency, you (the programmer) will finally start to understand what it is the users might actually want and what this thing actually does. Unfortunately, about the time you see exactly what needs to be done and the best, most flexible way to do it, the "two months past code complete" date will be hit. Then you will enter "crunch time". The weekly two-hour meetings will change to daily three-hour meetings. One of the agenda items on each meeting will be to discuss how to improve programmer productivity. To make up for the lost time spent in these meetings, you'll be required to work on the weekends. Fifty new programmers will be added to the effort to "help". You'll spend 10 hours of your 16 hour day getting them up to speed. At this point, you do whatever the hell it takes to get this monkey off your back.

Then, at the end of the year, after months and months of 80+ hour work weeks, a failing liver from your increased alcohol consumption (it's the only way you can actually manage to fall asleep) and a divorce since your wife and kids can't remember your first name or what you look like, the users complain about the stinking pile of poo that you finally were able to produce in spite of the "efficiency experts" driving the process. You see, you nee

Re:Walk a mile in their shoes...

[computational super]

And then an anonymous coward says, "Well, you should have said no", so you hunt him down, tie his hands behind his back like Kevin Spacey did to that fat guy he made eat the spaghetti sauce in "Seven", and force him to implement a fully SOAP and BPEL compliant Web Service-based SOA solution. In Lisp. Using Notepad. On Windows 3.1. In a shared cubicle.

Extreme Programming

[StarvingSE]

This is why I think certain elements of extreme programming should become prominent in the software industry, particularly user stories and incremental releases.

The non-technical customer can provide the programming team with "stories" about how they would like their software to function, and rate these stories based on priority. It is up to the programming team to figure out how to do the technical work in the software to accomplish make the story (or use case) a functional part of the software.

Then, the team can incrementally add these user stories and show the customer a working prototype, so that if the design isn't exactly what they expect, it is easier to change and hopefully to maintain.

User Centric Design!

[jma05]

That's the official name for it. Universities have been offering courses in it as part of HCI (Human Computer Interactions) for a while now.

The author has no clue...

[xxxJonBoyxxx]

"At the dawn of software history, programmers wrote software for other programmers."

I thought the first programmers wrote software to figure out ballistic missle trajectories, crack ciphers, count census figures and perform other useful work. What exactly is our clueless author whining about?

Re:The author has no clue...

[Vellmont]

You're right, for the most part programmers were writing software for other engineers, not other programmers. The author is using a shortcut though to describe "other people like themselves". The software programmers need is tools to create software.

Oh THAT kind of Product Design

[Saeger]

I was thinking of the other kind of actual Product Design that we're also evolving towards. :)

You believe in the user?

[Tackhead]

> Randomly throwing elements into software from the ground up rarely works well; everyone knows that early design is important for the software to be easy to use. In fact, intelligent design saves people from thinking about the software creation process at all, since the intelligent designer keeps the underlying processes of the software hidden from the user.

RAM: "You believe in the user?"
Crom: "Sure I do! If I didn't have a User, than who wrote me?"
RAM: "That's what you're doing down here. Master Control Program's been snapping up all us programs who believe..."

Re:You believe in the user?

[PitaBred]

Than != then. Pay fucking attention.
I think that you may have meant "Fucking pay attention" or "Pay the fuck attention." Spelling's all well and good, but it doesn't do anything without somewhat correct grammar, especially when cursing. Otherwise, you just sound like an inbred Tourette's kid.

Can it really be engineering right now?

[MikeRT]

People often treat code like a paper. You want something added, you type it in and flesh out the paper. I think that this is why software often ends up working so badly. If people would treat their programs more like engineered pieces of hardware than written works in progress, things would be better. It just seems to me that people all too often forget that software development requires real planning and that it's not as simple as "I want feature X" in many cases. Maybe the solution is extreme extensibility. Cars can be modded in pretty complex ways. Perhaps the solution for software development is building a consistent core that just works and then plugging new modules into it, sort of like Eclipse.

Re:Can it really be engineering right now?

[Vellmont]

If people would treat their programs more like engineered pieces of hardware than written works in progress, things would be better. It just seems to me that people all too often forget that software development requires real planning and that it's not as simple as "I want feature X" in many cases.


I think there's a lot of truth to this. Design requirements changing in the middle of development, or being extremely vague is a killer. It's like someone making a building and saying "I want this building to have between 1 and 3000 bathrooms"). Clearly that's insane.

He needs a new diagram at the end

[egarland]

There needs to be a new little diagram called 'The Open Source Era'. It would start with the programmer throwing the 'Biz Guy' out the window. Then he'd put up a wall labled 'Skinning' between him and the 'Designer' and the 'Interaction Dude' and going back to work.

Huh.

[Z0mb1eman]

I read this blog entry with a growing sense of unease, until I got to this point:

The benefits of a product design process are well documented. New products that deliver superior, unique benefits to the customer have a commercial success rate of 98% compared to 18.4% for undifferentiated products. These products reach an outstanding 53.5% market share.

As much as I wanted to finish reading the article, I just couldn't get past that. It is well documented that 83.8% of Slashdotters who share my interests and read the RTFA reacted the same way.

Cute illustrations, impressive list of references... but I haven't been able to extract any useful information from the article. Yes, writing software that people want to use is hard. Yes, listening to customers is very important, but also a lot trickier than "listen to your customers". Yes, to write successful software you need a mix of many different skills, not just programming, and yes, it is often difficult to even know what those skills are, let alone to find people who have them and to get them to work together productively.

Is any of this theory really that groundbreaking? I like to think that all these concepts are self-obvious to anyone involved in the software industry - the difficult part is actually translating them into reality.

Re:Huh.

[hawkeesk8]

The reason that it is so hard to "listen to your customers" is that far, far too often, customers don't know what they want - they just know that what they currently have is no good. I believe the comment below this advocating finding customer pains will lead to more success than trying to get any sort of reasonable story of what a customer wants.

I don't believe any customer went to Apple and said, "I want a digital music player that uses a really groovey touch sensitive circle as the user interface." There is a real lack of "designers" in the software industry that are akin to physical product designers. Yes, you have to listen to customer's desires but if you ask developers to code to some spec that was solely created from customer "requirements" you will end up with shit software. You need a designer interjected into that process that has vision and can recognize where customers are feeling pain.

Is it fair to say...

[skoaldipper]

...that linux is in "The Early Business Era" (stage 2)? When some distros like Ubuntu actually pay for and sponsor usability studies, I'd say Shuttleworth has this distro uniquely placed in "The Late Business Era" (stage 3). So, just grab yourself a middle man (for example) who scours linux forums for window migration user problems, relaying them back to the devs and artists, and that polish will become apparent in "The Product Design Era" (stage 4).

Hmm

[ENOENT]

So why is it that all these engineers wrote all of their ancient programs in COBOL?

Hmm???

Re:Hmm

[Vellmont]

That would fit into the second era of software design, where the guys with money thought that they (or other managers) could understand code, so they naturally wanted a language that was as much like english as possible. Thus COBOL became terribly (in more than one sense) popular.

Identifying user pain

[c0d3h4x0r]

I'm a developer on a product team at a huge software company. I've worked on 6 shipped releases of the same product, and I've worked within the same core feature areas for about 3 of those releases. More than half my time each release cycle is spent helping the feature team to identify user scenarios and optimize ways to solve them.

One of these core feature areas had frustratingly low usage and user-satisfaction ratings for years, until we got serious about feeling users' pain. It took lots of usability testing, using the right tests and asking the right questions, to finally expose a number of thematic problems users were having. It took even more usability testing on many iterations of designs to find approaches that really solved those problems well for most users.

The most educational lesson in all of this was that the things the product team suspected were user pain points were often not so, and the things the product team thought were fine were often problematic. In other words, the product team's very educated guesses were frequently wrong. These were people who had worked on the same product, on the same feature areas, for years, often looking into bugs and suggestions sent in by real end-users. If anyone was qualified to make an educated guess, it was these people, and yet they were often wrong.

We didn't make huge technical changes under the hood or introduce loads of new power-user functionality. We didn't just try to pile hacky bug fixes on top of the existing user experience. We didn't just try to optimize the performance or speed of the existing feature. We listened to what real users were telling us, and we squarely addressed their frustrations and confusions.

In the latest round of usability testing, the feature scored more than double the old user-satisfaction numbers, and there will be even more improvements made to address more user feedback gathered from that testing. We anticipate that when the next release ships, this feature area will have dramatically improved user-satisfaction and significantly reduced abandonment.

Now, I think about my Kubuntu installation on my PC at home, and about the variety of open-source applications that I use on it, and I skeptically wonder: is the same kind of feedback loop and concern for non-technical users applied in the open-source world? It seems like most developers of open-source software spend more time developing what they think is cool, or what other geeks might want, than trying to identify and eliminate the pain experienced by non-technical users. Even when some open-source projects (say, GNOME, KDE, or Firefox) are genuinely trying to make things easier for non-technical folk, they are often just flying blind, copying the UI of commercial software or taking wild personal guesses at what they think non-technical users want. Their guesses, although well-intentioned, are often completely wrong.

The moral of my story: you have to approach identifying user needs in a scientific way, or you'll almost never get it right. You have to perform your own research and perform it frequently as the design evolves/iterates. And no matter how crazy the results of that research seem to you, the software designer/developer, you should still trust in them.

Product "Designers"

[wrook]

Hrmph...

Seems to be yet another self-agrandizing Product Designer/Manager whose thinks that they "understand" the software development process. What's crazy about this article is that I haven't worked in a company that *doesn't* work like this. And yes, we have generally produced large quantities of user-useless poo.

The problem is largely due to the attitude of these guys. It's just another "throw it over the wall" to the programmers illusion. "If we just work out all the details before we write any code, we'll get it right the first time!"

The difference between most consumer goods and most computer software, is that computer software is a great deal more interactive than other things. It's therefore orders of magnitude more difficult to understand the subtleties of human interaction problems.

Usually, what I find is that these "product designers" have only a vaugue understanding of what they want, because they are not techinical (well, pedantic is a better word) enough to understand the difference. They figure out 10% of the problem and "throw it over the wall" to the programmers, saying "implement this, and I don't want any backtalk, you unwashed heathens".

So the programmers do what they do best -- solve problems. Only since they've never even seen a customer let alone talked to one, they get it all bass ackwards. The product "designers" then say, "Oh, well I told him what I wanted, but he insisted on doing his own thing -- it's not my fault".

Production pipeline indeed. It's some kind of pipe, but I'm not going to smoke it.

Re:Product "Designers"

[AaronBrethorst]

There are solutions to the (very real) issue you describe. For example, at Microsoft, the role I'm in as a Program Manager is meant to bridge gaps just like this one. I act as something of a technical liaison between my division's User Experience team (usability engineers, product designers, and interaction designers) and developers and PMs.

Once you have someone in a role where their entire job focuses in on ensuring that necessary conversations and information transfer occur it becomes much easier to make headway and avoid confusion between the Ux and Dev camps.

Think inside your user

[Opportunist]

One of the first things my "guru" told me. Your user is the one who will use your tool. It can be the best tool ever, if its interface sucks, nobody will use it.

Now, I don't enjoy UI design. I hate GUI design. I HATE flashy button design. But that's what makes your user happy. And I am happy when my programs are being used. Does it make sense that my new interface requires DirectX 9.0 (for a "normal" office application, mind you)? No. But it looks good, gives the user a fuzzy nice feel and he's feeling important and very smart that he can actually handle a tool that looks so terribly important and cool.

Yes, it's Fluffware. But that's what the customer wants.

Personally, I enjoy the CLI. Easy, fast, reliable, uses little to no resources, perfect. But then, I wrote that thing. I'm used to CLI. I grew up on CLI. I'm a 400 keys per minute typist. And most of all, I know what I want to do. I read manuals...

People today don't read manuals. They want to explore their software like children explore their toys. "Gee, what does this button do?" must not be from the famous last words collection when it comes to software, it's the way a lot of your users want to find out how their software works. It's appealing to the explorer in them.

So I give them stuff to explore. Make the tutorial a game. It sounds silly, allright. I know. But it sells...

It's a sad world we're living in.

HCI

[kevin_conaway]

HCI (Human Computer Interaction) principles are, in my opinion, THE toughest thing to learn properly in Computer Science. It is easily the thing I struggle most with in my coding.

You can't blame programmers for being programmers. Programmers are focused on programming principles like good architecture design, good algorithms etc. But making it look good? Thats tough. It requires that the programmer not think like a programmer, but a regular old end-user who has no concept of the internals of a program.

Building programs that are usable to other developers is a joy. Building programs that are usuable for developers and regular users is an outstanding feat.

Re:MDA

[discontinuity]

So, anyone else thinking that the AC poster of the parent just might be one of the authors of the linked-to book? ;)

Where the problem is

[Anonymous Coward]

Phase one, the developers worked with the users (hell, they WERE the users). Every phase since then, there is someone bewteen the programmers and the users, claiming that they know what the users want. Perhaps we should get rid of those guys, and let the programmers and users work together again?

Seeya in a Couple Years

[Bob9113]

Robert G. Cooper, a well known researcher on new product development, states that there are several core factors (listed in order of importance) for any successful new product design process:

      1. A unique, superior and differentiated product with good value-for-money for the customer.
      2. A strong market orientation - voice of the customer is built in
      3. Sharp, early, fact-based product definition before product development begins
      4. Solid up-front homework - doing front end activities like market analysis well
      5. True cross functional teams: empowered, resourced, accountable, dedicated leader
      6. Leverage - Where the project builds on business's technology and marketing competencies
      7. Market attractiveness - size, growth, margins
      8. Quality of the launch effort: well planned, properly resourced
      9. Technological competencies and quality of execution of technology activities.

Many companies in the Late Business era already emphasize a few of these factors. However, there are some differences. Notice how technical competencies are important, but last on the list. Notice also, how that creating a solution to customer needs is first on the list.

I look forward to seeing the resulting products in a couple years. Five years ago I started on a project that was initially focused on technical proficiency. Then a shift occurred. For the last three years the only thing that was allowed to guide work was customer feedback (I switched projects about two years ago). Technical profiency of the developers for the past three years has been considered nice, but not critical. Technical CRs were explicitly disallowed by the project management. It is a multimillion dollar application suite pushing two million lines, which is the primary application of about 8,000 people - roughly half our labor force. It is currently under review and may be scrapped.

Why? Because the code is dying. The features are there, and it looks good, but it is extraordinarily expensive to maintain, is overly tolerant of corrupt data, and every time someone looks at it funny it breaks. Adding new features has become an exercise in chasing an induced bug through dozens of classes.

The initial approach of technical isolation was wrong - we needed closer contact with the customers and it would have made the product much better earlier in the lifecycle. Now, however, it is precisely the lack of technical proficiency that is killing it. Relegating the user experience or the infrastructure to the back seat is a road to ruin. There is no point in creating software that doesn't solve the customer's problem. But there is also no point in creating software that is more costly than the problem the customer is trying to solve. Discounting technical proficiency is a direct path to cost-ineffective solutions.