The Economist Tackles Complexity in IT

yfnET writes "In recent weeks, The Economist has run a number of articles addressing the ever-increasing complexity of software systems. The magazine, with typical Economist wisdom, casts an eye towards past human endeavors for lessons on how today's IT industry can succeed in dealing with complexity. As part of last month's extensive survey of information technology (see Related Items sidebar), the magazine offers insight on the limits of real-world metaphors, the perils of managing a rat's nest of obsolescent systems, and the need for 'disappearing' technology. And hitting newsstands just today is an overview of development models for increasingly large and unwieldy software projects. Among other things, this article compares the open source model to Microsoft's efforts using a quasi-open license. It also describes the 'agile' programming movement and its potential to keep even the most gigantic of projects under control."

Comments

"Agile" works when you're located together...

[Not_Wiggins]

There's a serious problem with agile methodology and outsourcing (I didn't see any articles on Economist.com related to outsourcing, but may have missed it as I gave it only a cursory look).

Large and unwieldy projects benefiting from agile methodologies? Yeah... when you have easy communication between the "customers" (business partners) and your IT staff.

How does that happen when your developers are thousands of miles away, in a different timezone, with a totally different culture, and don't speak your native language?

Do simple things

[barcodez]

People tend to make things over complex if you let them. Bosses want that technology they read about in whatever PHB read these days. Such and such wants some TLA to do some other TLA.

I know it's hard but you have to tell them that these things don't add any value in and of themselves. You want the simplest possible system that will solve the problem at hand. Really, nothing more, don't implement something because you may want/need it tomorrow 'cos when tommorrow comes it won't be right (and if it is, hey, you can implement it then).

MAGIC, COMPLEXITY ARE INCOMPRESSIBLE

[museumpeace]

fascinating read that, but facile too.

All the management of complexity now takes place within the network, so that consumers no longer even know when their electricity or water company upgrades its technology. Thus, from the user's point of view, says Mr Benioff, "technology goes through a gradual disappearance process."

[from the article]
This push to make [for the user] simple what is after all increasingly complex, can only hide, not eliminate the role of the nerd class, a role the article disparages because nerds are presumed, as the inventors, to have foisted off complexity on the unwitting public. Was it Heinlein who said that "any sufficiently advanced [or was his word complex] technology is indistinguishable from magic"? The wish, on the part of typical users and marketers, that all the wonders of our age and those ages coming next should all just work like magic will in fact only ADD the complexity of UI technologies that are good at hiding the guts of the systems we depend upon. The the engineers and technicians will be as needed as ever and get even less sympathy from a public that never sees directly what it is that the "nerds" are doing for them.

Complexities aren't going anywhere

[GreenCrackBaby]

For an outsider looking in, perhaps it's easy to look at something like Google's 23-word front page and say "why can't they all be like that!" Too bad most systems need more than one form element to allow the user to interact with the system. Can you imagine someone telling Adobe to reduce their Photoshop interface down to one or two buttons? It would make no sense simply because editing a digital image is far more complex a process than 'search the web for these terms' to a user (though both may have similarly huge code bases behind them).

Complexity in IT isn't going to go away. In fact, I'd argue it is a necessity. There are some tasks that simply require complex systems and those complex systems require complex data and/or complex user interfaces.

Re:Complexities aren't going anywhere

[Billly Gates]

But here is the problem. All the complex systems don't talk to each other.

The IT manager who quit from JP Morgan was a perfect example. You have 450 applications talking to each other and a user calls the helpdesk and demands an answer right away. What caused the problem? Which layer? Which application was doing what to the data?

Microsoft was hot for awhile with the IT managers in corporations because all the dcom/com/ole applications can interact with each and become one. This can help the problem tremendously.

However there is no standard protocal between all the vendors. That needs to change before vendors start with their own proprietary versions that only work with their products.

If an application uses several layers and it screws up there has to be a way to trace and find out what happened.

Perhaps a new opensource protocal could help? I like that idea.

Re:Complexities aren't going anywhere

[GreenCrackBaby]

All the complex systems don't talk to each other.

I'd argue that is a near-impossible task. My background is in billing systems, so I'll give an example from that realm...

Our company makes billing systems that end up producing your telephone bill. Sounds simple, but the billing system alone clocked in at over 6 million lines of code. Then you have the other two big necessary systems: a CSR application (such as Siebel) and a provisioning system. Not to mention hundreds of smaller apps that feed/collect data from each application. You have no idea how complex the infrastructure can get!

But it's not just that complexity. In our billing system, a customer's account was important because we needed to know who to charge for each transaction. In a CSR application, they care about who to contact. In a provisioning app, they care about where the account is physically. This leads to a different approach to designing something as simple as the account structure in the database. It's not something that could be standardized because each application needs to look at the data differently.

There was some hope of standardization with middleware applications like Vitria, but what we found is that we'd spend insane amounts of time building code that translated our account between our billing system and some common model held by the middleware. The complexity didn't go away -- it got worse!

You won't ever see a standard vendor protocol. Not for lack of wanting one, but simply because it's impossible.

A Lot of Silliness, and Two Spectacular Points

[Onimaru]

So a lot of this space was spent explaining to Joseph P. Siquespack, Esq. what a "protocol" was and the like, but there were two points in here that I'm really glad my great-grandboss might be reading:

1. A system should be designed to fail in a predictable way. Much like a car body, it should crumple to protect its most valuable assets, and repairs should have obvious beginnings, middles, and ends.
2. Obsolete systems will cause you more downtime in the end than incremental upgrades. And, what's worse, it will be all at once instead of at 4am twice a month on Saturday morning.

Neither of the above are impossible goals! They can be done with a little thought and elbow grease. And the great part is, they're probably already being done! Next time you're reading over your IT department head's recommendations for a project, call them up and ask WHY. You might be amazed at how awesome the answer is, and it might even persuade you to put away the "my way or the highway" stamp.

Re:Why should we believe what they say?

[fiannaFailMan]

Aren't these the same people who tried to tell us that "comparative advantage" meant we should give up our manufacturing and all get degrees in high tech? Why should we believe ANYTHING they have to say after they've lied to us for so long?

Actually, I don't seem to remember that paper ever advocating any such thing. Nevertheless, a great many people fell for that one. A great many people still cling to it. And to say that they "lied" implies that they knew different but told us this anyway. Perhaps you can tell us what they hoped to gain from this grand deception. Or perhaps you'll just admit that you had an aversion to their endorsement of John Kerry.

Economics is a religion- and a failed mythology at that.

Nonsense. Economists, and the Economist newspaper, seldom have a problem admitting mistakes. Show me a religion that is willing to change its belief in a supernatural being after centuries of mounting evidence to explain away the mysteries that underpinned such superstition.

Just Engineering Taken to its Logical Conclusion

[smug_lisp_weenie]

Engineers love to break things into smaller parts, each part serving one and only one function, like pulleys, shafts, rotors, etc.

For really effective design each part has to serve multiple functions, like evolution is able to do: The human mouth can be used to eat, breathe, talk, etc etc.

That's why a robot can't compete with an animal- In a robot each part usually severs only one function, making the machine inefficient as a whole.

This problem is just magnified in computer software and will only get worse unless engineers start changing their tune. I think the worst offender of this philosophy is object oriented programming: It's the ultimate embodyment of this philosophy- Most big object-oriented software have only about 2-3% of code that performs any real work, with the rest only is window dressing to fulfill the engineer's urge to "modularize".

The best software I see seems to be written either in more pragmatic procedural styles, or uses better mathematical underpinnings for its structure, like you'll find in functional programming languages (Haskell, Lisp, etc.)

My apologies for living up to my user name!

Conrad Barski

Near Max Complexity of Industry

[Teancum]

The problem with trying to compare the software industry to other endeavors of human experience is that in the realm of the computer/electronics industry that most software developers are dealing with, you are near the pinacle of dealing with abstraction of complex systems. While there may be other very complex systems that on a large scale can compare to some computer networks or CPU designs, computer science is the practice of dealing with abstractions on many levels, sometimes simultaneously.

Indeed, electronic state machine digitial computing devices (also called computers) have proven so successful, with the software abstraction dealing with the various levels of abstraction, that they are used in the controlling of other complex systems, from air traffic management, urban water system management, freeway traffic monitoring, and law enforcement dispatching. You've seen them, and they are out there.

Some very talented engineers have done a surprisingly good job of simplifying the tasks and reducing the abstractions to the point that all you need to do for the most part is plug it in and watch the gizmo do its thing. What this article in the Economist seems to be doing is complaining that the job isn't finished, and that complexities in setting up a computer system for some project is more difficult than it should be. That is primarily due to the fact that the author is using products that don't comply with standards (a real problem if standards don't exist yet for a certain concept or technology) or they are using the wrong tool for the job, like using a hammer to put in a few screws. Sure, it will work, but it is aggrivating and sometimes takes quite a bit longer to get the job done, and can damage things around it as a side effect. How many software/electronic gizmos out there do you know get used like that?

While I'm willing to acknowledge that I don't know everything there is to know about the management and organization of complex systems, I would be more inclined to get the opinion on such a subject from a computer programmer than from a plumber.

Re:Lisp, Smalltalk and complexity.

[Anonymous Coward]

the advantage Lisp and Smalltalk have that other languages lack is that their syntax is so simple that extensions meld in as if they were part of the language.

I cry bullshit...

[Linegod]

I hear this every damn day. 'We need to make it simple', 'It is a simple service' or 'It is a low option service'. This may work fine for the sales and marketing drones that make their commission off selling unnecessary services to uninformed customers, but as long as there is _choice_ out there, the backend is going to be complicated, and somebody, somewhere is going to have to known how all of it (or at least a major part of it) works.

Try all you want, but unless the entire IT industry decides to switch to one massive global device that we all plug _directly_ into, you can't make video conferencing/VOIP/disaster recovery/etc through 2 LANS, 3 Service Providers and 10 different security layers a 'green/red' push button operation.

I've gotta go get drunk now....

Re:Lisp, Smalltalk and complexity.

[John Courtland]

Strong typing should be a damn requirement. Every language I use that doesn't allow strong typing seriously chaps my ass in a major way. The last time was when I was simply trying to implement a dumb calculator for RPM MPH for a car given gearing info and tire size. Simple function:

function get_tire_circumference_in_inches(tire_width, tire_profile, rim_diameter)
{
return (3.1415926535 * (rim_diameter + 2 * (tire_width * 0.01 * tire_profile)/25.4));
}

But javascript, when provided 16 as rim_diameter, returned some huge number (~5,000+), instead of what I expected (~84). Turns out JavaScript interpreted all the other numbers fine, but somehow wanted to make the 16 a literal, and added the damn thing right onto the end of the equation. That took me about an hour to figure out. I finally ended up getting Venkman and watched it fuck up. Then I tried implementing (and did get it to work, eventually with some code pinching) MD5 in JavaScript. Yeah, it's not worth it. Imagine if this were production work... VB (especially VB...) has done similar shit to me. To me, languages that do that are not languages, they're toys.

How aerospace does it

[Animats]

Aerospace has dealt with high complexity for decades, rather more successfully than the IT industry. Here's how.

• Interface specifications dominate If it doesn't work the way the spec says it does, fix the box, not the spec. If A won't talk to B, run the tests to check compliance with the spec. If you can't tell who's at fault, the spec is broken. This is why you can swap a Pratt and Whitney engine for a Rolls Royce engine.
• The buyer, not the vendor, decides what is a "defect". One of the fundamental problems in IT is that vendors have sole discretion to decide what is a defect and what isn't. That doesn't fly in aerospace.
• Fix blame. In aerospace, people get blamed for screwing up. You do not want your name or the name of your company to appear in an NTSB crash report. If you screw up big time, it will. Mistakes in aerospace are publicized. There's an NTSB database of 140,000 crashes. If it was a hardware failure, the vendor is named.
• Warranties have real meaning Airplanes come with good warranties, and so do all the parts that go into them. Commercial software doesn't.

This runs engineering costs way up, and the life cycles are longer, but in IT, most of the commercial products are sold in large numbers, so you get to spread that engineering cost over a large number of items.

It's time for computing to grow up and accept this kind of discipline. The automotive industry had to accept it in the 1960s, and cars got much better within a decade.

Re:Chinese Slave Labor

[tarunthegreat2]

Of course, I'm feeding a troll here, but India has been on a path to "westernization" for about 200 years now, ever since the British East India Company first set foot on the shores of Calcutta. India alreasy has a "western-style" political and economic system, so STFU. Our laws are based on British Common Law - newsflash - so are American laws. We VOTE our leaders in to power. When two companies have a dispute ove a contract they go to court. When we want to make laws, they have to passed by two houses of parliament. Parliament happens to be this big place where elected representatives gather - to pass laws. Oh, and the unofficial offical language of India is ENGLISH. Imposing this system on a culture which has been transforming and transitioning for the past 1900 years going from Hindu - to Buddhist- To Muslim - to British is going to produce results which will be very different from what a pitifully young country like America isn't used to seeing. So just get the fuck over it. Finally this commitment to free trade that you talk about - The developing countries are ready & waiting for it. It is America that can't handle freeing trade in agriculture and industries like steel. It is USA which puts quotas on garments. Look up any textbook: quota != Free Trade. If you think an Indian software engineer is cheap, wait'll you discover the price of an Indian orange, or an Indian T-shirt. But you won't know about those because trade in those items is not "FREE". And it's not free because the EU and America want it that way. Because Billy Bob with-mouth-in-straw living in a redneck county of a red state just voted the current monkey into the white house. Now go back guarding the bridge, trolly-wolly.