The Economist Tackles Complexity in IT

yfnET writes "In recent weeks, The Economist has run a number of articles addressing the ever-increasing complexity of software systems. The magazine, with typical Economist wisdom, casts an eye towards past human endeavors for lessons on how today's IT industry can succeed in dealing with complexity. As part of last month's extensive survey of information technology (see Related Items sidebar), the magazine offers insight on the limits of real-world metaphors, the perils of managing a rat's nest of obsolescent systems, and the need for 'disappearing' technology. And hitting newsstands just today is an overview of development models for increasingly large and unwieldy software projects. Among other things, this article compares the open source model to Microsoft's efforts using a quasi-open license. It also describes the 'agile' programming movement and its potential to keep even the most gigantic of projects under control."

Comments

Why should we believe what they say?

[Marxist Hacker 42]

Aren't these the same people who tried to tell us that "comparative advantage" meant we should give up our manufacturing and all get degrees in high tech? Why should we believe ANYTHING they have to say after they've lied to us for so long?

Economics is a religion- and a failed mythology at that. Economists need to learn to examine and reconfigure their basic axioms before ANYBODY should ever listen to them again.

Re:Why should we believe what they say?

[Marxist Hacker 42]

I trained to be a software ENGINEER- that includes coding far better than what any given code monkey can do. But since the code monkeys are doing it, my talents went to waste for over two years as I scrambled to find some way to make a living.

Science is about theory- ENGINEERING is about application of that theory. I'd much rather be doing the application of the theory than inventing the theory- but that's all been ripped away now.

Plus, if you haven't noticed- Honda has a computer that walks, and Microsoft has a computer that talks. It won't be long now.

Agile Programming Isn't a Silver Bullet Either

[Greyfox]

I've seen it work well and I've seen it work poorly. If all you have programming or managing on your project are chimpanzees, telling them to use scrum (or whatever) will not help you.

I'd say just hire good people and keep the managers focussed on just a few projects, but this seems to be beyond the capabilities of most companies. So you end up with programmers who write Java like it was fortran and managers who juggle so many projects that they barely know your name much less what you do for the company. There doesn't seem to be a fix for this which companies will be willing to accept.

Re:Agile Programming Isn't a Silver Bullet Either

[Qui-Gon]

Right on. Execution is everything (in any business). Fred Brooks tried informing the software community of this ~20 years ago.

People are everything on a software project. You can't put 1000 monkeys on 1000 type writters (pay them bananas) and expect the next great novel (or great piece of software). You can try but, it just ain't gonna happen. The reason IMHO is drive. Your typical "code-monkey" (or whatever derogatory slang term is used now a days) is only there for the "bananas". An engineer on the other hand, has two drivers - pride and fame. Every person who calls themself an engineer always wants to look like the smartest guy in the room. Don't get me wrong... engineers want money as well... but they know that the path to real fortune starts with credibility and acknoledgment.

Some might say outsourcing is an exception to this... That you can come up with great software without the need of strong enigneers (and people who really have an attachment to thier work). To me, outsourcing is the equivalent of fast food -- Its great for a quick meal. Its cheap. Its solves your immediate problem (aka hunger). But it isn't going to be good for you in the long run. Outsourcing companies are about one thing -- Cashing in on the slumming american economy that is still tech hungry.

Complexities aren't going anywhere

[GreenCrackBaby]

For an outsider looking in, perhaps it's easy to look at something like Google's 23-word front page and say "why can't they all be like that!" Too bad most systems need more than one form element to allow the user to interact with the system. Can you imagine someone telling Adobe to reduce their Photoshop interface down to one or two buttons? It would make no sense simply because editing a digital image is far more complex a process than 'search the web for these terms' to a user (though both may have similarly huge code bases behind them).

Complexity in IT isn't going to go away. In fact, I'd argue it is a necessity. There are some tasks that simply require complex systems and those complex systems require complex data and/or complex user interfaces.

Re:Complexities aren't going anywhere

[Billly Gates]

But here is the problem. All the complex systems don't talk to each other.

The IT manager who quit from JP Morgan was a perfect example. You have 450 applications talking to each other and a user calls the helpdesk and demands an answer right away. What caused the problem? Which layer? Which application was doing what to the data?

Microsoft was hot for awhile with the IT managers in corporations because all the dcom/com/ole applications can interact with each and become one. This can help the problem tremendously.

However there is no standard protocal between all the vendors. That needs to change before vendors start with their own proprietary versions that only work with their products.

If an application uses several layers and it screws up there has to be a way to trace and find out what happened.

Perhaps a new opensource protocal could help? I like that idea.

Re:Complexities aren't going anywhere

[GreenCrackBaby]

All the complex systems don't talk to each other.

I'd argue that is a near-impossible task. My background is in billing systems, so I'll give an example from that realm...

Our company makes billing systems that end up producing your telephone bill. Sounds simple, but the billing system alone clocked in at over 6 million lines of code. Then you have the other two big necessary systems: a CSR application (such as Siebel) and a provisioning system. Not to mention hundreds of smaller apps that feed/collect data from each application. You have no idea how complex the infrastructure can get!

But it's not just that complexity. In our billing system, a customer's account was important because we needed to know who to charge for each transaction. In a CSR application, they care about who to contact. In a provisioning app, they care about where the account is physically. This leads to a different approach to designing something as simple as the account structure in the database. It's not something that could be standardized because each application needs to look at the data differently.

There was some hope of standardization with middleware applications like Vitria, but what we found is that we'd spend insane amounts of time building code that translated our account between our billing system and some common model held by the middleware. The complexity didn't go away -- it got worse!

You won't ever see a standard vendor protocol. Not for lack of wanting one, but simply because it's impossible.

A Lot of Silliness, and Two Spectacular Points

[Onimaru]

So a lot of this space was spent explaining to Joseph P. Siquespack, Esq. what a "protocol" was and the like, but there were two points in here that I'm really glad my great-grandboss might be reading:

1. A system should be designed to fail in a predictable way. Much like a car body, it should crumple to protect its most valuable assets, and repairs should have obvious beginnings, middles, and ends.
2. Obsolete systems will cause you more downtime in the end than incremental upgrades. And, what's worse, it will be all at once instead of at 4am twice a month on Saturday morning.

Neither of the above are impossible goals! They can be done with a little thought and elbow grease. And the great part is, they're probably already being done! Next time you're reading over your IT department head's recommendations for a project, call them up and ask WHY. You might be amazed at how awesome the answer is, and it might even persuade you to put away the "my way or the highway" stamp.

Re:A Lot of Silliness, and Two Spectacular Points

[johnjaydk]

The problem is, that's at odds with the "if it ain't broke don't fix it" wisdom.

That guy is my boss: We don't do patches, they only break stuff. Build a stable system and never touch it again until we throw it out (years later).

Just today we had a box owned: Red Hat 7.3 (unpatched) and Tomcat 3.3. It took two years of neglegence with that box. We have other box'es that's older than that ..

Right now we are (i shit you not) migrating our self-service system to .not, including access to all your phone records. Combine loads of personal sensitive material (who calls who) with MS servers and anti-patching strategy. Of course the ISP section of the company is moving in exactly the opposite direction MS to Linux.

Chinese Slave Labor

[Anonymous Coward]

The problem is that American politicians want to integrate the Chinese economy and our economy. In this combined economy, whenever Beijing intervenes in the Chinese economy, that intervention also affects the American economy. Yet, American politicians continue lying, "The USA has a free market because Washington does not intervene."

What we should do is to terminate all trade (including goods and services -- which includes labor) with mainland China. Ditto for India and Mexico. Until the Chinese, the Indians, and the Mexicans commit to free trade and Westernization, we do not trade with them.

Further, we deport all the H-1B workers.

Re:Chinese Slave Labor

[Anonymous Coward]

Further, we deport all the H-1B workers.

Tell you what, at this point, there's no point in doing this. Instead, I say we let these people come over and enjoy our crappy healthcare, crappy retirement fate, crappy work environment, and crappy housing situations, while India accepts the same number of people to take their American money and live like kings (or at least moderately wealthy princes) over there. Or maybe I'll go move to Spain. Work all morning, work in the evening, sleep all afternoon. Sounds like the good life to me. Or Austraila, being "down under" sounds like fun, and the accent is pleasing to me.

The balance would instantly snap back to normal if people were allowed to be as global as companies Until then, the companies will simply move from population to population leaving a mess of people who are no longer trained to do anything but the jobs that the company took with it as it goes. Countries make loud noises about people "stealing the jobs of locals" but when IBM packs up a call center and moves it, a few million dollars to the local politicos' campaign funds and suddenly they're not stealing jobs, they're simply doing that thing corporations do.

Re:Chinese Slave Labor

[tarunthegreat2]

Of course, I'm feeding a troll here, but India has been on a path to "westernization" for about 200 years now, ever since the British East India Company first set foot on the shores of Calcutta. India alreasy has a "western-style" political and economic system, so STFU. Our laws are based on British Common Law - newsflash - so are American laws. We VOTE our leaders in to power. When two companies have a dispute ove a contract they go to court. When we want to make laws, they have to passed by two houses of parliament. Parliament happens to be this big place where elected representatives gather - to pass laws. Oh, and the unofficial offical language of India is ENGLISH. Imposing this system on a culture which has been transforming and transitioning for the past 1900 years going from Hindu - to Buddhist- To Muslim - to British is going to produce results which will be very different from what a pitifully young country like America isn't used to seeing. So just get the fuck over it. Finally this commitment to free trade that you talk about - The developing countries are ready & waiting for it. It is America that can't handle freeing trade in agriculture and industries like steel. It is USA which puts quotas on garments. Look up any textbook: quota != Free Trade. If you think an Indian software engineer is cheap, wait'll you discover the price of an Indian orange, or an Indian T-shirt. But you won't know about those because trade in those items is not "FREE". And it's not free because the EU and America want it that way. Because Billy Bob with-mouth-in-straw living in a redneck county of a red state just voted the current monkey into the white house. Now go back guarding the bridge, trolly-wolly.

Re:MAGIC, COMPLEXITY ARE INCOMPRESSIBLE

[Headw1nd]

This push to make [for the user] simple what is after all increasingly complex, can only hide, not eliminate the role of the nerd class, a role the article disparages because nerds are presumed, as the inventors, to have foisted off complexity on the unwitting public.

I think the consideration you should take in that article is distiguishing what we will call the "common nerd" and the "ubernerd".

When a technology is in an underdeveloped state, the use of that technology is very complex. It requires a certain level of knowlege to do anything, thus the employment of large numbers of people whose primary asset is the knowlege of technology X. As these are the gatekeepers of the technology, the interface is designed with their needs and knowlege in mind. As the technology develops, the interface to that technology simplifies. The gatekeepers obsolesce, even though the technology has grown more complex. Certainly, very talented individuals are at work behind the scences developing and maintaining the technology- but they are behind the scenes. You don't need to deal with them to use the technology.

I don't think the intent is to blame complexity on nerds, but to point out that most new technology is designed for consumption by nerds. The nerds are still gatekeepers, and access to technology in nerd-mediated. Today we see a huge demand for the "common nerd" in a very public role: maintaining IT systems. In the future, the gatekeepers will, like in technologies before, obsolesce. The "ubernerds" of the future will still be at work, but far from the public that uses their services.

forest through the trees ...

[wobblie]

Main problems in IT in the US:

1. Companies have gotten to big
2. Companies try to centralize everything, instead of delegating duties to competent people (this practice also encourages hiring incompetent people). Competent IT workers are also very unhappy when they hands are tied by bureaucratic BS
3. relating to (2), companies don't give raises or benefits anymore, which causes competent / motivated workers to hop jobs to get increases in pay
4. People doing all the "centralizing" are ignorant of standards

How aerospace does it

[Animats]

Aerospace has dealt with high complexity for decades, rather more successfully than the IT industry. Here's how.

• Interface specifications dominate If it doesn't work the way the spec says it does, fix the box, not the spec. If A won't talk to B, run the tests to check compliance with the spec. If you can't tell who's at fault, the spec is broken. This is why you can swap a Pratt and Whitney engine for a Rolls Royce engine.
• The buyer, not the vendor, decides what is a "defect". One of the fundamental problems in IT is that vendors have sole discretion to decide what is a defect and what isn't. That doesn't fly in aerospace.
• Fix blame. In aerospace, people get blamed for screwing up. You do not want your name or the name of your company to appear in an NTSB crash report. If you screw up big time, it will. Mistakes in aerospace are publicized. There's an NTSB database of 140,000 crashes. If it was a hardware failure, the vendor is named.
• Warranties have real meaning Airplanes come with good warranties, and so do all the parts that go into them. Commercial software doesn't.

This runs engineering costs way up, and the life cycles are longer, but in IT, most of the commercial products are sold in large numbers, so you get to spread that engineering cost over a large number of items.

It's time for computing to grow up and accept this kind of discipline. The automotive industry had to accept it in the 1960s, and cars got much better within a decade.

Religious awe

[TapeCutter]

The feeling of religious awe when looking at something with the beauty and complexity of DNA is the same regadless of your belief in God or lack thereof. Assigning "miricale status" to DNA gets logically messy since if DNA is so "special" that God must have developed it, then who developed the "special" God. The usual answer is that God's simply states "I am", but if God "just is" then why can't the Universe "just be"? Scientific dogma says nothing about the existence of God. It's just that as far as explaining how the Universe works, God is considered irrelevant.

Middleware...

[TapeCutter]

I have worked in a Telco on a large job dispatch system and what the parent says is spot on. I found that the middleware was indespensible since various changes would impact the transaction formats. The middleware was used to translate transactions into various versions so that upgrading the whole system could be managed one app/sub-system at a time. I don't know of any simpler way that gives you the flexiblity to "evolve" a system regardless of project deadlines being met or not. However the "evolution" by default gets more complex since you can easily add a field to the end of a transaction but it is difficult to remove them. Instead people tend to recycle old fields, ignore them or do something "clever" like variable length field with sub fields, all sorts of arcane rules and band-aids some going back 20 years! A Telco system keeps it's history in its transaction formats (kinda like DNA). Even though the original app may have died 3 generations ago the ability to "talk" to the system is dependent on the original transaction formats. This is because replacement systems are required to "fit in" with the existing formats either directly or by using middleware. In effect a new app (or the middleware) inherits the DNA from the app's it replaces or talks to. It then starts adding some of it's own mutations.

Re:you should have prototyped the argument types

[MagikSlinger]

I didn't have time to read the whole Pike manual, but from what I can tell, it's C++'s run-time type system implemented in a scripting language setting. But the thing I notice was that you can (and usually do) declare types, even if it's just the most general type that is accepted.

I remember in Comparative Programming Languages playing with an ML variant called Miranda. All the joys of LISP's data structure manipulation but the power of types. In fact, I remember Miranda could do some seriously cool stuff with the type signatures for functions to implement polymorphism more akin to Prolog than a functional language.

Declaring types is good. Making promises between modules and objects is good. LISP and SmallTalk don't believe in this. They say "the programmer can do all that discipline themselves". Yeah, and they can also do their own memory management too.

I sometimes think the advancement of the LISP concepts has been held back by the zealot LISP purists. "You must use brackets, Infidel!! ALALALALALALALA!!!"

Re:Lisp, Smalltalk and complexity.

[MagikSlinger]

You know, most trolls attribute goatse.cx to the appropriate URL, because they know plagiarism is bad. Give credit where credit is due.

You wish I was a troll. I was some one who spent 2 years learning Lisp, getting really good at it then discovering that I was 2x to 5x more productive in Perl & C and 20x more productive in PROLOG (which the Lisp zealots sniffed at as an "inferior" by-product of Lisp).

The paper you site is laughable:

It might be because the subjects were self-selected. It might be because the benchmark task involved search and managing a complex linked data structure, two jobs for which Lisp happens to be specifically designed and particularly well suited. Or it might be because Lisp programmers tend to be good programmers. This in turn might be because good programmers tend to gravitate toward Lisp, or it might be because programming in Lisp tends to make one a good programmer.

To be fair, I left the last line in even though it's completely unsubstantiated. To be further fair, I cannot prove or disprove it. But let's continue with the meat of it: compilation time. Before posting my comments, I looked for a productivity report and all I could find were anecdotes and self-serving quotes on Lisp pages. The productivity is due to interactive debugging & compilation time!?!? I saw no mention of meta-programming as a productivity boost (a Faustian bargain one at that), and just a lot of crap about how Lisp is the only language that can do incremental compilation. Oh geez, give me a fucking break!

The reason I am so anti-Lisp is because I was a Lisp zealot until I took comparitive programming languages. I am probably one of the few Lisp detractors who have written Lisp programs that can verify the correctness of Lisp programs! I have written "self-modifying" Lisp code. Dude, I know Lisp better than some of its adherents. I've written my own Lisp interpreter from scratch using K&R C.

I have never found Lisp to be that productive. It forces me to think in unnatural ways to express my solutions. To be productive, I usually end up doing ugly things like writing procedural code. For power, I prefer PROLOG. For raw speed, I prefer C. For OO, well, I'm still looking. But the point here is what does Lisp offer anymore? List handling and garbage collection is now a standard part of all modern languages. Meta languages is a Faustian bargain where if everyone is on the same page, great, but if someone has to pick it up from scratch, one had to depend on the previous programmer organizing their code in an easily comprehensible manner (which they usually don't). Switching projects becomes a headache because now you have to learn that local dialect. In short, you got a Tower of Babel of language extensions.

I feel like Diogeneyse looking for a wise man in the city of Lisp zealots.