Slashdot Mirror
The Economist Tackles Complexity in IT
yfnET writes "In recent weeks, The Economist has run a number of articles addressing the ever-increasing complexity of software systems. The magazine, with typical Economist wisdom, casts an eye towards past human endeavors for lessons on how today's IT industry can succeed in dealing with complexity. As part of last month's extensive survey of information technology (see Related Items sidebar), the magazine offers insight on the limits of real-world metaphors, the perils of managing a rat's nest of obsolescent systems, and the need for 'disappearing' technology. And hitting newsstands just today is an overview of development models for increasingly large and unwieldy software projects. Among other things, this article compares the open source model to Microsoft's efforts using a quasi-open license. It also describes the 'agile' programming movement and its potential to keep even the most gigantic of projects under control."
Powerful Languages Like Smalltalk, and Lisp help one handle complexity.
Aren't these the same people who tried to tell us that "comparative advantage" meant we should give up our manufacturing and all get degrees in high tech? Why should we believe ANYTHING they have to say after they've lied to us for so long?
Economics is a religion- and a failed mythology at that. Economists need to learn to examine and reconfigure their basic axioms before ANYBODY should ever listen to them again.
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
There's a serious problem with agile methodology and outsourcing (I didn't see any articles on Economist.com related to outsourcing, but may have missed it as I gave it only a cursory look).
Large and unwieldy projects benefiting from agile methodologies? Yeah... when you have easy communication between the "customers" (business partners) and your IT staff.
How does that happen when your developers are thousands of miles away, in a different timezone, with a totally different culture, and don't speak your native language?
Diplomacy is the art of saying, "Nice doggie!" until you can find a rock.
Translated from a a french newpaper: :-)
I'm not sur it will help...
"The historical operator, his Orange subsidiary company and Bouygues Telecom have all three known, in the last seven months, of the significant breakdowns, depriving their customers of telephone service for durations varying from a few hours at almost two days.The succession of these events clarifies the increasing complexity of the systems of telecommunications and the difficulties raised by the needs, news, of interconnection of many heteroclite networks". A minister in the french government as asked for a study about all these failures
le souvenir d'une certaine image n'est que le regret d'un certain instant (M.Proust)
Hardware complexity can be reduced with homogenous environments.
Good planning, documentation and standards reduce complexity in software.
How much more do they have to complicate the issue?
People tend to make things over complex if you let them. Bosses want that technology they read about in whatever PHB read these days. Such and such wants some TLA to do some other TLA.
I know it's hard but you have to tell them that these things don't add any value in and of themselves. You want the simplest possible system that will solve the problem at hand. Really, nothing more, don't implement something because you may want/need it tomorrow 'cos when tommorrow comes it won't be right (and if it is, hey, you can implement it then).
----
I'd say just hire good people and keep the managers focussed on just a few projects, but this seems to be beyond the capabilities of most companies. So you end up with programmers who write Java like it was fortran and managers who juggle so many projects that they barely know your name much less what you do for the company. There doesn't seem to be a fix for this which companies will be willing to accept.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
This push to make [for the user] simple what is after all increasingly complex, can only hide, not eliminate the role of the nerd class, a role the article disparages because nerds are presumed, as the inventors, to have foisted off complexity on the unwitting public. Was it Heinlein who said that "any sufficiently advanced [or was his word complex] technology is indistinguishable from magic"? The wish, on the part of typical users and marketers, that all the wonders of our age and those ages coming next should all just work like magic will in fact only ADD the complexity of UI technologies that are good at hiding the guts of the systems we depend upon. The the engineers and technicians will be as needed as ever and get even less sympathy from a public that never sees directly what it is that the "nerds" are doing for them.
SLASHDOT: news for people who can't concentrate on work or have no life at all and got tired of yelling back at the TV.
Complexity in IT isn't going to go away. In fact, I'd argue it is a necessity. There are some tasks that simply require complex systems and those complex systems require complex data and/or complex user interfaces.
"The market alone cannot provide sufficient constraints on corporation's penchant to cause harm." -- Joel Bakan
So a lot of this space was spent explaining to Joseph P. Siquespack, Esq. what a "protocol" was and the like, but there were two points in here that I'm really glad my great-grandboss might be reading:
Neither of the above are impossible goals! They can be done with a little thought and elbow grease. And the great part is, they're probably already being done! Next time you're reading over your IT department head's recommendations for a project, call them up and ask WHY. You might be amazed at how awesome the answer is, and it might even persuade you to put away the "my way or the highway" stamp.
adam b.
Sure, information technology is becoming ever increasingly complex; but it is also a great opportunity for companies that have the management able to deal with that complexity to excel and outperform their competition.
What we should do is to terminate all trade (including goods and services -- which includes labor) with mainland China. Ditto for India and Mexico. Until the Chinese, the Indians, and the Mexicans commit to free trade and Westernization, we do not trade with them.
Further, we deport all the H-1B workers.
Engineers love to break things into smaller parts, each part serving one and only one function, like pulleys, shafts, rotors, etc.
For really effective design each part has to serve multiple functions, like evolution is able to do: The human mouth can be used to eat, breathe, talk, etc etc.
That's why a robot can't compete with an animal- In a robot each part usually severs only one function, making the machine inefficient as a whole.
This problem is just magnified in computer software and will only get worse unless engineers start changing their tune. I think the worst offender of this philosophy is object oriented programming: It's the ultimate embodyment of this philosophy- Most big object-oriented software have only about 2-3% of code that performs any real work, with the rest only is window dressing to fulfill the engineer's urge to "modularize".
The best software I see seems to be written either in more pragmatic procedural styles, or uses better mathematical underpinnings for its structure, like you'll find in functional programming languages (Haskell, Lisp, etc.)
My apologies for living up to my user name!
Conrad Barski
The problem with trying to compare the software industry to other endeavors of human experience is that in the realm of the computer/electronics industry that most software developers are dealing with, you are near the pinacle of dealing with abstraction of complex systems. While there may be other very complex systems that on a large scale can compare to some computer networks or CPU designs, computer science is the practice of dealing with abstractions on many levels, sometimes simultaneously.
Indeed, electronic state machine digitial computing devices (also called computers) have proven so successful, with the software abstraction dealing with the various levels of abstraction, that they are used in the controlling of other complex systems, from air traffic management, urban water system management, freeway traffic monitoring, and law enforcement dispatching. You've seen them, and they are out there.
Some very talented engineers have done a surprisingly good job of simplifying the tasks and reducing the abstractions to the point that all you need to do for the most part is plug it in and watch the gizmo do its thing. What this article in the Economist seems to be doing is complaining that the job isn't finished, and that complexities in setting up a computer system for some project is more difficult than it should be. That is primarily due to the fact that the author is using products that don't comply with standards (a real problem if standards don't exist yet for a certain concept or technology) or they are using the wrong tool for the job, like using a hammer to put in a few screws. Sure, it will work, but it is aggrivating and sometimes takes quite a bit longer to get the job done, and can damage things around it as a side effect. How many software/electronic gizmos out there do you know get used like that?
While I'm willing to acknowledge that I don't know everything there is to know about the management and organization of complex systems, I would be more inclined to get the opinion on such a subject from a computer programmer than from a plumber.
All my programs would have a great interfaces if they only had 1 boolean variable in it. A light switch is very simple. But try connecting it to the internet, email a friend and do your accounting with a light switch.
Ever heard of BugMeNot?
To paraphrase oo doesn't make messy code, messy people who mis-use oo make messy code.
I think the main problem with what, aparently all of us, have seen with oo code, is the universities. The coding style that is taught in universities makes for really ugly, unmaintainable code.
If you cram 100 abstractions and modularization into a project in university, you get an A and every one says how clever you are to have used all of these features in your project. Do the same in the real world and you are left with unmaintainable blech.
People have to learn that the various oo features are there to help them simplify their code, not to make it more complex. If using a particular modularization technique, say an interface, doesn't remove complexity then don't add it.
Another really bad thing that people do is add code for some unspecified future purpose. Maybe they are creating a class that does some math, they need an add method and a subtract method, so they think, what the hell, I'll add a multiply and divide too. Why? All this does is make the code less readable. Never implement anything that you don't need right now.
I hear this every damn day. 'We need to make it simple', 'It is a simple service' or 'It is a low option service'. This may work fine for the sales and marketing drones that make their commission off selling unnecessary services to uninformed customers, but as long as there is _choice_ out there, the backend is going to be complicated, and somebody, somewhere is going to have to known how all of it (or at least a major part of it) works.
Try all you want, but unless the entire IT industry decides to switch to one massive global device that we all plug _directly_ into, you can't make video conferencing/VOIP/disaster recovery/etc through 2 LANS, 3 Service Providers and 10 different security layers a 'green/red' push button operation.
I've gotta go get drunk now....
-- I care not for your foolish signatures.
Yes it was fascinating reading. I was so looking forward to that last article (Maybe I will try bugmenot, been meaning to.) I even for a split second thought about giving up my information and registering. But then, I thought... you know, if I was having trouble getting people to register to my website, wouldn't a trail of breadcrumbs in a ./ article do the trick?
Sorry, no nibbles from this fishy.
Someone had to do it.
> That doesn't mean that all layers are actual code,but can be just proxies for a possibility.
.NET, it has 18 levels of inheritance above it, all designed for these kinds of hooks. Are you able to break the concept of a button into 18 separate levels? I can't. Do you think this is a good design? I don't.
If you look at a button control in the latest iteration of
> an engineer makes things as modular as they need to be and no more. But they design so that they can replace with better technology later.
If that was really the case, I think that line of reasoning would be justifiable. Personally, though, I find a lot of the complex software written that I have to deal with is filled with good modularization intentions that would have been a lot less problematic if people had thought to themselves "Don't write what I don't need right now, today."
> You don't know what you are talking about
You make some good points. Why follow them with attacks?
I like how this post instntly soawned a flmewar pn economy, mostly fueled by people that either don't understand
1. that The Economist is a magazine, not the people referred to in the sentence "economists say".
2. anything about economics.
3. both.
The Economist is a very good news magazine full of reasonable articles and opinions, in all senses of the word "reasonable". There is not enough praise on Slashdot to make it justice. You should all subscribe, assuming that you are interested of knowing what happens in the world from a political, economical and yes, technical standpoint.
Code for rm could be implemented in C with handfull of lines. Todays alternatives take thousands of lines of code, but to an end user the second alternative is simple. User doesn't have to know what the commands are, just toss the file away, as you would with solid objects.
So there we have it, simple problem becomes complex from implementation point of view. I once had a customer who joked when I delivered them a new system that calculated the price and basic layout of the systems they were manufacturing, that inspite the fact that now it took less than tenth of the time to do the calculations, that we could still improve it so it would do the calculations when he pressed a button while thinking of something else.
It should also be noted that what was impossible few years ago is now possible, because of improvement to hardware. This adds to the layers of complexity, because implementers can actually use modular approach instead of optimizing at lowest possible level.
Main problems in IT in the US:
1. Companies have gotten to big
2. Companies try to centralize everything, instead of delegating duties to competent people (this practice also encourages hiring incompetent people). Competent IT workers are also very unhappy when they hands are tied by bureaucratic BS
3. relating to (2), companies don't give raises or benefits anymore, which causes competent / motivated workers to hop jobs to get increases in pay
4. People doing all the "centralizing" are ignorant of standards
-
Interface specifications dominate If it doesn't work the way the spec says it does, fix the box, not the spec. If A won't talk to B, run the tests to check compliance with the spec. If you can't tell who's at fault, the spec is broken.
This is why you can swap a Pratt and Whitney engine for a Rolls Royce engine.
-
The buyer, not the vendor, decides what is a "defect". One of the fundamental problems in IT is that vendors have sole discretion to decide what is a defect and what isn't. That doesn't fly in aerospace.
-
Fix blame. In aerospace, people get blamed for screwing up. You do not want your name or the name of your company to appear in an NTSB crash report. If you screw up big time, it will.
Mistakes in aerospace are publicized. There's an NTSB database of 140,000 crashes. If it was a hardware failure, the vendor is named.
-
Warranties have real meaning Airplanes come with good warranties, and so do all the parts that go into them. Commercial software doesn't.
This runs engineering costs way up, and the life cycles are longer, but in IT, most of the commercial products are sold in large numbers, so you get to spread that engineering cost over a large number of items.It's time for computing to grow up and accept this kind of discipline. The automotive industry had to accept it in the 1960s, and cars got much better within a decade.
Eh... unconvincing. Organic systems are great if you don't mind a little fudging of the numbers. For example, human memory is notoriously unreliable. Combining features tends to create single points of failure and organic systems have a very narrow optimal operating environment. Try dropping the temperature by a hundred degrees or so and see how you fare. The lack of validation, if you will, on inputs, creates a significant susceptibility to viruses, bacteria, food poisoning and the like.
There are plenty of successful organic systems, but there's a whole lot more failures, which you don't see, because they are all dead. And failure is pretty expensive.
Also, you aren't taking into account the fact that complex computer systems perform sophisticated analysis on other complex information systems that organic systems are ill-equipped to handle. They are really best at operating on other organic systems. So if you were to design organic financial software, it would work best on an organic financial system. When you want near-instant, reliable, repeatable and accurate information, an organic system won't work for you at all.
Its like the difference between designing a bridge over a river, and waiting for a rockslide to fill in the river.
"It's Dot Com!"
The premium article, naturally, is the worst of the bunch. It reads like the introduction to a Software Engineering textbook, padded with endless definitions of "regression testing", "open source", et al and contains almost no actual content. It may be interesting to the Economist's readers, but I doubt it is to very many people on Slashdot.
you can't slam modularity w/o being self-inconsistent.
why? because to slam something means to put yourself outside of it in the first place (and pour invective on it in the second). putting yourself outside of something (drawing a boundary around it) is the essence of encapsulation, which is one of the techniques of modularity.
so, let's be charitable and assume you don't want to slam modularity full-force, but rather some of the other techniques for modularity that often find themselves badly understood and/or badly implemented.
btw, lisp is modular, object-oriented, etc etc, too.
as for the invective you poured, not bad. just don't waste it in the gutter the next time 'round.
... which is a magazine, and not to be confused with the profession. In fact, I know of a few newspapers with that sort of name: The Economist and Sun, the Economist-Tribune.
Newspaper and magazine names are usually historical, and the words they contain have often changed meaning. For example, there is a local newspaper north of Toronto called "The Liberal". This newspaper has, for example, no ties to the Liberal Party, is not particularly philosophically Liberal (whatever that means), and is basically a local community newspaper that was named almost a century ago {more or less). What's in a name? Often, when it comes to media titles and tag-lines... nothing.
Just rememeber, "fair and balanced" is the tagline of Fox News.
i - This sig provided by
I think the problem is usually implicit type conversions rather than operator overloading. In particular, allowing almost-random conversions between a concrete type such as a number and a string form is just asking for trouble: it's great for quick 'n' dirty scripts, but a child's toy in a grown-up world for serious projects.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Complexity may be inevitable, but for a moment lets put on our economist hat and look at what motivates companies to tolerate complexity. While some complexity in inherent, I believe that a lot of this complexity is often accidental, sometimes born out of naivete and occasionally deliberate. However, at the end of the day companies are not in a big hurry because the added cost of complexity often doesn't cost them anything, it is merely added to the cost of doing business and paid for by the consumer.
Furthermore, in some cases complexity can be quite convenient. For example, in my city the gas company outsources its billing so that one company provides gas and the other bills for it. Sounds great, but when ever there is a discrepancy between the bill and the amount of gas you've used, good luck trying to get something resolved. In my case the gas had long been cut off but the billing continued. After fighting through automated telephone systems to speak to real live people they quite happily point the finger at the other guy and leave it at that. In effect the complexity that they have engineered into their business model provides them with a very handy excuse for inaction.
Perhaps you expect capitalism and free markets to address this problem? I think that the much of our technical infrastructure is so monstrous and well entrenched that, just like your local utilities, there no room for upstart firms to enter the market. For the consumer (end consumer or other business entities) there are very few legitimate choices. More often than not I think that complexity is often just another excuse to maintain the status quo and an excuse that is held up to 'educate' consumers not to expect any better.
The feeling of religious awe when looking at something with the beauty and complexity of DNA is the same regadless of your belief in God or lack thereof. Assigning "miricale status" to DNA gets logically messy since if DNA is so "special" that God must have developed it, then who developed the "special" God. The usual answer is that God's simply states "I am", but if God "just is" then why can't the Universe "just be"? Scientific dogma says nothing about the existence of God. It's just that as far as explaining how the Universe works, God is considered irrelevant.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
I have worked in a Telco on a large job dispatch system and what the parent says is spot on. I found that the middleware was indespensible since various changes would impact the transaction formats. The middleware was used to translate transactions into various versions so that upgrading the whole system could be managed one app/sub-system at a time. I don't know of any simpler way that gives you the flexiblity to "evolve" a system regardless of project deadlines being met or not. However the "evolution" by default gets more complex since you can easily add a field to the end of a transaction but it is difficult to remove them. Instead people tend to recycle old fields, ignore them or do something "clever" like variable length field with sub fields, all sorts of arcane rules and band-aids some going back 20 years! A Telco system keeps it's history in its transaction formats (kinda like DNA). Even though the original app may have died 3 generations ago the ability to "talk" to the system is dependent on the original transaction formats. This is because replacement systems are required to "fit in" with the existing formats either directly or by using middleware. In effect a new app (or the middleware) inherits the DNA from the app's it replaces or talks to. It then starts adding some of it's own mutations.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
I didn't have time to read the whole Pike manual, but from what I can tell, it's C++'s run-time type system implemented in a scripting language setting. But the thing I notice was that you can (and usually do) declare types, even if it's just the most general type that is accepted.
I remember in Comparative Programming Languages playing with an ML variant called Miranda. All the joys of LISP's data structure manipulation but the power of types. In fact, I remember Miranda could do some seriously cool stuff with the type signatures for functions to implement polymorphism more akin to Prolog than a functional language.
Declaring types is good. Making promises between modules and objects is good. LISP and SmallTalk don't believe in this. They say "the programmer can do all that discipline themselves". Yeah, and they can also do their own memory management too.
I sometimes think the advancement of the LISP concepts has been held back by the zealot LISP purists. "You must use brackets, Infidel!! ALALALALALALALA!!!"
The bitter lessons of a veteran coder: http://bitterprogrammer.blogspot.com
Most people can only keep/visualize/instantiate about 7 distinct objects in short term memory at any one time. You might want to stick to a max of 5 to cater the slightly below average. (Maybe someone might want to be a hero and try increase the population average to 10 items or something ;) ).
There are workarounds if you can easily get people to group a bunch of items as one object.
If people have to remember 7 or more important things during the learning period where short term memory is used then the thing is hard.
Given the amount of choices and options possible with software, it's going to be hard.
So the workaround is to split the learning phases to small absorbable chunks and give time between these phases for people to commit them to long term memory.
If you use common/defacto UI standards, many of the users would be familiar them and so the effective number of learning phases required drops.
I'd prefer something like: "Don't bother implementing things you don't know you'll ever need." That still rules out the time-wasting feature creep, while acknowledging that it's often far more efficient to plan for likely future developments from the start rather than constantly evolving a system without any sort of "grand plan".
All evolutionary development causes overhead, most of which is unnecessary if you can anticipate major future developments with reasonable accuracy. Despite all the nay-sayers, I have never yet worked on a real project where this was not the case.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
On the lighter side of things, I find it funny that IT departments often "re-invent" themselves by changing their name and acronyms--a complexity in themselves. You find acronyms such as MIS, CLC, ITC, CIT, CITS, etc. But in essence they all stand for the same thing.
My university's acronym is CITS (Computer and Information Technology Systems), and before that they were just the CLC (Computer Learning Center). Imagine if they kept the name "Learning" in the acronym somewhere, it could've been: Computer Learning and Information Technology Systems (CLITS). But somehow I don't see that happening.
Linux at home
This has to be a wind up!! East Timor! The Australian Government of Gough Whitlam stood by in 1974 whilst the Indonesian Government invaded East Timor then condoned Indonesia's control of the country. Until 1999 the Australian governments from both parties were happy to keep diplomatic ties with Jarkarta and sell weapons, and military training to the TNI (indonesian miliary). Weapons and training they had been made plainly aware was being used to suppress civilan dissent in East Timor, including torture and murder.
The brief intervention in East Timor was aberrant, rather than standard behaviour for Australia and was a tiny step back towards reasonable and respectable treatment of a loyal wartime (WWII) ally, East Timor.
Perhaps while you take the time to "condemn India, China and Mexico" you might read an international news paper... this is all in the public record.
Who's with me?! I SAID... WHO'S WITH ME!!??