Slashdot Mirror


Lessons From a Decade of IT Failures (ieee.org)

New submitter mixed_signal writes: IEEE Spectrum has an online set of articles, or "lessons," on why big IT projects have failed, including analysis of the impacts of failed systems and the life cycles of failed projects. From the summary: "To commemorate the last decade's worth of failures, we organized and analyzed the data we've collected. We cannot claim—nor can anyone, really—to have a definitive, comprehensive database of debacles. Instead, from the incidents we have chronicled, we handpicked the most interesting and illustrative examples of big IT systems and projects gone awry and created the five interactives featured here. Each reveals different emerging patterns and lessons. Dive in to see what we've found. One big takeaway: While it's impossible to say whether IT failures are more frequent now than in the past, it does seem that the aggregate consequences are worse."

32 of 118 comments (clear)

  1. LESSON NUMBER #1 by Lisias · · Score: 5, Insightful

    You will never write good code without writing bad code first.

    And you will never stop writing bad code without being accountable for the results of writing bad code.

    Experience is not how long you spend writing code. Is about how much time you spend fixing code, learning how to avoid having to do it again,

    --
    Lisias@Earth.SolarSystem.OrionArm.MilkyWay.Local.Virgo.Universe.org
    1. Re:LESSON NUMBER #1 by Dragonslicer · · Score: 4, Funny

      Also summarized by the saying, "Some people have 10 years of experience. Other people have 1 year of experience 10 times."

    2. Re:LESSON NUMBER #1 by MoarSauce123 · · Score: 2

      In theory, Agile does produce better software. In reality, all that management takes away from the one day long Agile Coach session is "If we tell DevQA to be Agile we can deliver software faster!" So release dates are set even more arbitrarily ignoring any amount of work that needs to happen no matter which approach one takes, which means nothing more than increasing workload and stress, forcing to cut corners, encouraging the business to not make any decisions or commit to anything (because we can iterate over it and do it next time). There are surely plenty of Agile implementations that worked out fine, but most that I came across deliver the same or worse results than before while increasing the amount of rework drastically, and especially with scrum introduce excessive overhead with meetings that require so many resources that teams have to hire scrum masters to manage all that overhead.All the while management stays stuck in old patterns, scheduling decisions to happen during the monthly managers meeting (very agile!). At the same time, business analysts are no longer allowed to write down any requirements because Agile supposedly forbids any kind of documentation. Developers just start coding the way they like with the goal to get something to demo as quickly as possible. That facade impresses the product owner and inclusion in the release coming up a week later is demanded. Lastly, when things fail product owner, managers, developers, support, and C-levels turn around to QA and ask "Why didn't you test this right?" Agile is the death of quality!

  2. Reasons things fail by JustAnotherOldGuy · · Score: 4, Insightful

    There are a million reasons why things fail, but they fall into a few broad categories:

    Failure to plan ahead ("we'll worry about demand later, once we have a viable product"),
    Failure to adapt to changing circumstances ("buggy whips will always be essential to our lives"),
    Failure to avoid predictable or likely failures (i.e. "develop a perpetual motion machine")
    Failure to manage resources properly ("have everyone working on this and not that).

    There are millions of others, but most of them fall under one of these primary categories.

    --
    Just cruising through this digital world at 33 1/3 rpm...
    1. Re:Reasons things fail by Archangel+Michael · · Score: 2

      who cares it isn't really my money being spent

      While I suspect that you're a tad sarcastic here, I would like to point out, yes, it is your money being spent. It is your money. It is my money. It is our kids and grandkids money (debt).

      The fact that I have to say it is really sad. And yet, enough people don't care that it is still a reality. "Not my money" is a big fat lie.

      --
      Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
    2. Re:Reasons things fail by Rob+Y. · · Score: 2

      Here's another:

      Decision makers deferring technical decisions to project managers with a stake in defending their past bad decisions. Results in doubling down on mistakes long after they proved to lead to dead ends.

      And another:

      Corporate business plans focused on 'selling the company' or 'an IPO in a few years'. Creates a perverse incentive. An incomplete project with the promise of 'enormous success just around the corner' is an easier sell than the finished, quantifiable result. So the above 'doubling down on failure' isn't failure at all. It's just a series of bumps in the road that are behind us now. And success - just around that corner we've just turned.

      --
      Posted from my Android phone. Oh, I can change this? There, that's better...
    3. Re:Reasons things fail by JaredOfEuropa · · Score: 3, Insightful

      I would add to that: stakeholder apathy, or failure to generate sufficient buy-in. I have seen this apathy in big projects from everyone involved: the project team, business stakeholders, steering group, sponsors, focus groups, and vendors. There are many small things that cumulatively will cause this; it certainly doesn't set in only after the project is already on the fast track to failure. The result is sloppy work, an increased tolerance for shortcomings (in systems as well as people), mutual acceptance of missed deadlines and broken agreements, leading towards a project where people will be happy to deliver anything, no matter how crappy, just to be done with it.

      --
      If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
    4. Re: Reasons things fail by Anonymous Coward · · Score: 3, Insightful

      That's actually 2 more entries, not just 1.

      1) Poorly defined project specifications (the specs say to build a Chevy, but the customer/user is expecting a Ferarri)
      2) Scope creep (the customer asked for a no-frills Ford, then says they need air, cruise, and a high-end stereo/GPS)

    5. Re:Reasons things fail by Noah+Haders · · Score: 2

      I think the point is, one driver for govt IT failures is lack of accountability or concern over finances. If the govt wastes a lot of money, no worries, there will be more money next year. Maybe a head will roll just for the good publicity. but otherwise things keep moving forward BAU.

    6. Re:Reasons things fail by LoyalOpposition · · Score: 5, Informative

      While I suspect that you're a tad sarcastic here,...

      I agree with you that Impy the Impious Imp was speaking sarcastically. It reminds me of the four types of spending Milton Friedman classified, and the value of its results. I'm working from memory here, so please forgive me my mistakes. Type 1 spending is where you spend your own money on yourself. This type of spending has the greatest results because you take care to spend as little as possible, and to purchase the things you want most. Type 2 spending is where you spend someone else's money on yourself. This has worse results than type 1 spending because, while you still take care to purchase what you want most, you are more likely to try to spend the entire amount. Type 3 spending is where you spend your money on someone else. In type 3 spending you try to conserve funds, but rather than getting someone what they most want, you get them what you think they should want. Type 4 spending is where you spend someone else's money on someone else. In type 4 spending you neither try to conserve money nor purchase what's most needed or wanted. I interpret Impy to be saying that all government spending is type 4 spending.

      ~Loyal
       

      --
      I aim to misbehave.
    7. Re:Reasons things fail by __aaclcg7560 · · Score: 5, Interesting

      Not at my government job. A newly hired I.T. guy who expects to get paid for doing nothing because he thinks this is a "government job" will find himself on the unemployment line within a month. Most of my coworkers are ex-military who tolerate zero crap from each other. We worked very hard to provide the best services to our users despite taking abuse from the public for being government employees.

    8. Re:Reasons things fail by JaredOfEuropa · · Score: 2

      Failure to control comes in different flavours: one is scope creep resulting from a failure to control requirements. Another cause is controlling requirements too tightly. I have never seen a large IT project defined correctly and completely from the get-go; during project execution there will be new insights, errors and inconsistencies are uncovered, and external cirumstances will change. Having control of your project means that you are prepared for change, equiped to separate necessary or sensible changes from unnecessary ones, and able to manage those changes and the impact on budget and timelines. The answer certainly isn't to clamp down on all suggested changes, that kind of stubbornness may lead to a successful project that no one wants, or a project done fully to spec but utterly useless in real life.

      --
      If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
    9. Re:Reasons things fail by joss · · Score: 2

      Do you think the people running corporate IT programs are spending their own money ?

      No, ok.. well, do you have any evidence that large government run IT programs are more prone to failure than large commercial sector IT programs ?

      I think it's more a question of people are not very smart and large scale software development is hard.

      --
      http://rareformnewmedia.com/
    10. Re:Reasons things fail by plopez · · Score: 2

      For as much money as the Gov't has wasted, I have seen the private sector waste more. I have work for small, midsized, and fortune 500 companies. I have worked for the US government, and have helped people, and state government. An there is nothing like a fortune 500 company when it comes to throwing money down a rat hole. They waste money like crazy and then lay people off. Small and mid-sized businesses are the most efficient and nimble; and the difference between working for state governments and the Feds is that state governments seem to be more politically driven and less performance driven. See Scott Walker as an example.

      --
      putting the 'B' in LGBTQ+
    11. Re:Reasons things fail by chipschap · · Score: 3, Insightful

      As an ex-government person I support your statement. There were/are some of us, quite a few in fact, who actually did care and actually believed in the mission and tried despite all obstacles to carry it out. We understood that it was American taxpayer money we were spending and that we were morally accountable.

      The biggest problem I saw was the army of Beltway Bandits anxious to land contracts and then bill for millions while producing nothing of value.

    12. Re:Reasons things fail by JustAnotherOldGuy · · Score: 4, Insightful

      This is an older article, but according to the research, 68% of IT projects fail.

      I'm not surprised. The more people involved and the more moving parts you have, the less likely anything will ever come to completion.

      SAP projects are a perfect example of this. Those clowns could fuck up a guestbook script, all 30 lines of it. By the time they got does it would be 550 megs of object oriented code (java, C++, Oracle, COBOL, and maybe some perl just to help make it unreadable).

      --
      Just cruising through this digital world at 33 1/3 rpm...
    13. Re:Reasons things fail by AF_Cheddar_Head · · Score: 2

      Having worked in/with the US military on and off for 30+ years I can honestly say I have never encountered this attitude or heard some say "Who cares it isn't really my money being spent."

      Now some contractors will try to rip off the government and perhaps they have that attitude but none of the GS civilians or GIs or the actual peon contractors have ever exhibited that attitude. A lot of them get pissed at waste because they know it is their own tax money being wasted.

      There is the bad thing where if you don't spend this year's budget you may start will less next year but normally they really do try to spend the money on necessities.

      YMMV

    14. Re:Reasons things fail by AF_Cheddar_Head · · Score: 2

      Beltway bandits AKA Contrators not the little guy contractor that is actually on the ground doing the work but the three-letter types. Most of the little guys doing the work are conscientious about doing a good job and saving money.

    15. Re:Reasons things fail by chipschap · · Score: 2

      Good point, thanks for posting the clarification. The little guys competed and survived on merit alone. They ended up doing a lot of good work for just enough money to get by[1]. The big guys had the right friends in the right places. They did a lot of bad work, if you can even call it work, and collected the maximum possible. Two different worlds.

      [1] There was one major exception to this, but I'll leave it unsaid and avoid all the flames from the SJWs.

    16. Re: Reasons things fail by Bender0x7D1 · · Score: 2

      2) Scope creep (the customer asked for a no-frills Ford, then says they need air, cruise, and a high-end stereo/GPS)

      Or, they say they don't need air, cruise and a high-end stereo and then complain it is too hot, doesn't maintain its speed and they aren't able to hear any music during their test drives.

      --
      Reading code is like reading the dictionary - you have to read half of it before you can go back and understand it.
  3. Software is not 'fire and forget' by roman_mir · · Score: 2

    The list of failures from TFA proves that software cannot be created and then not maintained, not constantly observed, not updated. Complex software that interacts with real life systems cannot be treated as if it is a 'fire and forget' thing. It is not.

    The "disappearing warehouse" case is perfect, nobody was keeping an eye on the system, nobody at all was actually personally invested, personally responsible. I have created a lot of software over my life, I built and own a retail chain management system, store management, supply chain management, customer relations management, logistics, shipping, payment, warehouse management and some other systems. These are used by medium sized companies (actually in the world they could be called small, only dozens of stores, only hundreds of employees and a hundreds of suppliers, but actually tens of thousands of SKUs)

    I know this: the owner of the business does not forget anything. The owner of the business is the person whose personal wealth is tied into the business, that person does not miss anything and if he or she is personally involved in the project, they understand the system (not entirely, but the parts that are important to running the business), no warehouse would go missing, no store would go missing, no supplier would go missing.

    The reality is that combining complex software with lack of personal responsibility leads to real world issues.

  4. It is all about project management by guus_deleeuw · · Score: 2

    It all comes down to the time, money, quality triangle. The more you focus on one aspect the more you lose on the other two. And because in the end projects are always focused on time and money, quality is almost always neglected. Especially in highly political environments where the ideas have to be implemented with low cost and this year projects are bound to fail.

  5. Re:Root cause analysis by tomhath · · Score: 4, Insightful

    Suggests that management hubris plays a big part in IT Failures.

    I think it's a combination of hubris and naiveté. Management and architects look at legacy systems and think all the complexity is unnecessary - that they can implement a "modern" system with the methodology that is in vogue (OOA/OOD, SOA, whatever). Anyone who tries to point out that the complexity is there for a reason is branded a naysayer and ignored. Years later management and architects are still struggling to deal with all the complexities they didn't want to see at the beginning, then the money runs out.

  6. Re:Dock the severance pay of the old IT guys / HB1 by __aaclcg7560 · · Score: 3, Informative

    Actually, you're the biggest idiot here. The OP was referring to HB1 pencils. Only the old I.T. guys knows what those are and how to use them.

  7. Re:Dock the severance pay of the old IT guys / HB1 by Tablizer · · Score: 2

    The OP was referring to HB1 pencils

    Hey, it's only a stereotype that visa workers are skinny.

  8. Reason 1: Magical Thinking by ErichTheRed · · Score: 3, Insightful

    I have worked for a lot of large companies, and one of the things I've seen cause a lot of failures is thinking a problem will disappear by throwing Magic at it.
    - Cripplingly-slow WAN speeds? Vendor X is the Gartner Magic Quadrant leader in WAN Optimization, we'll just use that! Here's $2 million, Vendor X. Just put it in, you're smart IT guys, how hard could it be?
    - Developers and IT guys are expensive. I know, let's call Infosys/Tata/Accenture/HP/IBM, all I have to do is write them a check and all my IT problems disappear offshore!
    - I don't want to pay for equipment. I know, let's put it in the cloud! The cloud makes all problems disappear for a low low monthly fee!

    I'm a pretty avowed generalist, but my two "specialties" are end user computing stuff and systems management. EUC is rife with magic solutions -- I can't tell you how many thin client/zero client/cloud desktop/VDI/Citrix/Whatever iterations I've been through where the CIO didn't realize that the problems don't go away. Problems just get moved around and may be more expensive to solve in the new configuration. Systems management is a whole other ball game. In this field more than others, vendors like CA, Microsoft and some of the startups have the art of the stunning sales demo down pat. As a result, people like me have spent untold hours and company dollars on expensive vendor consultants getting even a fraction of that sales demo working in the real world.

    I love the constant innovation that our field serves up, but one needs to temper that with the reality that most innovation is a rehash of something done before, with the underlying pieces improved. I think the IT field is long overdue for at least some standardization where we don't let vendors run the show.

  9. Mismanagement by DarthVain · · Score: 2

    You can't have mismanagement without "management".

    Anyway there are plenty of reasons, much of them boring like budgets and staff resources.

    One however that isn't talked about much, is the ability to say "No" when talking about requirements analysis. Usually this is where nobody wants to say no to a manager who has seen things like the internets and iphones.

    Typically an application is created to solve a business problem. There is a tendency to want to throw everything and the kitchen sink into the project, more less because you can. I think if a lot of projects concentrated on producing a simple product that solves the core business problem in a very stable way without a lot of bells and whistles increasing the complexity of the project they would be a lot more successful. Nothing wrong with collecting the bells and whistles as requirements, that might be added at a later date, once the core business requirements have been met, deployed, and proven. If more time was dedicated to core than on fluff towards something that is functional, I think it would pretty much eliminate project failure, at least in that there would be some usable results, and not just a huge pile of code and documentation that is non-functional. Big healthcare systems come to mind.

  10. Failures 1A and 1B: Offshoring and Outsourcing by sethstorm · · Score: 3, Insightful

    They overpromise, underdeliver, and screw everyone when all is said and done.

    --
    Twitter supports and protects racists - by smearing their critics with the "Hate Speech" label.
  11. Failure rate? by tomhath · · Score: 2

    TFA lists a few big failures. But in isolation that list doesn't mean anything. Are we getting better or worse at implementing big projects? How many multi-million dollar projects were completed successfully versus failed? Humans are not perfect so there will always be failed projects, but there have also been many, many successful ones in the past decade.

  12. Projects are much much more than code by billstewart · · Score: 2

    I've worked on big IT projects, and I've worked with government people who've worked on them, or managed or procured them. One director at Livermore Labs in the late 80s commented that he'd never seen a billion-dollar computer project succeed - it's just too big to do the communications that are needed to make it work, through the requirements, design, and management parts, and he was trying to work on how to break projects down into things that were small enough that they could be managed and implemented. Even the successful things are messy at large scale.

    This was long before Agile (which is pretty tasty Kool-Aid, for some kinds of projects, but has its own limitations).

    --

    Bill Stewart
    New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
    1. Re:Projects are much much more than code by Lisias · · Score: 2

      You underestimate the importance of good coders. Or perhaps, overestimate the importance of managers.

      Good developers can delivery a viable product besides bad management. But the best management of the World can't deliver a viable product without a minimum threshold of good code!

      I agree and understand the problematic of big projects, I had my share of it too. But when the worst happens, and it eventually happens (more than once I saw a project being trashed by external causes, as a legislation that was changed without notice that fsck up our funding) it was always possible to salvage something from the mess, specially good and well written artifacts that were reused on other projects. Bad code is always trashed.

      However, it is clear that the failure rate in large IT projects is higher than in projects, for example, from Civil Engineering. The Project Management Theory is essentially the same for all areas, so I have serious reservations to believe that this apparent crisis in IT is merely managerial.

      --
      Lisias@Earth.SolarSystem.OrionArm.MilkyWay.Local.Virgo.Universe.org
  13. Natural Selection by aberglas · · Score: 2

    When small projects fail, the contractors move on. By the time large projects fail senior managers need to be promoted.

    Over time, people that work on smaller projects are the competent ones, whereas the people that work on large projects have fantastic skills in working in a bureaucracy, but none in actually developing software.