Slashdot Mirror
World's Biggest 'Agile' Software Project Close To Failure
00_NOP writes "'Universal Credit' — the plan to consolidate all Britain's welfare payments into one — is the world's biggest 'agile' software development project. It is now close to collapse, the British government admitted yesterday. The failure, if and when it comes, could cost billions and have dire social consequences. 'Some steps have been taken to try to rescue the project. The back end – the benefits calculation – has reportedly been shifted to a "waterfall" development process – which offers some assurances that the government at least takes its fiduciary duties seriously as it should mean no code will be deployed that has not been finished. The front end – the bit used by humans – is still meant to be “agile” – which makes some sense, but where is the testing? Agile is supposed to be about openness between developer and client and we – the taxpayers – are the clients: why can’t we see what our money is paying for?'"
But it might make it clear that it will fail much earlier and then at a lower cost.
Agile assumes you have smart, talented, dedicated individuals doing the work. Then again if you have that pretty much anything works.
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
I refer to it as "Monkey's Paw Development" (And before anybody asks agile to me ends up being "Hey let's ask developers what they'd wish for in an awesome development environment. Then give them that but give it to them in such a way that they regret ever asking for it.")
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
"Before I learned the art, a punch was just a punch, and a kick, just a kick.
After I learned the art, a punch was no longer a punch, a kick, no longer a kick.
Now that I understand the art, a punch is just a punch and a kick is just a kick."
I've worked with a few different methodologies in my time, but now that I'm older I realize all you have to do is follow common sense. It's really not that difficult.
That question would indicate to me that they're doing Agile wrong. Agile development ought to include a short feedback loop that includes not only the client, but automated tests. So, if this question is legitimate, then something is very wrong with how this project has been run.
Even if I knew that tomorrow the world would go to pieces, I would still plant my apple tree. -Martin Luther
Agile Needs Testing? Open Source It.
It's perfect for that. Open an issue tracker, let folks dump in the requirements. Give us the backend API, and I'll work on it for free an hour or two here and there, just for grins.
Oh, it's closed source? Well, I hope it's a massive disaster and you learn your lesson. You can't pay for the brightest minds. They wouldn't be caught dead slaving away at those software houses. But let them do it "for the good of mankind", they'll be brow bashing each other for a chance to get their beautiful bit of brilliance in the code base.
The government is the client. There is no reason for the public to be involved in an unfinished project.
"World's biggest" and "agile" don't really go together. One of the core tenants to agile is to break things down into small chunks. Multi year contracts for a predetermined end product are waterfall by definition. Either way, I have seen waterfall work just fine and I have seen True Agile[tm] fail hilariously miserably (to which most Agilistas respond with some form of the "No True Agile" fallacy). The most important thing is tight iterations. If a 2 week sprint fails, then it is not that big of a deal. If a 2 year death march fails? Someone's getting fired, since its the equivalent in agile-land of failing 52 sprints straight.
It's the usual crowd screwing money out of the government
HP/IBM/Accenture/BT
https://www.whatdotheyknow.com/request/130677/response/322518/attach/html/3/FOI%203648%20Response%20181012.pdf.html
Just because you're agile, doesn't mean you crap daisies and unicorns. I often see inept upper managers latch onto agile as the latest magic bullet which will solve all their problems with no other changes on their part. Except they keep all the micromanagement bits, discard all the engineer empowerment bits and hand their scrum team a year's worth of priority 1 stories to implement in the next sprint. Good managers will likely be successful no mater what methodology they use, bad ones will likely fail no matter what methodology they use and the ones in between will have mixed results no matter what methodology they use.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
All Agile methodologies really are are different ways of implementing the Spiral Model of development. If used correctly, any of them can work fine. Unfortunately, that's only in theory. In practice, Agile generally becomes an excuse to use Code and Fix, which is the worst methodology and the most prone to failure. Beware anyone who claims that Agile is the solution to anything.
To me, "Agile" isn't meant to be free for all. Rather, it's meant to decouple the development of large software into more manageable chucks and teams.
Someone still has to helm this and there has to be a rather obvious set of goals and objectives to attain.
The problem I've seen with Agile, is that often, the software is being developed as it goes and to me, that's poor use of the "Agile" experience.
To me, Agile would mean that the software architect(s) know what they are doing, and have a bloody good idea of the road map to take to get there.
The "Agile" part is that for each of these goals to achieve, there is a separation of tasks and an agreement on implementation specs.
Any single team has full knowledge of the objectives of other teams and thus, this team can concentrate of their specific task and at the same time, mockup whatever is required to support their work based on any of the dependencies which depends on other teams work.
Unit Testing, Functional Testing, etc, all of that must come into play.
The Agile process should NOT be 'on-the-fly', which is what I've often seen, for that is the reason why we get delays and development costs going over budgets.
You wouldn't understand it.
You would exploit it.
You wouldn't do anything to make it better.
You would waste time complaining about everything.
Burma Shave!
The UK Government used to have its own internal computer consultancy. This was called the Central Computer Agency - the Central Computer and Telecommunications Agency after it took over running the government phone network.
CCTA was staffed by a mixture of experienced civil servants and expert contractors, and provided support to all UK government department's computer projects. It had procurement experts, sizing experts, code, architecture, the lot. When these experts were not working for Departments they worked on UK and international standards and methodologies. Some of you may remember the OSI, ITIL, PRINCE, PROMPT, SSADM, CRAMM, BS7799/ISO 27001/2, BS5750, etc...
In those days UK Government projects ran to time and came in on budget, with CCTA project managers. But CCTA was constantly under attack from the computer industry, who saw CCTA as an expert gatekeeper, stopping them from making major profits out of government projects. They lobbied for its closure constantly, and in 2000 they got their wish. CCTA's major functions were closed down and the rump moved to OGC
Interestingly the CCTA Security and Privacy Group (the only UK Government computer security organisation at the time) had been closed down earlier at the request of GCHQ and MI5, who wanted to take its budget and responsibilities. The SPG arguably ran the first CERT (though not called by that name) in 1984, and was influential in developing, inter alia, early AV company liaison and security accreditation with initiatives like Common Criteria.
The UK government departments, encouraged by a variety of groups with ulterior motives, cut off its nose to spite its face. They are now a byword for incompetence and overspend. The collection of experts which existed in Riverwalk House during the 1980s and 1990s will never be replicated again...
Government department + software project = total failure. .
I would love to know how cheaply this same project could be done. Probably by one person. Probably a $10,000 project with the final project size 100 times smaller, run 100 times faster, 100 times more accurate. [That is what I achieved after a payroll application they tried to force on our dept. was discarded and we rolled our own.]
Not a real biggie, just a replacement for Jobseeker’s Allowance, Income-related Employment and Support Allowance, Income Support, Child Tax Credits, Working Tax Credits, Housing Benefit for the whole of the UK population. I mean how hard can it be? Given your obvious talents I am sure you could knock something together using a few Excel macros by next Tuesday.
One of the things about this is that it is being driven by an ideologue who doesn't give a toss about evidence, not quite the person who thinks all government sponsored software development but pretty close.
A lot of the 'agile' models reward talkers and people who take immediate action and can rattle off buzzwords, at the expense of more introverted engineers who like to investigate and plan before they act. In a continuously moving environment such as a social networking site, that reward system might be appropriate. In a financial back end doing mission critical work, that sounds like a disaster. So, no surprises here. One size does not fit all companies.
Open source developers will not flock to work on a bureaucracy-ridden welfare allocation and tracking platform, of all things.
I will if I get to work on the 'open source developers benefits module'.
Have gnu, will travel.
Agile can be as foolish a project management method as any other but it seems to have a religious component to it.
Like everything else about software development, from methodology to choice of language to how you name variables and format code.
I find it hard to imagine that people in other fields are as opinionated as we are, with so little evidence to back up their opinions.
(But they're people too, so they probably are.)
Sheesh, evil *and* a jerk. -- Jade
Agile/Extreme programming is the alternative medicine of software development. It's a collection of mostly unrelated and sometimes contradictory concepts, the only common thing between them being lack of widespread adoption. Like alternative medicine, it has components that are useful in some circumstances. These give unearned credibility to Agile, even though they were there well before it. The problem is also similar, these components are taken to the extreme and are claimed to be universal solutions to every situation. For example, acupuncture is useful to ease pain, but no matter what the charlatans say it doesn't help against cancer. Similarly, frequent testing during development can be useful, but test-driven development is taking it to the extreme, and it doesn't work at all in for example web development. And like alternative medicine has homeopathy, Agile/Extreme has their own set of ideas that are total bullshit like pair programming.
Having seen both the waterfall model and the agile model I would say that both models needs to be applied correctly to work. The agile is working for incremental development in small steps where you can adjust and adapt as you go, the waterfall works for long iterations but it requires that each step is thoroughly analyzed for problems. It works fine for slow projects but not for quick to market and cheap projects.
Changing process method in the middle of a project is a sure sign that it's time to abandon ship because the management doesn't have a clue what they are doing.
One of the primary reasons for a large project failure is that people aren't working with the same goal in sight. Even the single worker needs to know what the overall picture is and what they are supposed to do and why.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
Basic problem here is non-programmers in the management visualize software "product" being manufactured somewhat like a very sophisticated assembly line or teams of workers putting together a product like they did before the assembly line was invented. But software is organic, it is grown. It is not grown like rows and rows of corn in a field but more like growing a city from a village. You are not going to plunk down a new 80 story skyscraper in downtown without huge disturbance to existing structures and users near by. You are not going to add a new Lorentz transform based asymptotic wave form expansion feature in the middle of the simulation sequence without huge disturbance. Or add payroll processing of German employees to you American HR system without serious disturbance.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Its been done before :D
Build a Man a Fire, and He'll Be Warm for a Day. Set a Man on Fire, and He'll Be Warm for the Rest of His Life.
Being called 'Agile' doesn't mean that it is in the spirit or letter of 'Agile'.
The reality is that 'Agile' is in practice more of a brand than anything else. Project Managers love to apply the terminology to their projects. This does not mean they actually meaningfully follow a consistent set of behaviors, just that they use similar sounds words.
'Agile' is like 'Cloud' and 'Web 2.0'. While each phrase may have coined with a particular specific concept in mind, they became more hype than anything usefully descriptive.
XML is like violence. If it doesn't solve the problem, use more.
This should be suppose an article about "agile" and the Universal Credit. After reading the article there is no information what-so-ever, except that the Universal Credit project has been admitted to be failing.
So why is Universal Credit an "agile" project?
Why it is failing?
What is Universal Credit anyway?
Maybe that is why Twitter is so successful, the whole article is just a Twitter message: "Universal Credit, suppose to be biggest Agile Software Project, is failing".
Here is some more information:
http://www.guardian.co.uk/society/2013/apr/29/universal-credit-pilot-scheme
http://www.guardian.co.uk/commentisfree/2013/apr/30/universal-credit-iain-duncan-smith
Is it called "agile" because it's a "step-by-step approach" ?
http://www.mueller-public.de - My site http://www.anr-institute.com/ - Advanced Natural Research Institute
Whether it was 'waterfall', 'agile'', or whatever, every project that I've worked with that seemed to put more effort into using the most hyped phrasing to describe their process than into actually developing the project has been doomed.
I liken it to religion. The spirit of most holy texts is quickly lost in the actions of adherents as they focus on the specific content rather than the message. For Christianity, specific belief in the divinity of Jesus seems to often be more important than adhering to his teachings. Similarly, in Agile, managing to map words like 'scrum', 'sprint', 'epic', 'user stories', and so on to what you do is more important than internalizing the original intent behind those words.
Projects that don't make a lot of effort to 'conform' to any specific renowned fad tend to do well. They also tend to do the sorts of stuff Agile advocates without using the words.
XML is like violence. If it doesn't solve the problem, use more.
Over my career, i've worked in the UK Benefits Agency processing claims, i've worked in their IT departments, i've worked for the outsourced departments later supporting them, and i've worked for a software company which loves agile (but will do waterfall if pushed).
The problem here isn't waterfall/agile. The problem here isn't .Net/Linux.
The problem here is the parties involved. On one side you have a government agency where people obtain seniority largely through age, not skills, and the main skill that is relevant is passing the buck when things go wrong and taking the credit when things go right (really, this is government agencies through and through - not to mention, most people with real skills/brains get out as soon as they can). On the other side you have the dinosaurs of development (not necessarily age, but sizewise).
Somebody earlier in the thread stated this whole project could have been delivered with much lower cost, with just a few devs, in a much shorter time. I'm 100% in agreement with them. The only real complexity of most government systems is the labyrinthine workflows, but they are documents and strictly followed in their paper variants, its just a matter of getting an understanding of this and turning it into software.
My recommended development approach for this project would have been as follows:
1) Hire some decent devs. They don't need to be hotshots, what is being developed is fairly simple from a technical standpoint. Mainly guys who can follow a spec.
2) Take a bunch of people who actually do the work for real, the paper pushers. Take them down the pub and get them rat-arsed. Listen to them bitch and whine about all the idiotic things they have to do in day-to-day operations.
3) Take notes of their bitching! It may help if you are drunk!
4) Any requirements given to you through the official channels are probably worthless. Dump them. They will simply mislead you from what is actually required.
5) Build the system based on what you learned from the drunk employees.
6) Demo it to the stakeholders and hand over.
7) Contract fulfilled.
Oh, and of course.... 4) Profit... erm...
Two principles were key and could be used in any methodology.
1) Address risky (new technology, undefined specs, etc.) first
2) Regular time boxed functional builds.
If you can't address the risks successfully, then at least you can cancel the project early.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
Just because the failure of one project is caused by .NET does not mean that a project not based on .NET is going to succeed. In fact, traditionally 80% of software projects fail.
This project is clearly failing for the second from last reason. It is also failing because it is not an "agile project". An "agile" project cannot fail and cost Billions because it must always deliver runnable software with a maximum of a few weeks delay if you use some "semi agile" process like scrum or immediately any point if you use some true agile process.
Once you deviate from "Working software" for more than a couple of sprints (everybody can make a mistake) then you are no longer doing agile. I have seen so many "agile" projects which seem to define "Working" as meaning something like "a prototype which would never work at full scale" and so they have never addressed the major problems of their class of system.
If they are "billions" of pounds down whilst doing agile, then they should have already delivered plenty of working systems and have hundreds of happy users. In this case they are a "success" even if they were a bit slower and more expensive than some other projects. If, however, they really haven't delivered anything then what they were doing was an unplanned disaster using "agile" as an excuse for not having a proper plan.
Whilst I know that the "waterfall" method of development is famed for it's failures. Whilst I know that those failures are spectacular and huge. I really don't see how you deliver, for example, 5% of a working mobile phone network. You just have to have a big interlocked plan with a working phone, transmitter, backend, management and interconnection all planned together. I don't believe that such a thing can be done in a true "agile" way and pretending that you are doing it in an agile way is a dangerous fantasy. Only once you have a working network can you start to improve it in Agile increments.
=~ s,(.*),<sarcasm>$1</sarcasm>,g if any_point_you_wish();
Indeed. No MBA will ever be on the level of insight that a good developer has. As the MBAs know this they consistently hire bad developers, going for the fallacy that more bad developers are better than less good developers and not wanting to hire people that are smarter than they are. While it is true that the number of subordinates you have determine your importance in a bureaucracy, bad developers cannot do complex projects, no matter how many you have. Something these MBAs cretins are not equipped to understand. As long as these people are put in charge of complex software projects, these projects will fail. I have seen it several times now. And no, 100 bad developer hours are not even worth one god developer hour. The bad developer hours have negative worth, as you can only throw away what they have produced.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
An "agile" project cannot fail and cost Billions because it must always deliver runnable software with a maximum of a few weeks delay if you use some "semi agile" process like scrum or immediately any point if you use some true agile process.
The trouble is, you can have software that runs and passes some tests, yet still does not meet all of the mandatory requirements for the project and therefore may have no value at all in the real world. You don't get any credit for meeting 90% of the mandatory requirements on a job like this. The idea that having software that runs and maybe passes some tests has some sort of inherent value might be the biggest fallacy perpetuated by the whole Agile movement. It's just not true, and therefore neither is the claim that agile projects can't fail as a result.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
I have never, not once, seen a requirements document that accurately captures exactly what the system will do. [...] The whole point of stories is to do things in small, end-to-end slices to produce functionality quickly, let the product owner see it and play with it and then get a better idea of what they really want.
Well, in this case, "what they really want" tends to be defined literally by Acts of Parliament and/or policies set by the highest legal authorities in the government. I know it's popular to mock politicians in Europe for having no idea what they're doing economically, and it seems there's some truth to that in light of recent events, but you don't implement software to automate a major component of your national tax and benefits system in incremental changes with one guy designated as the project owner who can change his mind every now and then as you go along. That guy is the Chancellor of the Exchequer, and he has a few other things to do apart from responding in timely fashion to Bob the Programmer's request for clarification of how to implement the national tax rules.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
You wouldn't steal a keyboard? Would you steal a handbag? A DVD?
Oops. Sorry. Wrong thread.
Check out my sci-fi/humor trilogy at PatriotsBooks.
Stop being dumb and bigoted.
I think you failed to notice the shill posting that I was responding to. I didn't bring up .NET to bash it, rather to respond to someone who was trying to compare it to RedHat which is a system that .NET clearly doesn't approach in any of maturity, flexibility or stability.
There are tens of thousands of .NET projects if not more that have succeeded.
That's hardly a recommendation. There are hundreds of thousands of projects that have been based on PHP. There are probably millions of such projects which are based on excels. The fact is that almost every problem system has lots of "success" stories. Many of those would have been much more successful if they chose a less problematic system but that doesn't mean that they aren't in their own terms successful.
Look at some key criteria for choosing a good system
When people choose .NET, they are choosing a system which is locked into one vendor. They are choosing a system which has a record of a number of serious disasters and lack of performance at higher system demand levels. Most of all, they are choosing a system from a vendor which is more willing to pay shills to turn up on forums astroturfing their success than they are willing to solve the underlying problems of their system. This doesn't guarantee failure any more than choosing a solid system guarantees success. However, it can be a single decision point which causes some given project to fail.
Compare the decision to choose .NET to the decision to choose SQL or C or even Java. Even though they have their problems, each of those systems has multiple suppliers and it will be possible, if for example your Java supplier has a bad security record, to migrate to a different one which is more responsive. That gives a more solid environment and a better chance of long term success.
=~ s,(.*),<sarcasm>$1</sarcasm>,g if any_point_you_wish();
There are three things I have learned never to discuss with people... Religion, Politics, and Agile.
I could have told you in advance, just from that list, that the project was going to fail.
Fail, that is, from the perspective of the agency and its taxpayers. From the perspective of the consulting companies, it worked just fine. They wanted big fees and got them.
They are choosing a system which has a record of a number of serious disasters and lack of performance at higher system demand levels.
What serious disasters? Do you have any references that of any, especially ones that say it was because of .NET? Those sound like typical things that people who only read Slashdot for news seem to believe. I remember the London Stock Exchange fiasco, but that was because of incompetent people rather than the technology itself. In the last quarter, Microsoft's Server and Tools division recorded an increase in revenue of 11% despite completely free alternatives available.
if for example your Java supplier has a bad security record, to migrate to a different one which is more responsive
Like who?
This space for rent.