Australian State Bans IBM From All Contracts After Payroll Bungle

renai42 writes "If you don't follow Australian technology news, you're probably not aware that over the past few years, the State of Queensland massively bungled a payroll systems upgrade in its Department of Health. The issues resulted in thousands of hospital staff being underpaid or not paid at all, and has ballooned in cost from under $10 million in budget to a projected total cost of $1.2 billion. Queensland has now banned the project's prime contractor, IBM, comprehensively from signing any new contracts with any government department, until it addresses what the state says are IBM's project governance issues."

Comments

Language Barrier

[Anonymous Coward]

Obviously, you Aussie blokes need to learn Hindi if you want to partner effectively with IBM.

Re:Language Barrier

[Vinegar Joe]

The Aussies would have learn English first.

Re:Language Barrier

[Anonymous Coward]

^ to

pot / kettle

Re:Language Barrier

[Nyder]

Not likely given China's propensity towards spying on everyone.

Did you say that with a straight face?

Re:Language Barrier

[Entropius]

Just because the Americans are spying on people doesn't mean the Chinese aren't doing it too.

Re:Language Barrier

[Entropius]

As an American, I find it easier to understand Australian English than many American-born English speakers.

Re:Language Barrier

[nospam007]

"Today Chinese build roads and buildings as the locals don't want to bake in the desert sun,..."

Perhaps one day they will even build a railway through America...

Re:Language Barrier

[Entropius]

:P

I don't see why we don't just outsource all our snooping to the Chinese. The outcome will be the same but it'll be cheaper for the American taxpayer, and the Chinglish translations will be hilarious.

Lol

[Anonymous Coward]

Deflection, Qld health are the worst run bureaucracy in country. I've heard first hand they put non IT on the project and were forever changing scope then pushing forward with little or no testing.

Re:Lol

[Elindor]

That came up in the various stories I read yesterday about this issue - and I'm more inclined to believe IBM's side of the story.

Re: Lol

[Anonymous Coward]

Ditto. I know one of the IBM Admins for this job, she said Qld health signed off at every stage before going live. I'd like to see who has the greater budget for a court battle - the qld govt is broke.

Re:Lol

[Anonymous Coward]

I worked on a large project for a quasi-government body building software for Queensland Health as a customer.

They had reservations about us being able to deliver. We delivered a rock-solid piece of software on time and budget. They, however, took 8 _years_ to take that piece of software and put it into production.

Yes, they are that bad.

They were a basketcase _at least_ a decade before the payroll bungle.

Re: Lol

[Anonymous Coward]

The telling part is that IBM only got $25M for their efforts. I say this as a government PM. We are absolute, miserable failures at buying software. We don't know what we want, which begs IBM, SAIC, SAP, et al, to bid low and then increase the price every time we go "shit, we didn't really mean that."

Re:Lol

[erroneus]

Indeed. IBM's reputation is pretty well established. They are slow, tedious and yet effective. They are a glacier in IT. But I see it everywhere -- people making decisions in an IT project that have know knowledge of what it takes to make things happen. The illusion that "it's all so easy" has really gotten buried too deep in someone's head somewhere.

Re:Lol

[wisty]

Requirements:

Make it better than the old system.

Make it work the same way as the old system.

Make it compatible with every else's system.

The only trade-off allowed is cost, since it's just tax dollars.

Re:Lol

[RabidReindeer]

Indeed. IBM's reputation is pretty well established. They are slow, tedious and yet effective. They are a glacier in IT. But I see it everywhere -- people making decisions in an IT project that have know knowledge of what it takes to make things happen. The illusion that "it's all so easy" has really gotten buried too deep in someone's head somewhere.

The magic phrase is "All You Have To Do Is..."

Those six words have destroyed more IT projects than anyone can count.

Re: Lol

[RabidReindeer]

I am Mordac, Preventer of Information Services.

Re:Lol

[sjwt]

IBM Pulled out after the costs really started to balloon.. as in the 10's of Millions... And then it really went down hill.

Re:Lol

[__aaeihw9960]

The illusion that "it's all so easy" has really gotten buried too deep in someone's head somewhere.

I think it's because PC's are the new 'old car'. In my youth, when men were bored, they would go tinker around with their cars. This tinkering began and ended at home, simply because there was no translation to the workplace. Today, though, with all the gee-gaws and doohickeys that are on modern cars, men have less to tinker with. What we do have, though, is a home PC. We can tinker, we can figure, we can play with the home PC and not really screw stuff up. SO, to people like that, it really is a simple transition between home PC tinkering, and systems design.

Or, it could be because most people HAVE to have say in what goes on around them, regardless of skills or knowledge.

One of those two things.

Re:Lol

[tibit]

And the man-month. The mythical man-month. Let's not forget the man-month. Have I mentioned the man-month yet? :)

Re: Lol

[plover]

That Dilbert cartoon was based on a reality I think we all share. The business comes in and says "we want the WhizBang package." The giant IT wheels kick in and someone says "you must follow procurement procedures, specify requirements, find vendors, get bids, select vendor, etc." So IT asks "what are your requirements", and the business says "we want what WhizBang does."

From then on facts no longer seem to matter. Some analyst copies WhizBang's brochure into a spreadsheet and labels it Requirements.xls. Someone from IT adds a few rows that have technical requirements the business doesn't care about. "Hey, it says here that WhizBang supports up to 5 simultaneous users. How many users will we have? 10,000? OK, we'll mark that one as a 'doesn't precisely match expectations', let's move on to the next requirement." Someone adds up the tally marks at the end of the process, and surprise, WhizBang matches 80%, while the other packages only come in at 50%.

Business buys WhizBang. WhizBang fails to deliver. Business blames WhizBang.

Re:Lol

[BrokenHalo]

The illusion that "it's all so easy" has really gotten buried too deep in someone's head somewhere.

Sure, but payrolls are easy. They were easy when I was doing it for local and national governments in Europe 30 years ago, and I don't see that they could possibly have got much harder since then, even if staff do get demoted/promoted five times a day.

Re:Lol

[BrokenHalo]

India Business Machines is among the worst information technology outsourcing and consulting services in the world.

You're talking about a completely different company. "IBM" in this context refers to "Itty Bitty Machines". Or the opposite, as in "an elephant is a mouse with an IBM operating system". Take your pick...

Re:Lol

[sjames]

It does seem unbelievable that IBM and IBM alone is responsible for the project's failure given the fairly small piece of the pie they had.

Re:Lol

[sjames]

That's half the problem. The other half is the 'business logic' that never made any sense in the first place. What is actually being done is multiple individuals' interpretation of a tremendous mass of confusing and conflicting rules combined with unwritten assumptions and word of mouth 'folk wisdom' that may or may not bear any relation to what is documented. Further, there is a good chance that nobody actually knows where all of the documentation is or how inclomlete what they have may be. Implicit in that is a bunch of stuff that was 'agreed upon' in an ad-hoc way over a period of years to decades without ever being written down and people doing things they do not see the sense of because '"that's just how we're supposed to do it".

Trying to implement the current ad-hockery in an all encompassing computer system exposes each and every one of those bits of insanity. Worse, rather than taking the opportunity to rationalize and normalize the system, management typically insists that these systems built on logic perfectly implement the current illogic on a special case by special case basis ad-nauseum.

Wrong!

[TheRaven64]

That's not how government procurement is supposed to work! A company that has failed to deliver on multiple contracts in the past should be given priority, because it has significant experience in government contracting work!

Re:Wrong!

[K. S. Kyosuke]

"Working closely" being a technical term for the intersection of pockets being non-empty?

Re:Wrong!

[Thanshin]

A company that has failed to deliver on multiple contracts in the past should be given priority, because it has significant experience in government contracting work!

And a company that was ready to sell a $1.2 billion product for only $10 million should be praised as benefactors of Australia!

$1.2 billion payroll system

[Agent ME]

If I were paying $1.2 billion for something as rote as a payroll system, it better be fucking amazing. It's estimated that the entirety of Linux could be recreated from scratch for $600 million. A payroll system twice as complex as the entire Linux operating system! Think of the possibilities! I have no idea what the possibilities are, but they must be amazing to justify that cost!

Re:$1.2 billion payroll system

[skovnymfe]

You're not taking into account all the middle management and project management such an endeavor requires. That alone easily accounts for 90% of said budget. After all if you don't hire at least 3 managers per developer, how can you make sure they're doing their work properly all 16 hours of the work day?

Re:$1.2 billion payroll system

[Anonymous Coward]

In fact IBM did comment on this:

As the prime contractor on a complex project, IBM must accept some responsibility for the issues experienced when the system went live in 2010, however, as acknowledged by the commission’s report, the successful delivery of the project was rendered near-impossible by the state failing to properly articulate its requirements or commit to a fixed scope.

IBM’s fees of $25.7 million accounted for less than 2 per cent of the total amount. The balance of the costs is made up of work streams which were never part of IBM’s scope.

Re:$1.2 billion payroll system

[Joce640k]

If I were paying $1.2 billion for something as rote as a payroll system, it better be fucking amazing.

The real WTF is that IBM still don't have an off-the-shelf payroll system.

Paying people's wages is almost the original computer application.

Re:$1.2 billion payroll system

[orlanz]

Every consulting company out there has multiple off the shelf, turnkey payroll options. Just that no one wants them. Most of the time, the "consultants" just customize one of the options as per the customer's unique needs. Then the customer has even more extremely special and unique needs. Some clearly poor practices and some just not feasible. About 1/2 way through the project people realize that the customer never wanted an off the shelf, turnkey solution. They want a custom built solution. But they just keep going cause its hard to stop a train; even thou they all know the wreck that is coming.

Funny thing is that if people just bit the bullet and understood the limits of a turnkey or that they wanted a custom solution, they would certainly save a lot of money. It would cost more than the original budget but it would cost a LOT less than the end result. This is why people just don't be honest up front. No one likes approving a $100k project while there is a $90k option. No matter how wrong the second is, they just spend $9.9k figuring out how make the later look good in the summary reports.

I have spent an unfortunately amount of time & cost convincing and proving to the decision makers what basically to me was 2+2 can not equal 5. It feels insulting most of the time cause they bring us in for our "expert" opinions, but don't trust said opinions. Until there is a cost that is big enough to show up as a line item in a report or some high up gets red in the face. Its sad, but just the way of the world.

Re:$1.2 billion payroll system

[pipedwho]

In fact IBM did comment on this:

As the prime contractor on a complex project, IBM must accept some responsibility for the issues experienced when the system went live in 2010, however, as acknowledged by the commission’s report, the successful delivery of the project was rendered near-impossible by the state failing to properly articulate its requirements or commit to a fixed scope.

IBM’s fees of $25.7 million accounted for less than 2 per cent of the total amount. The balance of the costs is made up of work streams which were never part of IBM’s scope.

There is an expectation that engaging a large professional specialist contractor would avoid the problems of using a smaller outfit or running the project in-house. You'd expect a specialist mega-corp would be able to help you define the scope and requirements of the project, as it's something of which they supposedly have prior experience.

IBM should have been the one asking the right questions at the start, and requesting access and authority to do their job. It's not like a health care payroll system is something new that no one's ever seen before. The QLD government is essentially employing IBM to be the experts in this area to deliver a suitable system.

I see this crap from these big end of town software outfits all the time. They sell products and customisations that the client doesn't need, features that in most cases just get in the way and make the systems unusable. They charge 10s to 100s of millions to build websites that are unstable and too cumbersome to maintain and use. And generally overcharge for a final product that they shoehorn to fit the actual requirements of the customer (and by extension, the customer's customers).

I don't think the general tendering/bidding process helps much either, as it doesn't always give enough access to scoping and requirements gathering to be able to generate a valid cost estimate. In many cases it comes down to the sales team getting a huge bonus contingent to signing off on the sale. And they'll say and promise anything upfront, and let the weasel^wlegal team rewrite the contracts to make every request for something that should have been included seem like an out-of-scope up-charge.

Australia could have learned from New Zealand

[ernest.cunningham]

IBM were the contractor for New Zealand's largest IT cock up INSIS (Integrated National Crime Information System, http://en.wikipedia.org/wiki/INCIS) which was a total flop and cost $110,000,000.

Funny thing is though, we didn't learn from our own mistakes and hired an Australian company called Talent2 for our Education Payroll. It has been a runaway failure (with more new bugs being found than being fixed over any given time period).

Re:Australia could have learned from New Zealand

[LordLucless]

Geography lesson: New Zealand is NOT a state of Australia.

However, we've got provision for you in our constitution, just waiting :P

6..."The States" shall mean such of the colonies of New South Wales, New Zealand, Queensland, Tasmania, Victoria, Western Australia, and South Australi

Re:Australia could have learned from New Zealand

[Entropius]

In the US, there was a "three-fifths compromise" in the drafting of the Constitution, where the South got to count 3/5 of the slave population in determining representation in Congress. Could NZ do something similar with sheep?

Queensland Health Payroll were a joke already

[Anonymous Coward]

My wife is a doctor who works for Queensland health. To be honest, they had comprehensively mucked up her payroll numerous times prior to the IBM System. Unfortunately, they now feel the need to deduct her pay based on shifts she did 4 years ago (as the new system has slightly different data than the old one). The staff of QH are basically comprehensively useless, and even prior to the new system they would do things like email her other people's personal details and salary information. The staff always have been lazy and careless, and the new system couldn't handle users that didn't give a shit about doing a good job. IBM has undoubtedly ballsed things up, but QH Payroll are genuinely amongst the least competent people in the world. In fact, pretty much anyone in a government position in Queensland is useless, which is why they are in the process of firing 16,000 of them...

Same thing here

[vikingpower]

IBM bungled a massive project for the re-automation of the AKH, Vienna's main hospital. Got banned. It seems, however, that IBM does not care: such "missers" are like flies to such an elephant - yet.

Re:Perspective

[girlintraining]

* Footnote: The average meal at mcdonald's costs around $6. The ratio is accurate: This is like going to McDonald's to order a happy meal and winding up spending more than you do on rent for it. Whups.

Project governance issues

[DrXym]

I wonder if any government has *ever* had a good experience after signing a major contract with a supplier to implement one of these systems. A single time ever where a project was delivered on time, on budget and performed at or beyond the expectations set down in paper.

I thought these contracts were just an excuse by suppliers to wildly overcharge governments on the daily rates, software licences and support fees knowing that once the ink has dried on the contract they basically have them by the balls.

I wonder given the expense of these systems if governments wouldn't be better off to hire teams in-house to write this stuff.

Re:Project governance issues

[jonbryce]

The London Congestion Charging System was delivered on time, within budget and with no major flaws.

Re:Project governance issues

[benjfowler]

You NEVER hear about the successes.

An example is the Docklands Light Railway here in London. It's conspicuous by the lack of whingeing that went on about the cost of its construction. When they extended the line to Lewisham, they had to dig tunnels under the Thames, etc. Delivered on time and under budget, and nary a complaint.

Contrast that with the Jubilee Line Extension...

Re:Project governance issues

[maroberts]

Hmm, about that successful Docklands Light Railway

While the first five years were plagued by unreliability and operational problems,[55] the system has now become highly reliable.[55] In 2008, 87% of the population of North Woolwich were in favour of the DLR

i.e. it took five years to fix the issues with it.
It's also overcrowded and the level of demand was grossly underestimated.

Re: Lol x 2

[DeathElk]

They should have gone with Pronto. An Australian ERP company that is quickly responsive to changes in legislation (for Australian and overseas customers) and very flexible during implementation.

I've recently finished a two year planning and implementation of Pronto for my employer, and we are impressed with the outcome. We were on budget, accomplished within a reasonable timeframe (given some feature creep) and the post implementation support is great.

Re:Perspective

[Somebody Is Using My]

"Hey honey, I'm going to McDonald's to grab a bite to eat, be back in 10!"
(A few hours later)
"... Umm, honey, how did you manage to spend $710 dollars at McDonalds?"

But let's be fair, the actual breakdown is probably more along these lines:

            $6 Happy meal (expected budget)
            $250 consultants and managers haranguing you about how you are hungrier than expected
            $200 to replace provided hamburger with a specialty burger
            $250 "expert eating" trainers who advise you on the how to insert hamburger into mouth
            $4 extra hamburger you ate because the above three took so much time lecturing you that you got hungry again

IBM only got $25 million of that $1.2 billion. The rest was a result of "the state failing to properly articulate its requirements or commit to a fixed scope."

$10 million to $1.2 billion

[CadentOrange]

That's off by more than two orders of magnitude. Heads need to roll.

Re:$10 million to $1.2 billion

[msobkow]

But you'll note only 2% of that money went to IBM. A 25 million final cost on a 10 million dollar project is only a 150% overrun, and quite reasonable given the spec churn that occurs in government. The specs are never final at the time of bidding, and everyone knows that.

It would seem the bigger consumer of resources was by far the "out of scope" costs that the goobermint conveniently ignored while setting the initial budget. There are always costs involved with large deployments, and they usually dwarf the cost of development, especially if hardware and infrastructure costs get rolled into it, such as upgrading everyone's PCs from XP at the same time, but "sneaking" that expense into the budget of the large project. And that happens All The Time.

Right. IBM Needs More Process.

[C0C0C0]

Poppycock. I used to work for Big Blue. It was the most process bound organization on Earth. It's entire business model is to sell, not innovation, not cutting edge, not feature set, but a complete and utter lack of surprises. If there is anything I can't imagine blaming on IBM, it is a lack of governance.

Re:Sounds like what IBM did in PA

[tibit]

I think that those big failures indicate something about human nature at a much lower level. The green screen (CICS and such) was something that every developer could reasonably comprehend in its entirety, and there weren't really a 100 ways to do anything and everything (AFAIK based on 2nd hand lore). It was "limited", but those limitations made it mesh with human limitations. Bazillions of CICS and architectural lookalike systems were successfully deployed, and are still in use and under maintenance.

Then we've got all the web technologies and suddenly you can do everything in an almost infinite number of ways, and the tech involved can be selected to be almost arbitrarily complex. All this while the people who actually design and implement the stuff haven't magically become any more intellectually-able. They are the same species; there's no particular pressure to select truly cream-of-the-crop uber-developers with extraordinary mental capabilities. This impedance mismatch between the breadth of technology and the limitations of the implementers is a surefire recipe for disaster, especially given the demonstrably natural tendency for PHBs to jump onto "new" technologies for no reason other than those technologies happen to be out there.

There is a reason why there are many very successful projects that run on "legacy" technologies and purposefully limit themselves to what they use in the implementation. Had they not done it, they'd not be successful, it's that simple. I agree with Joel Spolsky that such legacy implementation should never be scrapped by a "total rewrite". You've got something that's CICS but you want to move on - port it to something more modern, web2py for all I care, but do it by a series of easy to understand (perhaps automated, even) transformations. Only when that's done, and you've got exactly the same system on a platform you feel like using in the longer term, should you start adding new features and doing deeper architectural changes.