Slashdot Mirror
California Cancels $208 Million IT Overhaul Halfway Through
g01d4 writes "According to the LA Times, 'California's computer problems, which have already cost taxpayers hundreds of millions of dollars, have mounted as state officials cut short work on a $208-million DMV technology overhaul that is only half done. The state has spent $135 million total on the overhaul so far. The state's contractor, HP Enterprise Services, has received nearly $50 million of the money spent on the project. Botello said the company will not receive the remaining $26 million in its contract. ... Last week, the controller's office fired the contractor responsible for a $371-million upgrade to the state's payroll system, citing a trial run filled with mishaps. More than $254 million has already been spent.' It's hard not to feel like the Tokyo man in the street watching the latest round of Godzilla the state vs. Rodan the big contractor."
And this is the state that has Silicon Valley...you would think there would be a lot of good expertise in the computing arena for the state to tap in to. However, in their defense, this happens constantly in the federal government too. So much money wasted...
I'm glad to see that they didn't fall prey too badly to the fallacy of sunk costs. Too many places wouldn't realize they've already lost the money they threw at the project, and no amount of extra spending in the hopes that it will eventually succeed will get that back.
HP screwed over Vermont: http://governor.vermont.gov/newsroom-Vermont-HP-reach-DMV-settlement-gov-shumlin in its attempt to redo the VT DMV.
Of course, we end up paying for the incompetence that drives the grossly misnamed Department of Information and Innovation...
It's not about the government. It's all about the useless IT consulting companies. Pretty much every single flashy consulting company billboard/AD that you see at an airport is just a way to milk the gullible and not deliver. This is an across-the-board problem. Nobody wants to fucking do their jobs. The government thinks they don't need the right people to do it, so they hire a contractor. The contractor doesn't want to do the job either, it's not their core competency (nobody knows what it is anyway), so they hire subcontractors. Subcontractors have very little vested interest in anything, and they maybe deliver, maybe not, but due to multiple layers of clueless management, it's of no use anyway. So there you go.
A successful API design takes a mixture of software design and pedagogy.
We see this all the time in the military. A low estimate is given on a minimally speced out project. Then as the project money is spent, the agencies go back to the congress and ask for more money, saying we already spent this money, and it won' really work the way we need it to. Instead of firing the con artists, and suing the contractors, and accepting the money as lost, we fund it more thus encouraging the fraudulent behavior.
"She's a scientist and a lesbian. She's not going to let it slide." Orphan Black
I was going to use my mod points to mod up the first person who questioned the new math behind how a $208 million dollar project cancelled halfway through already cost $254 million dollars.
Alas, nobody had yet... and it's just about beer-o-clock here.
The Digital Sorceress
"The contract was awarded in 2007 to the Texas-based Electronic Data Systems. The company was later bought by Hewlett-Packard and renamed HP Enterprise Services. Hewlett-Packard is now run by Meg Whitman, who during her failed campaign for governor in 2010 promised to save California money with better computer technology."
I smell something going on here. I'm thinking this may have been a bit too convenient.
I thought it was an "emoticon" of a large breasted woman giving the finger.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
This is by far the best line of the article....
"Hewlett-Packard is now run by Meg Whitman, who during her failed campaign for governor in 2010 promised to save California money with better computer technology."
Okay, the system presumably has to handle about 30 million drivers and vehicle statistics, as well as other information such as traffic citations. I assume it's only accessed through a few hundred offices plus allow access to authorised systems (police etc) at any one time. Obviously it's got to be reasonably secure and perhaps operate at more than one site to cater for disaster recovery and redundancy. This is not beyond the capabilities of a few large servers to handle (I presume that cloud storage may be out due to security issues). Such a system could supply the information to Windows/Unix or even phone app clients. I assume driving licenses and vehicle ownership records have to be printed and sent from an office somewhere.
What else is in the scope of the project? Why does it cost several hundred million bucks to develop a new system? I can understand perhaps 10 million to develop and install. The biggest problem I can see is porting the data from the "40 year old antiquated system" to the new one. Someone must be able to explain where the extra £198 million has to go, apart from the contractors pockets.
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
California of course is a behemoth of State agencies spread everywhere, not to mention hundreds more various County and Municipal agencies and departments. Just within the scope of the State of California there are massive agencies like the DMV, Health and Human Services (i.e., Welfare), State Parks, Department of Insurance, Franchise Tax Board, and dozens of regulatory agencies and sub-agencies, and the Legislature itself. Across these numerous agencies and departments there are hundreds of thousands of employees and a huge and frequently antiquated technological infrastructure. Most agencies are running independent IT silos and there's very little, if any, connectivity and coordination between these usually very large IT groups. In spite of all this for years the State's CIO was only in his position part-time (huh?) and, while he has since been replaced with a full-time CIO (probably a few times over, by now), none have been successful overhauling the State's horrific IT issues. The State's payroll system is among the most notorious in the nation and believed to be at least 30 years old and running on rock-solid but extremely EXTREMELY antiquated hardware. This is why certain mainframe programmers and administrators will NEVER lose their jobs - lifetime, guaranteed employment maintaining an archaic piece of hardware. It's so bad that when then-Governor Arnold Schwarzenegger threatened an across-the-board 20% pay cut to State workers to balance the budget (don't laugh; the State HAS to have a balanced budget but you know what it does? Never passes a budget or passes it 6-12 months after it was supposed to). State Controller John Chiang fired back, proclaiming the State's payroll system "couldn't handle" an across-the-board wage adjustment. Can you imagine? Over the last 10-15 years you're easily looking at billions thrown at overhauling California's ancient IT infrastructure, with likely tens if not hundreds of thousands of unique, probably very hard to support applications "vital" to its hundreds of State departments and agencies. The progress it has made with these billions? Save the overhaul of the HHS system - a huge, mega-hundred-million expense that was also fraught with major major problems - the State is showing no signs it is making serious progress to refine its systems and infrastructure.
This is entirely normal when you take a government that chronically under-staffs on IT and relies on consultants. They go and try to do something big, and they don't have the expertise in house to deal with it. Enter more consultants, particularly of the variety that like to write a lot of powerpoint presentations and bill a lot of hours but never actually deliver a bloody thing. Of course, since the government doesn't have enough IT expertise to actually figure that out, the high level senior managers that love powerpoint and high-level mumbo jumbo MBA talk think everything is going well.
And then, scope creep happens. It follows one of three lines:
1. Election happens. New government comes in, with new priorities and a new way they want to do things. This is obviously bad for a huge project in progress.
2. The existing project has a new department join in, which means new managers and thus a new set of demands. Instead of starting up a new project, they try to shoehorn those into the current project to satisfy management's desire for design by a giant committee of managers.
3. Someone realizes that the project didn't actually have all the requirements properly captured in the first place, which is pretty much inevitable in my experience.
You'd think at some point the government would learn that they can't manage projects in this way and rely on consultants to sort it out, but they never do. Of course, in the case of #1 or #2 even in house IT doesn't really save you, but in my experience they tend to be more flexible than a giant Enterprise consulting outfit (mostly because there's no contract they can hide behind to deliver X, even if X doesn't actually solve the problem that prompted the project in the first place).
The whole process is a giant mess.
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
obamacare is not contractor and it fixes a lot of stuff.
Also the obamacare exchanges are like new stores with more choice then in the past.
>"The DMV project began in 2006, according to the California Technology Agency. Instead of using 40-year-old, "dangerously antiquated technology," DMV staffers were supposed to get a modern, user-friendly system that minimized the risk of "catastrophic failure," according to a DMV report on the project."
This encapsulates solving multiple problems at the same time. This cannot be done. You update large systems by plotting a path through incremental improvements that get deployed, tested and fixed before the next increment, so that get you to where you need to be. It might not seem like the optimal path, but anything involving a switch over of technology, UI, back end, infastructure and buckets of code all at the same time is simply never going to work.
In the case of the DMV, it might involve unifying disjoint databases pair by pair until you have only one, while maintaining the same interface to the heterogeneous clients. Then one by one converting the heterogeneous clients to a standard back end interface. Then one by one adding the features of a client to the grand unified client and switching over that system, until the GUC has all the features for all the clients are new client. Then one by one, updating the organizational procedures to make them better, and updating the GUC while doing so. You can make these changes one by one. You can roll back one step if it doesn't work right the first time. You can measure progress by the number of working updates, not in how much less non-working the global-replace-systems is today.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
Look, when you're an IT consultant company, your first job is to do a discovery project that is aimed precisely at defining exactly what the project is -- in absence of proper requirements. Heck, if it's a bidding process, you must also do due diligence -- that would typically involve talking to the customer(s), talking to their employees, etc. It's your own fault, as a consulting company, if you can't deliver. You must be able to tell that the customer is too clueless to make it work. It's your fucking job.
A successful API design takes a mixture of software design and pedagogy.
Nobody wants to fucking do their jobs.
That about sums it up right there.
"If a nation expects to be ignorant and free in a state of civilization, it expects what never was and never will be."
Having worked on govt projects before, it's all spent on :
a) Management. Lots of it. About 5 times as many managers/sub-contract managers/advisors etc than there will be coders. Because the more management a project has, the harder it is to blame any one person.
b) Paper. Lots of paper. The amount of pages generated on specifications, revisions, reports, recommendations will be able 10 times the number of _lines_ of code created. All to show that no taxpayers money was wasted.
c) Tendering. It costs a lot to tender a bid, which reduces the competition to only the big ones who can afford to throw a million at a 1in5 chance. Whereas, if they were allowed to go to a small consultancy who only has 30 employees, they'd be able to get a much better price.
d) Changes. The requirements are often so written in very complex language that noone really understands it, and then they come along with changes every 2 months which require 3 months of recoding because they didn't fully understand what they were asking for to start with.
e) User acceptance. Don't underestimate the ability of a low level govt employee to refuse to use the new system because 'I've done it this way for 30 years and it worked just fine! This doesn't work like the old one did.'
Oh good grief what a bunch of bunk. Do you know why big contracting companies exist? It's because companies have lost faith in their internal organizations to deliver. Plain and simple. I've seen it in dozens of organizations in my career where an entrenched group builds a castle of "can't." So, the execs hire an outside firm not only to get what they want but to also force these little castles to actually deliver something. Lazy contractors? Please, how about lazy employees who feel that they hold the keys and as long as they keep pushing back and feel empowered so nobody is going to mess with their careers even though they may be doing such innovative things like writing System 360 Assembly Language. It's everywhere and it's not just in IT, people in this country have become lazy and foolish relying on attitude rather than customer service and trying to do a good job.
Every large organization has this problem and IT isn't just one of the areas where dead space can occur. So, the big companies come in, push change, make big promises that sometimes are overblown. In the case of California I could probably guess that the specifications of what were required were done by bureaucrats who have no clue on how to spec out requirements or were based on something that wasn't possible to build. Are there bad firms? yes, but are there bad customers? hell yes and they can make it absolutely impossible to deliver anything because the same people who have to approve or test anything are usually overworked, or not committed to the project leaving the contractor and subsequently the whole project in limbo. That usually leaves to failure despite the best efforts of all parties involved.
So before you blame consulting companies for this failure, remember they wouldn't exist if people were doing their job or came to the realization that their skills and abilities are out of alignment with what their management expects.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
I'll never look at that word the same way again :)
Therefore, by the (faulty) logic you're using, you're just a cow with a keyboard - osu-neko (2604)
This. I hired a popular 'big' construction company to do some renovations on my business. They got as far as fucking everything up and then holding on to the money as long as possible. It took a lawsuit and a solid WEEK in civil court to get them to give ANY of the money back. Once they were court ordered to give the money back, they received extension after extension and after all was said and done, I had my money back and then had to shop for other people to do the work.
Guess what? Very few of them would even continue the conversation after they learned who my company was. So I sat there for three months without a physical location for my business, while the city told me over and over I had to do something with it. Only through extensive paperwork and documentation was I able to prove that I was pretty much blackballed from all the local construction companies. I ended up paying something like 3x the original cost (basically 3x more than market value) for the work.
So you're basically screwed. If you retaliate legally for them not doing their job JUST to end up with your money back, you get blackballed and no one will deal with you unless you pay them a ton more. If you don't retaliate legally, you've basically wasted the money. This is probably where they stand right now.
Could it be that the way the government contracts are structured and micromanaged by government agencies is the problem and not the contractor or their programmers? I work for a company that provides government services under contract to the State of California and the government agency that oversees us micromanages us so much that it is often impossible to to develop systems properly. The 4 biggest problems I see are 1)constantly changing requirements that are written by government employees with little or no IT/web knowledge 2) contracts secured by being the lowest bidder which do not allow us to have the resources to properly design or test the system we are building 3) forcing us to work with other contractors including non-profit ones that are "donating" their services (very strange to me really) and that provide inferior IT systems we must use or integrate. 4) Requirements, features and design being dictated by government agencies or advocacy groups with little knoweldge of system design & development. For example, we are currently forced to support an application written by one of these "non-profits" that uses ASP classic and violates every current IT standard. My company has the IT staff & talent to completely rewrite the application but we are not allowed to and must instead support and integrate the badly written one that was donated to the state. It is unclear why this non-profit is allowed to force the agency & us to use their product, but it seems they have political connections that make it so. I believe also that government contracts almost always go to the lowest bidder and not the company with the most expertise. Often a contractor is the lowest bidder because they plan to cut corners and not follow good IT practices, or have not estimated costs correctly. Also as a web developer for a company that works under government contracts, I cannot count the number of times we have received requirements for a website from people that have little or no computer skills, let alone web skills or experience. You would think in this day and age that the government employees providing requirements for government IT systems would have at least basic IT knowledge, but this is often not the case. I am not exaggerating that I have received requirements from people that have no Excel, Word or even email skills and have obviously barely even used the Internet. Many people in the top levels of government management are older (baby boomers) or were promoted for reasons other than great IT skills. They often have no professional experience with developing IT Systems, ADA or other required standards and yet they are the one writing the criteria for the contracts and the system requirements. State agencies also often demand that large amounts of money be spent on "usability studies" or other commitees where a lot of people discuss and dictate what the IT contractor should do in building the new system. The people running these studies often have very poor IT skills themselves and have little experience designing IT systems, but they often have an enormous say in how the system is designed. By the time the IT contractor's development staff is involved in the project, everything is already specified by non-IT government people and between that and the contractor management trying to save every dime (therefore not providing resource for testing), it is not really possible to build a quality system. I say all of this inspite of the fact that the State of California actually has a good Department of Technology Services that provides great ADA compliant web templates. The California State government is so large that even with a good DTS department, the management and staff at specific agencies providing the requirements for a new system may have no knoweldge or interaction with that department and never involve them in creating the contract or project requirements. I think the solution to this is the state should be involving its DTS department in creating all contracts and requirements for new systems projects and ind
Companies lost faith in their internal organizations? No, chicken shit unknowledgeable managers (not companies) don't want to take responsibility for doing anything more complex than a CRUD service, so they send it to Accenture and IBM. It is a big blame game and a web of CYAs.
I think you've been reading too much fiction. The discovery and due diligence processes are there for the benefit of the customer, in order to make it appear that you know what you're doing with their project, even when their project is a mess - which is most of the time.
If you tell the customer that they are clueless, you probably lost the contract. So, you have to pussyfoot around and figure out how you can make any progress at all, despite the fact that your management, and their management are both fighting you every step of the way.
You probably believe in the ghost called Minority Shareholder Lawsuits too.
Oh, I see. It started under EDS auspices a few years back. Pretty light on details other than that, but let me guess, EDS proposed Citrix as a solution right out of the gate and set up the server on some 286 that they found in Ross Perot's attic. Am I getting warm? I'm pretty sure I'm getting warm, because EDS is a one-trick pony, and their trick sucks. Doesn't matter if you're setting up an accounting system or a next generation war ship, EDS will find SOME way to install Citrix on it. I'd say "and make it suck" but that's kind of redundant when you're talking about Citrix!
Too bad for the Government EDS is pretty much the only game in town if you need some IT contracting done. Enjoy your Citrix!
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
And companies like HP, L3, Cisco, SAIC, and others, make Michelangelo look like my dog with a crayon.
The problem is that you are both right!
Just look at the new California high speed rail system and how it went from a simple concept to a bloated monster.
Both teams are screwing up and they will both blame the other team for the failure.
For every problem there is a solution that is simple, obvious and wrong.
There are some cases also where this misalignment of skills and management expectations is more of a management deficiency. Many organizations have technical people who are quite willing and capable, but they have been pigeon-holed and beaten down by policies which incentivize apathy. I have worked with long-time developers in quasi-government jobs who have skills only on legacy systems, and I have had the pleasure of helping them participate in the development of modern SOA interfaces. Most of these people just need an opportunity to learn, grow, and feel like their contributions will be meaningful. And it is not expensive, in fact if it looks expensive you are doing it wrong! You don't need or want conventional "training" for them, and if it is done right, it can cost little or nothing in extra time. These people have a goldmine of lost productivity to tap into -- productivity that poor management has beaten out of them and that good management can cash in on.
Strange things are afoot at the Circle-K.
Problem 1: Sales team says yes to everything the customer wants.
Problem 2: The business managers think anything tech-related is trivial. Commoditized. Already been done before. Simple to implement.
Problem 3: Uncooperative government employees who don't want to be bothered or want to protect their little empire.
These are very difficult issues.
Re: Problem 1 - if our sales team doesn't nod eagerly at every requirement, someone else's team will. The customer can't tell if he's being lied to or not.
Re: Problem 2 - business managers like to look at things in the aggregate. BUT - the devil's in the details. And getting the right people is very difficult. You need a talented business manager to understand what what is complex, what is possible, what is impossible.
Re: Problem 3 - You have to find a way to circumvent many of the government employees / chieftains with this attitude. It requires buy-in from top management. But if it's a powerful sysadmin chieftain who doesn't want his empire f--ked with, well... you're going to need a full separate testbed on which to deploy and test the system. Hard to do if you're working almost totally on the client site.
But ultimately, to the business, it's about getting paid. And someone's been paid 100's of millions so far. So, from the business side, that would be considered a success. That's a problem too, if you're the customer trying to get a quality final product..
My experience is that the problem is too much govt. oversight. At least in my industry they don't say okay we need a system to do this, how much will it cost, how will you implement it, okay, that looks fine, now go ahead and do it! It's no I want it done this way, and then two weeks later I want it done that way, and then I want it totally changed to this, and now that.
The unethicalness of govt. contracting co.'s is in saying sure whatever to all of that, because they get paid all the same. I get paid all the same too, but I want to get something done. And so do my coworkers.
The problem is that for the govt. reps, it's not their money, they don't give a shit, and they're unaccountable so they don't have to. They can use it as an ego-stroking fest, because the contracting company will worship their every word as long as the funding spigot stays open. So none of the big-wigs in either party to the transaction care if anything ever gets done.
Attention zealots and haters: 00100 00100
Well, a contractor experienced supporting a particular industry would find the processes in place and help the customer learn about what can be done to improve them. Most customers moving to new tech won't know what to implement, they just know things need to be more efficient. As a contractor it is up to you to NOT do what the customer tells you, but to convince them to use the best solution for the problem.
"If the customers don't know what they want, you teach them" - Umm...if the customer doesn't know what they want then the project is in deep and immediate trouble. It is the customers business to run and if they don't know what they want they had damn well better figure it out before they spend a boatload of money on a team of consultants.
I have worked on many large projects, some with government agencies some with large private companies, and the most successful ones have a common thread. The common thread is that the critical project decisions are a collaborative effort, not simply punted to the consultants to say "here, you figure it out". If a company has management that is unwilling or unable to make decisions that company will fail. I don't care how many consultants you bring in, if the management cannot articulate how the business is currently run and what the future goals are the project will fail.
Are there shitty consultants out there? Sure. Just like there are company executives that have no business running a lemonade stand, never mind a multi-billion dollar company or large government office. To succeed you need talent on both sides.
Some of the government IT systems that I have been exposed to are arcane, to say the least. Often they have little or no documentation and are extremely complex. Many of the agencies are under funded and under staffed. Talent and motivation, frankly, are sometimes lacking. Basically it's a difficult environment to work in.
Really big projects like this have a high failure rate. There are a multitude of reasons, some of them above. I suspect that both sides share in the blame.
If your management is fighting you, you have already lost the battle -- you're working for the wrong bosses. As for their management: that's why you hire the right people who can set the customer straight while the customer is thinking all the time that they are in charge. Manipulating career bureaucrats is easy -- they are all the same, after all. You only need those who know how to pull it off working for you. That's all. I've heard first hand from a small European consulting company that undertook a few out of the view but significant projects, and they had two psychologists on staff -- one specializing in negotiations and another one specializing in profiling. I think a couple private investigators were also temporarily retained to figure out the exact political pressures in play. A couple weeks into the bid review process they had a pretty good handle about the personalities of all the key people who they were interfacing with on the customer end, and the bosses above those people. The competitors didn't even stand a chance.
A successful API design takes a mixture of software design and pedagogy.