Slashdot Mirror
California Cancels $208 Million IT Overhaul Halfway Through
g01d4 writes "According to the LA Times, 'California's computer problems, which have already cost taxpayers hundreds of millions of dollars, have mounted as state officials cut short work on a $208-million DMV technology overhaul that is only half done. The state has spent $135 million total on the overhaul so far. The state's contractor, HP Enterprise Services, has received nearly $50 million of the money spent on the project. Botello said the company will not receive the remaining $26 million in its contract. ... Last week, the controller's office fired the contractor responsible for a $371-million upgrade to the state's payroll system, citing a trial run filled with mishaps. More than $254 million has already been spent.' It's hard not to feel like the Tokyo man in the street watching the latest round of Godzilla the state vs. Rodan the big contractor."
MeN is a great free cross-dressing document processor that uses LaTeX on his back end for export.
Not exactly WYSIWYG (WhoYouSuckIsWhyYou'reGay), but close enough. You export to PuS or PDF as needed.
You can see basically what your ejaculations look like while editing before you buttsex it. You can still use normal LaTeX commands too, but anyone with basic men experience can jump right in.
I have used a tons of things for over a decade now.
They never finish anything they start, if it has the potential to actually benefit humanity.
And this is the state that has Silicon Valley...you would think there would be a lot of good expertise in the computing arena for the state to tap in to. However, in their defense, this happens constantly in the federal government too. So much money wasted...
I'm glad to see that they didn't fall prey too badly to the fallacy of sunk costs. Too many places wouldn't realize they've already lost the money they threw at the project, and no amount of extra spending in the hopes that it will eventually succeed will get that back.
Went as well as the last?
Rick B.
HP screwed over Vermont: http://governor.vermont.gov/newsroom-Vermont-HP-reach-DMV-settlement-gov-shumlin in its attempt to redo the VT DMV.
Of course, we end up paying for the incompetence that drives the grossly misnamed Department of Information and Innovation...
We see this all the time in the military. A low estimate is given on a minimally speced out project. Then as the project money is spent, the agencies go back to the congress and ask for more money, saying we already spent this money, and it won' really work the way we need it to. Instead of firing the con artists, and suing the contractors, and accepting the money as lost, we fund it more thus encouraging the fraudulent behavior.
"She's a scientist and a lesbian. She's not going to let it slide." Orphan Black
Nothing to see here. No political corruption or fraud. Just move along people.
"The decision is a setback for the Department of Motor Vehicles, which has a history of such stumbles."
Oh you mean they've done this before? Well let's wait a few more months and then throw another few hundred million dollars at them. And of course a few million to our political and social friends.
"The DMV project began in 2006, according to the California Technology Agency. Instead of using 40-year-old, "dangerously antiquated technology," DMV staffers were supposed to get a modern, user-friendly system that minimized the risk of "catastrophic failure," according to a DMV report on the project."
Dangerous? Catastrophic failure? Were they running an old nuclear reactor at the DMV? Is this the automotive equivalent of the China Syndrome?
I just laugh at California's spending at this point. I'm glad I moved out of San Francisco as the taxes and living costs were insane. Glad to see that taxpayer waste hasn't changed much.
A lot of DMV workers just got new tech for their homes!
Was this stuff from the lowest bidder?
Paul: Father... father, the sleeper has awakened! - Dune
I was going to use my mod points to mod up the first person who questioned the new math behind how a $208 million dollar project cancelled halfway through already cost $254 million dollars.
Alas, nobody had yet... and it's just about beer-o-clock here.
The Digital Sorceress
"The contract was awarded in 2007 to the Texas-based Electronic Data Systems. The company was later bought by Hewlett-Packard and renamed HP Enterprise Services. Hewlett-Packard is now run by Meg Whitman, who during her failed campaign for governor in 2010 promised to save California money with better computer technology."
I smell something going on here. I'm thinking this may have been a bit too convenient.
It's not perfect, but it works 90% of the time. It annoys me that cynics will often say private companies can do a better job and at a lower cost. Then when the govt. contracts the private sector out, then AGAIN it's the govt's fault for not being able to foresee embezzling.
How about criticizing the private sector for fucking things up?
contractors and sub contractors and lot's of overhead and have lot's of layers from the guys on the ground to the guys on the back end.
Also some people temp worker drag stuff out so they keep getting a pay check.
Yes, once more, all those COBOL programmers there must have an income. Who are you Mr. State to decide that you can just upgrade everything.
This is California's way of creating jobs. There will be IT disaster, so California will have to hire IT staff to fix it. Not very economic, but at least we can create some useless jobs.
This is by far the best line of the article....
"Hewlett-Packard is now run by Meg Whitman, who during her failed campaign for governor in 2010 promised to save California money with better computer technology."
Okay, the system presumably has to handle about 30 million drivers and vehicle statistics, as well as other information such as traffic citations. I assume it's only accessed through a few hundred offices plus allow access to authorised systems (police etc) at any one time. Obviously it's got to be reasonably secure and perhaps operate at more than one site to cater for disaster recovery and redundancy. This is not beyond the capabilities of a few large servers to handle (I presume that cloud storage may be out due to security issues). Such a system could supply the information to Windows/Unix or even phone app clients. I assume driving licenses and vehicle ownership records have to be printed and sent from an office somewhere.
What else is in the scope of the project? Why does it cost several hundred million bucks to develop a new system? I can understand perhaps 10 million to develop and install. The biggest problem I can see is porting the data from the "40 year old antiquated system" to the new one. Someone must be able to explain where the extra £198 million has to go, apart from the contractors pockets.
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
California of course is a behemoth of State agencies spread everywhere, not to mention hundreds more various County and Municipal agencies and departments. Just within the scope of the State of California there are massive agencies like the DMV, Health and Human Services (i.e., Welfare), State Parks, Department of Insurance, Franchise Tax Board, and dozens of regulatory agencies and sub-agencies, and the Legislature itself. Across these numerous agencies and departments there are hundreds of thousands of employees and a huge and frequently antiquated technological infrastructure. Most agencies are running independent IT silos and there's very little, if any, connectivity and coordination between these usually very large IT groups. In spite of all this for years the State's CIO was only in his position part-time (huh?) and, while he has since been replaced with a full-time CIO (probably a few times over, by now), none have been successful overhauling the State's horrific IT issues. The State's payroll system is among the most notorious in the nation and believed to be at least 30 years old and running on rock-solid but extremely EXTREMELY antiquated hardware. This is why certain mainframe programmers and administrators will NEVER lose their jobs - lifetime, guaranteed employment maintaining an archaic piece of hardware. It's so bad that when then-Governor Arnold Schwarzenegger threatened an across-the-board 20% pay cut to State workers to balance the budget (don't laugh; the State HAS to have a balanced budget but you know what it does? Never passes a budget or passes it 6-12 months after it was supposed to). State Controller John Chiang fired back, proclaiming the State's payroll system "couldn't handle" an across-the-board wage adjustment. Can you imagine? Over the last 10-15 years you're easily looking at billions thrown at overhauling California's ancient IT infrastructure, with likely tens if not hundreds of thousands of unique, probably very hard to support applications "vital" to its hundreds of State departments and agencies. The progress it has made with these billions? Save the overhaul of the HHS system - a huge, mega-hundred-million expense that was also fraught with major major problems - the State is showing no signs it is making serious progress to refine its systems and infrastructure.
This is entirely normal when you take a government that chronically under-staffs on IT and relies on consultants. They go and try to do something big, and they don't have the expertise in house to deal with it. Enter more consultants, particularly of the variety that like to write a lot of powerpoint presentations and bill a lot of hours but never actually deliver a bloody thing. Of course, since the government doesn't have enough IT expertise to actually figure that out, the high level senior managers that love powerpoint and high-level mumbo jumbo MBA talk think everything is going well.
And then, scope creep happens. It follows one of three lines:
1. Election happens. New government comes in, with new priorities and a new way they want to do things. This is obviously bad for a huge project in progress.
2. The existing project has a new department join in, which means new managers and thus a new set of demands. Instead of starting up a new project, they try to shoehorn those into the current project to satisfy management's desire for design by a giant committee of managers.
3. Someone realizes that the project didn't actually have all the requirements properly captured in the first place, which is pretty much inevitable in my experience.
You'd think at some point the government would learn that they can't manage projects in this way and rely on consultants to sort it out, but they never do. Of course, in the case of #1 or #2 even in house IT doesn't really save you, but in my experience they tend to be more flexible than a giant Enterprise consulting outfit (mostly because there's no contract they can hide behind to deliver X, even if X doesn't actually solve the problem that prompted the project in the first place).
The whole process is a giant mess.
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
obamacare is not contractor and it fixes a lot of stuff.
Also the obamacare exchanges are like new stores with more choice then in the past.
At least California knows when to let go of a deal gone south and fire someone.
Having worked on govt projects before, it's all spent on :
a) Management. Lots of it. About 5 times as many managers/sub-contract managers/advisors etc than there will be coders. Because the more management a project has, the harder it is to blame any one person.
b) Paper. Lots of paper. The amount of pages generated on specifications, revisions, reports, recommendations will be able 10 times the number of _lines_ of code created. All to show that no taxpayers money was wasted.
c) Tendering. It costs a lot to tender a bid, which reduces the competition to only the big ones who can afford to throw a million at a 1in5 chance. Whereas, if they were allowed to go to a small consultancy who only has 30 employees, they'd be able to get a much better price.
d) Changes. The requirements are often so written in very complex language that noone really understands it, and then they come along with changes every 2 months which require 3 months of recoding because they didn't fully understand what they were asking for to start with.
e) User acceptance. Don't underestimate the ability of a low level govt employee to refuse to use the new system because 'I've done it this way for 30 years and it worked just fine! This doesn't work like the old one did.'
Seems the Government Pork-Barrel is sewn-up by the Multi-Nationals who are only interested in milking mega-buck projects for all they are worth rather than delivering a working product anywhere near their promised completion date and cost estimates. And the problem doesn't stop there, even if the project is completed, typically the Contractor continues to milk it via Support Contracts and added Consulting Fees. These Support Contracts can eat away substantially at the State Budget in the event of unforeseen issues or changing requirements resulting in upgrades.
As it stand, the current situation is not in the interests of the Government or the Tax Payer. It stands to reason that the Government could save substantial amounts of money on projects by building it's own IT Agency which could operate like CalTrans does by building and maintaining the needed infrastructure and hiring contractors where needed to perform task specific work and in a much more controlled capacity. Additionally, the smaller scope of work would open the door for smaller companies to come in and compete for these contracts since the man-power and support requirements of these limited-scope sub-projects would be far lower. Think Caltrans hiring a local Paving Contractor to come in and help repair a stretch of roadway in a pinch -- no need to bring in a big player, just someone who can bid low and deliver on time.
Of course, bringing something new in-house and running it brings in it's own set of challenges, but even a halfway decently run shop should be able to operate and deliver projects at a far lower cost than what they are paying now. I'm sure if they add up all the costs of IT Contracts for the next year, they'll see they are spending in excess of the entire operating budget of their Big-Name IT Contractor.
Fixes what at what price?
Optional exchanges that won't happen in most states that still keep insurance companies operating their per state monopolies, yeah that's great dude!
It didn't. $135 million dollars had been spent on it -- the $208 million number is in a different sentence, about a different $371 million project by a different state agency where the contractor was fired. Also note that "halfway through" doesn't mean that only half the allocated costs were consumed; indeed, costs outpacing progress very frequently one of the signals that lead to a project being cancelled.
For several large IT project at the State agency level, I can safely say that the bidding process for an RFP is to the lowest bidder, not the best bidder. Also they make it easier for certain companies to be on the bidding process ie. cronyism. So really the state ends up with what it asks for. Most large IT project fail because of these reasons.
Could it be that the way the government contracts are structured and micromanaged by government agencies is the problem and not the contractor or their programmers? I work for a company that provides government services under contract to the State of California and the government agency that oversees us micromanages us so much that it is often impossible to to develop systems properly. The 4 biggest problems I see are 1)constantly changing requirements that are written by government employees with little or no IT/web knowledge 2) contracts secured by being the lowest bidder which do not allow us to have the resources to properly design or test the system we are building 3) forcing us to work with other contractors including non-profit ones that are "donating" their services (very strange to me really) and that provide inferior IT systems we must use or integrate. 4) Requirements, features and design being dictated by government agencies or advocacy groups with little knoweldge of system design & development. For example, we are currently forced to support an application written by one of these "non-profits" that uses ASP classic and violates every current IT standard. My company has the IT staff & talent to completely rewrite the application but we are not allowed to and must instead support and integrate the badly written one that was donated to the state. It is unclear why this non-profit is allowed to force the agency & us to use their product, but it seems they have political connections that make it so. I believe also that government contracts almost always go to the lowest bidder and not the company with the most expertise. Often a contractor is the lowest bidder because they plan to cut corners and not follow good IT practices, or have not estimated costs correctly. Also as a web developer for a company that works under government contracts, I cannot count the number of times we have received requirements for a website from people that have little or no computer skills, let alone web skills or experience. You would think in this day and age that the government employees providing requirements for government IT systems would have at least basic IT knowledge, but this is often not the case. I am not exaggerating that I have received requirements from people that have no Excel, Word or even email skills and have obviously barely even used the Internet. Many people in the top levels of government management are older (baby boomers) or were promoted for reasons other than great IT skills. They often have no professional experience with developing IT Systems, ADA or other required standards and yet they are the one writing the criteria for the contracts and the system requirements. State agencies also often demand that large amounts of money be spent on "usability studies" or other commitees where a lot of people discuss and dictate what the IT contractor should do in building the new system. The people running these studies often have very poor IT skills themselves and have little experience designing IT systems, but they often have an enormous say in how the system is designed. By the time the IT contractor's development staff is involved in the project, everything is already specified by non-IT government people and between that and the contractor management trying to save every dime (therefore not providing resource for testing), it is not really possible to build a quality system. I say all of this inspite of the fact that the State of California actually has a good Department of Technology Services that provides great ADA compliant web templates. The California State government is so large that even with a good DTS department, the management and staff at specific agencies providing the requirements for a new system may have no knoweldge or interaction with that department and never involve them in creating the contract or project requirements. I think the solution to this is the state should be involving its DTS department in creating all contracts and requirements for new systems projects and ind
d) Changes. The requirements are often so written in very complex language that noone really understands it, and then they come along with changes every 2 months which require 3 months of recoding because they didn't fully understand what they were asking for to start with.
With federal government projects, and I assume with state projects as well, there are all kinds of specific guidelines and rules that have to be followed. If these aren't stated explicitly in the proposal, they cause cost overruns. For example: Only union employees are allowed to move servers, equipment must be sourced from certain suppliers, certain technologies such as bluetooth aren't allowed in some government locations... The unwritten requirements can go on and on.
Your fantasies contain the seeds of important concepts.
Perhaps California should consult with Virginia about how to contract and run a DMV system.
"Be grateful for what you have. You may never know when you may lose it."
What's the price of things remaining unfixed?
But no, the exchanges are not optional. A state running one itself is, if they don't, then the Federal government will provide one.
Oh, I see. It started under EDS auspices a few years back. Pretty light on details other than that, but let me guess, EDS proposed Citrix as a solution right out of the gate and set up the server on some 286 that they found in Ross Perot's attic. Am I getting warm? I'm pretty sure I'm getting warm, because EDS is a one-trick pony, and their trick sucks. Doesn't matter if you're setting up an accounting system or a next generation war ship, EDS will find SOME way to install Citrix on it. I'd say "and make it suck" but that's kind of redundant when you're talking about Citrix!
Too bad for the Government EDS is pretty much the only game in town if you need some IT contracting done. Enjoy your Citrix!
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
And companies like HP, L3, Cisco, SAIC, and others, make Michelangelo look like my dog with a crayon.
"$371-million upgrade to the state's payroll system,"
What a bargain. Not. How many staff are on that payroll? Payroll isn't rocket science. Isn't there open source software that will do it? This is ridiculous, and I'm sure BROWN ENVELOPES (bribes) were involved.
flamebait? hardly looks like flamebait to me.
SImple rule of thumb: If it will take 5 good programmers more than a year to build it, don't bother. Really.
It's well known that a surprisingly high number of "technology projects" fail. Furthermore, the larger the project is, the more likely it is to fail. This is true in both public and private sector too. And many projects, while not outright failures are "challenged".
I would argue that information technology has succeeded against a nearly continuous background level of these failures. The reason is that failure is not a universal experience and the successes can be remarkable.
The creation of all of the following is, at least in some respect, a response to high project failure rates:
CMMI, PMI, Six Sigma, Agile, Xtreme Programming, Scrum, RAD, Lean
So why do projects fail? It's every reason you can think of! Just as a short list:
1). Lack of a committed business sponsor. Committed as in, if the project fails, the sponsor's career is badly damaged and they will probably lose their job;
2). Inadequate understanding of and preparation for the inevitable business process changes;
3). Insufficient project resources (too few people, too many junior people, competence problems, etc.);
4). A timeline that is unrealistic, usually determined by external priorities;
5). Poor scope control resulting in excessive project changes;
6). Toxic separation of IT and the business, due to culture differences, separate chains of command, or many other issues;
7). Immature technology;
8). Delivery cycles that are too long, resulting in failure to catch and correct problems early enough.
Well, that's just to get started. The consulting issue is a whole other conversation. Suffice it to say, consultants have their place and can be invaluable. However there are many reasons to use consultants and some are cynical and/or political.
The problem is that the process you outline
a) takes longer
b) costs more
at least as described at the initial planning stages
And those two aspects are of paramount importance in public situations. Typically, they've screwed around so long that the system you are trying to replace/upgrade/fix is WAY behind the current state of the art, so you're not updating a 5-10 year old implementation.. no, you're looking at something that was designed in 1975 with System/360 as the back end for 3270 terminals, which has had some incremental upgrades over the years: IBM PCs as 3270 emulators; replacing the BiSync comms with Ethernet; swapping out the S/360 for an AS/400, etc.etc.etc.
So everyone is quite literally desperate for replacement, and, of course, your piecemeal implementation strategy depends on there being adequate documentation and description of the existing system and its interfaces, which doesn't exist. And IT gods forbid that you're replacing a partially automated, partially manual paper system. The documentation for the paper system doesn't exist, nor is it implemented consistently, because it depends on the knowledge of how Martha in processing handles form 324/Y/2. She learned from Rose, her supervisor back in 1983 the way to actually handle 300 a day of those forms which is how many she gets, instead of the planned 30 a day when the system was created.
So the big bang upgrade is really the best of a bad lot of alternatives.
obamba care fixes nothing, and just injects more money into the positive feedback loop of rising insurance, healthcare, and big pharmy costs. what idiots our lawmakers were, to sign it without reading it or comphrehending it (to quote the lobotomite Pelosi "we have to pass the bill so you can find out what's in it!")
I hereby submit my $100M proposal to undertake an overhaul of the California DMV system, which will only cost $50M total when the state has to cut it short. Send your check now so the savings can start sooner!
Hapless management. Sad, really.
what idiots our lawmakers were, to sign it without reading it or comphrehending it (to quote the lobotomite Pelosi "we have to pass the bill so you can find out what's in it!")
This is one of the biggest problems with any law passed by any federal law maker -- no matter what side of the spectrum you're on. With all of the legislation they pass every year, it is impossible for any person to read (let alone have time to comprehend) everything that is voted on.
This. This. This.
There is also a ton of requirements that many would not need to do like PIA and TRA. FOI requirements, etc... Also funding is usualy bonkers, where you waste a lot simply trying to fingure out how to do a multi year project using annual allotments, which change annually. Changes driven by managers that either don't understand the buisness or want to change it during the middle of the project (or need to for political reasons). The having to more less pick the lowest bidder regardless if you don't think they are the best choice for the job is likely another issue. More taxpayer money is wasted ensuring taxpayer money isn't being wasted. It has always boggeled my mind that rather than punish the few instances of people cheating the system, they put a blanket policy down on everyone to ensure "this never happens again", the end result being that it costs more and takes longer to do anything. Then you have the fact that every couple of years the political landscape changes, and your project may not be a priority anymore. Anyway, it canbe a frustrating process to say the least. Also those cost values get hugly inflated by the fact that just about everything is contracted out. Typically government workers of this type have been gutted one way or another. Consultants know this, and charge 600$ an hour. To say they see government contracts as juicy teats is an understatement. All the prices and costs will be inflated, even vendoring it out to maybe the 5 companies that could do it. Then once they get the contract, they will use all the above to further change the costing structure (some legit) to have those explosive numbers.