Slashdot Mirror
Ships Turned Away As Aussie Customs' IT System Melts Down
An anonymous reader writes "Urgent shipments of medicine and goods for the holiday season have been turned away by customs officials due to a massive computer problem. The initial budget for the system upgrade was said to be A$80 million but has since blown out to A$250 million. Customs officials and the government have been forced to admit that they might actually have to revert to the old system if things don't improve. One cargo user said on national TV that he used to process 300 orders daily but the new system is so complex and unusable, he's happy if he can manage 100 orders per day. The system failure is expected to have a massive impact especially on the retail sector this Christmas."
Simple solution. Push it back 6 months till when it's actually cold!
Grump.
Is it true that more people vote for the winner of American Idol, than vote for the president? -Ali G.
I hope my shipment of inflatable Love Dolls makes it through customs, otherwise it's going to be a lonely new year.
This is news generally because of the rarity of this sort of thing. The only real issue I can think of is that our Health Minister has recently announced plans to immunise ALL Australians against bird flu. This could disrupt that (if it was realistic anyway). I guess this is all a part of ever increasing control.
What OS do they run?
What software do they use?
And how will their IT people and/or management continue to justify said choices in the wake of this?
This is the sort of thing that needs "big iron". Machines that have uptimes measured in decades. Why do I have the sneaking suspicion that they're running it all on a bunch of commodity PCs (or the like) with off-the-shelf software?
With spending like this, exactly what are "conservatives" conserving?
It'll be interesting to see what the ultimate culprit is. (overpriced IBM/Accenture contractors, Indian outsourcing, Windows, Linux, etc)
But I'm 99% sure it'll have something to do along the lines of:
"Mate, we need a new Customs software system."
"No prob. We'll do it in [whiz bang technoterm du jour]"
"That's it?"
"That's it. [whiz bang technoterm du jour] using [whizbang development process du jour]"
"But what about things like useability? Proof of concept? Customer Support if the design proves unwieldy?"
"Top. Men."
All the news and radio reports I've read and heard (including TFA) have made no mention of ships being actually turned away at this stage. So far they're just saying that the storage space at the ports is rapidly filling up, so if the processing rate doesn't improve soon they will have to look at turning ships away. But as far as I can tell, they're planning to roll back to the old system before that becomes necessary...
For the AU government to let goods travel freely until they fix or bring up the old system. There really is no excuse for what is going on. Yes, that means that the AU government doesn't get its cut of taxes but them's the breaks. The money lost from import fees would be DWARFED compared to the lossess incurred by *not* letting goods through the ports.
--
BMO
http://www.customs.gov.au/site/page.cfm?c=6361
Partial quote...
"Customs is doing everything possible to resolve technical and business issues arising from the introduction of the new Integrated Cargo System (ICS) for imports.
"Contrary to some media reports, the new IT system for imports has not failed, nor is its performance solely responsible for the problems that have occurred.
"The problems experienced in part, flow from inaccurate and incomplete information being submitted by some users, which the new system is designed not to accept for security reasons," the spokesman said.
I just can't stand it when they don't post whether it's a windows-based or a unix/linux-based implementation. I need to know whether I can indulge in schadenfreude or whether I have to make excuses.
I was actually part of a company a couple of years ago which put through a proposal to assist with tracking firearms imported into Australia. We were shocked at what we found when we consulted several customs offices.
There was no integrated network system between interstate customs offices.
Sure, they e-mailed each other and did some odd bits of communication, but there was nothing solid in place. Part of our proposal was to put in a system where if a shipment of firearms was sent from Melbourne to Sydney the Sydney office would actually know that one was going to arrive. A step up from their existing system at the time, where the firearms actually left Melbourne, turned up at the Sydney customs depot without prior knowledge and then processed!
Task Mangler
I'm no grizzled guru by any means, but damn, I know by now that though it *may* seem cheaper to upgrade all in one fell swoop, you're gonna get hosed every time. The bigger the system, the more likely, just because there's no way you can *test* the thing at that scale.
Software is *complicated*. Large-scale software rollouts are even *more* complicated, just because now you've involved hundreds or thousands of non-debuggable, unpredictable people into the equation. No matter how many meetings you have about it, no matter how many different people assure you that they will do "whatever it takes" to make sure it goes smoothly, keep in mind that they probably don't have "what it takes", which would often be some kind of deity-level power.
Let's look now at the "largest e-government projects ever undertaken", introduced "despite industry protests that Customs had not allowed them ample time for the changeover." It's not hard to guess how it's going to go.
Sometimes, you gotta go the slow way... replace the old system bit by bit, make sure you can flip the switch back every step of the way if something goes wrong. At the very least you have to plan it from the start so that you can roll out piecemeal, just in one site, or run the old/new in parallel, etc..
This method results in a more expensive *estimate* at the start of the project. But the actual *cost* in the end can be much, much lower.
Just my 2c...
80mil AUD = approx 50mil Euro = 60mil USD
250 = 156 = 188
Computer World Article
ICS is a cornerstone of Customs' massive Cargo Management Re-engineering (CMR) project. This was intended to replace the export and brokerage industry-developed EDI system Customs Connect with a Web-based model co-developed by Customs and a consortium of IT vendors led by Computer Associates. The project aims to facilitate all aspects of Customs involvement in the import and export process including declarations and GST transactions collected at port.
Nother Article
More than seven years to this point of readiness, ICS is a cornerstone of Customs' massive Cargo Management Re-engineering (CMR) project, which will replace the export and brokerage industry-developed EDI system, Customs Connect. CMR is a Web-based model co-developed by Customs and a consortium of IT vendors led by Computer Associates, EDS, IBM and Telstra nee Kaz.
Funny how to the state, free commerce isn't an option, but blowing $250 million that isn't even yours on a computer system that doesn't work is okay.
"Your papers, citizen! Whoops, my citizen-authorization-scanner just went dead. You'll have to be detained while I get fresh batteries. Oh, and that'll cost $10 - batteries aren't free, you know."
He who lights his taper at mine, receives light without darkening me.
Ah, keynote also has some nice data. Apparently Level3 & Verio have both been having severe problems in the last few hours, and have a high latency connecting to most of the other Level-2 & Level-3 providers:
x ?Login=Y&Username=public&Password=public
http://scoreboard.keynote.com/scoreboard/Main.asp
94% of Repubs and 21% of Dems voted to renew the Patriot Act
Yes
EDS == Everything Done Slowly
But in the Aussie case it could be changed to
EDS == Everything Done Shithousely
The real problem with this system is that it used the principle of "Big Design Up Front". Ask Joel Spolsky about the benefits of "Big Design Up Front" - you get to make all kinds of assumptions about the environment to simplify development, then find when you turn on the switch that this $80M system just doesn't work right.
The little things that get you down? Oh... date formats, validating input, units for measurement, using a communications system intended for overnight batch operations to support real-time interactive operations.
As other posters have mentioned, the bid that got the nod was the lowest one. The bid that should have received the goahead was the one that recommended incremental changes. The one that recommended introducing a new means for handling import declarations - and not cutting over, but rather letting the old one die the natural death of user migration.
The final nail in the coffin was Customs insisting that more detail be included in these reports - no longer can you submit 300 reports in a day saying that what you're importing is "1 Box of parts", you actually have to specify what the parts are and how many are in the box - I suspect this is what is causing the problem as the system rejects "invalid" submissions and forces the importers to rework and resubmit their import declarations.
Apparently the issue is that the data coming in (mainly from ships) is quite crufty, whereas the system expects nice clean data (GIGO anyone?).
Also, apparently a lot of these Brokers have a vested interest in the old system, as the new one will allow major importers (eg. supermarkets) to clear goods themselves, meaning less money for the brokers.
As for delays and ships being turned back- appears to be mainly FUD, with a little bit of lack of foresight and poor planning.
Seems like a change management failure to me.
One factor seldomly taken in account is the user's reluctance to the new system.
You may have a 100% working new system, with a 1000% improvement over the old system, but if users are not excited about the new system and they do not want to use it for whatever-the-reason (maybe just because he/she now has to learn new things), the new system is going to fail. Users will make sure it fails. I have seen that many times.
Comment removed based on user account deletion
The rumour on the grapevine is that the problems don't entirely stem from the software. The data entry now requires details (you want what now?) and that makes it impossible to process cargo as quickly as before. The software is just a convenient scapegoat. The reality is that the old system allowed the data entry to be sloppy (and effectively useless).
They say that in a few years a human-engineered microorganism will be created with a selected set of genes. All very well, and I suppose that won't be released into the wild. But I bet that if they ever do it (release it into the wild), it'll last about 5 minutes against its evolution-designed competitors and generally hostile environment.
The same happens to the IT systems. Legacy systems may be old (how can software be old, anyway?), incompatible, user-unfriendly, and whatever else. But a basic fact so often overlooked is that they have for many years been adapting (or rather being adapted) to their environment (users, other programs, etc). If you look at legacy code you always find odd-looking "if's" with comments like "It must do this to work", or "The other program expects it that way", or no comment at all. The point is that all this spaguetti code has beed polished, adapted and perfected by the work of programmers guided by the reality, as opposed to designers guided by their own desires and incomplete knowledge of the problem.
So the point is that _all_ scratch designed systems will lose all that ancient knowledge embedded into the code, and there is nothing you can do about it (inspecting all the code would be impossible, and the knowledge can sometimes be into OS parameters, shell scripts, scraps of paper with procedures in the drawers of remote users, or even in the brains of world-scattered users) So the only thing to do is to have it into account when designing a new system of some complexity, and knowing that it will take you like a year at least of real running till it's at the same level of functionality as the old. So probably you'll need a year of overlaping systems (perish the thougth).
When presented with that reality most managers will think again if they really need the new system, and at least will be prepared for the problems ahead.
But of course that might not sell the new system, so who's interested in telling those truths to management. Certainly not the seller's marketing dept, their concealing habilities much helped by the fact that they are themselves blissfully unaware of the problem.
Rome taught me patience and assiduous application to detail. Virtues which temper the boldness of great, general views.
$20 says there will be in a few days/weeks.
When you're afraid to download music illegally in your own home, then the terrorists have won!
I love open source software too, but isn't the budget blowout on this (triggered by scope creep etc. like most projects) going to be the cost of services (ie. people), rather than the software itself? If anything, it would be harder to find enough people skilled up OSS people in Australia and that would make the project cost even higher than with proprietry systems.
This Customs IT project is definately a disaster, but I haven't seen too many stories about open source projects on a similar scale that have been under budget and on time to balance it out. Anyone got stories/sites out there about OSS large-scale project success stories? I need ammo to convince my boss on some upcoming work :)
In a closely related current issue federal Agriculture Minister Peter McGauran displayed the scientific illiteracy so recently evident in more governments than ours by getting all in a tizz about some Canadian pigeons that flew in ahead of the customs slow down only to be discovered to have viral antibodies but not live viruses and be sentenced to immediate death for having beaten the dreaded avian flu or, in four cases, Newcastle disease.
If only we could do the same to politicans carrying antibodies, let alone their sick computer systems.
Better not think about juxtaposing the importation of pigeons from the other side of the world with the wish of local authorities to wipe out the feral pigeons already settled in here.
Don't worry, it gets worse. Just check out the support for teaching "intelligent design" from the general practitioner our over-tired and under-opposed federal government have given responsibility for education.
-- Our systemic servants do not good masters make.
after EDS let their old mainframes walk out the door.
Is it a big suprise that EDS fucked the upgrade as well?
POKE 36879,8
Dear Project Director,
...I think it's time you opened the third envelope. Good luck in your future endeavors.
Your situation reminds me of the old IT parable that goes something like this...
On his first day on the job a new IT Director has a meeting with the outgoing one. At the end of the meeting the ougoing IT Director hands the new on 3 envelopes and tells him to use them to get out of his first 3 major meltdowns, "just make sure you wait to open them until you need them."
About 3 months later the new IT Director has his first major disaster and remembers the envelopes. Opening the first one he sees, "Blame Me" in big bold letters. Which he does and it works.
Six months after that the second blow up happens and the second letter reads, "Blame the Vendors" which also works.
One year later when everything falls apart the new IT Director opens the third letter full of hope. It reads, "Write 3 Letters."
Sincerely,
The Guy Before You
"Never limit what you know to what you do", Me
The article also answers other posters questions about the platform it was delivered on. Certainly no cheap linux stuff used here !
But really interesting is this:
With so many cooks in the kitchen, shouldn't problems be expected ? How could you ever figure which one can is responsible for the mess now emerging ?
Open-source projects sometimes have more cooks, but could the commercial agendas in a closed source project with patents etc.,. destroy the synergies ?
plurality should not be posited without necessity. - William of Occam
I have seen this so many times. Big project, Big Budget, Big Names, Big Price, Big Stuffup.
I believe that a system like this is reasonably simple and can be created by a very small team.
With big projects you end up with teams of project managers micro managing everything. This is why it gets so diffiult. I was once on a project where my part was to copy files intact from remote locations to a central site. What a mess. The project manager had designed a process that failed every time. Not to mention the bandwith upgrades that happened after the file transfers. All they needed was one person with the know how to get it done and a small team of switched on IT persons to manage the entire thing.
Companies are concetrating too much on process and management than getting the work done. These types of projects are not that difficult.
Look, I know you're just a troll, but I have to ask anyway:
You say: "It was disproven long ago. There is NO CREATOR, and there never was."
I'm very curious about the when, where, who and how of that proof.
Do you have links or references? Can you explain the proof to me?
I ask because I have never before heard anyone claim that there is PROOF of the nonexistence of a Creator before.
(I have heard many people say that there is no proof of the existence of a Creator, but I hope you see the difference.)
Exam 4/C again. Maybe I'll do better this time.
Used guns. A few years ago, the relative strength of our dollars made it possible to get a SIG P240 in .38 Special from Australia for about 850 Australian dollars. IIRC, that was about 500 American. At the time, the same gun in the U.S., if you could find one, would have easily topped a thousand dollars. Same story on a Hammerli 120, which is rare and pricy in the U.S. Those were my two failures.
To be fair, though, the ATF-related problems were serious, too. Theoretically, it's not that difficult, but the ATF has been such a pain in the ass to so many gun dealers over the years that those dealers are terribly gun-shy. (I say, that's a joke, son.) They aren't willing to do any out-of-the-ordinary paperwork and draw attention to themselves. The only one I found who would do the job (and I limited the search to my state since a concurrent state-to-state transfer adds exponentially to the complications) quoted a minimum of $350 to start the paperwork and wanted me to sign a contract saying that once they had the gun in hand they could sell it to me at any price they wanted, nevermind that I would have already paid the foreign dealer directly. No thanks. FFLs are limited forms of government-granted monopoly and come with all the problems that implies.
Slightly offtopic - If anyone has a production-class silhouette pistol made by Unique, any chambering, anywhere in the world, at anything approaching a reasonable price, then I'm buying. (French readers, I'm talking to you.) Please post back here.
When large new systems like this one wreak this much havoc, I think lessons learned need to be disseminated to the entire industry.
I've seen many interesting posts about why Australia failed in this new system, but it's mostly conjecture. They should (and I'm guessing, will) conduct a deep and thorough post-mortem and find out what went wrong, down to the lines of code, scheduling decisions, rollout decisions, etc.
And (here's the controversial part) they should provide every single document to the public.
When projects gone amok have international impacts like this one why can't the rest of the industry learn from the mistakes by having access to the post-mortem. Involved companies want to maintain control of their Intellectual Propert, but in cases like this, EVERYTHING should be made public. Actually at this point companies involved really aren't protecting IP, but would be hiding behind that canard to deflect the embarrassment of public scrutiny.
Many similar failures wrought similar havoc. Denver International Airport (DIA) spent millions (don't remember exact numbers, but I'm guessing it was in the $100's of millions) of dollars for their dramatically failed automatic baggage handling system. Today DIA not only handles baggage the old fashioned way (carts and tow-tractors), they have to do it through too-small tunnels not designed for the task because of the hubris of the project they wouldn't need to.
So, for now, all we have is conjecture from government officials and slashdotters, one demographic of which already shows some deep insights and possible explanations. But that's all we have.
I hope cause and effect is investigated, and I hope the IT industry gets the opportunity to understand the failures and learn from them.
Integrated Cargo System (ICS)
The cornerstone of CMR, ICS is an integrated system giving enhanced risk assessment at the border and allowing more efficient cargo tracking. Its software suite has 23,000 function points.
It operates on an IBM OS390 mainframe [they mean zSeries] running z/OS with transactions in a CICS environment with DB2 database management. MQ-series provides the mainframe interfaces with the CCF gateway and other business applications. [CCF is a Customs communication system, I believe].
Customs' Web-based user interface, Customs Interactive (CI) has a WebSphere Java application server front end. CI system software is hosted on infrastructure managed as part of the CCF gateway.
...
Design detail in the 19,000 pages of analysis for ICS includes 800 screens, 16,000 business rules, 70 complex business messages, 850 database tables, 3700 executable load modules, 1800 CICS transaction types, 55 batch jobs, 90 reports and 35 system interfaces.
So they certainly didn't pick a few cheap PCs running the latest whizbang toyware. This is solid, proven hardware. CICS is the "old faithful" of massive transaction processing, DB2 is an old workhorse learning new tricks these days, and WebSphere is a good J2EE app server (if quite complex) with good support. And MQ is a robust guaranteed-delivery messaging system on which you can run JMS and other messenging frameworks. Overall, good choices.
I'd say that the problem is the complexity of the software... 23,000 function points? 1800 different transactions? A system of this complexity cannot reasonably be created in such a short time frame (2 years). They probably had a Mongolain hord of the lower bidding coders develop this thing without time to do any cross-project concertation, and it smells of overburdened teams working in isolation, trying to implement paper specs that aren't waterproof.
You want slow integration with a succession of prototypes for such a project. I would bet this prototyping phase was too short and that integration of parts written by isolated teams was rushed.
If you know an IBMer working in WebSphere or MQSeries on z/OS, you can ask him to bring you back a souvenir from AU, 'cuz chances are he'll be there a lot soon...
--
Mad science! Robots! Underwear! Cute girls! Full comic online! http://www.girlgeniusonline.com/
The movement to teach creationism or intelligent design in a science classroom is ill advised because it presents a non-scientific hypothesis as a scientific theory. Creationism and intelligent design represent a system of religious beliefs founded upon faith. They do not provide an experimentally verifiable or falsifiable set of ideas. They are not scientific theories, and we cannot teach them to our children as science if we hope to compete in an ever more technological world.
Evolution, on the other hand, has resulted in a great number of experimentally verifiable ideas. Through the fossil record, scientists have evidence of natural selection. By examining creatures with very short live cycles, scientists have been able to directly observe and maniuplate natural selection. The structure of our own DNA is the strongest evidence yet that we, too, are subject to natural selection. New ideas are only called theories if they can be verified or falsified. It is a very different definition of theory than exists for the general public, who confuse "theory" with "hypothesis".
Many people believe that to teach evolution is to teach that there is no God. Evolution does not explicitly discuss God because we cannot test for God, and this is evidence for some that evolution teaches atheism. I know religious people who take evolution and natural selection as evidence of God, and have heard them call DNA "God's fingerprints".
What does evolution mean? Is it evidence that miracles do not happen, or is it evidence that God was here? That is an interesting theological question, and one for which there is no experimental test. It is not a scientific question, so it should not be taught in a science classroom.