Slashdot Mirror
Ships Turned Away As Aussie Customs' IT System Melts Down
An anonymous reader writes "Urgent shipments of medicine and goods for the holiday season have been turned away by customs officials due to a massive computer problem. The initial budget for the system upgrade was said to be A$80 million but has since blown out to A$250 million. Customs officials and the government have been forced to admit that they might actually have to revert to the old system if things don't improve. One cargo user said on national TV that he used to process 300 orders daily but the new system is so complex and unusable, he's happy if he can manage 100 orders per day. The system failure is expected to have a massive impact especially on the retail sector this Christmas."
All the news and radio reports I've read and heard (including TFA) have made no mention of ships being actually turned away at this stage. So far they're just saying that the storage space at the ports is rapidly filling up, so if the processing rate doesn't improve soon they will have to look at turning ships away. But as far as I can tell, they're planning to roll back to the old system before that becomes necessary...
http://www.customs.gov.au/site/page.cfm?c=6361
Partial quote...
"Customs is doing everything possible to resolve technical and business issues arising from the introduction of the new Integrated Cargo System (ICS) for imports.
"Contrary to some media reports, the new IT system for imports has not failed, nor is its performance solely responsible for the problems that have occurred.
"The problems experienced in part, flow from inaccurate and incomplete information being submitted by some users, which the new system is designed not to accept for security reasons," the spokesman said.
80mil AUD = approx 50mil Euro = 60mil USD
250 = 156 = 188
Computer World Article
ICS is a cornerstone of Customs' massive Cargo Management Re-engineering (CMR) project. This was intended to replace the export and brokerage industry-developed EDI system Customs Connect with a Web-based model co-developed by Customs and a consortium of IT vendors led by Computer Associates. The project aims to facilitate all aspects of Customs involvement in the import and export process including declarations and GST transactions collected at port.
Nother Article
More than seven years to this point of readiness, ICS is a cornerstone of Customs' massive Cargo Management Re-engineering (CMR) project, which will replace the export and brokerage industry-developed EDI system, Customs Connect. CMR is a Web-based model co-developed by Customs and a consortium of IT vendors led by Computer Associates, EDS, IBM and Telstra nee Kaz.
Ah, keynote also has some nice data. Apparently Level3 & Verio have both been having severe problems in the last few hours, and have a high latency connecting to most of the other Level-2 & Level-3 providers:
x ?Login=Y&Username=public&Password=public
http://scoreboard.keynote.com/scoreboard/Main.asp
94% of Repubs and 21% of Dems voted to renew the Patriot Act
What OS do they run?
What software do they use?
CA, NCR and IBM are the service providers; Novell's providing the directory service.
The ICS (Integrated Cargo System) application is running on an IBM OS390 mainframe; the OS is ZOS, the database is DB2. The web interface is Java, using WebSphere.
The CCF (Customs Connect Facility) runs on Sun Solaris Unix platforms (using a variety of other servers for validation and transformation). Again, the database is DB2 and the interface uses WebSphere Java.
More information here.
Comment removed based on user account deletion
I listened to a report about it on the radio this morning, and the system was started in 1994 and ran on Windows 3.1. Then they upgraded it to Windows 95. I takes 25 minutes to process what used to take 25 seconds on the old system. 135 million dollars from an initial bugdet of 25 odd million.
Makes me feel a bit better about my job.
http://pcblues.com - Digits and Wood
This is the info
"It operates on an IBM OS390 mainframe running ZOS with transactions in a CICS environment with DB2 database management. MQ-series provides the mainframe interfaces with the CCF gateway and other business applications. "
And the CCF is run on
"Communication channel management and CI runs on Sun Solaris Unix platforms and Cisco routers, with validation and transformation processed on IBM P- and SP-series Unix platforms and Wintel servers running IBM AIX, Win2K, DB2 , WebSphere, Tivoli WebSeal and Baltimore's FormSecure. "
I see only 2 weakpoints the win2k systems and the implementation of the Java handling in websphere. Other than that theres no reason any individual part should be failing, the entire thing is built from dependable parts.
I just guess no one properly considered that it would be handling
"3 million import entries, 1.2 million export clearances, 4 million container and 100,000 flight movements, and the collection of nearly $7.5 billion in Customs duties."
And remember... each one of those is likely to involve between 10 and 25 or more individual forms and checks and clearances...
This isnt realy unexpected
XML - A clever joke would be here if
It's true that the main problem isn't the software (although the bugs don't help): it's the way the new system was implemented
According to transport.nsw.gov.au Botany transfers 1.1 million 20' containers a year or about 3,000 containers per day. So, no, you are right it won't be a PC in a basement room. It'll be some big iron running this web based app.
Orationem pulchram non habens, scribo ista linea in lingua Latina
I have seen this so many times. Big project, Big Budget, Big Names, Big Price, Big Stuffup.
I believe that a system like this is reasonably simple and can be created by a very small team.
With big projects you end up with teams of project managers micro managing everything. This is why it gets so diffiult. I was once on a project where my part was to copy files intact from remote locations to a central site. What a mess. The project manager had designed a process that failed every time. Not to mention the bandwith upgrades that happened after the file transfers. All they needed was one person with the know how to get it done and a small team of switched on IT persons to manage the entire thing.
Companies are concetrating too much on process and management than getting the work done. These types of projects are not that difficult.
What OS do they run?
The same OS they've been using for a while (WinXP)
What software do they use?
Is a custom built system - written by EDS I believe.
And how will their IT people and/or management continue to justify said choices in the wake of this?
"Their" IT people didn't make the choices - Customs IT is provided by EDS (which is why I believe EDS also developed the system). The choices would have been made by higher management - but ultimately it doesn't matter, if the system is failing then it's the design of the system or the hardware in use - which I would expect is top dollar equipment, charged for at higher than retail prices (it's a government contract). The IT experts in Customs are more for retrieving data of hard disks after they've been seized etc. Customs hasn't managed their own IT for years now.
This is the sort of thing that needs "big iron". Machines that have uptimes measured in decades. Why do I have the sneaking suspicion that they're running it all on a bunch of commodity PCs (or the like) with off-the-shelf software?
This is laughable at best. How many "off the shelf" packages have you seen for handling Customs? The new package (and the old I expect) is a custom built piece of software (heck even the summary pointed this out - A$80 million but has since blown out to A$250 million - that is not "off-the-shelf")
The system itself was written specifically for customs and has great features like it was too big to fit on all the monitors that customs was using (so naturally EDS upgraded all the machines - at a price - to have 19" LCD's).
dnuof eruc rof aixelsid
This being the former Federal Science Minister that refuses to accept that Near-Earth Objects (NEOs) pose any possible problem to us and thus cut all funding for catalogueing and studying such objects in the Southern Hemisphere.
Well, the author (of the article, Peter Davidson) doesn't know mainframes. "...IBM OS390 mainframe running ZOS ..." OS390 is an operating system (the precursor to zOS). Maybe they're running on a z900 or z990 or z9 (or 2064-yada yada). So, the way I interpret this, is they are running some CICS (presumably TS 1.3, 2.3, or 3.1) which talks to DB2, does some messaging with WebSphere MQ, and all the web interface (WebSphere, whatever) is on some Unixy (Solaris?) front end. Speaking as a IBM mainframer, any zOS or CICS systems programmer worth their salt would be able to tell you how long all of those transactions took, and where the problem was. I guarantee we run a much smaller box here, and we push through millions of transactions a day (just business hours!), and we guarantee the majority of the transactions are done in 1/4 second or less. So, shitty code, shitty performance, but on the backend they should be able to pinpoint any problems. I would guess that the bottleneck (and money pit) showed up on the front end (or everyone on the project is clueless - 50/50).
Integrated Cargo System (ICS)
The cornerstone of CMR, ICS is an integrated system giving enhanced risk assessment at the border and allowing more efficient cargo tracking. Its software suite has 23,000 function points.
It operates on an IBM OS390 mainframe [they mean zSeries] running z/OS with transactions in a CICS environment with DB2 database management. MQ-series provides the mainframe interfaces with the CCF gateway and other business applications. [CCF is a Customs communication system, I believe].
Customs' Web-based user interface, Customs Interactive (CI) has a WebSphere Java application server front end. CI system software is hosted on infrastructure managed as part of the CCF gateway.
...
Design detail in the 19,000 pages of analysis for ICS includes 800 screens, 16,000 business rules, 70 complex business messages, 850 database tables, 3700 executable load modules, 1800 CICS transaction types, 55 batch jobs, 90 reports and 35 system interfaces.
So they certainly didn't pick a few cheap PCs running the latest whizbang toyware. This is solid, proven hardware. CICS is the "old faithful" of massive transaction processing, DB2 is an old workhorse learning new tricks these days, and WebSphere is a good J2EE app server (if quite complex) with good support. And MQ is a robust guaranteed-delivery messaging system on which you can run JMS and other messenging frameworks. Overall, good choices.
I'd say that the problem is the complexity of the software... 23,000 function points? 1800 different transactions? A system of this complexity cannot reasonably be created in such a short time frame (2 years). They probably had a Mongolain hord of the lower bidding coders develop this thing without time to do any cross-project concertation, and it smells of overburdened teams working in isolation, trying to implement paper specs that aren't waterproof.
You want slow integration with a succession of prototypes for such a project. I would bet this prototyping phase was too short and that integration of parts written by isolated teams was rushed.
If you know an IBMer working in WebSphere or MQSeries on z/OS, you can ask him to bring you back a souvenir from AU, 'cuz chances are he'll be there a lot soon...
--
Mad science! Robots! Underwear! Cute girls! Full comic online! http://www.girlgeniusonline.com/
The movement to teach creationism or intelligent design in a science classroom is ill advised because it presents a non-scientific hypothesis as a scientific theory. Creationism and intelligent design represent a system of religious beliefs founded upon faith. They do not provide an experimentally verifiable or falsifiable set of ideas. They are not scientific theories, and we cannot teach them to our children as science if we hope to compete in an ever more technological world.
Evolution, on the other hand, has resulted in a great number of experimentally verifiable ideas. Through the fossil record, scientists have evidence of natural selection. By examining creatures with very short live cycles, scientists have been able to directly observe and maniuplate natural selection. The structure of our own DNA is the strongest evidence yet that we, too, are subject to natural selection. New ideas are only called theories if they can be verified or falsified. It is a very different definition of theory than exists for the general public, who confuse "theory" with "hypothesis".
Many people believe that to teach evolution is to teach that there is no God. Evolution does not explicitly discuss God because we cannot test for God, and this is evidence for some that evolution teaches atheism. I know religious people who take evolution and natural selection as evidence of God, and have heard them call DNA "God's fingerprints".
What does evolution mean? Is it evidence that miracles do not happen, or is it evidence that God was here? That is an interesting theological question, and one for which there is no experimental test. It is not a scientific question, so it should not be taught in a science classroom.