Slashdot Mirror
Ships Turned Away As Aussie Customs' IT System Melts Down
An anonymous reader writes "Urgent shipments of medicine and goods for the holiday season have been turned away by customs officials due to a massive computer problem. The initial budget for the system upgrade was said to be A$80 million but has since blown out to A$250 million. Customs officials and the government have been forced to admit that they might actually have to revert to the old system if things don't improve. One cargo user said on national TV that he used to process 300 orders daily but the new system is so complex and unusable, he's happy if he can manage 100 orders per day. The system failure is expected to have a massive impact especially on the retail sector this Christmas."
What OS do they run?
What software do they use?
And how will their IT people and/or management continue to justify said choices in the wake of this?
This is the sort of thing that needs "big iron". Machines that have uptimes measured in decades. Why do I have the sneaking suspicion that they're running it all on a bunch of commodity PCs (or the like) with off-the-shelf software?
With spending like this, exactly what are "conservatives" conserving?
For the AU government to let goods travel freely until they fix or bring up the old system. There really is no excuse for what is going on. Yes, that means that the AU government doesn't get its cut of taxes but them's the breaks. The money lost from import fees would be DWARFED compared to the lossess incurred by *not* letting goods through the ports.
--
BMO
I'm no grizzled guru by any means, but damn, I know by now that though it *may* seem cheaper to upgrade all in one fell swoop, you're gonna get hosed every time. The bigger the system, the more likely, just because there's no way you can *test* the thing at that scale.
Software is *complicated*. Large-scale software rollouts are even *more* complicated, just because now you've involved hundreds or thousands of non-debuggable, unpredictable people into the equation. No matter how many meetings you have about it, no matter how many different people assure you that they will do "whatever it takes" to make sure it goes smoothly, keep in mind that they probably don't have "what it takes", which would often be some kind of deity-level power.
Let's look now at the "largest e-government projects ever undertaken", introduced "despite industry protests that Customs had not allowed them ample time for the changeover." It's not hard to guess how it's going to go.
Sometimes, you gotta go the slow way... replace the old system bit by bit, make sure you can flip the switch back every step of the way if something goes wrong. At the very least you have to plan it from the start so that you can roll out piecemeal, just in one site, or run the old/new in parallel, etc..
This method results in a more expensive *estimate* at the start of the project. But the actual *cost* in the end can be much, much lower.
Just my 2c...
Funny how to the state, free commerce isn't an option, but blowing $250 million that isn't even yours on a computer system that doesn't work is okay.
"Your papers, citizen! Whoops, my citizen-authorization-scanner just went dead. You'll have to be detained while I get fresh batteries. Oh, and that'll cost $10 - batteries aren't free, you know."
He who lights his taper at mine, receives light without darkening me.
The real problem with this system is that it used the principle of "Big Design Up Front". Ask Joel Spolsky about the benefits of "Big Design Up Front" - you get to make all kinds of assumptions about the environment to simplify development, then find when you turn on the switch that this $80M system just doesn't work right.
The little things that get you down? Oh... date formats, validating input, units for measurement, using a communications system intended for overnight batch operations to support real-time interactive operations.
As other posters have mentioned, the bid that got the nod was the lowest one. The bid that should have received the goahead was the one that recommended incremental changes. The one that recommended introducing a new means for handling import declarations - and not cutting over, but rather letting the old one die the natural death of user migration.
The final nail in the coffin was Customs insisting that more detail be included in these reports - no longer can you submit 300 reports in a day saying that what you're importing is "1 Box of parts", you actually have to specify what the parts are and how many are in the box - I suspect this is what is causing the problem as the system rejects "invalid" submissions and forces the importers to rework and resubmit their import declarations.
How about...'it doesn't matter'.
This is probably the result of a crappy design, with little interaction between the developers and the eventual users.
It does what it was designed to do. The problem is the design and implementation does not match what it NEEDS to do.
The drug companies have quite successfully pwned the tabloid newsmedia in Australia (and I suspect in plenty of other places on the planet) to the extent that every time they feel the need for an injection of cash, they prime the tabloids (newspapers, today tonight, current affair, sixty minutes and all of the similarly unreliable sources) with rumours of an outbreak of something-or-other, then it's all hands on the cash registers as the general public launches into a flurry of panic over whatever is $biohazard of the month.
The best known of the recent efforts has been the meningitis scare here in Australia. The tabloid press/radio/tv has worked the public into a lather, and the drug companies are laughing all the way to the bank. Somehow the bit where the death rate from meningitis and related diseases is exactly the same this year as it was the year before and the year before that while (1) { and the year before that } seems to have been conveniently ignored.
The connection back to the politicians is, of course, that there's nothing a politician likes more than a plethora of panicked punters to pacify, and that's exactly what's happening right now.
What should the thinking Australian do right now? Buy pharmaceutical shares, that's what!
I find your ideas intriguing and I wish to subscribe to your newsletter.
They say that in a few years a human-engineered microorganism will be created with a selected set of genes. All very well, and I suppose that won't be released into the wild. But I bet that if they ever do it (release it into the wild), it'll last about 5 minutes against its evolution-designed competitors and generally hostile environment.
The same happens to the IT systems. Legacy systems may be old (how can software be old, anyway?), incompatible, user-unfriendly, and whatever else. But a basic fact so often overlooked is that they have for many years been adapting (or rather being adapted) to their environment (users, other programs, etc). If you look at legacy code you always find odd-looking "if's" with comments like "It must do this to work", or "The other program expects it that way", or no comment at all. The point is that all this spaguetti code has beed polished, adapted and perfected by the work of programmers guided by the reality, as opposed to designers guided by their own desires and incomplete knowledge of the problem.
So the point is that _all_ scratch designed systems will lose all that ancient knowledge embedded into the code, and there is nothing you can do about it (inspecting all the code would be impossible, and the knowledge can sometimes be into OS parameters, shell scripts, scraps of paper with procedures in the drawers of remote users, or even in the brains of world-scattered users) So the only thing to do is to have it into account when designing a new system of some complexity, and knowing that it will take you like a year at least of real running till it's at the same level of functionality as the old. So probably you'll need a year of overlaping systems (perish the thougth).
When presented with that reality most managers will think again if they really need the new system, and at least will be prepared for the problems ahead.
But of course that might not sell the new system, so who's interested in telling those truths to management. Certainly not the seller's marketing dept, their concealing habilities much helped by the fact that they are themselves blissfully unaware of the problem.
Rome taught me patience and assiduous application to detail. Virtues which temper the boldness of great, general views.
In a closely related current issue federal Agriculture Minister Peter McGauran displayed the scientific illiteracy so recently evident in more governments than ours by getting all in a tizz about some Canadian pigeons that flew in ahead of the customs slow down only to be discovered to have viral antibodies but not live viruses and be sentenced to immediate death for having beaten the dreaded avian flu or, in four cases, Newcastle disease.
If only we could do the same to politicans carrying antibodies, let alone their sick computer systems.
Better not think about juxtaposing the importation of pigeons from the other side of the world with the wish of local authorities to wipe out the feral pigeons already settled in here.
Don't worry, it gets worse. Just check out the support for teaching "intelligent design" from the general practitioner our over-tired and under-opposed federal government have given responsibility for education.
-- Our systemic servants do not good masters make.