Slashdot Mirror
Denver Airport Automated Baggage System Abandoned
cherylchase writes "Denver International Airport opened in 1995 with an ambitious fully automated baggage system: 26 miles of underground track, thousands of small gray carts, all controlled by a mainframe programmed for just in time delivery. But the system never worked well; bugs delayed the airport's opening for months (at $1M/day). The system has now been abandoned as a cost cutting measure." From the article: "Technology, too, has brought change. Back then, the big-brained mainframe doing it all from command central was the model of high tech. Today the very idea of it sounds like a cold-war-era relic, engineers say. Decentralization and mobile computing technology have taken over just about everything, allowing airlines, warehouse operators and shippers like FedEx to learn with just a few clicks the whereabouts of an item in motion, a feature that was supposed to be a chief strength of the baggage system."
I've been in airports all over the place, I would think 26 miles of track underground wouldn't speed up the process, especially if it is unmanned. I trust eyes on my luggage more than nobody knowing if it is really being moved or not. I've had luggage take forever in JFK airport, and the fastest was in another country!
Look, you have to store the data somewhere. Just because your FedEx guy clicks his little wireless dealie when you sign for a package, doesn't mean that his little wireless dealie is the datastore for all info about the package.
Why is the 'big central mainframe' the cause of the problems here?
I want to delete my account but Slashdot doesn't allow it.
Go ahead and Google Denver International Airport and look into some of the conspiracy-theories surrounding the building, murals, underground facilities, etc. It's pretty wierd stuff, interesting to say in the least.
Whether or not it's true, I don't know. You decide.
I can see it now. A bunch of guys staring at their PDAs wondering why the luggage sitting in front of them isn't going anywhere. On a side note, anyone else ever want to ride those tracks a la Toy Story 2?
suck my ping!
When's the opening of the electronica club that is replacing it?
** "It's not my job to stand between the people talking to me, and the ones listening to me." -- Pego the Jerk
This was on the world news (well nightly network news) almost 3 weeks ago, bleck.
Also, they mentioned that this system was the first one run by PCs! Wikipedia has had this up for quite some time as well.
Reading up on it, it appears more that the lack of PLANNING was more at fault. The system was designed AND implemented with only 2 years left before opening, and with the majority of construction on the airport already completed, meaning the physical aspects of it had to be squeezed in where ever spae was available, given that, the results are not to surprising.
If anything this represents a massive failure on the part of management to allocate enough time for a project, implementations of far smaller systems than the one at Denver spent two years alone in just the research phase!
Need help treating your acne? Come here!
Just as the implementation of IBM's OS/360 forms part of the "history" section of many Computer Science texts, so the Denver Airport baggage system is fast becoming history. The big difference of course being, OS/360 was a spectacular success, wheras Denver was a catastrophic failure.
Writing this stuff up is fine and good, but I think it would be worthwhile to try to learn from it. What was done differently?
If folklore serves me correctly, IBM was not afraid to throw money at the problem. I seem to remember they put two separate teams on the problem and took the best from each, fully conscious that half the effort would be thrown away. They sank as much money on it as was required, and ultimately succeeded.
Denver probably ate many more Dollars than OS/360, though I wouldn't know. But:
Apparently, this last has become a self-fulfilling prophecy.
I work in software development for an airline. It's amazing how much of a megaproject a reservation system is proving to be these days, and how many past attempts have failed. That's why one of the world's major reservation systems still runs in assembler on an IBM mainframe.
I think we're talking over-engineering, Big Design Up Front, profiteering, and (attempted, far too late) price-gouging.
Either that, or the only way to make a very large project successful is to code it in Assembler on an IBM mainframe.
When one person suffers from a delusion, it is called insanity. When many people suffer from a delusion it is called Rel
It's bad when your luggage is stuck in an infinite loop and the airport can't claim that the luggage was lost when it whizes by.
Not mentioned much these days was that the huge delays in getting the Denver Airport baggage handling system was a huge black eye to IBM who had been bragging loudly about how their OS/2 operating system was running it.
"Automation always looks good on paper," said Veronica Stevenson, a lead baggage handler for United Airlines and president of the union local that represents United's 1,300 or so baggage handlers in Denver. "Sometimes you need real people."
A system that would have streamlined and reduced the need for union employees has been found to not be very good by those union employees? Shock and awe, gentlemen. Shock and awe.
Robots do exactly what you tell them to. It only damaged luggage if the luggage wasn't loaded onto the robot correctly, it only misplaced luggage if the robot was told to go to the wrong place.
Can you blame me for wondering if the failure of this system was not entirely because of technical reasons?
Living in Denver and flying in and out of DIA, I can say it's better that any of the other Big City airports I've used. (Dulles, Seatac, Atlanta, DFW, Las Vegas, etc.)
It was accomplished on a scale and timeframe that was hard to imagine before the project. As a Student in Civil Engineering, I got a behind the scenes tour in college.
As the automated baggage system a f*ckup? Oh yeah, most certainly. Did they recover well? I'd say so.
Course, DIA is a political animal, and in all things politics, you're guaranteed to piss off more than half your constituents. But it's a damn sight better than Stapleton was.
Funny thing is, I saw a newspaper article about Denver's new airport, how it was in the middle of nowhere, and had cost overruns, and how it was nothing but a boondoggle.
It was written about Stapleton in the lates 1930's. The switchover in 1985 meant that Stapleton was useful for more than _50_ years. I suspect in another 40 years, DIA won't be in the middle of nowhere anymore.
"Draco dormiens nunquam titillandus."
From reading the article, it sounds like the problems had almost nothing to do with the software aspect of the system, whether on a mainframe or not, and everything to do with the physical design of the tracks.
The fact that bags fell off the tracks because the corners weren't banked has nothing to do with the control system. Same for using unstable pallets to hold the bags.
This whole article seems to be based on a flagrant redefinition of the term "bug" as we understand it. It wasn't software bugs that caused the problems, it was crap engineering.
Which begs the question why, when other airports (such as Heathrow) have miles of tracks that work just fine, couldn't Denver do the same?
You're exactly right. I really wish the NYT reporter would have done a better job figuring out why the baggage system didn't work. Instead we're left guessing, with vague anecdotes about carts tipping over and barcodes not being properly scanned -- all of which has nothing whatsoever with the computer in the back office. By the way, mainframes are at the heart of every major package shipper's operations, tracking every little bit of package-related data. Not so surprising: it's a job that must run reliably, without downtime. Mainframes do that well. And if the mainframe was to blame, why didn't United (or the airport or whomever) just drop in one of those wonderful Peecees to do the job? That would have fixed everything, right? Obviously no, that wasn't the problem.
Now they employ thousands of little grey bug eyed people to push the little grey carts around the 26 miles of dark underground tracks...
Oh well, what the hell...
Mars rover for one.
I think you are only seeing the negative and assuming that is all that is out there. The problem with constant media is that we really do lose our sense of proportion. Yeah, one airport luggage system failed because of bad planning. You don't think anything like this ever happens in Europe? Or that there aren't success stories in the US? Think again.
You mention outsourcing that is another story that has been blown out of proportion by the media, including the self-promiting asshats..I mean "researchers" at Gartner. Yeah, some jobs have gone over to India, and they may not be coming back, but it's not nearly as big of thing as NeoIt, Gartner, or the Washington Tech Alliance(is that their name? Can't remember, the group in Washington State who is organizing against outsourcing) would have you believe.
The media only reports on what is new and interesting. Remember the huge SARS scare? Worldwide that killed about 800 people. That is about the number of people who die on America's highways PER WEEK, and yet whenever the Transportation Safety Board issues it's report on how 40,000 people died last year in car accidents, the media gives it a blurb and then turns it's attention to whatever the scare tactics of today are.
The US economy isn't nearly as bad as the naysayers claim it is, nor is it nearly as strong as the Bush apologists boast. The hardest thing to find in this sea of information is the truth.
Monstar L
the article does not draw any correlation between mainframe programming, software, or the failures of the system. a major flaw, according to the article was that: "The whirring baggage carts, programmed to pick up and drop off bags in a perfectly coordinated ballet, often just tipped over and dumped their loads." it also speaks vaguely about some "lizard tongue conveyor" whose failures would hardly seems the domain of software development. the denver baggage system fiasco sounds more like a failure in regular engineering, or at best, robotics programming. i hardly see why mainframe architecture, or any piece of software code, should be blamed as the primary culprit.
From a RTFA between the lines it would appear that they started on this project late, hadn't factored in where they were going to put the necessary IT equipment, almost as if it were an after thought. Essentially their customers baggage was well well down the list of priorities.
And then they blame it on the computers.
Typical.
Some companies/public services really do give the distinct impression that they consider their customers/clients a major inconvenience as they attempt to make a profit/index linked pension.
threadeds blog
I'm an automation systems engineer. I always find the failure of systems such as this very interesting. I've done firefighting operations on many jobs where they were on their way down the toilet. Most of the time failures are caused by only one weak area in a project.. usually it's mechanical design problems, or software (logic) problems. I have seen an instance once where it was a union sabotage problem. It was interesting how that particular line would run perfectly well on it's own during the weekends; but during the week it was a disaster. Since I spend most of my time writing automation logic and robot programs, I tend to get stuck with developing software workarounds for bad mechanical designs. The worst that I recall was a tread booker for a tire plant. It was one of the most crude machines I've ever worked on. My favourite part was a coupling that tended to slip; I was asked to put code in that 're-homed' the servo axis every few minutes automatically. I was paid by the hour; I'll software patch the hell out of bad mechanics if you want! I'm not sure what the problem(s) "really" were in this instance, but it's kind of sad; what airport will be brave enough to try it again?
The BAE design includes a number of high-tech components. It calls for 300 486-class computers distributed in eight control rooms, a Raima Corp. database running on a Netframe Systems fault-tolerant NF250 server, a high-speed fiber-optic ethernet network, 14 million feet of wiring, 56 laser arrays, 400 frequency readers, 22 miles of track, 6 miles of conveyor belts, 3,100 standard telecars, 450 oversized telecars, 10,000 motors, and 92 PLCs to control motors and track switches.
Let's make some educated guesses here. How many PCs? 300? Good grief! Later in the article it says the PCs were running OS/2. So what? This is just bad architecture, regardless of OS. So many parts, so many points of potential failure. And the NetFrame Systems "fault tolerant" server is simply...a glorified PC. (It's X86, ~300 MHz P2, and likely running Windows NT, according to other sources.) This article has more on the sad fate of NetFrame.
There's nothing even close to a mainframe computer in this baggage handling system. The New York Times sucks again!
Ben Hocking
Need a professional organizer?
They should have built the airport in downtown Denver.
there is a section on this fiasco In "waltzing with bears' a book on software risk management by demarco and lister.
they point out that although the software is blamed for the 1.1m / day cost of lateness, the reality is that many other contractors unrelated to the baggage handling system hid their own lateness behind the very public software problems. Even if the baggage system was on time the airport likely wouldn't have opened when it was supposed to.
The book is pretty interesting and uses the Denver thing to show how a lack of risk management played a big part in the software woes.
Other than the opening days a decade ago, the system has one of the best records of any US airport. The problem is that when trying to decide where to send it to, it would send it down the wrong ramp. basically, the system could not detect the bags correctly. Sadly, with RID coming online with baggage, the system would have been made reliable. Since this system will NOT be ripped out, it may still be brought back in the future.
I prefer the "u" in honour as it seems to be missing these days.
But since you mentioned it, Kunstler is a hypocrite and jackass. He flies all over the country promoting his book in the least-efficient way possible. He came to speak at Google and stayed in San Francisco, took no public transportation to Google campus (which is served by Caltrain amongst others), then rode in a taxi from Mountain View to Berkeley, a trip that could easily be made by a Caltrain-to-BART transfer. It's pretty ridiculous. He is apparently uninterested in taking advantage of the kinds of technology he advocates. If he's so advanced, and if flying and driving are so stupid, why doesn't he just give his talk by remote video?
I was living in the area when DIA was being built (and life sucked badly after it was complete - pretty much everyone I talked to preferred Stapleton for many reasons, the simplest of which was that you didn't have to drive 5 miles at 25mph after getting your short-term parking ticket that charges by the tenth of an hour).
Anyhow, I remember they held a press conference when they finally started the baggage system, and it was one of the funniest things I've ever seen in my life. Suitcases were flying every which way, often ripped in half, and the reporters were all hitting the deck! Of course, this was funny to me because I wasn't down there dodging flying Samsonites; one of the problems with the baggage system was the startlingly high rate of Workers Compensation claims of the workers who had to deal with it, and the most-common cause of injury was, unsurprisingly, falling items.
If anyone has a link to that video, I'd love to see it again. I've tried, but no luck. Maybe some enterprising soul in one of the Denver local news channels can put it up on their website as part of the story of the system's closure?
political_news.c: warning: comparison is always true due to limited range of data type
I personally know the conveyor mfg'r in NZ. His co is the only serpintine conveyor patent holder worldwide. If your bags go around, that's his system. Early-on he could not get US engineers to respect design limits of his product's radii limitations.
It wasn't just a botched set of expectations. Blatently they designed away in full-face of specifications to the contrary that components had working limitations. The attitude was fix-it, rather than design to product spec.
Hardware Lies.
Which means, those laser scanners don't always read the label as they should, boxes and things get caught on edges and don't move even when the conveyor is on, electro-mechanical equipment doesn't always work, switches sometimes stick, etc, etc. Your job as a software engineer is to anticipate these and to try to make sense out of the information the hardware's giving it, even though something may be garbled, and write your program so that the system can keep running and that operators are made aware of the mechanical problems the software is seeing so they can correct the situation.
A few years ago, K-Mart introduced automated checkouts (with all the buzz words of "convenience", "automatic", "quick"). It caused a lot of problems. The solution: K-Mart started putting notices that said, "To improve customer service, we are opeing more checkout lanes with a cashier" (the same ones that they closed earlier). In MBA speak, they have made two "improvements" in a space of a year!
S
For more background see Software Runaways: Monumental Software Disasters, ISBN: 013673443X, 1997 by Robert P. Glass includes the history of the Denver airport baggage handling system and 15 other desasters in large software systems, e.g. the FAA Air Traffic Control system (death by committee), American Airlines reservation system and others.
Be aware that this is not a technical book and mostly concerned about project management and the problems of defining the requirements of large projects years ahead of their finalization. All the project failures described are very large, complex projects including lots and lots of politics.
As a whole the book is rather depressing, because although in review the cause of failure seems rather obvious, but there is no obvious way to avoid them. It's also a rather dry subject, do not expect to many laughs. But it is great for a large picture on software development, a kind of "how not to do it" guide.
memomo: free web based language trainer DE-EN-ES-FR-IT
I'm posting anonymously, because I was a maintaince guy on the baggage system until a couple weeks ago when I decided I'd better look for somewhere else to be because it became painfully clear that my job was going away. I've been there for ten years, and I'll admit that the machine has had some problems... But it very rarely goes down to the point to dosen't work.
The fact is, it's *the biggest machine in the world*, bar none. There's almost 30 miles of completely automated track, more motors and linear accelerators than you can shake a stick at. A machine that big is going to require lots of time to shake out, and that happened about 5 years ago.. It's been running very smothly since then, because we've established protocols to cataloge and rank priority of repairs. You can't imagine the dynamic loads on it. 1/4 inch thick track pieces can snap in two if they weren't repaired correctly, and yeah, 5 years ago we were having problems. It's all but solved today, it's very smoth running and despite it's costs, it's STILL the cheapest way to move bags around in the world.
Those baggage handlers are full of shit, it should be known. One of the main reasons it eats baggage is because those asshats load the baggage like morons. I've seen more panties strewn about than you'd like to know, and it's almost always a women's bag that gets ate. You know why? They pack the fuckers like sausages, and the baggage handlers just plain don't load them correctly. They won't put them flat, one end will be hanging out--and it's always the heavy end. The machine has close tolerances in some places where tracks intersect go over-under, and tight turns that can fling the bags out if they're loaded poorly, I mean they're going 30mph at some points, an improperly loaded bag will get tossed, regardless of it's weight or size.
It is a mechanical monster, no doubt, any machine that big is bound to be... But it baffles me why they've got to shut it down at the peak of it's opperating efficeincy. It's never run so good, and they decide to kill it after we tamed the beast. You should realize that the command and control system they have in place that operates the machine is always being optomized, and sometimes poor programming has led to breakdowns and increased baggage eating.
Conveyors will be much less efficient, and the airport dosen't have the infrastructure in place to handle the entire load of bags by hand, and even if they did it will be far more expensive. There are 4 turnstyles that will need to be built soon--and airport construction is anything but fast.. Like I said, it dosen't make anything but political sense to shut the machine down.
I've had luggage take forever in JFK airport, and the fastest was in another country!
This may come as a surprise to you, but some of us out here in the rest of the world don't consider US technology to represent the absolute pinnacle of human achievement.
That's not to say that you don't lead in some areas, just not all of them. Judging by the horror stories one hears about US airports, I'm lead to think that aerial transportation is in the latter category.
If your comment title says 'Re: Foo', I'm not likely to read it.
Sure, deep blue and other contendant for the 1000000 googlyflop use mainframe for computer power. but most of the world do not use mainframe for that, but for transaction based system which handle a LOT of transaction per second with real time big database (for example airline RES system, bank system...) and security (transaction termin properly and start properly, and save data properly on disk). Processing power is your LEAST problem. Plus those mainframe system are so old they have been debugged by 2-3 generations of programmer 10 times over. Due to the size of the code and the constant debugging I doubt you can get this kind of quality on PC with a new software and a lot of investement...
C. Sagan : A demon haunted world:
http://www.amazon.com/gp/product/0345409469/
visit randi.org
... mental note
be
more
careful
of
the
word
PROTOTYPE
There were a whole host of problems, including late starts, moving specs, a plan for a small system that was changed to make a plan for a large system by simply multiplying the spec for the small system, construction interferance, etc.
Software Runaways has lots of information about this projects problems. And lots of good info about other runaway projects such as the new ATC system that hasn't gotten off the ground yet.
I took a picture of my punk rock cousin standing next to a mural of a soldier in a gas mask stabbing a white dove in the ass with what looks like a scimitar. Freakin weird. Who on earth would paint such a thing in the airport??
According to the Fine Article:
BAE Automated Systems of Carrollton, Tex., which designed the system, has since been liquidated, and no one associated with the effort could be reached for comment.
I think you're confusing it with BAe, formerly British Aerospace.
If your comment title says 'Re: Foo', I'm not likely to read it.
With all of the cost overruns, the wierd artwork, and the abandoned baggage system, DIA is still the single most usable airport in the United States.
1: There is more room for security which leads to shorter lines. Additionally, connecting flights don't require going through security again, further decreasing the load.
2: The airport design is simple and easy to understand. There is only one terminal building to arrive at, and the concourses are arranged logically.
3: The terminal is very nice - well lit and refresingly open. There is a distinct "open air" feeling that doesn't exist in many airports. There is a wide range of services as well - plenty of food, bookstores, coffee, etc.
4: Unlike Stapleton, snow doesn't shut down DIA.
5: The train system is fast and effective.
6: There is room for expansion, which is particularly important as Frontier expands (DIA is a major United hub, and the only Frontier hub).
7: The large size of the airport and openness of the runways make it easier to land and eaiser to route traffic.
DIA is the world's 10th largest airport. Give it a bit of credit.
I agree with you, and I hope the following example can contribute.
The inventor of the japanese subway tickets system had the same problem (regarding users not being precise enough, sometimes the tickets would go sideways, etc). People were sick tired of having the machines eat their tickets just because they weren't in the right position.
He was so pressured that he almost gave up, so to clear his mind, he took a walk in the park. Then, as he was on a wooden bridge over a small river, he saw a leaf floating on the river moving against a rock. The leaf was perpendicular to the river flow, but then it collided with a small rock, that made it turn parallel with the flow.
Based on this idea, he implemented a small device consisting of a round piece of metal that would rotate the tickets to the correct order, so they would pass the magnetic scan. Currently this magnetic ticket system is implemented in many countries, including the mexican subway which is over 25 years old now.
So, in the end, it all comes to this: A well-designed system will pass even the worst conditions. The Denver Airport Baggage design team certainly needed to work more, and think of the worst cases - i.e. quasi-spherical (i.e. bloated) luggage.