Computer Failure Causes Gridlock In MD County
Uncle Rummy writes "A central traffic control computer in Montgomery County, Maryland failed early Wednesday morning, leading to widespread gridlock across the entire county. The computer, which dates to the 1970s, is the single point of unified control for all traffic signals in the county, which comprises a number of major Washington DC-area suburban communities. When the system failed, it caused all signals to default to stand-alone operation, rather than the highly-tuned synchronization that usually serves to facilitate traffic flow during rush hours. The resulting chaos is a yet another stark reminder of how much modern civilization relies on behind-the-scenes automation to deliver and control basic services and infrastructure. The system remains down Thursday, with no ETA in sight."
All the lights have turned blue. And near equal proportions of drivers are interpreting blue lights as go and stop. Very messy.
Maybe all the meter maids could direct traffic for a while?
I smell foul play...
Quick, someone get Bruce Willis!
Wooo Fire Sale!!
"I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)
Waiting for the Hackers movie references in 3...2...1...
back in the day i read a "tfile" by Sunspot IIRC that explained how to break into those boxes attached the stop lights at intersections and make every light stay green all the time. Not sure if it was legit or not but it sounded a little far fetched.
As for the single computer, i bet a coke no one knows the root password, the system administrator is long gone and the programmers are very long gone. I bet the staff tried to power cycle it thinking it was just like a PC and now they've made the problem 3x worse.
I came to the datacenter drunk with a fake ID, don't you want to be just like me?
I have an old Pentium 75. They are welcome to it, it it will help. :) Of course with the diffrence in processing power between a Pentium and a 1970's era computer, you could probably run the entire countries traffic lights with a P75.
If I were God, wouldn't I protect my churches from acts of me?
I live around this area and I hadn't noticed anything. I haven't heard anyone complain either..
I guess traffic was a little heavier yesterday. Traffic sucks all the time, if it rains it is the apocalypse.
Hmm.. Strange.
~Mekkah
http://www.symantec.com/netbackup
Just throwing that out there.
~Mekkah
Most of those boxes have a "conflicting green" detector circuit that automatically puts the signal in "safe mode" when it detects two conflicting green lights.
On simpler systems, "safe mode" is all-way flashing red lights.
I guess if you knew what wires to mess with you could disable this safety feature.
Physical access is root access.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
No lights is better than badly times lights.
Go green: turn off your refrigerator.
So the default behavior is basically traffic lights in Los Angeles on a normal day? I feel soooo sorry for them. ;)
If there had been a widespread power outage, most of the lights would have been dark.
Um, what's the protocol when the traffic light is all-ways flashing black/black? *groan*
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
According to this it was a Data General main frame from the 1980s and not from the 1970s. Anyone know what model?
For those who aren't familiar with Montgomery County, MD. It is one of, if not the richest counties in the nation. I find it amazing that even in a county like this, the public infrastructure is crumbling.
They had a massive water main break earlier this year that made the national news.
on the same day, the PDU for the D.C. Metro (subway and bus) communication system failed, leaving no communication for the bus system (including fair collection machines), leading to more travel trouble.
Got all those conspiracy theorist wondering if they are related.
A wood rowboat from the '60s develops a leak and sinks to the bottom of lake Woebegone.
But you'll never make it to the interview on time, as traffic moves at the speed of my old TRS80... wait a minute...
No comprende? Let me type that a little slower for you...
The upside in this is that the lights still work when the controller is down. They don't go flashing red, stay red, turn off or something worse.
I hereby place the above post in the public domain.
Fire him.
Damn! 70s? Talk about Return on Investment.
The WashPost, in another article touts Fragile Technology.
I reach for my 70's era calculator and estimate the operational life of 34 years for this system. Some Fragility. Who or what at the Post has been there that long.
Wonder if its some ancient PDP version or an small IBM mainframe. The article is scarce on details. Parts for either are getting hard to find except in the scrap market.
Still you have to wonder why it wasn't ported to some other platform if nothing else as an exercise in disaster preparedness. Any commodity computer could do the job.
There is a lot of stuff like this still in service. I saw a PDP 8 monitoring turbines in a hydro Power station, and asked about where they get that fixed. The reply was it never broke down, but they had stockpiled 6 replacements, tested each yearly, just because they realized how old it was. Nobody knows exactly what it does anymore.
Sig Battery depleted. Reverting to safe mode.
it's "password"!
This is government, you know.
It's NOT me! It's the meds! I'm on 1000mg of Fukitol.
Just across the river in Fairfax County, Virginia, this is the normal behavior for lights. In fact, i suspect some of them are timed so as you get released from one light, the next one (200m away) turns red.
Quote "The county is in the second year of a six-year program that will bring in more modern and reliable equipment."
Reliable?!?!?! If this thing was built back in the 70s and just now has crashed badly....how much more reliable could you be? I can't get a modern motherboard to last more then 5-7 years before something goes wrong with it....if i'm lucky.
"You're only supposed to blow the bloody doors off!"
Let's not go crazy here, I mean, there's not a whole lot that's relatively modern about this if the thing was developed in the friggin' '70s and operates without an efficient backup system. If anything, it's an example of how much society would _BENEFIT_ from a modern system.
That's a sane default at least. Never overestimate a large software system...
... such as "having a simultaneous green phase for bikes to go in all directions at once."
Here's a piece about traffic lights optimized for furry bicyclists... http://hembrow.blogspot.com/2008/09/default-to-green.html
From TFA: "They know where the problem is, but they just don't know what it is," she said. "The server seems to be sending the signal, but the conduit is not transferring the information to the signal lights."
I can tell you where it is. Right there on layer 4. Does that help? Then try layer 8.
I judt got a nre Kinesis keybiartf so please excusr ant egregiou typos.
..that Charlie Croker wasn't involved?
I was going to say we should blame this on Windows Vista, until I saw the part about the computer system dating back to the 1970s, so that wouldn't work. Still, there's got to be some way we can put the fault on Micro$oft? Maybe the computer was in need of some necessary maintenance, and the technician whose responsibility that was was too tied up in a game of Minesweeper or Solitaire, or something?
...you can always call work and say you'll be late. Unless you've got T-Mobile.
never under estimate the ability of some fucking pro central control wanker to fuck you over along with millions of others. this stands as an omen of doom for anything too centralized like OBAMANUSIM or OBAMACARE
The last time I remember being at a light that was out, it was night and the nearby street-lamps were also out.
Unless you knew there was a cross street there, you were likely to plow right though it.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Its started .. EBE Entity self aware 18:23 hours
The resulting chaos is a yet another stark reminder of how much modern civilization relies on behind-the-scenes automation to deliver and control basic services and infrastructure.
Just Skynet trying to figure out how to bunch up targets when it seizes control of our Predator and Reaper UAV's.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
"firesale, stage one" now please excuse me while i talk to the nice FBI guys arriving at my door in the next 5 minutes.....
Smoking cures cancer. Smoking also cures stupidity. check darwinawards . com for some stupid stuff
The traffic here in NYC hasn't suffered at all!
Don't you have roundabouts in the US ?
This is so much more efficient than traffic lights... and no computers are involved.
I work at the NIH in Bethesda, MD and live in Baltimore County, north of Baltimore City. It normally takes me an hour and a half to get home, but lately it's been three hours or more. That's okay though; I carpool and get OT for working in the car via MiFi.
He seem to know how the machine works.
From "Enemy of the State" :P
... who cares about nation-wide traffic light outages ... especially in comparison with nation-wide power grid outages. Exactly why it is not always a good thing for the big bad government to control everything at one central point ... I personally think that for items like this, we need it piece meal ... so that we ensure we don't have nation-wide issues that could potentially put us at risk.
We barely have driver education to get your license. Actually in many states there is no requirement for drivers education if you are 18 and can pass the driving test. There are no requirements for ongoing education/training unless you count the silly 8 hour defensive driver course for those that get traffic tickets. People have trouble with the concept of yielding here.
We have a few roundabouts. Believe it or not, frequently every entrance has a stop sign instead of a yield sign. Kinda defeats the purpose.
At least from the article, it sounds like they have just one server set up to do this.. I guess I shouldn't be surprised, especially given the things I've heard about other types of infrastructure but isn't one of those things that should really have some hard core built in redundancy? They should really have some backup servers that are ready for this sort of thing to happen and can take over when one of the systems fail.
...except for a bunch of Mini Coopers driving down the sidewalks.
Have gnu, will travel.
I have two words when it comes to roundabouts in D.C.:
Dupont Circle
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
... to mediate traffic instead of traffic signals, they wouldn't have needed the aging old single-point-of-failure computer in the first place, because roundabouts (a) require no computers, (b) require no electronics at all, (c) require no electricity, and (d) don't require maintenance. What's more, since they allow motorists to preserve some momentum in all but the most congested traffic, gas consumption from forced arbitrary deceleration and acceleration is reduced. The only intelligence they require isn't of the artificial sort at all, only a smidgen of it from the motorists using them. They are un-powered and self-adjusting to traffic flow.
Would anyone like to take a stab at how much energy and man-hours is expended on the traffic signal network in the United States every year?
Most likley it's an IBM 1800 process control computer. This was popular during the early to mid 70's for traffic light control. The program was written in 1800 ASM. Parts must be scarce as are programers and language docs.
I had the color version of the TRS-80. I thought saving code on cassette tape was better than sliced bread!
I only look human.
My mother is a halfling and my dad is an ogre, so that makes me an Ogreling
modern civilization
relies on
modem serialization
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Traffic here is normally awful, but thanks to this it was really bad this morning.
This is my home county for the past eight years, and I think the default traffic patterns are actually probably going to be a significant improvement. I am not a traffic engineer but I am amazed at the number of lights in the county that have clearly wrong behavior where most of the time there is a green light going in the direction there is no one at all traveling at that time of day.
Brian Fundakowski Feldman
Did anyone else read
When the system failed, it caused all signals to default to stand-alone operation
as a pretty good thing?
I mean, any time you have a central system you will have failures in it, that is the nature of systems, but the fact that the central system failure initiated decentralized operation, instead of outright failure, sounds like good design to me. Perhaps the autonomous light program could be made more intelligent to handle central failures better (operate in approximately the same way that you were instructed to exactly a week ago, for instance), but while this is obviously not ideal, things worked out about as well as they could have IMO.
Nothing new here. If you have only two highways (495 and 270), with no east-west flow, you get gridlock, and you get it often. This is no different than a semi flipping over on 495, or a stalled car on Rockville pike, causing bailout to a traffic grid that hasn't been able to handle it for over 20 (30) years. The fact that Potomac was affected is a good thing. Those entitled citizens would never consider a bridge to cross the Potomac River into Tyson Corner to ease such congestion, which is exactly the choke point in this entire region.
Maybe they could make some use of the GPL code from the SUMO project? http://sumo.sourceforge.net/
"I object to doing things that computers can do." -- Olin Shivers, lispers.org
"Intelligent Transport Systems: Cases and Poliies" by Roger Stough on Google Books: http://books.google.com/books?id=fs-SYjIS88oC&lpg=PA110&ots=HcpCMSKdgw&pg=PA110#v=onepage&q=&f=false
When I was a teen, I had a car with the license plate RED SHFT. Most people saw it and thought, "Red car, stick shift, who cares?" but I did get that enthusiastic thumbs up of understanding from maybe one person in ten thousand.
Space game using normal deck of cards: http://BattleCards.org
"The American Association of State Highway Officials (AASHO) was founded on December 12, 1914. Its name was changed to American Association of State Highway and Transportation Officials on November 13, 1973" ... after members became fed-up with the acronym.
Did the computer "fail" or did the engineers blow off "High Availability" This is 2009 most legacy systems can be emulated.
When the system failed, it caused all signals to default to stand-alone operation, rather than the highly-tuned synchronization that usually serves to facilitate traffic flow during rush hours.
So now they work like the lights in Northern Virginia?
> When the system failed, it caused all signals to default to stand-alone operation,
Here I thought it was gonna be some stupid design flaw only exposed when the system broke down, but defaulting to stand-alone behavior when the controlling master system breaks down is the proper thing to do at that point.
It's interesting how said efficiency allows for fewer roads and lanes than otherwise would be needed to handle that much flow.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Welcome to Florida in hurricane season, guys.
I mean, Wheeljack must have had some kind of computer failure if he thought it was a good idea to give the most physically powerful Autobots the most feeble brains available...
And, really, he could have skipped the whole transformation thing. It was quite awesome enough having a big metal dinosaur tromping around kicking ass...
Bow-ties are cool.
Being from Buffalo I was curious that I'd never heard of that - turns out it's actually in Syracuse, which is two cities east of Buffalo (Rochester in between) and about a two and a half hour drive :)
Here's some info,
And here's a photo.
I was *just* out on the affected roads in Montgomery County and I can tell you exactly why this failure of this system IS an issue of safety -- more than a few of the people who live/work/commute in this County are self-important idiots who refuse to wait 5 minutes for the traffic light to turn from red to green during rush hour so they just stomp on their car horn and proceed to drive through the red lights!!! This wasn't happening one or two cars at a time either, it was walls of them all acting as though the traffic lights were off rather than just changing more slowly than they should be.
THAT is an issue of safety.
I also saw a Montgomery County Police car drive through one such intersection while people were doing that and he/she did not stop to deal with the situation.
If you read TFA, down a ways, it seems the problem is not in the computer at all, it's in the conduit that distributes the signals. Maybe just a dope with a backhoe.
they do more then that they are linked to other lights and some areas with 3-ways / ramps lights need to be linked or things will back up real bad.
... for the convenience of the red-blue color blind (the commonest type of color-blindness).
Typo: For the convenience of the red-GREEN color blind ...
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
He's just repeating the official figures from the government office, where an ex Verizon accountant tallied up the contract specs...hell, dots, zeroes, commas..just throw a few in there, it doesn't matter
"Big computer in Rockville, MD" ... from the 70's ... XDS Sigma series perhaps? Xerox had a big presence in Rockville way back then, and their computers were definitely big. Not particularly powerful, but they did have hardware interrupt and that was fairly new in the early 70's and they did a bit of traffic control work (generally via TRW). In the late 70's, it could have been a PDP-11 of some stripe or even a DEC 10. Anybody have any particulars?
Do not mock my vision of impractical footwear
"The system remains down Thursday, with no ETA in sight.""
Can't find those 5.25" floppy disks for DOS 3.1 I see. Time to check ebay.
I've seen a lot of projects like NES, Apple ][, Amiga... emulators so that old games can be played on modern machines...
Are there not such projects for old Mainframes? Might be an excellent way to replace such systems... unless the costs of the required stability testing, software licensing issues and such would be too much... also custom hardware I/O may be difficult to adapt... (Anyone know where to get a USB 8 inch floppy disc drive or 110 baud modem?)
Probably better to replace such incredibly old applications anyway.
...is that the state of Maryland does not have a proper driver's test. There isn't even a ROAD TEST. You go to the DMV and drive around the goddamned parking lot for few minutes. As a transplant living in the state, it scares the living hell out of me driving on the road with these people. Driving here, you either learn to anticipate all manner of stupidity, or you die.
--------
This isn't the sig you're looking for. Move along.
Props to whoever mantains this system that the issue does occur more frequently, but this is a problem with a simple solution. Redundancy. The system is probably too old to be configurable to automatically swap, but a simple setup with two servers allowing a manual hot-swap, is all you need. Not a difficult problem to solve.
In fact when I've a floppy of a maximum diameter,
When I can call a subroutine of infinite parameter,
When I can point to registers and keep their current map around,
And when I can prevent the need for mystifying wraparound,
When I can update record blocks with minimum of suffering,
And when I can afford to use a hundred K for buffering,
When I've performed a matrix sort and tested the addition rate,
You'll marvel at the speed of my asynchronous transmission rate.
Though all my better programs that self-reference recursively
Have only been obtained through expert spying, done subversively,
But still for input vegetable, animal, and mineral,
I've built a better model than the one at Data General.
Paging Bill Gates!
Believe it or not, frequently every entrance has a stop sign instead of a yield sign. Kinda defeats the purpose.
So, you are saying you live in Washington, D.C. You mention you are surrounded by self-important idiots. And you also mention that your traffic is controlled by idiotic laws.
I fail to see what you are surprised by.
It's interesting how said efficiency allows for fewer roads and lanes than otherwise would be needed to handle that much flow.
Kind of like the newer heavy day versions with wings...
The county's chief traffic engineer, Emil Wolanin, said replacing the 1970s-vintage system would not be easy.
And will the replacement system be Y2K safe?
One line blog. I hear that they're called Twitters now.
As for the single computer, i bet a coke no one knows the root password, the system administrator is long gone and the programmers are very long gone.
Reading TFA (but you have to go to the second page, oooh), it seems that they're a bit more clueful than that.
"The server seems to be sending the signal, but the conduit is not transferring the information to the signal lights."
Maybe in a day or two they'll contact a network engineer :-)
Just switch all the traffic lights off.
Seriously. I have never, in decades of driving, seen longer queues at broken traffic lights than when they were working. This is especially true at a busy crossroads on my old route to work, working traffic lights meant a queue of 12 - 20 cars in each direction whereas broken traffic lights had less than 5 cars in any direction. Same roads, same time of day.
The traffic light manufacturers must be bribing the local councils and highways departments pretty good.
They whose government reduces their essential liberties for temporary security, receive neither liberty nor security.
Remember that studies have shown cycling may cause impotence in men and sexual dysfunction in women.
http://www.latimes.com/news/obituaries/la-he-cycling9apr09,0,5220104.story
More likely you're seeing two sets of lights, but then you're probably seeing two roads as well.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Okay look, technology has advanced to the point this probably works for 90% of the time awesomely. Just get a modern server, and someone fueled with Coke in front of it. The combo is way more powerful than whatever they had originally.
First make an api to the switching network, then map the nodes to match the geography roughly, using google if you must. Throw a machine learning algorithm that simulates lots of ants (okay cars for this project) going through the traffic signal graph and each can have a few characteristics like average speed to destination, gas consumption, frustration, etc. The traffic signals can be tweaked by hand or you can give them genes for different oscillatory patterns.
If you start with the main arteries first I bet you could quickly develop a traffic signal plan that works great. How much you want to bet this could be done by a bunch of suitably competent types with a nice big prize in a hackathon? Now I'm not saying to plug the network into Wolfram Alpha's server farm but it just might work...
>The resulting chaos is a yet another stark reminder of how much modern civilization relies on behind-the-scenes
>automation to deliver and control basic services and infrastructure
No the real problem is the fact that someone decided that keeping a computer from 1970 running till today to run the traffic system!!!
We have dual core processors for 300$ at futureshop, seriously, what could ever make someone decide to leave such an old system in place to run things....hopefully they learned their lesson, and will be installing the new technology soon.
Maryland county people should ask for the city council to report back any other such situations to pass a vote on whether to replace all the aging technology!
Now I have to change all of my passwords. Thanks.