Slashdot Mirror
Verizon's NYC 911 System Shutdown
Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
← Back to Stories (view on slashdot.org)
I think this is a shining demonstration of why monopolies shouldn't be allowed to run the phone system. Speaking of monopolies, I wonder what the 'software' concerned was?
Was it a "what does this little red button do?", or a misconfiguration somewhere by a sleep-deprived sysop at 3am?
Am I the only one who has never heard of 311?
I know about 411, but that's not a non-emergency number.
Any other *11 numbers we should know about (and what do they do/where are they available?)
So, they've just announced to all the terrorists in the world - this problem still exists, is going to remain there and if any terrorist organisation can get one of their members on the inside, they'll be able to take the 911 lines down anytime they want. great.
"Free software as in beer, copy protection as in racket" - Telsa Gwynne
Here a few years ago there was a sting of robberies where the thieves called a whole lot of people and convinced them to "test" the emergency response system at a specified time a few days later.
All of a sudden there were hundreds of simultaneous calls reporting accidents, fires, muggings, heart attacks, rapes, robberies, etc. The thieves robbed two banks and a big-box store while the police were tied up.
Thanks for bringing up 311. I'm amazed at how few people even know about 311. It seems the public's adoption of 911 has to be 90%+, yet I bet the adoption of 311 has to be 10% or less.
:cough: gentleman fleeing the scene looked around for a good 30 seconds to see if anyone noticed before making a break for it. (Read: he sped away.)
I recently sat on hold with 911 for literally 20 minutes after I watched a vehicular hit-and-run accident from my car, where the
Luckily I was close enough to get the a-hole's license number before he did, and ultimately I got thru, but wow am I glad it was just damage to another vehicle and that there were no injuries/fatalities.
Also worth noting is that I was twice transferred to other divisions/operators after speaking with the first one.
BTW, In case you're wondering it happened in the middle of broad daylight on a Saturday last Fall--not exactly "peak" 911 time.
This has to make you wonder how many of those calls in front of me were BS, and why not allow people to route themselves based upon an automated menu system?
Oh, you're calling about your neighbor's dog barking? We'll get back to you... eventually. Leave your number.
-OR-
Oh, you're on fire? No problem. I'll put you right through.
----------
Nope. Not gonna do it. Wouldn't be prudent. Not at this juncture.
Won't work, even with antiquated mechanical switches. Phone lines are traditionally connected to lightning protectors to save the switches from (as if you couldn't have guessed) lightning strikes at endpoints. Usually the protectors were part of the frame where you run patches from switch ports to the actual cables that hold X pairs.
My wife,daughter and I happened to be wandering around Brooklyn during the outage. We noticed an increase in the number of uniformed officers on the street. This made us slightly nervous in a 9/11 (not a 911) sort of way. It wasn't until we read Saturday's NYT that we realized what was up. To increase visibility during the outage, NYPD had ordered all plainclothes officers to put on uniforms.
As we strolled through some of the city's busiest commercial sectors, everybody as oblivious as we were.
I second this. I Live in PA. I would have no idea about a NY phone system crashing if it wasn't for this website posting the info...
I'm not a prophet or a stone-age man,
I'm just a mortal with potential of a super man.
Thanks to the government mandated breakup and unfair reprisals against AT&T, the current phone system in the USA is **NOT** a monopoly.
This is yet another example of why the breakup was a bad thing.
---- Booth was a patriot ----
Yeah, either a 5E or a DMS 100 (or the 100/200 tandem combo) - Verizon has lots of both.
As a 100 design engineer for the last 10 years, I can say that this type of problem is very difficult to detect by software. Routing and translations datafill is highly decentralized with no interswitch communications to verify the impact of a single switch configuration change on the network as a whole. You gain a great deal of reliability with such a design but loose the ability for interswitch cross checking.
One of the more interesting problems in this class is the 'translations loop'. This problem occurs between two offices, usually connected by a two-way trunk. A translations mistake on both ends cause incoming calls to be routed out the same incoming trunk. A single call can chew up all circuits on the trunk group in a matter of seconds - until the caller gets a 'all-circuits busy' treatment. They remain busy until the caller goes on hook. It's always fun to watch happen in the lab. (Why am I getting treatment? aww shit haha)
I feel sorry for their technician. The tools he had available would not have detected this nor are any improved tools on the horizon.
Hell, I live in NY state, and this is the first i've heard of it, online or the radio.
Having worked in private line provisioning and maintenance at Ma Bell I can say without equivocation that this is a direct result of the breakup and not really 'human error'.
I've seen the exact same thing being done at all the locals and the long distance companies.
Manpower is being drawn down, redundancies eliminated, and a talent and brain drain that causes errors like this.
The reason is always given that automation is allowing the company to maximize the remaining workforce and competition makes is neccesary. BS.
Best Practices are -gone- everything is driven by sales and bean counters. Engineers, Technicians and Managers who complain are moved, removed or eliminated from the loop because facts are not going to be allowed to get in the way.
It used to take weeks to get a misdirected line corrected in some instances.
The fault was blamed on too many layers, and union incompetence.
Now with all the improvements brought about by divestiture and competition it is a near imposibility unless it affects a major source of income or government.
This type of error was prevented by human redundancy and a workforce able to put the breaks on before the damage was done because they could stand on the strength of regulations and the union and tell the idiot boss in charge that things were wrong.
Get used to it, rapid reorder will be the order of the day.
I worked 911 for 14 years........here in the midwest, we have tornados around this time of year. The warning sirens usually go off to warn the public. Ok, I'm sitting at my console, waiting for people to call about wind damage etc......and the line rings....911, do you have an emergency?....yeah, can you tell me what the sirens are going off for?.........it's a shame that the line is recorded....there have been times, when we get idiot calls like that, that you really really want to say....well, I'm sorry to tell you this, but the russians have changed their minds, the bombs will be here in 10 minutes. And then, hang up on them LOL.... Heck, we've had calls from people that ask us where Bass Pro Shops is.....(Springfield Missouri)
Damn, where is the 11 button!
That's not as much of a joke as it appears at first glance. Emergency service instructors (like the people who give fire drills in schools and such) always refer to 911 as nine-one-one, not nine-eleven for just that reason.
If you're a zombie and you know it, bite your friend!
I dunno where you live, but the last time I called 911, I was talking to an operator within about 15 seconds.
We just recently (about a year ago) got 911 service here. Before that there were separate local phone numbers to call for police, ambulance, fire, what-have-you.
Using the "old" system, I have over the course of time reported a couple of fires, a few burglaries/robberies, and a disoriented old lady to the appropriate emergency services. The fire emergency phone and the police emergency phone were always answered by a real person within two rings.
I called 911 to report a burglary in progress across the street a couple of months back and got a recording when I called and had to wait on hold before someone got around to talkign to me.
I find that completely unacceptable so I wrote a letter of complaint to our city council.
Everyone else should do the same when that kind of thing occurs.
Don't think that a small group of dedicated individuals can't change the world. it's the only thing that ever has.
Your sig is appropriate here.
If you're a zombie and you know it, bite your friend!
===--===
Together, we will drive the rats from the tundra.
1. False redundancy: Although the NYC system has a backup central offices and call centers, it apparently routes all calls from the affected area through a single Verizon subsystem. Their system is fully redundant except where its not.
No, the calls always have multiple paths they can travel. The problem came when all of the routers were given the same mis-information. No number of redundant routers can protect from that.
How often do people use the phone during a severe storm to report dammage etc. How often does a high tension line (7200 volt an up) come down and cross a phone line or cable TV line? Ever seen a report of a wire down taking out an entire area code? Lines do come down all the time due to Ice and Wind storms. I've never heard of this taking out an area code. I have had to do repair work in a home office after a power line crossed a phone line. Anything connected between the phone line and the power line was toast due to arcing. This included a computer with modem, answering machine, cordless telephone base, and fax machine. The POTS was undamaged. The other phones in the neighborhood were OK. Only the line to the house died even though there was enough fault current to burn the line in two and drop it into the street.
The truth shall set you free!