Slashdot Mirror
Verizon's NYC 911 System Shutdown
Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
← Back to Stories (view on slashdot.org)
"Quick! Give me the number for 911!" - Homer J. Simpson
Do you think someone called 911 to report the emergency outage?
well the calls were routed to 311, so the calls got answered, just maybe not as quickly. Yes this sucks, but the calls didn't go nowhere.
I think this is a shining demonstration of why monopolies shouldn't be allowed to run the phone system. Speaking of monopolies, I wonder what the 'software' concerned was?
For those who don't want to register:
username: slashdot2003
password: slashdot2003
---
Never criticize religion on Slashdot. You will be modded down for "Troll" no matter how factual it is.
You read stores like this and think, "Oh, it was just error."
But you fail to realize that a big heist probably took place. I saw George Clooney and Matt Damon do it to Las Vegas. They let off this big ass EMP and shut down the power to the whole city! Long story short, they stole a shitload of money and got away with it.
Don't let these stories fool you, that is exactly what happened here.
We have secretly replaced these Slashdot mods' sense of humor with a rusty nail. Let's see if they notice!!
And to think everyone was worrying about the terrorists, fer chrissakes.
I'd "call the wahmbulance," but 911 isn't currently available.
(Are you new here?)
Do you like German cars?
Considering Slashdot doesn't write their own articles and it is basically an aggregate of postings from other sites, I'm not sure what you're complaining about.
:P
Technically, any news on Slashdot is old news since it has been reported already.
Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.
The emergency system broke down about 7:20 p.m. after a Verizon engineer who was making service changes to a bank's telephone numbers in Brooklyn inadvertently included numbers that are used to carry 911 calls, city and telephone company officials said. The numbers were close in sequence, the officials said.
The 911 calls then ended up being rerouted to the bank's phone system, and callers heard a busy signal. City and Verizon officials said that while the backup system in place for 911 was functioning properly, it failed to pick up the calls because it was designed to catch a technical error, not a human error that would be interpreted as simply a change of instruction.
Daniel Diaz Zapata, a Verizon spokesman, said the telephone company would now require a second person to double-check any entry of data that could affect the 911 system, and said the company planned a thorough review of its procedures that would be documented in a report to the city within a few days.
"We determined that a human error resulted in the accidental rerouting of phone calls during a procedure to upgrade service for a corporate client," Mr. Zapata said. "We have immediately altered our processes to ensure this type of situation does not reoccur. We have assured the city that we took immediate steps to make sure this doesn't happen again."
Citing privacy concerns, Mr. Zapata declined to identify the Verizon engineer, except to say that he was a veteran of the company. Mr. Zapata said it was unlikely that disciplinary action would be taken against him.
Police and fire officials said yesterday that they had no reports of injuries during the 911 failure. Fire officials said that about 60 firefighters responded to a major fire, at 3301 Foster Avenue in Brooklyn, which was called in at 8:49 p.m. by someone using a fire alarm box on the street. There were no injuries in the fire.
Paul J. Browne, the Police Department's deputy commissioner for public information, said the department immediately adopted emergency procedures, like requiring e officers on patrol to turn on their flashing lights so people could find them easily and increasing staffing at precinct station houses to answer phone calls. But he said there was no reported increase in crime.
"This didn't present an opportunity for the criminally minded - like the blackout did - because probably most people were unaware that it was out of service," he said.
However, several City Council members expressed anger that the 911 system could have been so easily disabled, and called for creating a more effective backup procedure.
"It's an emergency wakeup call," said Councilman Peter F. Vallone Jr., the chairman of the Public Safety Committee, who plans to hold a hearing about the incident. "We don't have an adequate backup system for 911, which is more important than ever as we fight the war against terrorism."
Gino P. Menchini, the commissioner of the Department of Information Technology and Telecommunications, said city officials were working with Verizon to ensure that the emergency system's numbers were clearly identified, and that its software and equipment were protected from similar human errors.
But Mr. Menchini emphasized that the emergency system already had many built-in safeguards, such as the ability to route 911 calls through either of two central offices and their 911 answering centers. "The bottom line is, 911 works very well, and it's worked very well for a long time," Mr. Menchini said.
Several emergency services experts agreed yesterday with Mr. Menchini, saying that New York 911 system compared favorably with those in other large cities and that an error like the one made by Verizon could not necessaril
Jay | http://oldos.org
Non-reg link: here
"You got the wrong number, this is nine one...two"
-Chief Wiggum
I once built an HA cluster that had a role in the 911 system, at a major telco switch vendor. Apparently the fines for a vendor bungle in this are in the millions of dollars per minute of downtime. I dont know which switches these are, but such penalties could affect that vendor in fines and their bottom line. It looks like Verizon shot itself in the foot, but keep an eye out for a dip in share prices for some of the switch vendors in case the blame gets spread around.
The 911 system seems to have continued to work correctly, but the regular numbers that calls to 911 get directed to were redirected to a bank. So the issue is really that there aren't safeguards against the wrong phone numbers getting changed accidentally, and the phone lines used by the 911 system aren't immune.
Shouldn't the interface for the system prevent you from accidentally modifying similar but unrelated numbers when you're modifying a set of numbers?
Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.
Now imagine what a genuinely malevolent person could accomplish. Perhaps a single individual shouldn't be capable of disabling such a critical system.
Do you like German cars?
This type of error is a classic problem of the computer assuming that the human was right. We create machines to give us power, but use that power to cavalierly.
The idea of having a second person "double-check" is nice in theory, but I will wager that the second person will let errors through too. If the first person is careful, the second person is faced with a long list of matching, correct entries to check. The second person soon becomes fatigued and keeps hitting the "OK" button even when there is a discrepancy. Unless the second person is offered an outsized reward (and the first person is penalized by an even greater amount), its to easy to become apathetic or non-vigilant. (Also, the double-checking process assumes that the original set of command directives was correct).
The real solution is a meta system that logs any changes to the system (like a config change), monitors dependencies of that change, and cross-checks them during exceptions. When an exception occurs, such as a bunch of 911 busy signals, the system would trace through the code and config files and correlate the fact that the onset of busy-911 calls corresponded with the insertion of the erroneous numbers. The system would then either roll-back the changes that caused a fault or alert someone of the list of likely culprits.
Two wrongs don't make a right, but three lefts do.
is that there is a very finite number of customers for things like electricity and phone service. You need a very large customer base in order to be able to charge a reasonable amount.
When you allow competition those that attempt to compete are forced to either charge less than it costs to supply the service or charge more. If they charge less but can't get the customer base they go out of business. If they charge more people tend not to switch. And if you don't charge enough less, nobody cares enough to switch.
Cox saves us all of a buck or two over Qwest on phone service. We never bothered to switch until we switched to Cox for high speed internet.
And of course the only reason Cox had the money to implement phone service is because they're the monopoly on cable service.
In cases like this it's actually better for the government to force the monopoly to act in the best interest of the people than to allow competition which just gives people the false impression that it'll lead to cheaper prices and better service.
Competition in these cases are almost always forced to either cut corners to survive or charge more.
Ben
Work Safe Porn
311 is the non-emergency number for police/fire/medical services in some areas.
Visit the
Last I checked 511 will tell you your own phone number. It sounds silly, but it came in handy for me in college when they would often forget to tell us the phone numbers for our dorm rooms.
That number is handy, but I have to admit that the time I called the pizza place that I knew had caller id and asked them what my phone number was makes a lot better story!
NOTE: I just tried this on my home for and it worked. My phone company is CenturyTel.
Probably because folks were trying to use 911 to report them.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
Here a few years ago there was a sting of robberies where the thieves called a whole lot of people and convinced them to "test" the emergency response system at a specified time a few days later.
All of a sudden there were hundreds of simultaneous calls reporting accidents, fires, muggings, heart attacks, rapes, robberies, etc. The thieves robbed two banks and a big-box store while the police were tied up.
The political types saying that they don't have a good enough backup 911 system failed to understand the root cause of this failure.
A Verizon tech who was re-routing a customer's numbers accidently made a numerical error that ended up re-routing lines that were meant to go to 911 to a bank. Therefore, the backup system never got a chance to kick in, people were being routed to a very poor selection for a primary destination.
The safety valve that I'm sure is being installed now is requiring a higher degree of password to change the routing instructions for the 911 lines... because this tech should not have been able to mess with them, and didn't mean to, he just typoed the numbers he was supposed to type in. He at least should have seen a "You're trying to reroute 911! Are you sure you want to do that? N" prompt.
As I read the article, it is obvious that NYC's system is fraught with deep flaws in its design and management. These include:
1. False redundancy: Although the NYC system has a backup central offices and call centers, it apparently routes all calls from the affected area through a single Verizon subsystem. Their system is fully redundant except where its not.
2. Organizational silos in a coupled system: The City claimed that its 911 system was fine because "an error like the one made by Verizon could not necessarily have been prevented because it was not a flaw in the 911 system itself." Yet the Verizon circuits, systems, and procedures are an integral part of the 911 system. The City (and Verizon) maintain a fiction that they are independent entities when, in fact, they are tightly coupled. This division of responsibility is fine for playing the CYA Blame Game, but does not create a robust system.
3. User Interface Flaws I don't know what kind of user interface that technician was using, but it obviously has some terrible flaws if it did not warn him of the implications of the data entries. I also suspect that he was manually retyping some numbers off a computer print-out when he should have had some mechanism to download a set of proofread, verified, double-checked entries.
I don't fault NYC or Verizon in particular, they are probably no worse that anyone else. I only get angry that these types of structural insecurities are probably more widespread than anyone realizes.
Two wrongs don't make a right, but three lefts do.
Eh? Did NYPD and NYFD need the 911 system to find out about the WTC strikes? Terrorism isn't about killing people, it's about getting publicity.
The councilman can rest assured that the terrorists will helpfully keep their activities high-profile enough that 911 notification will not be required.
[Sheesh, why does *everything* have to be about terrorism these days.]
Slashdot monitor for your Mozilla sidebar or Active Desktop.
Blame you guys.
Won't work, even with antiquated mechanical switches. Phone lines are traditionally connected to lightning protectors to save the switches from (as if you couldn't have guessed) lightning strikes at endpoints. Usually the protectors were part of the frame where you run patches from switch ports to the actual cables that hold X pairs.
The NYT has a habit of catching these things. It won't even let you register cypherpunks/cypherpunks. However, there is a work around -- get some sort of proxy, such as Proxomitron, and configure it so the referer is news.google.com. The same trick works with the WaPo's new registration service.
Les Miserables Volume 1 now up with my reading of
My wife,daughter and I happened to be wandering around Brooklyn during the outage. We noticed an increase in the number of uniformed officers on the street. This made us slightly nervous in a 9/11 (not a 911) sort of way. It wasn't until we read Saturday's NYT that we realized what was up. To increase visibility during the outage, NYPD had ordered all plainclothes officers to put on uniforms.
As we strolled through some of the city's busiest commercial sectors, everybody as oblivious as we were.
I second this. I Live in PA. I would have no idea about a NY phone system crashing if it wasn't for this website posting the info...
I'm not a prophet or a stone-age man,
I'm just a mortal with potential of a super man.
Yet at the same time, daytime Long Distance cost over a dollar a minute from NY to California. Phones and telephone equipment had to be rented from the phone company, so technological development in many areas (faxes, answering systems, business telephony) came at a snail's pace. Also, if you think DSL rollout was slow and overpriced under the Baby Bells, just imagine what it would have been like under Ma Bell. All things have their price, and this was a high one.
Furthermore, AT&T had their problems as well. On at least one occasion, they had massive network failures due to a combination of-- guess what-- human error and software failure.
Hell, I live in NY state, and this is the first i've heard of it, online or the radio.
Having worked in private line provisioning and maintenance at Ma Bell I can say without equivocation that this is a direct result of the breakup and not really 'human error'.
I've seen the exact same thing being done at all the locals and the long distance companies.
Manpower is being drawn down, redundancies eliminated, and a talent and brain drain that causes errors like this.
The reason is always given that automation is allowing the company to maximize the remaining workforce and competition makes is neccesary. BS.
Best Practices are -gone- everything is driven by sales and bean counters. Engineers, Technicians and Managers who complain are moved, removed or eliminated from the loop because facts are not going to be allowed to get in the way.
It used to take weeks to get a misdirected line corrected in some instances.
The fault was blamed on too many layers, and union incompetence.
Now with all the improvements brought about by divestiture and competition it is a near imposibility unless it affects a major source of income or government.
This type of error was prevented by human redundancy and a workforce able to put the breaks on before the damage was done because they could stand on the strength of regulations and the union and tell the idiot boss in charge that things were wrong.
Get used to it, rapid reorder will be the order of the day.
I worked 911 for 14 years........here in the midwest, we have tornados around this time of year. The warning sirens usually go off to warn the public. Ok, I'm sitting at my console, waiting for people to call about wind damage etc......and the line rings....911, do you have an emergency?....yeah, can you tell me what the sirens are going off for?.........it's a shame that the line is recorded....there have been times, when we get idiot calls like that, that you really really want to say....well, I'm sorry to tell you this, but the russians have changed their minds, the bombs will be here in 10 minutes. And then, hang up on them LOL.... Heck, we've had calls from people that ask us where Bass Pro Shops is.....(Springfield Missouri)
As far as the software is concerned, I'm glad it's getting fixed. Sounds like your typical permissions problem to me. Some guy out in the field shouldn't be able to redirect the phone number for 911. Just like some e-mail attachment I run shouldn't be able to modify HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\Run.
Seriously, this is something that should be top priority in all kinds of software. Even if the person at the keyboard should be able to modify stuff, doesn't mean they want to. And by default, they should not have the ability to modify certain data.
Let's learn from this and crack down on insecure code. PLEASE!!
</troll> please.
And, ummm, stop reading Phrack from 1985.
First off, these days, most lines are served off 'digital loop carriers (DLCs),' which take the analog lines from your home, and multiplex it onto high-capacity lines (often running over fiber-optic SONET loops) back to the central office.
Even if your 220 volts made it back to the DLC (which is fairly unlikely, considering 220 VAC at any dangerous ampreage will probably overheat and melt the copper, anyway), the worst you'd do would be to burn out the service area the DLC is handling.
And even if your unlikely scenerio of getting 220 VAC back to a central office, and through the fuses, and the main distribution frame, and even if you hit the switch, you wouldn't affect anything more than that local exchange. Central offices aren't "daisey chained" down copper lines.
(and yes, I do work for a telco)
Chris -- http://www.bitter.net/
It may have been someone who recently moved to the area and didn't know about the tornado warning system. I'm old enough that when I hear a test of civil defense sirens, the first thing that comes to mind is "Oh shit. We're going to be nuked by the Russians."
Mea navis aericumbens anguillis abundat
Plan 9 was an utter failure. Just three zombies? What were the aliens thinking?
The ______ Agenda
===--===
Together, we will drive the rats from the tundra.
People seem to think that the 911 system is directly connected to the systems that run the telephone network. This really isn't so.
Databases at telcos contain what they think is accurate data about how the telephone switches are configured. Telcos may occassionaly audit these two systems to see how well they match up. Then again, they may not do such audits. In modern soft switches, the DB controls the switch, so they do match exactly. But most of the world (including the U.S.) still use legacy telephone switches that are not well integrated with the customer DBs.
Every so often the telcos query their DBs to create 911 update reports. Those reports are passed on to whoever maintains the 911 systems. Then the 911 systems are updated. Maybe.
Overtime, inconsistencies between the telco's system and the 911 system build up. Every so often (once a year or so, maybe) the 911 system gets purged and reloaded from the telco's system. Between reloads, it is not uncommon for a police department to call the 911 system maintainer or the phone company or both (often the 911 maintainer and the telco are completely unrelated entities) to let them know that there were 911 calls last night from such and such phone numbers which had missing or erroneous address info. Steps may or may not be taken to manually correct the info for those individual phone numbers.
You might be surprised to see the percentage of 911 calls that come in with bogus subscriber data.
911 has been a mess for many years and it hasn't been a secret. Eventually, some homeland security committee is going to pass a lot of legislation to address this. The legislation will cost a lot of money, and impose silly requirements. It will likely be drafted by the clowns who have created the bureaucratic 911 system of today. That will only make things worse.
A handful of good IT folks could clean things up very quickly. But that will never happen.