Slashdot Mirror
Verizon's NYC 911 System Shutdown
Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
← Back to Stories (view on slashdot.org)
Do you think someone called 911 to report the emergency outage?
well the calls were routed to 311, so the calls got answered, just maybe not as quickly. Yes this sucks, but the calls didn't go nowhere.
I think this is a shining demonstration of why monopolies shouldn't be allowed to run the phone system. Speaking of monopolies, I wonder what the 'software' concerned was?
And to think everyone was worrying about the terrorists, fer chrissakes.
Considering Slashdot doesn't write their own articles and it is basically an aggregate of postings from other sites, I'm not sure what you're complaining about.
:P
Technically, any news on Slashdot is old news since it has been reported already.
Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.
The emergency system broke down about 7:20 p.m. after a Verizon engineer who was making service changes to a bank's telephone numbers in Brooklyn inadvertently included numbers that are used to carry 911 calls, city and telephone company officials said. The numbers were close in sequence, the officials said.
The 911 calls then ended up being rerouted to the bank's phone system, and callers heard a busy signal. City and Verizon officials said that while the backup system in place for 911 was functioning properly, it failed to pick up the calls because it was designed to catch a technical error, not a human error that would be interpreted as simply a change of instruction.
Daniel Diaz Zapata, a Verizon spokesman, said the telephone company would now require a second person to double-check any entry of data that could affect the 911 system, and said the company planned a thorough review of its procedures that would be documented in a report to the city within a few days.
"We determined that a human error resulted in the accidental rerouting of phone calls during a procedure to upgrade service for a corporate client," Mr. Zapata said. "We have immediately altered our processes to ensure this type of situation does not reoccur. We have assured the city that we took immediate steps to make sure this doesn't happen again."
Citing privacy concerns, Mr. Zapata declined to identify the Verizon engineer, except to say that he was a veteran of the company. Mr. Zapata said it was unlikely that disciplinary action would be taken against him.
Police and fire officials said yesterday that they had no reports of injuries during the 911 failure. Fire officials said that about 60 firefighters responded to a major fire, at 3301 Foster Avenue in Brooklyn, which was called in at 8:49 p.m. by someone using a fire alarm box on the street. There were no injuries in the fire.
Paul J. Browne, the Police Department's deputy commissioner for public information, said the department immediately adopted emergency procedures, like requiring e officers on patrol to turn on their flashing lights so people could find them easily and increasing staffing at precinct station houses to answer phone calls. But he said there was no reported increase in crime.
"This didn't present an opportunity for the criminally minded - like the blackout did - because probably most people were unaware that it was out of service," he said.
However, several City Council members expressed anger that the 911 system could have been so easily disabled, and called for creating a more effective backup procedure.
"It's an emergency wakeup call," said Councilman Peter F. Vallone Jr., the chairman of the Public Safety Committee, who plans to hold a hearing about the incident. "We don't have an adequate backup system for 911, which is more important than ever as we fight the war against terrorism."
Gino P. Menchini, the commissioner of the Department of Information Technology and Telecommunications, said city officials were working with Verizon to ensure that the emergency system's numbers were clearly identified, and that its software and equipment were protected from similar human errors.
But Mr. Menchini emphasized that the emergency system already had many built-in safeguards, such as the ability to route 911 calls through either of two central offices and their 911 answering centers. "The bottom line is, 911 works very well, and it's worked very well for a long time," Mr. Menchini said.
Several emergency services experts agreed yesterday with Mr. Menchini, saying that New York 911 system compared favorably with those in other large cities and that an error like the one made by Verizon could not necessaril
Jay | http://oldos.org
Non-reg link: here
As I read the article, it is obvious that NYC's system is fraught with deep flaws in its design and management. These include:
1. False redundancy: Although the NYC system has a backup central offices and call centers, it apparently routes all calls from the affected area through a single Verizon subsystem. Their system is fully redundant except where its not.
2. Organizational silos in a coupled system: The City claimed that its 911 system was fine because "an error like the one made by Verizon could not necessarily have been prevented because it was not a flaw in the 911 system itself." Yet the Verizon circuits, systems, and procedures are an integral part of the 911 system. The City (and Verizon) maintain a fiction that they are independent entities when, in fact, they are tightly coupled. This division of responsibility is fine for playing the CYA Blame Game, but does not create a robust system.
3. User Interface Flaws I don't know what kind of user interface that technician was using, but it obviously has some terrible flaws if it did not warn him of the implications of the data entries. I also suspect that he was manually retyping some numbers off a computer print-out when he should have had some mechanism to download a set of proofread, verified, double-checked entries.
I don't fault NYC or Verizon in particular, they are probably no worse that anyone else. I only get angry that these types of structural insecurities are probably more widespread than anyone realizes.
Two wrongs don't make a right, but three lefts do.
Posting login info in /. doesn't work, because some asshole always goes and changes the password.
It's hard to be religious when certain people are never incinerated by bolts of lightning.
My wife,daughter and I happened to be wandering around Brooklyn during the outage. We noticed an increase in the number of uniformed officers on the street. This made us slightly nervous in a 9/11 (not a 911) sort of way. It wasn't until we read Saturday's NYT that we realized what was up. To increase visibility during the outage, NYPD had ordered all plainclothes officers to put on uniforms.
As we strolled through some of the city's busiest commercial sectors, everybody as oblivious as we were.
I worked 911 for 14 years........here in the midwest, we have tornados around this time of year. The warning sirens usually go off to warn the public. Ok, I'm sitting at my console, waiting for people to call about wind damage etc......and the line rings....911, do you have an emergency?....yeah, can you tell me what the sirens are going off for?.........it's a shame that the line is recorded....there have been times, when we get idiot calls like that, that you really really want to say....well, I'm sorry to tell you this, but the russians have changed their minds, the bombs will be here in 10 minutes. And then, hang up on them LOL.... Heck, we've had calls from people that ask us where Bass Pro Shops is.....(Springfield Missouri)
People seem to think that the 911 system is directly connected to the systems that run the telephone network. This really isn't so.
Databases at telcos contain what they think is accurate data about how the telephone switches are configured. Telcos may occassionaly audit these two systems to see how well they match up. Then again, they may not do such audits. In modern soft switches, the DB controls the switch, so they do match exactly. But most of the world (including the U.S.) still use legacy telephone switches that are not well integrated with the customer DBs.
Every so often the telcos query their DBs to create 911 update reports. Those reports are passed on to whoever maintains the 911 systems. Then the 911 systems are updated. Maybe.
Overtime, inconsistencies between the telco's system and the 911 system build up. Every so often (once a year or so, maybe) the 911 system gets purged and reloaded from the telco's system. Between reloads, it is not uncommon for a police department to call the 911 system maintainer or the phone company or both (often the 911 maintainer and the telco are completely unrelated entities) to let them know that there were 911 calls last night from such and such phone numbers which had missing or erroneous address info. Steps may or may not be taken to manually correct the info for those individual phone numbers.
You might be surprised to see the percentage of 911 calls that come in with bogus subscriber data.
911 has been a mess for many years and it hasn't been a secret. Eventually, some homeland security committee is going to pass a lot of legislation to address this. The legislation will cost a lot of money, and impose silly requirements. It will likely be drafted by the clowns who have created the bureaucratic 911 system of today. That will only make things worse.
A handful of good IT folks could clean things up very quickly. But that will never happen.