Slashdot Mirror
Verizon's NYC 911 System Shutdown
Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
← Back to Stories (view on slashdot.org)
well the calls were routed to 311, so the calls got answered, just maybe not as quickly. Yes this sucks, but the calls didn't go nowhere.
Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.
The emergency system broke down about 7:20 p.m. after a Verizon engineer who was making service changes to a bank's telephone numbers in Brooklyn inadvertently included numbers that are used to carry 911 calls, city and telephone company officials said. The numbers were close in sequence, the officials said.
The 911 calls then ended up being rerouted to the bank's phone system, and callers heard a busy signal. City and Verizon officials said that while the backup system in place for 911 was functioning properly, it failed to pick up the calls because it was designed to catch a technical error, not a human error that would be interpreted as simply a change of instruction.
Daniel Diaz Zapata, a Verizon spokesman, said the telephone company would now require a second person to double-check any entry of data that could affect the 911 system, and said the company planned a thorough review of its procedures that would be documented in a report to the city within a few days.
"We determined that a human error resulted in the accidental rerouting of phone calls during a procedure to upgrade service for a corporate client," Mr. Zapata said. "We have immediately altered our processes to ensure this type of situation does not reoccur. We have assured the city that we took immediate steps to make sure this doesn't happen again."
Citing privacy concerns, Mr. Zapata declined to identify the Verizon engineer, except to say that he was a veteran of the company. Mr. Zapata said it was unlikely that disciplinary action would be taken against him.
Police and fire officials said yesterday that they had no reports of injuries during the 911 failure. Fire officials said that about 60 firefighters responded to a major fire, at 3301 Foster Avenue in Brooklyn, which was called in at 8:49 p.m. by someone using a fire alarm box on the street. There were no injuries in the fire.
Paul J. Browne, the Police Department's deputy commissioner for public information, said the department immediately adopted emergency procedures, like requiring e officers on patrol to turn on their flashing lights so people could find them easily and increasing staffing at precinct station houses to answer phone calls. But he said there was no reported increase in crime.
"This didn't present an opportunity for the criminally minded - like the blackout did - because probably most people were unaware that it was out of service," he said.
However, several City Council members expressed anger that the 911 system could have been so easily disabled, and called for creating a more effective backup procedure.
"It's an emergency wakeup call," said Councilman Peter F. Vallone Jr., the chairman of the Public Safety Committee, who plans to hold a hearing about the incident. "We don't have an adequate backup system for 911, which is more important than ever as we fight the war against terrorism."
Gino P. Menchini, the commissioner of the Department of Information Technology and Telecommunications, said city officials were working with Verizon to ensure that the emergency system's numbers were clearly identified, and that its software and equipment were protected from similar human errors.
But Mr. Menchini emphasized that the emergency system already had many built-in safeguards, such as the ability to route 911 calls through either of two central offices and their 911 answering centers. "The bottom line is, 911 works very well, and it's worked very well for a long time," Mr. Menchini said.
Several emergency services experts agreed yesterday with Mr. Menchini, saying that New York 911 system compared favorably with those in other large cities and that an error like the one made by Verizon could not necessaril
Jay | http://oldos.org
Non-reg link: here
People seem to think that the 911 system is directly connected to the systems that run the telephone network. This really isn't so.
Databases at telcos contain what they think is accurate data about how the telephone switches are configured. Telcos may occassionaly audit these two systems to see how well they match up. Then again, they may not do such audits. In modern soft switches, the DB controls the switch, so they do match exactly. But most of the world (including the U.S.) still use legacy telephone switches that are not well integrated with the customer DBs.
Every so often the telcos query their DBs to create 911 update reports. Those reports are passed on to whoever maintains the 911 systems. Then the 911 systems are updated. Maybe.
Overtime, inconsistencies between the telco's system and the 911 system build up. Every so often (once a year or so, maybe) the 911 system gets purged and reloaded from the telco's system. Between reloads, it is not uncommon for a police department to call the 911 system maintainer or the phone company or both (often the 911 maintainer and the telco are completely unrelated entities) to let them know that there were 911 calls last night from such and such phone numbers which had missing or erroneous address info. Steps may or may not be taken to manually correct the info for those individual phone numbers.
You might be surprised to see the percentage of 911 calls that come in with bogus subscriber data.
911 has been a mess for many years and it hasn't been a secret. Eventually, some homeland security committee is going to pass a lot of legislation to address this. The legislation will cost a lot of money, and impose silly requirements. It will likely be drafted by the clowns who have created the bureaucratic 911 system of today. That will only make things worse.
A handful of good IT folks could clean things up very quickly. But that will never happen.