Slashdot Mirror
Verizon's NYC 911 System Shutdown
Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
← Back to Stories (view on slashdot.org)
well the calls were routed to 311, so the calls got answered, just maybe not as quickly. Yes this sucks, but the calls didn't go nowhere.
Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.
The emergency system broke down about 7:20 p.m. after a Verizon engineer who was making service changes to a bank's telephone numbers in Brooklyn inadvertently included numbers that are used to carry 911 calls, city and telephone company officials said. The numbers were close in sequence, the officials said.
The 911 calls then ended up being rerouted to the bank's phone system, and callers heard a busy signal. City and Verizon officials said that while the backup system in place for 911 was functioning properly, it failed to pick up the calls because it was designed to catch a technical error, not a human error that would be interpreted as simply a change of instruction.
Daniel Diaz Zapata, a Verizon spokesman, said the telephone company would now require a second person to double-check any entry of data that could affect the 911 system, and said the company planned a thorough review of its procedures that would be documented in a report to the city within a few days.
"We determined that a human error resulted in the accidental rerouting of phone calls during a procedure to upgrade service for a corporate client," Mr. Zapata said. "We have immediately altered our processes to ensure this type of situation does not reoccur. We have assured the city that we took immediate steps to make sure this doesn't happen again."
Citing privacy concerns, Mr. Zapata declined to identify the Verizon engineer, except to say that he was a veteran of the company. Mr. Zapata said it was unlikely that disciplinary action would be taken against him.
Police and fire officials said yesterday that they had no reports of injuries during the 911 failure. Fire officials said that about 60 firefighters responded to a major fire, at 3301 Foster Avenue in Brooklyn, which was called in at 8:49 p.m. by someone using a fire alarm box on the street. There were no injuries in the fire.
Paul J. Browne, the Police Department's deputy commissioner for public information, said the department immediately adopted emergency procedures, like requiring e officers on patrol to turn on their flashing lights so people could find them easily and increasing staffing at precinct station houses to answer phone calls. But he said there was no reported increase in crime.
"This didn't present an opportunity for the criminally minded - like the blackout did - because probably most people were unaware that it was out of service," he said.
However, several City Council members expressed anger that the 911 system could have been so easily disabled, and called for creating a more effective backup procedure.
"It's an emergency wakeup call," said Councilman Peter F. Vallone Jr., the chairman of the Public Safety Committee, who plans to hold a hearing about the incident. "We don't have an adequate backup system for 911, which is more important than ever as we fight the war against terrorism."
Gino P. Menchini, the commissioner of the Department of Information Technology and Telecommunications, said city officials were working with Verizon to ensure that the emergency system's numbers were clearly identified, and that its software and equipment were protected from similar human errors.
But Mr. Menchini emphasized that the emergency system already had many built-in safeguards, such as the ability to route 911 calls through either of two central offices and their 911 answering centers. "The bottom line is, 911 works very well, and it's worked very well for a long time," Mr. Menchini said.
Several emergency services experts agreed yesterday with Mr. Menchini, saying that New York 911 system compared favorably with those in other large cities and that an error like the one made by Verizon could not necessaril
Jay | http://oldos.org
Non-reg link: here
I once built an HA cluster that had a role in the 911 system, at a major telco switch vendor. Apparently the fines for a vendor bungle in this are in the millions of dollars per minute of downtime. I dont know which switches these are, but such penalties could affect that vendor in fines and their bottom line. It looks like Verizon shot itself in the foot, but keep an eye out for a dip in share prices for some of the switch vendors in case the blame gets spread around.
is that there is a very finite number of customers for things like electricity and phone service. You need a very large customer base in order to be able to charge a reasonable amount.
When you allow competition those that attempt to compete are forced to either charge less than it costs to supply the service or charge more. If they charge less but can't get the customer base they go out of business. If they charge more people tend not to switch. And if you don't charge enough less, nobody cares enough to switch.
Cox saves us all of a buck or two over Qwest on phone service. We never bothered to switch until we switched to Cox for high speed internet.
And of course the only reason Cox had the money to implement phone service is because they're the monopoly on cable service.
In cases like this it's actually better for the government to force the monopoly to act in the best interest of the people than to allow competition which just gives people the false impression that it'll lead to cheaper prices and better service.
Competition in these cases are almost always forced to either cut corners to survive or charge more.
Ben
Work Safe Porn
311 is the non-emergency number for police/fire/medical services in some areas.
Visit the
Last I checked 511 will tell you your own phone number. It sounds silly, but it came in handy for me in college when they would often forget to tell us the phone numbers for our dorm rooms.
That number is handy, but I have to admit that the time I called the pizza place that I knew had caller id and asked them what my phone number was makes a lot better story!
NOTE: I just tried this on my home for and it worked. My phone company is CenturyTel.
The NYT has a habit of catching these things. It won't even let you register cypherpunks/cypherpunks. However, there is a work around -- get some sort of proxy, such as Proxomitron, and configure it so the referer is news.google.com. The same trick works with the WaPo's new registration service.
Les Miserables Volume 1 now up with my reading of
Yet at the same time, daytime Long Distance cost over a dollar a minute from NY to California. Phones and telephone equipment had to be rented from the phone company, so technological development in many areas (faxes, answering systems, business telephony) came at a snail's pace. Also, if you think DSL rollout was slow and overpriced under the Baby Bells, just imagine what it would have been like under Ma Bell. All things have their price, and this was a high one.
Furthermore, AT&T had their problems as well. On at least one occasion, they had massive network failures due to a combination of-- guess what-- human error and software failure.
People seem to think that the 911 system is directly connected to the systems that run the telephone network. This really isn't so.
Databases at telcos contain what they think is accurate data about how the telephone switches are configured. Telcos may occassionaly audit these two systems to see how well they match up. Then again, they may not do such audits. In modern soft switches, the DB controls the switch, so they do match exactly. But most of the world (including the U.S.) still use legacy telephone switches that are not well integrated with the customer DBs.
Every so often the telcos query their DBs to create 911 update reports. Those reports are passed on to whoever maintains the 911 systems. Then the 911 systems are updated. Maybe.
Overtime, inconsistencies between the telco's system and the 911 system build up. Every so often (once a year or so, maybe) the 911 system gets purged and reloaded from the telco's system. Between reloads, it is not uncommon for a police department to call the 911 system maintainer or the phone company or both (often the 911 maintainer and the telco are completely unrelated entities) to let them know that there were 911 calls last night from such and such phone numbers which had missing or erroneous address info. Steps may or may not be taken to manually correct the info for those individual phone numbers.
You might be surprised to see the percentage of 911 calls that come in with bogus subscriber data.
911 has been a mess for many years and it hasn't been a secret. Eventually, some homeland security committee is going to pass a lot of legislation to address this. The legislation will cost a lot of money, and impose silly requirements. It will likely be drafted by the clowns who have created the bureaucratic 911 system of today. That will only make things worse.
A handful of good IT folks could clean things up very quickly. But that will never happen.