Slashdot Mirror
Verizon's NYC 911 System Shutdown
Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
← Back to Stories (view on slashdot.org)
well the calls were routed to 311, so the calls got answered, just maybe not as quickly. Yes this sucks, but the calls didn't go nowhere.
For those who don't want to register:
username: slashdot2003
password: slashdot2003
---
Never criticize religion on Slashdot. You will be modded down for "Troll" no matter how factual it is.
Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.
The emergency system broke down about 7:20 p.m. after a Verizon engineer who was making service changes to a bank's telephone numbers in Brooklyn inadvertently included numbers that are used to carry 911 calls, city and telephone company officials said. The numbers were close in sequence, the officials said.
The 911 calls then ended up being rerouted to the bank's phone system, and callers heard a busy signal. City and Verizon officials said that while the backup system in place for 911 was functioning properly, it failed to pick up the calls because it was designed to catch a technical error, not a human error that would be interpreted as simply a change of instruction.
Daniel Diaz Zapata, a Verizon spokesman, said the telephone company would now require a second person to double-check any entry of data that could affect the 911 system, and said the company planned a thorough review of its procedures that would be documented in a report to the city within a few days.
"We determined that a human error resulted in the accidental rerouting of phone calls during a procedure to upgrade service for a corporate client," Mr. Zapata said. "We have immediately altered our processes to ensure this type of situation does not reoccur. We have assured the city that we took immediate steps to make sure this doesn't happen again."
Citing privacy concerns, Mr. Zapata declined to identify the Verizon engineer, except to say that he was a veteran of the company. Mr. Zapata said it was unlikely that disciplinary action would be taken against him.
Police and fire officials said yesterday that they had no reports of injuries during the 911 failure. Fire officials said that about 60 firefighters responded to a major fire, at 3301 Foster Avenue in Brooklyn, which was called in at 8:49 p.m. by someone using a fire alarm box on the street. There were no injuries in the fire.
Paul J. Browne, the Police Department's deputy commissioner for public information, said the department immediately adopted emergency procedures, like requiring e officers on patrol to turn on their flashing lights so people could find them easily and increasing staffing at precinct station houses to answer phone calls. But he said there was no reported increase in crime.
"This didn't present an opportunity for the criminally minded - like the blackout did - because probably most people were unaware that it was out of service," he said.
However, several City Council members expressed anger that the 911 system could have been so easily disabled, and called for creating a more effective backup procedure.
"It's an emergency wakeup call," said Councilman Peter F. Vallone Jr., the chairman of the Public Safety Committee, who plans to hold a hearing about the incident. "We don't have an adequate backup system for 911, which is more important than ever as we fight the war against terrorism."
Gino P. Menchini, the commissioner of the Department of Information Technology and Telecommunications, said city officials were working with Verizon to ensure that the emergency system's numbers were clearly identified, and that its software and equipment were protected from similar human errors.
But Mr. Menchini emphasized that the emergency system already had many built-in safeguards, such as the ability to route 911 calls through either of two central offices and their 911 answering centers. "The bottom line is, 911 works very well, and it's worked very well for a long time," Mr. Menchini said.
Several emergency services experts agreed yesterday with Mr. Menchini, saying that New York 911 system compared favorably with those in other large cities and that an error like the one made by Verizon could not necessaril
Jay | http://oldos.org
Non-reg link: here
I once built an HA cluster that had a role in the 911 system, at a major telco switch vendor. Apparently the fines for a vendor bungle in this are in the millions of dollars per minute of downtime. I dont know which switches these are, but such penalties could affect that vendor in fines and their bottom line. It looks like Verizon shot itself in the foot, but keep an eye out for a dip in share prices for some of the switch vendors in case the blame gets spread around.
The 911 system seems to have continued to work correctly, but the regular numbers that calls to 911 get directed to were redirected to a bank. So the issue is really that there aren't safeguards against the wrong phone numbers getting changed accidentally, and the phone lines used by the 911 system aren't immune.
Shouldn't the interface for the system prevent you from accidentally modifying similar but unrelated numbers when you're modifying a set of numbers?
is that there is a very finite number of customers for things like electricity and phone service. You need a very large customer base in order to be able to charge a reasonable amount.
When you allow competition those that attempt to compete are forced to either charge less than it costs to supply the service or charge more. If they charge less but can't get the customer base they go out of business. If they charge more people tend not to switch. And if you don't charge enough less, nobody cares enough to switch.
Cox saves us all of a buck or two over Qwest on phone service. We never bothered to switch until we switched to Cox for high speed internet.
And of course the only reason Cox had the money to implement phone service is because they're the monopoly on cable service.
In cases like this it's actually better for the government to force the monopoly to act in the best interest of the people than to allow competition which just gives people the false impression that it'll lead to cheaper prices and better service.
Competition in these cases are almost always forced to either cut corners to survive or charge more.
Ben
Work Safe Porn
311 is the non-emergency number for police/fire/medical services in some areas.
Visit the
Last I checked 511 will tell you your own phone number. It sounds silly, but it came in handy for me in college when they would often forget to tell us the phone numbers for our dorm rooms.
That number is handy, but I have to admit that the time I called the pizza place that I knew had caller id and asked them what my phone number was makes a lot better story!
NOTE: I just tried this on my home for and it worked. My phone company is CenturyTel.
> A combination of human error and software that
> didn't anticipate...
Several "human errors", one of them be excessive centralization.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
Not a desirable situation, but better then it not working at all..
The NYT has a habit of catching these things. It won't even let you register cypherpunks/cypherpunks. However, there is a work around -- get some sort of proxy, such as Proxomitron, and configure it so the referer is news.google.com. The same trick works with the WaPo's new registration service.
Les Miserables Volume 1 now up with my reading of
Yet at the same time, daytime Long Distance cost over a dollar a minute from NY to California. Phones and telephone equipment had to be rented from the phone company, so technological development in many areas (faxes, answering systems, business telephony) came at a snail's pace. Also, if you think DSL rollout was slow and overpriced under the Baby Bells, just imagine what it would have been like under Ma Bell. All things have their price, and this was a high one.
Furthermore, AT&T had their problems as well. On at least one occasion, they had massive network failures due to a combination of-- guess what-- human error and software failure.
If you think you can provide the story much faster and sumit it to slashdot earlier than other people. Please do so, or stop complaining.
I remember the "good ol' days" of Ma Bell's monopoly, before the big breakup. I'd take exception to the use of the phrase "*real* quality of service".
The only good thing I recall about those days was that most Western Electric phones were virtually bulletproof, the telco equivalent of an IBM Type M keyboard. But that's about it.
For starters, since the telco owned everything on the network, adding an extension phone was a violation of their terms of service, and they'd come down hard on you if they found one in use (and don't think they didn't check up on people; they did). The sound quality was vastly inferior to what we now have: long distance sounded like long distance, but even local calls could sometimes be rendered unintelligible by the monopoly's antiquated switching system. Service in rural areas pretty much sucked hind tit; even in the late '70s it was party lines or nothing in certain towns in upstate New York.
Even worse, Ma Bell's responsiveness to consumer complaints was a national joke. Remember Lily Tomlin's character Ernestine? One ringy-dingy... That was a caricature, of course, but one grounded in truth.
After divestiture, things really changed for the better overall. The relaxation of restrictions on what could be placed on the network meant a boom in devices like answering machines, fax machines, and modems. Had the old pre-1984 restrictions been in place, what do you think the effect would have been on BBSs and dial-up access to the Internet? Imagine having to pay extra in order to have a modem connected to your phone line. Sound quality improved largely due to technological advances, but had the monopoly still been in place, would there have been any incentive to upgrade the telco network?
Yes, AT&T had been on the cutting edge of computer science and electronic engineering for decades. But had the break-up not taken place, we'd still be using a phone system worthy of the movie Brazil.
k.
"In spite of everything, I still believe that people are really good at heart." - Anne Frank
Orleans/Essex Counties Vermont BY SETH POWERS And ROBIN SMITH, Staff Writers - A dump truck hit an aerial fiber-optic cable in Barton Tuesday morning, knocking out most local and all long-distance phone service for 18,000 people in Essex and Orleans counties. The dump truck was working on a new driveway off Route 5 near Crystal Lake and clipped the line at 8:30 a.m., said Beth Fastiggi, spokeswoman for Verizon. After the damage, residents were able to dial within their own exchange, but not to neighboring exchanges or long distance. "People in Island Pond could call people in Island Pond, but not out of their area," Fastiggi said. Those calling into Essex and Orleans counties found the phones gave a busy signal for hours, as workers attempted to fix the problem. Phone service was restored by 1:40 p.m., after phone crews spliced a new cable into the damaged line, Fastiggi said. The damaged line is a connector between seven northern Vermont towns and St. Johnsbury - and then to the rest of the world, Fastiggi said. For an Island Pond resident to dial out of Island Pond, the resident's call goes through the St. Johnsbury host station, and out to Barton or Boston, depending on where the individual wants to call. Other affected exchanges were in Albany, Barton, Derby and Derby Line, Morgan, Newport City, Norton, Orleans and Troy, according to Evelyn Bailey, executive director of the Vermont emergency 911 board. The 911 service worked. However, people in these exchange areas, with the exception of Derby, were unable to reach it, she said Tuesday afternoon. Derby callers were connected directly to the local state police barracks and its 911 dispatchers. They would never have known that they were not tied into the 911 network, she said. The damage affected some Internet and cellular phone service as well. Both rely on land lines, depending on the service, Bailey said. Bailey, as Vermont's emergency 911 contact, and the hospital were contacted by Verizon when the network went down. Vermont State Police and other emergency services are also alerted. North Country Hospital in Newport City had working e-mail and Internet service all morning, but was limited like everyone else to calls within the city exchange, said Larry Labor, acting chief executive officer. The hospital internally had no problems after the phone system failure, he said. Emergency connections with rescue and other crews are radio-based. Also, there were plenty of cell phones on hand to connect to other health services outside the area. These kinds of failures are not rare, Bailey said. "It's so common for someone to put a backhoe into the ground without calling Dig Safe." Anyone preparing to dig is required by law to call the Dig Safe number: 888-344-7233. With Tuesday's phone system failure in mind, Bailey has two recommendations for local residents: Along with 911, have the seven-digit number for your local police or fire department handy. And don't call 911 when there are problems just to see if it is working. "You'd be surprised how many people can't resist," Bailey said.
Bob is at home in the Northeast Kingdom.
People seem to think that the 911 system is directly connected to the systems that run the telephone network. This really isn't so.
Databases at telcos contain what they think is accurate data about how the telephone switches are configured. Telcos may occassionaly audit these two systems to see how well they match up. Then again, they may not do such audits. In modern soft switches, the DB controls the switch, so they do match exactly. But most of the world (including the U.S.) still use legacy telephone switches that are not well integrated with the customer DBs.
Every so often the telcos query their DBs to create 911 update reports. Those reports are passed on to whoever maintains the 911 systems. Then the 911 systems are updated. Maybe.
Overtime, inconsistencies between the telco's system and the 911 system build up. Every so often (once a year or so, maybe) the 911 system gets purged and reloaded from the telco's system. Between reloads, it is not uncommon for a police department to call the 911 system maintainer or the phone company or both (often the 911 maintainer and the telco are completely unrelated entities) to let them know that there were 911 calls last night from such and such phone numbers which had missing or erroneous address info. Steps may or may not be taken to manually correct the info for those individual phone numbers.
You might be surprised to see the percentage of 911 calls that come in with bogus subscriber data.
911 has been a mess for many years and it hasn't been a secret. Eventually, some homeland security committee is going to pass a lot of legislation to address this. The legislation will cost a lot of money, and impose silly requirements. It will likely be drafted by the clowns who have created the bureaucratic 911 system of today. That will only make things worse.
A handful of good IT folks could clean things up very quickly. But that will never happen.