Slashdot Mirror
Verizon's NYC 911 System Shutdown
Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
← Back to Stories (view on slashdot.org)
Hey I hadn't heard about this yet. I bet alot of other people havn't either.
And to think everyone was worrying about the terrorists, fer chrissakes.
we could, uh, do without the, uhmmmm, simpsons quotes. especially if they have very little to do with the story.
yeeeah, you'll be uh, getting that +5 funny regardless.
Considering Slashdot doesn't write their own articles and it is basically an aggregate of postings from other sites, I'm not sure what you're complaining about.
:P
Technically, any news on Slashdot is old news since it has been reported already.
So instead when you call 911 you have a random chance of the software working? Yes, lets put small understaffed local companies in charge of the emergency phone system.
Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.
Now imagine what a genuinely malevolent person could accomplish. Perhaps a single individual shouldn't be capable of disabling such a critical system.
Do you like German cars?
This type of error is a classic problem of the computer assuming that the human was right. We create machines to give us power, but use that power to cavalierly.
The idea of having a second person "double-check" is nice in theory, but I will wager that the second person will let errors through too. If the first person is careful, the second person is faced with a long list of matching, correct entries to check. The second person soon becomes fatigued and keeps hitting the "OK" button even when there is a discrepancy. Unless the second person is offered an outsized reward (and the first person is penalized by an even greater amount), its to easy to become apathetic or non-vigilant. (Also, the double-checking process assumes that the original set of command directives was correct).
The real solution is a meta system that logs any changes to the system (like a config change), monitors dependencies of that change, and cross-checks them during exceptions. When an exception occurs, such as a bunch of 911 busy signals, the system would trace through the code and config files and correlate the fact that the onset of busy-911 calls corresponded with the insertion of the erroneous numbers. The system would then either roll-back the changes that caused a fault or alert someone of the list of likely culprits.
Two wrongs don't make a right, but three lefts do.
Been reading the "blotto box" chapter of the Jolly Roger cookbook? :-)
Like most things in that book, they either don't work, or will probably cause you to kill yourself if you tried it.
The phone company equipment would obviously have surge detection/supression systems in place to isolate any box that had something like that done to it.
What'd be really funny is if a guy hooked one up and then went home and answered his phone...
If you ask anyone from IT who has to deal with Verizon, they'll all tell you that Verizon and incompetence go together.
In any case, New York is the suxx0rz because city services don't work.
The political types saying that they don't have a good enough backup 911 system failed to understand the root cause of this failure.
A Verizon tech who was re-routing a customer's numbers accidently made a numerical error that ended up re-routing lines that were meant to go to 911 to a bank. Therefore, the backup system never got a chance to kick in, people were being routed to a very poor selection for a primary destination.
The safety valve that I'm sure is being installed now is requiring a higher degree of password to change the routing instructions for the 911 lines... because this tech should not have been able to mess with them, and didn't mean to, he just typoed the numbers he was supposed to type in. He at least should have seen a "You're trying to reroute 911! Are you sure you want to do that? N" prompt.
As I read the article, it is obvious that NYC's system is fraught with deep flaws in its design and management. These include:
1. False redundancy: Although the NYC system has a backup central offices and call centers, it apparently routes all calls from the affected area through a single Verizon subsystem. Their system is fully redundant except where its not.
2. Organizational silos in a coupled system: The City claimed that its 911 system was fine because "an error like the one made by Verizon could not necessarily have been prevented because it was not a flaw in the 911 system itself." Yet the Verizon circuits, systems, and procedures are an integral part of the 911 system. The City (and Verizon) maintain a fiction that they are independent entities when, in fact, they are tightly coupled. This division of responsibility is fine for playing the CYA Blame Game, but does not create a robust system.
3. User Interface Flaws I don't know what kind of user interface that technician was using, but it obviously has some terrible flaws if it did not warn him of the implications of the data entries. I also suspect that he was manually retyping some numbers off a computer print-out when he should have had some mechanism to download a set of proofread, verified, double-checked entries.
I don't fault NYC or Verizon in particular, they are probably no worse that anyone else. I only get angry that these types of structural insecurities are probably more widespread than anyone realizes.
Two wrongs don't make a right, but three lefts do.
Eh? Did NYPD and NYFD need the 911 system to find out about the WTC strikes? Terrorism isn't about killing people, it's about getting publicity.
The councilman can rest assured that the terrorists will helpfully keep their activities high-profile enough that 911 notification will not be required.
[Sheesh, why does *everything* have to be about terrorism these days.]
Slashdot monitor for your Mozilla sidebar or Active Desktop.
Posting login info in /. doesn't work, because some asshole always goes and changes the password.
It's hard to be religious when certain people are never incinerated by bolts of lightning.
No, it's not "almost is bad", it's many times worse. Not being able to switch your phone number is an inconvenience. Not being able to get help when you're being robbed, your house is burning, or your spouse just had a heart attack is catastrophic.
It's hard to be religious when certain people are never incinerated by bolts of lightning.
As far as the software is concerned, I'm glad it's getting fixed. Sounds like your typical permissions problem to me. Some guy out in the field shouldn't be able to redirect the phone number for 911. Just like some e-mail attachment I run shouldn't be able to modify HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\Run.
Seriously, this is something that should be top priority in all kinds of software. Even if the person at the keyboard should be able to modify stuff, doesn't mean they want to. And by default, they should not have the ability to modify certain data.
Let's learn from this and crack down on insecure code. PLEASE!!
</troll> please.
And, ummm, stop reading Phrack from 1985.
First off, these days, most lines are served off 'digital loop carriers (DLCs),' which take the analog lines from your home, and multiplex it onto high-capacity lines (often running over fiber-optic SONET loops) back to the central office.
Even if your 220 volts made it back to the DLC (which is fairly unlikely, considering 220 VAC at any dangerous ampreage will probably overheat and melt the copper, anyway), the worst you'd do would be to burn out the service area the DLC is handling.
And even if your unlikely scenerio of getting 220 VAC back to a central office, and through the fuses, and the main distribution frame, and even if you hit the switch, you wouldn't affect anything more than that local exchange. Central offices aren't "daisey chained" down copper lines.
(and yes, I do work for a telco)
Chris -- http://www.bitter.net/
It may have been someone who recently moved to the area and didn't know about the tornado warning system. I'm old enough that when I hear a test of civil defense sirens, the first thing that comes to mind is "Oh shit. We're going to be nuked by the Russians."
Mea navis aericumbens anguillis abundat
God forbid that you ever have a problem with a data line (higher quality line in those times) for the tech at AT&T would simply say that it was your equipment causing the problem and that their line check verified it. It was a monumental lesson in frustration.
I ran a 10 state wide call center and the data processing branch of our company and dealing with AT&T was not easy even though we had nearly 100 lines with them. The customer service representives though at AT&T were outstanding, however they also had frustration in dealing with the repair group.
When MCI lines arrived we switched and in doing so saved half of the expense and most of the problems encountered and all of the headaches.
Competition lowered the price, improved quality, and made the company (especially the techs) more responsive.
Breaking up AT&T was the best thing that ever happened to improve the telephone systems overall.
And in the end, the love you take is equal to the love you make
This is half right.
There were too many layers at Ma Bell. But with all the "manpower reductions" and "elimination of redundancies", there are still too many layers.
A 911 correction today requires a local telco in Portland to send an update to Florida, so that it can be uploaded to Kansas, and retrieved by someone in Ohio for review, then batched on a system in Denver that will finally load the data into the 911 system in Portland.
All the people from Florida to Denver are non-techy bureaucrats fighting for job security. The drone doing the "check" in Ohio has never been to Portland, let alone the suburb in which this small telco operates. They don't know if Johnsons just moved or not. They just check that Broadway Avenue uses the correct abbreviation for Avenue, as approved by the Portland City Council, and that the street number is within approved bounds. If not, they inform Kansas that there is an error. Oh, and they probably _are_ union.
Meanwhile, the small telco in Portland can't convince anybody in Florida, Kansas, Ohio, or Denver that a new subdivision has gone up on Broadway, and the address is actually correct. Even if someone in Denver believes them, there is nothing they can do about it.
If they were allowed to interact directly with EMS personnel in their local area, they might straighten things out very easily.
But Ma Bell was not replaced by small friendly companies. It was replaced by more layers of unfriendly megacorps with less technical people in the middle.
I don't agree. An eventuality occurred which the original software designers hadn't thought of. This happens all the time in embedded systems - the world is not only more complicated than we think, it is more complicated than we can think.
This failure required two failures in a row - one human, one software. After this failure, both systems will be fixed, so this failure won't happen again on this system. Furthermore, it won't happen again on any system those involved in cleaning up work on - we learn by our mistakes. Unfortunately, because they appear to be going to keep the details secret, none of the rest of us will have chance to learn from their mistakes.
This is nothing to do with monopolies. Any man-made system will hav failures. But it does have a bit to do with open/closed: if the system were open source, those interested would see the fixes even if they dsidn't get a complete report (which they paobaly would). "Many eyballs" would have made this failure less likely (though not impossible). A case for wanting open source in safety criticasl software: maximise the clooegiate understanding.
Consciousness is an illusion caused by an excess of self consciousness.
Remember, everytime a stupid person dies the average intelligence of the world goes up.
Remember this next time you have a cardiac arrest and the only person with you is such a "retard" (if you have enough time, that is). He will live, you will die.
Under stress situations everyone could have this problem, your brain may just stop working as it is overhauled by all the shocking information around you.
Seriously, parent was referring to fire drills *in schools*. I don't think it hurts to be clear in these things - it's entirely possible that a young child would find 9-1-1 clearer than 9-11. We're not born with these skills, you know.