Slashdot Mirror
Verizon's NYC 911 System Shutdown
Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
← Back to Stories (view on slashdot.org)
And to think everyone was worrying about the terrorists, fer chrissakes.
Considering Slashdot doesn't write their own articles and it is basically an aggregate of postings from other sites, I'm not sure what you're complaining about.
:P
Technically, any news on Slashdot is old news since it has been reported already.
So instead when you call 911 you have a random chance of the software working? Yes, lets put small understaffed local companies in charge of the emergency phone system.
Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.
Now imagine what a genuinely malevolent person could accomplish. Perhaps a single individual shouldn't be capable of disabling such a critical system.
Do you like German cars?
This type of error is a classic problem of the computer assuming that the human was right. We create machines to give us power, but use that power to cavalierly.
The idea of having a second person "double-check" is nice in theory, but I will wager that the second person will let errors through too. If the first person is careful, the second person is faced with a long list of matching, correct entries to check. The second person soon becomes fatigued and keeps hitting the "OK" button even when there is a discrepancy. Unless the second person is offered an outsized reward (and the first person is penalized by an even greater amount), its to easy to become apathetic or non-vigilant. (Also, the double-checking process assumes that the original set of command directives was correct).
The real solution is a meta system that logs any changes to the system (like a config change), monitors dependencies of that change, and cross-checks them during exceptions. When an exception occurs, such as a bunch of 911 busy signals, the system would trace through the code and config files and correlate the fact that the onset of busy-911 calls corresponded with the insertion of the erroneous numbers. The system would then either roll-back the changes that caused a fault or alert someone of the list of likely culprits.
Two wrongs don't make a right, but three lefts do.
The political types saying that they don't have a good enough backup 911 system failed to understand the root cause of this failure.
A Verizon tech who was re-routing a customer's numbers accidently made a numerical error that ended up re-routing lines that were meant to go to 911 to a bank. Therefore, the backup system never got a chance to kick in, people were being routed to a very poor selection for a primary destination.
The safety valve that I'm sure is being installed now is requiring a higher degree of password to change the routing instructions for the 911 lines... because this tech should not have been able to mess with them, and didn't mean to, he just typoed the numbers he was supposed to type in. He at least should have seen a "You're trying to reroute 911! Are you sure you want to do that? N" prompt.
As I read the article, it is obvious that NYC's system is fraught with deep flaws in its design and management. These include:
1. False redundancy: Although the NYC system has a backup central offices and call centers, it apparently routes all calls from the affected area through a single Verizon subsystem. Their system is fully redundant except where its not.
2. Organizational silos in a coupled system: The City claimed that its 911 system was fine because "an error like the one made by Verizon could not necessarily have been prevented because it was not a flaw in the 911 system itself." Yet the Verizon circuits, systems, and procedures are an integral part of the 911 system. The City (and Verizon) maintain a fiction that they are independent entities when, in fact, they are tightly coupled. This division of responsibility is fine for playing the CYA Blame Game, but does not create a robust system.
3. User Interface Flaws I don't know what kind of user interface that technician was using, but it obviously has some terrible flaws if it did not warn him of the implications of the data entries. I also suspect that he was manually retyping some numbers off a computer print-out when he should have had some mechanism to download a set of proofread, verified, double-checked entries.
I don't fault NYC or Verizon in particular, they are probably no worse that anyone else. I only get angry that these types of structural insecurities are probably more widespread than anyone realizes.
Two wrongs don't make a right, but three lefts do.
Eh? Did NYPD and NYFD need the 911 system to find out about the WTC strikes? Terrorism isn't about killing people, it's about getting publicity.
The councilman can rest assured that the terrorists will helpfully keep their activities high-profile enough that 911 notification will not be required.
[Sheesh, why does *everything* have to be about terrorism these days.]
Slashdot monitor for your Mozilla sidebar or Active Desktop.
Posting login info in /. doesn't work, because some asshole always goes and changes the password.
It's hard to be religious when certain people are never incinerated by bolts of lightning.
As far as the software is concerned, I'm glad it's getting fixed. Sounds like your typical permissions problem to me. Some guy out in the field shouldn't be able to redirect the phone number for 911. Just like some e-mail attachment I run shouldn't be able to modify HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\Run.
Seriously, this is something that should be top priority in all kinds of software. Even if the person at the keyboard should be able to modify stuff, doesn't mean they want to. And by default, they should not have the ability to modify certain data.
Let's learn from this and crack down on insecure code. PLEASE!!
</troll> please.
And, ummm, stop reading Phrack from 1985.
First off, these days, most lines are served off 'digital loop carriers (DLCs),' which take the analog lines from your home, and multiplex it onto high-capacity lines (often running over fiber-optic SONET loops) back to the central office.
Even if your 220 volts made it back to the DLC (which is fairly unlikely, considering 220 VAC at any dangerous ampreage will probably overheat and melt the copper, anyway), the worst you'd do would be to burn out the service area the DLC is handling.
And even if your unlikely scenerio of getting 220 VAC back to a central office, and through the fuses, and the main distribution frame, and even if you hit the switch, you wouldn't affect anything more than that local exchange. Central offices aren't "daisey chained" down copper lines.
(and yes, I do work for a telco)
Chris -- http://www.bitter.net/
It may have been someone who recently moved to the area and didn't know about the tornado warning system. I'm old enough that when I hear a test of civil defense sirens, the first thing that comes to mind is "Oh shit. We're going to be nuked by the Russians."
Mea navis aericumbens anguillis abundat
Remember, everytime a stupid person dies the average intelligence of the world goes up.