Slashdot Mirror
Vulnerability of Telco Switching Equipment
call -151 writes: "Interesting New York times article about the Sept 11th attacks' effect
on the Verizon switches in lower Manhattan. Turns out there
was a problem in that much of the network switching was in one
building and it has taken a while to restore service. Sounds like there
is lots of pondering about the vulnerability of the network,
even when it is distributed across many physical locations.
Of course the attacks are making lots of people rethink their
vulnerabilities, but the estimate is for five years' work before there
could be redundant paths for the lines into their switches in
the one building, with no plans to spend the money to do it.
Maybe someone should send them a few hundred thousand 'self-install'
kits like they do with their DSL service ..."
the problem is when you have a small metro area that is very dense and a high concumer of telco services. Even if you had redundant services, it setill makes economic sense (from the service point of view) to locate both (say) switches in the same area therefore, it would only somewhat help with an attack such as this.
Would this problem be easier to solve with a large wireless network? Considering the coverage of antennas these days, we could have some major overlappage for a fraction of the comparable cost.
Did you just grab my ass?
... Just because I have havinging to rego for the NYTimes site.
Attacks Expose Telephone's Soft Underbelly
By SIMON ROMERO
oseph Pennell, the prolific illustrator who often depicted the cityscape of Lower Manhattan in his prints, called the New York Telephone Building "the most impressive modern building in the world" when it was completed in 1926.
How antiquated it now seems.
The 32-story structure at 140 West Street, one of the city's first Art Deco skyscrapers, is now owned by New York Telephone's descendant, Verizon Communications (news/quote). And the heavy damage the building sustained on Sept. 11 underscores the vulnerability of communications networks operated by Verizon and other telephone companies -- sprawling systems that rely heavily on critical hubs.
In the days after the Sept. 11 attacks, it became commonplace to comment on how well the Internet performed because it was designed to route traffic around damage. But the telephone network, including the dedicated data lines that are used by big corporations, financial institutions and others, does not have the Internet's self-detouring abilities.
When they work, the telephone network's voice and data lines can be superior in quality and carrying capacity to the Internet. Yet when the telephone network is damaged, it cannot heal itself.
And while Verizon has worked almost around the clock the last month to restore operations at 140 West Street and service to its customers, the company has indicated that significantly reducing the building's network vulnerabilities would require more time or money than Verizon is willing to expend.
Domingo Mones/Verizon
Falling steel girders pierced the exterior of 140 West Street.
The Security: Rivals Worry About Access as Verizon Seeks Buffer (October 12, 2001)
Get Stock Quotes
Look Up Symbols
Portfolio | Company Research
U.S. Markets | Int. Markets
Mutual Funds | Bank Rates
Commodities & Currencies
Verizon's building was near the north tower of the World Trade Center and next door to 7 World Trade Center, which collapsed several hours after the attacks. Falling rubble and steel girders tore into 140 West Street, which housed one of the nation's busiest telephone central office switching stations. When fully operable, it serves a customer base comparable in number with all the telephone lines in a city the size of Cincinnati.
After electric power for the building was interrupted, service was temporarily disrupted for more than 300,000 telephone lines and 3.6 million high-capacity data circuits, many serving the New York Stock Exchange, large financial institutions and other companies in lower Manhattan. A gaping hole was torn in a seventh-floor exterior wall, exposing and damaging huge communications switches dedicated to the information needs of the banking company J. P. Morgan Chase.
In the last month, Verizon has labored to restore service or provide new service for customers that have moved to other parts of the city or to New Jersey. Virtually all of the fiber optic lines and copper strands that had wound their way under the streets and sidewalks and into 140 West Street are being replaced. Some circuits have been rerouted to other Verizon central offices in Lower Manhattan.
"The ideas we previously had about diversifying our networks have become much more important," Lawrence T. Babbio Jr., Verizon's vice chairman, said in an interview last week as he led a small group of journalists on a tour of 140 West Street.
Until last month, the most obvious reasons for network disruptions were natural disasters like hurricanes or floods. Now, though, Verizon and other telephone companies must worry about the possibility of physical attacks on their installations. Mr. Babbio warned last week that significant harm could be done to the nation's communications system if terrorists destroyed the 50 or 100 most important central offices.
Verizon, which is the dominant telephone company on the Eastern seaboard and operates in 30 states overall, is seeking to increase security at its central offices, where it is required by federal law to lease network access to its competitors. After Mr. Babbio issued his warning last week, competitors said they would resist tighter security measures if it made it more difficult for them to conduct operations within Verizon's central offices.
Beyond physically shielding their switching centers, phone companies can protect their communications networks from direct attacks or peripheral damage from nearby attacks by routing voice and data traffic to other parts of their own networks or those of other companies.
But Mr. Babbio said that it would take Verizon five years to build alternate pathways for all the telephone lines that wind their way into and out of the New York Telephone building. And Verizon has no plans to do so.
The reason may be a simple cost- benefit analysis. Despite its primacy to Lower Manhattan's communications network, the central office at 140 West Street accounted for less than 1 percent of the traffic on Verizon's nationwide network.
"So much of the activity on networks takes place at dispersed locations," said Roy A. Maxion, a system scientist at Carnegie Mellon University. "But the fact remains that we're vulnerable even after putting redundancy systems in place due to the physical nature of connecting to our networks. The issue should be what level of risk you're willing to live with."
Assuming they are willing to spend the money, business customers can achieve redundancy, or surplus and backup capacity, by running cables to several different central offices or, in some cases, by using several different communications carriers. Several of Verizon's competitors, in fact, have benefited from the disruptions by signing up new customers in Lower Manhattan.
"Identifying potential failures in networks is not easy," said Joe Flach, vice president of the Eagle Rock Alliance, a consulting company that provides advice on disaster planning. "The most important thing to avoid is putting all of your eggs in one basket."
Only after Sept. 11 did executives from the financial services industry in Lower Manhattan come to realize just how many of its eggs were in that one 75-year-old building.
Mr. Babbio recalled having to explain the situation at a meeting in Midtown Manhattan on Wednesday, Sept. 12, at the Park Avenue offices of the investment bank Bear, Stearns. Executives and government officials present included Richard A. Grasso, chairman of the New York Stock Exchange; Harvey L. Pitt, chairman of the Securities and Exchange Commission; Richard S. Fuld, chief executive of Lehman Brothers (news/quote); John A. Thain, a president of Goldman Sachs (news/quote); and Peter R. Fisher, under secretary for domestic finance at the Treasury Department.
The group was not happy when Mr. Babbio said how long it might take to restore basic service. Mr. Grasso had been hoping to reopen the stock exchange on Thursday or Friday. The following Monday now seemed ambitious.
"It was not an easy meeting," recalled Mr. Babbio, who spoke with the group immediately after visiting the disaster site, where his clothes had picked up the odor of smoke and ash. "I smelled awful after coming back from downtown. No one wanted to sit next to me."
So they are saying that if you take out a large telephone networks Central Office, people connected to this office will suffer lost connections. Infact some long distance connectivity will suffer as well.
Why does this suprise anyone. Hmmm let me see, if you take out your ISP, all of the sudden you will loose connectivity to the internet unless you pay A LOT of money to have a second line put in. Even then the chance that both of those lines run through some common area is pretty high.
Things are easy to engineer with fully redundancy, what isn't easy is to do it cheaply enough that people will still be willing to pay for it.
This makes perfect sense: the Internet did well because it relies on smart endpoints (computers) and unintelligent routes. The best routing, then, is equal speed routes from and to every endpoint and we see something approaching this with multiple routes connecting small groups of hosts.
The phone company relies on dumb endpoints (phones) and a smart system in the middle. The best (simple) routing solution would be every phone connected by a line to a central switching station. In an urban area, this is exactly what we see- one or two central switching stations or point of failure.
This really shouldn't be any surprise at all.
As a side note, this is also why growth and development has been much faster than on the phone- to change the phone system you have to change one place - but no one will let you, because you might break it for every other customer. On the Internet I can tinker with one or two machines and everyone else is unaffected.
Physical vulnerabilites (location, etc...) aren't the biggest worry.
Not too long ago, Wired ran an article about the apparent h4x0ring of phone lines in and around Las Vegas. It seems that a certain escort service (prostitution is legal there) would stop receiving phone calls, especially on busy nights. The employees would call their number from another line, but the phone wouldn't ring. When the authorities came to investigate, the phones miraculously started working again. So the mobsters are in it with the telco employees or the cops or the h4x0rz. Anybody with a copy of phrack or 2600 can probably hijack a switch. This has been known for years. Perhaps there is a large-scale secret phone net that dries up when the telcos or feds try to dial in?
Regardless, the telco infrastructure is hopelessly inadequate.
"What is the sound of one belly slapping?"
If this equipment is that important - and we know it is from the cost to replace it - why isn't it even worth the cost of one clerk at minimum wage around the clock to be able to check on things there? Someone once pointed out that Illinois Bell Telephone ended up spending millions because of the fire, hundreds of times more than it would have cost to have have had a single person present on each of 3 shifts, to provide a 24/7 presence in that building for the next 100 years.
Someone who claims that telephone service is distributed should look again; I've never found a telephone company that operated more than one central office for an area and in some cases trying to combine them in larger and ever larger buildings until the central office for an area might be 40 miles away, yet still continuing the previous rate structure - which may have been created 30, 40, or 50 years ago or more - so that a call to another phone connected to a different switch in the same building is a toll call because it's in a different rate center.
If all the mergers and acquisitions of telephone companies by each other was supposed to benefit the consumer, why is phone service more expensive than ever?
Paul Robinson < Postmaster@paul.washington.dc.us
The lessons of history teach us - if they teach us anything - that nobody learns the lessons that history teaches us.
I don't suppose anyone else remembers the
infamous fire in a Bell Canada phone exchange in Toronto. This fire knocked out phones in much of the city for a couple of days as the crews scrambled to fix things. It was interesting trying to do business....
In my company's case, we still had working Internet via ISDN, so we were still able to go about our business. Some cell phones weren't working, however.
You can accomplish anything you set your mind to. The impossible just takes a little longer.
for reasons below.
Might sound like a troll, but here goes. If you would like more specific info on the tech, reply and I'll reply to you.
I work for a large regional telco in Canada.
I routinely work on various switching and transport equipment. I think I'm finally somewhat qualified to post to slashdot on atleast one topic.
Modern telco equipment is VERY expensive. Large transport shelves will range anywhere from $150 000 to $400 000 per shelf (Canadian, transport being the fibre equipment) There will be several of these shelves per Central Office.(found in every neighborhood) Cards to fill these shelves will range from $8000 to $70000. (they burn out WAY more than I like, usually at 3 in the morning) Switching equipment is even more expensive, the prevelent DMS technology from Nortel Networks is per capita is even more expensive. I would imagine their competitors prices are about the same, although don't quote me. You will have several of these shelves per office as well.
As well, any good telco will have spare equipment on hot standby - major components at 1:1 and lesser at maybe 1 to 10 or 1 to 8 depending on manufacturing
Incidently, you also need expensive people to program and maintain the equipment. A good example is a DMS technician who will get paid the same as an excellent UNIX admin. (and rightfully so, the DMS is a convuluted enviroment to work in)
Each Office needs to be built to the highest standards, physical security, enviromental controls, backup battery plant and huge power systems to feed the equipment
Outside Plant, (that being the fibre and copper cable), is expensive as well, and even more expensive to maintain, this is why you see very few redundant routes, possibly only within a city. Often there is only 1 redundant route, in the classic SONET ring configuration, and often both sides of the ring have to terminate in one physical location. (office building collapses, phones don't work)
I don't know anything about the telco in manhatten, but I can imagine the catastrophe of losing a major office. If they were cutting corners on redundandcy, (which thankfully happens very seldom in Canada due to the regulations here) I could see major routing problems.
For those of you who thing telephone networking is like IP routing, it's not even similar. It's a hiearchy, you cut off the head, it suffers. Many companys may only have 1 or 2 hosts (a host being the "CPU" of the network.) This is due to the expensive of running a host. Telco equip manufactures charge an arm and a leg and your first born, and the liscensing is microshod style draconian.
What I'm saying after all that is - if you want total redundancy everywhere, it's going to cost more money for service. I don't know what the competition is like in Manhatten - but if you're not paying much for your cell phone, there might be a reason.
Just a thought. Flame away.
Redundancy for the casual consumer is just not practical. In order to do it right, you need fully diverse cables and conduits to/from *each* residence, each entering the residence in different parts of the building, and terminating into different CO's. You want your phone costs to double? I don't.
If you are a hospital, gov't office ( police, fire, ... ) you're phone service is on a priority restore. IE, anything that's not priority gets whacked until all critical service is restored.
It dosn't matter whenter you use voice over cowboy neal, if you haven't provided 100% diversity to every piece of the path between you and the phone switch, you are susceptible to exactly this type of catastrophe when something happens to the piece that isn't fully redundant.
For the business or really rich person who decides that they simply cannot afford to be down, even if a 757 hits their CO, you *can* get diversity. Be prepared to pay a lot of money for it, though, because it's not cheap. For the rest of us, between my POTS ( plain old telelphone service ) and my Cell, I'm comfortable that I've done pretty much all I can. Anything more and you're hitting the wall of diminishing returns for the money you're expending.
Remember, buzzwords do not a problem solve.
---
Segmentation Fault ( core dumped )
Old-time linemen and phone phreaks will tell you the same thing: a Telco's idea of a redundant circuit is two cables in the same conduit. About the only disaster-resistant construction Telcos undertake is replacing wind-blown-down telephone poles with underground cables.
Shutting down free speech with violence isn't fighting fascism. It IS fascism!
Not really. SS7 hardware has some buggy software, and after fixing a HLR/VLR (databases) and rebooting it, you just dropped your fivenine uptime.
Nobody really has fivenine, you can fake fivenine, if you exclude your maintence windows.
Part of my University-Sponsored Employment means I work for Communications Services--dealing with the phones, computers, and backbones as needed to keep them up. What I've come to find out is that most Administration don't want to plan for emergency situations.
We were looking at disaster planning. Since we use NEC Phone Switches, we were taking a look at what would be the first thing to go. Take a fire...you could get a switch in a semi trailer sent up overnight (or something like that), but your Main Distribution Frame (MDF) would be crud--you'd have to re-splice every cable pair that you have in order to restore service to everyone; depending on how bad the fire is, you'd have to resplice your RDF's as well
There are some things that we've thought of...like having a bit of redundancy in our wire plant, but the administration shoots us down every time we bring it up.
I guess what I'm getting at is that there isn't a whole lot of redundancy with SS7. Get into things like Voice Over IP, you'll have some flexability, but if your switch gets royally hosed, you're going to be down unless you've got an extra one sitting in another building with a backup MDF that is current.
I disable sigs...do you?
As long as you have lots of wire going back to an endpoint, the endpoint is vulnerable. Most CATV systems have the same weakness, too. About the only thing that isn't as vulnerable to a single point of attack is the power grid at the plant level, and that's because of grid interconnection (there were some interesting power grid-related articles in IEEE Spectrum a few months back). But at the local level, a few substations feed large portions of a city - in my city of 40,000 or so a single squirrel took out a large portion of the town earlier this year. And we have our own generating station here, too.
In any tree-shaped network taking out the trunk takes down all the branches. Verizon is just doing what makes (in the pre-9/11 world) good economic sense in not having full redundancy, with multiple paths. What you might see someday in the not-too-distant future is a few areas (like Wall Street) get second switching stations further uptown, but really the best solution for a business that really never thought about the phone network is a dish pointed to a CLEC that isn't in the same CO as the primary circuits from the ILEC.
If Winstar had remained viable they might well be seeing a big demand spike hit about now as corporate DR people realize their potential weakness.
-- Josh Turiel
"2. Do not eat iPod Shuffle."
I was/am in the middle of converting a federal agency in 26 Federal Bldg (about 6 blocks away from WTC) from analog to ISDN phones. We had half a floor converting on 9/13- needless to say, it has been postponed. 26 Fed has about 16000 phone lines, some ISDN, some analog. Analog service is being restored quicker, but almost no ISDN lines have been restored. Overall, Verizon is restoring about 200 lines a day in the building. 3 major problems with telecom after the attacks: 1) There were COs in the WTC and the Amex building, both of which are totally destroyed. 2) The Verizon CO building was damaged, including water and shock damage (I wonder how well an E5 switch handles water). 3) Several major trunk lines were cut to downtown Manhattan. Basically, too many COs were too close together, and every CO in the bottom half of Manhattan have their circuits maxed out, so numbers can only be restored when trunk lines are re-connected. This disaster has shown how vulnerable our infrastructure can be, especially in metropolitan areas.
Vote monkeys into Congress. They are cheaper and more trustworthy.
My brother is a Verizon installation manager downtown, and he told me one thing that isn't being publicised about the WTC tragedy.
When the towers collapsed, hat large antenna that was ontop of one of the towers pierced the Verizon bldg. on 140 West St. and travelled through the wall, down through several floors, through the basement into the cable vault, which is 2 stories deep there. It proceeded to annihilate a few racks of cable in the vault before coming to a hault lodged into the floor of the cable vault. As a former Outside plant tech for verizon (lineman) who used to pull cables into vaults - I can vouch that this one event alone caused considerable ammounts of damage. Go look at http://newscenter.verizon.com/wtc/ to take a look at the damage done to the 140 West St. Central office.
There was over 30 feet of rubble covering the outside service holes to feed cables into the vault too... the switches were also pretty much destroyed from the debris, the antenna, and water damage from broken pipes and the sprinkler system. The vault flooded from broken pipes, sprinklers, and the water used by the NYFD.
With all things considered, Verizon got circuits rerouted and are restoring them in a rather timely fashion. There is redundancy in the WTC area via SONET rings and other things, which helped get limited service back up as quick as it did... but Slashdotters must realize that MILLIONS of circuits were annihilated during that attack, including CO's in the basements of the WTC too.
Those old telco buildings built during the Bell System years are tough!!! They're built strong!
They weren't made to have 110 stories dropped on them tho... no buildings are. A tragedy like this is hard to be prepared for... .
[Connection closed by foreign host]
5 nines are required for wireline telco hardware. You might expect less than that other applications, but if you're talking about telco hardware made by the big companies (Alcatel, Lucent, Nortel), that kind of uptime is taken seriously. This equipment includes local exchanges, access tandems, long distance switches, and the SS7 network. So switches designed for wireline telco usage must meet the fewer than 5 minutes of downtime per year requirement.
SS7 networks are some of the most reliable in the industry. They're designed to be completely redundant, with the specialized switches (called STPs) set up in mated pairs, located in different parts of the country in the event of a catastrophic disaster. HLRs are typically run in mated pairs as well, so if you're updating the software in one, you still won't lose that kind of service because the mate can take over any functions.
Insert simplistic political, ideological, or personal proselytization here.