Slashdot Mirror
Vulnerability of Telco Switching Equipment
call -151 writes: "Interesting New York times article about the Sept 11th attacks' effect
on the Verizon switches in lower Manhattan. Turns out there
was a problem in that much of the network switching was in one
building and it has taken a while to restore service. Sounds like there
is lots of pondering about the vulnerability of the network,
even when it is distributed across many physical locations.
Of course the attacks are making lots of people rethink their
vulnerabilities, but the estimate is for five years' work before there
could be redundant paths for the lines into their switches in
the one building, with no plans to spend the money to do it.
Maybe someone should send them a few hundred thousand 'self-install'
kits like they do with their DSL service ..."
Telco switches and networks are the most reliable. 99.999% (5-9's) uptime. Better than IP, cable, wireless... Just ask the dorkwads trying to get VOIP to work...
Paul Robinson <Postmaster@paul.washington.dc.us>
The lessons of history teach us - if they teach us anything - that nobody learns the lessons that history teaches us.
for reasons below.
Might sound like a troll, but here goes. If you would like more specific info on the tech, reply and I'll reply to you.
I work for a large regional telco in Canada.
I routinely work on various switching and transport equipment. I think I'm finally somewhat qualified to post to slashdot on atleast one topic.
Modern telco equipment is VERY expensive. Large transport shelves will range anywhere from $150 000 to $400 000 per shelf (Canadian, transport being the fibre equipment) There will be several of these shelves per Central Office.(found in every neighborhood) Cards to fill these shelves will range from $8000 to $70000. (they burn out WAY more than I like, usually at 3 in the morning) Switching equipment is even more expensive, the prevelent DMS technology from Nortel Networks is per capita is even more expensive. I would imagine their competitors prices are about the same, although don't quote me. You will have several of these shelves per office as well.
As well, any good telco will have spare equipment on hot standby - major components at 1:1 and lesser at maybe 1 to 10 or 1 to 8 depending on manufacturing
Incidently, you also need expensive people to program and maintain the equipment. A good example is a DMS technician who will get paid the same as an excellent UNIX admin. (and rightfully so, the DMS is a convuluted enviroment to work in)
Each Office needs to be built to the highest standards, physical security, enviromental controls, backup battery plant and huge power systems to feed the equipment
Outside Plant, (that being the fibre and copper cable), is expensive as well, and even more expensive to maintain, this is why you see very few redundant routes, possibly only within a city. Often there is only 1 redundant route, in the classic SONET ring configuration, and often both sides of the ring have to terminate in one physical location. (office building collapses, phones don't work)
I don't know anything about the telco in manhatten, but I can imagine the catastrophe of losing a major office. If they were cutting corners on redundandcy, (which thankfully happens very seldom in Canada due to the regulations here) I could see major routing problems.
For those of you who thing telephone networking is like IP routing, it's not even similar. It's a hiearchy, you cut off the head, it suffers. Many companys may only have 1 or 2 hosts (a host being the "CPU" of the network.) This is due to the expensive of running a host. Telco equip manufactures charge an arm and a leg and your first born, and the liscensing is microshod style draconian.
What I'm saying after all that is - if you want total redundancy everywhere, it's going to cost more money for service. I don't know what the competition is like in Manhatten - but if you're not paying much for your cell phone, there might be a reason.
Just a thought. Flame away.
All the wireless "basestations" have frame relay connections into one building. Same point of failure.
I was/am in the middle of converting a federal agency in 26 Federal Bldg (about 6 blocks away from WTC) from analog to ISDN phones. We had half a floor converting on 9/13- needless to say, it has been postponed. 26 Fed has about 16000 phone lines, some ISDN, some analog. Analog service is being restored quicker, but almost no ISDN lines have been restored. Overall, Verizon is restoring about 200 lines a day in the building. 3 major problems with telecom after the attacks: 1) There were COs in the WTC and the Amex building, both of which are totally destroyed. 2) The Verizon CO building was damaged, including water and shock damage (I wonder how well an E5 switch handles water). 3) Several major trunk lines were cut to downtown Manhattan. Basically, too many COs were too close together, and every CO in the bottom half of Manhattan have their circuits maxed out, so numbers can only be restored when trunk lines are re-connected. This disaster has shown how vulnerable our infrastructure can be, especially in metropolitan areas.
Vote monkeys into Congress. They are cheaper and more trustworthy.
In larger areas, for instance Cincinnati, the Bell has a number of switches (I think it's 30). Each are connected with Sonet rings. I have toured the Cincinnati Bell NOC, where they have electronic maps of the rings and the network. Each CO may be on a number of rings. There are something like 300 rings in the area, all interconnecting different sets of COs, with plenty of overlap on each CO.
For those of you not familiar with Sonet, it is a ring of nodes, with a fiber pair running in each direction (four fibers instead of the normal two). If a cut happens, traffic is instantly routed in the opposite direction, around the break.
Cincinnati Bell uses their Sonet network for all voice, ATM, etc. LD carriers can connect to the network at any point (or multiple points).
Hello little man. I will destroy you!
Your argument would be valid if it even came close to the size and breadth of the VZ network at 140 West Street. Besides the facilities that existed in the Towers themselves, West Street was our sole serving facility for most of downtown. You can't compare 30 C.O.s in Cincy to one at West Street just because the NY Times article compared 140 West to Cincinnati in terms of data travelling through it. You can't make a direct comparison from an analogy like that. It's not as if West Street is the only C.O. in Manhattan- it's just the one that was damaged when a major catastrophe occurred next door.
The biggest problem at West Street is not necessarily the damage, or the flooding in our basements- it's the fact that West Street does not have reliable records. Their switching facilities and relay racks were not mapped reliably, and as a result, they are expending just as much manpower to figure out what was going where as to run new cables out from the 4th floor.
As for why there is so much damage to begin with, we were told that either a huge internal beam or the antenna on top the north tower pierced 140 West Street, causing most of the damage. The side of the building that faced the Towers was the side that most of the switching equipment was kept on. Combine that with the fact that the basement was flooded with water and diesel fuel, and you have a building that can't really support any kind of telephone service, at least immediately after the attacks.
Besides the huge amount of data lines that are served out of West Street, most other low speed data lines were routed through there. So, if the little bodega near your apartment in the Bronx or Brooklyn stopped selling Lotto Tickets right after the attacks, that's why.
Hhope ths helps.