Slashdot Mirror
Sprint Routers Stolen; NYC Internet Outage Ensues
cbnet2004 writes "This story on eWEEK reports that late Sunday night a number of Sprint's DS-3 network cards were stolen from a Verizon colocation center at 38th St in Manhattan. Some customers apparently have service back but a number remain down -- it could be a while. The latest rumor on this situation is that some fiber optic cables were cut as well; this could put the affected customers out for days more."
That's the truth. I've worked in places where the management is more concerned with the security of the accounting department (which is paperless btw) than the security of the computer room down the hall. The accounting department had key card access, restricted hours, etc. while the computer room just had a regular old lock and the keys were kept in an office managers drawer.
You're not kidding. My employer's voice provider [a CLEC, not Sprint] blamed Verizon techs for "stealing some boards from the CO" when I called to ask about our phone outage today.
VZ, especially in NYC, has a bad reputation for these antics.
Imagine my surprise to see this up on slashdot. Last night around 10 PM mountain I saw a couple circuits go down in NYC. So being the enterprising person I am, I immediately decided that it must be a higher level service problem with our Provider. I call them up, tell them what's going on, and they (Qwest) complete my suspicions and confirm they are having a higher level outage problem. About 4 AM Mtn I called Qwest for an update. They informed me that they were still waiting access as the site was currently cordoned off as a police crime scene and they were still awaiting access. Wow... Cool... never heard that one before during the night shift.
100K or so a year for fixing fiber optic cables... I'm definently in the wrong field. Of course, those technicians have to be very precise or else you get refraction in the wire.
I imagine fiber repair guys earn their pay, especially when it's out in some muddy hole and they have a couple hundred strands to fuse together while everyone is breathing down their necks.
Just wondering out loud... It would be really cute if some of the fiber repair guys worked for the NSA. "Of course there's a blip in the TDR, that's where we repaired it." "Okay, nevermind."
A dingo ate my sig...
I had two routers go missing from a transport room, which should have less people in and out. My name and phone number was all over the cabinet and the routers.
To be fair, I hadn't connected them yet, so they were just in the cabinet not powered up, and I was going to bring them up the next time I returned to that location, which was going to be in about 3 months. All to often, in a production environment, when there is an emergency, anything not powered up is often considered fair game. I'm sure that the routers are still in use at the company, I just can't find them.
Most colo space in our company is pretty secure. You'd have a pretty tough time getting in if you weren't supposed to be there. Even if you did get in to the colo space, most customers keep the stuff that they manage themselves in locked cages, inside the already secured colo space. Perhaps it was Verizon employees just trying to screw over Sprint. Or perhaps Sprint didn't secure their stuff properly.
------Can you hear me now?.
-- -- Warning. Do not stare directly at the sun.
Security for these places should be like fortx knox, and the second the card was removed there should be of been a notification to the current on-site physical security detail.
:)
Ar....that remind me of my days in a research lab.
Security guards downstair would be 'notified' whenever someone is attempting to reboot those SGI workstations at night. The problem was that SGI hanged up quite often. When this happen, we should either move to another workstation, wait til tomorrow morning. Sometime we had no choice but to trouble those security guards when we ran out of unhanged SGI.
Initially those security guards were nice to us as we didn't do reboot very often - until someone decided to replace all those SGI workstations with NT Alpha. You imagine how irritating to have been called 2-4 times every night.
Soon after the SGI were replaced by NT Alpha, those reboot-alarms were removed for obvious reason.
One of those companies was our NY office ;) ;)
We lost our direct extension phone dialing ability to them (could use the full 10 numbers though) and they completely disappeared from our network. They still had regular internet connectivity through a 100mbit cogent line and were able to access other company resources through our other offices Citrix metaframe farms [note 1] with almost full capacity but we still recieved numerous calls at our office as only the road warriors were actually used to using that method for access. We have the licences, horsepower, redundancy, and data sharing ability for this exact reason, well actually in case of another terrorist attack but it works for this too
[note 1]
One thing stood out above all of this. About a year ago, a discussion at a network/desktop meeting lead to a disagreement but eventually a gadget VBS workstation AD weenie created a script on the pc's to "automatically" select connections to our fellow offices Citrix servers through the internal network if you were plugged into the internal network. It was to "eliminate" any http or https confusion as you technically did not need https if you were already on the company WAN, I guess the KISS approach was not a challenge. That was all fine and dandy until today when the route was down. They eventually pushed out an undo so you could connect either way but I wanted to call up and laugh and say I told ya so but I decided not too. What comes around goes around.
Bad boys rape our young girls but Violet gives willingly.
Sorry for the AC.
My business initally heard stolen equipment but we were later told that it was caused by damaged equipment from a "Verizon union employee".
I was not on the call but that type of information is VERY specific and there is no gray area or room for interpetation there. I assume this is either totally 100% completely false or someone else knows something more.
The physical security is usually pretty good. About on par with a normal Fortune 500 company, where you scan into areas that you have a reason to be in. The switch room is usually a little harder to get in, especially since 9/11. At Nextel, they actually hired armed guards for a short while when we almost hired an alleged Felon. A competitors security guard recognized him and tipped off our security. Turns out he was supposedly part of a crew that carted off entire racks of telecom equipment.
Getting back on topic. The cards sound like they are the DS3 that pop into a larger fiber demark, like an OC12, 48 or 192. The cards are pretty small and just have coax-looking DS3 plugs on the front (in, out, and monitor). These aren't cards you could really ever use anywhere else. It almost sounds like someone accidentally yanked the wrong cards during maintenence. Although, most telecoms are very religious about not doing maint during the day (if the outage started at night, tho, I'd say it was a switch tech who screwed up).
The reason I'd assert this is the theft was too small to be of any other value. Three DS3 cards aren't going to fetch much, and they're tainted goods. If you're malicious, you're not going to just grab 3. If you're damaging a competitor, grabbing 3 cards is somewhat silly. We commonly have a backhaul path in preparation for things such as this. For example, when I worked at Nextel a fiber dig broke a couple DS3s we had going through PacBell. Within 4 or 5 hours, we swung the traffic over to other DS3s that bypassed the carrier and area with the break.
On a side note, it was also an eye opener that the "Protected, Redundant" Ring-topology that we were paying extra for was not being provided by the Telco. Let's just say there were some very colorful conversations going on between companies at the VP level.