Slashdot Mirror
Sprint Routers Stolen; NYC Internet Outage Ensues
cbnet2004 writes "This story on eWEEK reports that late Sunday night a number of Sprint's DS-3 network cards were stolen from a Verizon colocation center at 38th St in Manhattan. Some customers apparently have service back but a number remain down -- it could be a while. The latest rumor on this situation is that some fiber optic cables were cut as well; this could put the affected customers out for days more."
This quote sort of caught me off-guard as I imagine some customers might disagree:
Fleckenstein said that the outage was "not major," and not large enough to require a report to the Federal Communications Commission.
The beginning of the article states:
A handful of corporate customers were left without e-mail and Internet access Monday after the theft of networking equipment from a New York City office late Sunday.
So, I would guess that the "handful" of corporate customers who lost service probably felt it was major to them. I understand the notion that it was not major in the sense of being more widespread, I just think his comment could have been worded better.
Happy Trails!
Erick
http://www.busyweather.com/
Much of the time, thefts at locales like this are often done by people with at least some inside knowledge of the site's security.
It'll be interesting to see where this investigation goes.
"You spoony bard!" -Tellah
I guess you could say the bigger the internet gets, the greater the chance it becomes for real life to come slapping it down. Somebody steals expansion cards from a CO = loss of service for 10's of 1000's of people. It's pretty interesting, almost reminds me of that Real Life DDoS schtuff.
You have to assume they are going to sell them, but even so, it seems like if they show up on Ebay... If you're the kind of person that can pay for a DS3 connection to begin with, it's doubtful you need to be stealing that kind of hardware.
We are one consciousness experiencing itself subjectively. Back to you with the weather, Bob!
1. An employee stole the stuff and cut the wires to make it look like a vandal.
2. A vandal actually did it and will soon sell the goods on ebay
3. Spring is making the whole thing up to cover up their incompetence
4. The entire world is on crack.
Personally, I would vote choice 1.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
The article mentioned that the fiber optics cables were cut, which is a great business opportunity for people who "join" those cables together.
100K or so a year for fixing fiber optic cables... I'm definently in the wrong field. Of course, those technicians have to be very precise or else you get refraction in the wire.
Yes! I listen to NYC Speedcore and do math at 3AM. I suggest you try it too.
That's the truth. I've worked in places where the management is more concerned with the security of the accounting department (which is paperless btw) than the security of the computer room down the hall. The accounting department had key card access, restricted hours, etc. while the computer room just had a regular old lock and the keys were kept in an office managers drawer.
You're not kidding. My employer's voice provider [a CLEC, not Sprint] blamed Verizon techs for "stealing some boards from the CO" when I called to ask about our phone outage today.
VZ, especially in NYC, has a bad reputation for these antics.
Imagine my surprise to see this up on slashdot. Last night around 10 PM mountain I saw a couple circuits go down in NYC. So being the enterprising person I am, I immediately decided that it must be a higher level service problem with our Provider. I call them up, tell them what's going on, and they (Qwest) complete my suspicions and confirm they are having a higher level outage problem. About 4 AM Mtn I called Qwest for an update. They informed me that they were still waiting access as the site was currently cordoned off as a police crime scene and they were still awaiting access. Wow... Cool... never heard that one before during the night shift.
I don't think this is off-topic at all.
If the parent was trying to elliptically reference physical security at collo sites and what damage could be wrought at the hand of insider terrorists by knocking out a few BFRs, it's very much ON topic.
I had two routers go missing from a transport room, which should have less people in and out. My name and phone number was all over the cabinet and the routers.
To be fair, I hadn't connected them yet, so they were just in the cabinet not powered up, and I was going to bring them up the next time I returned to that location, which was going to be in about 3 months. All to often, in a production environment, when there is an emergency, anything not powered up is often considered fair game. I'm sure that the routers are still in use at the company, I just can't find them.
Most colo space in our company is pretty secure. You'd have a pretty tough time getting in if you weren't supposed to be there. Even if you did get in to the colo space, most customers keep the stuff that they manage themselves in locked cages, inside the already secured colo space. Perhaps it was Verizon employees just trying to screw over Sprint. Or perhaps Sprint didn't secure their stuff properly.
------Can you hear me now?.
-- -- Warning. Do not stare directly at the sun.
Security for these places should be like fortx knox, and the second the card was removed there should be of been a notification to the current on-site physical security detail.
:)
Ar....that remind me of my days in a research lab.
Security guards downstair would be 'notified' whenever someone is attempting to reboot those SGI workstations at night. The problem was that SGI hanged up quite often. When this happen, we should either move to another workstation, wait til tomorrow morning. Sometime we had no choice but to trouble those security guards when we ran out of unhanged SGI.
Initially those security guards were nice to us as we didn't do reboot very often - until someone decided to replace all those SGI workstations with NT Alpha. You imagine how irritating to have been called 2-4 times every night.
Soon after the SGI were replaced by NT Alpha, those reboot-alarms were removed for obvious reason.
My point is, Seth was not special, there are many many places to off high end network gear.
"Who are in control, they are not in control of anything - they don't even control themselves!" - Glen Beck
One of those companies was our NY office ;) ;)
We lost our direct extension phone dialing ability to them (could use the full 10 numbers though) and they completely disappeared from our network. They still had regular internet connectivity through a 100mbit cogent line and were able to access other company resources through our other offices Citrix metaframe farms [note 1] with almost full capacity but we still recieved numerous calls at our office as only the road warriors were actually used to using that method for access. We have the licences, horsepower, redundancy, and data sharing ability for this exact reason, well actually in case of another terrorist attack but it works for this too
[note 1]
One thing stood out above all of this. About a year ago, a discussion at a network/desktop meeting lead to a disagreement but eventually a gadget VBS workstation AD weenie created a script on the pc's to "automatically" select connections to our fellow offices Citrix servers through the internal network if you were plugged into the internal network. It was to "eliminate" any http or https confusion as you technically did not need https if you were already on the company WAN, I guess the KISS approach was not a challenge. That was all fine and dandy until today when the route was down. They eventually pushed out an undo so you could connect either way but I wanted to call up and laugh and say I told ya so but I decided not too. What comes around goes around.
Bad boys rape our young girls but Violet gives willingly.
I dont know about the no market comment. This type of thing happens in the CATV industry all the time. One rural town had a cable system built entirely with stolen equipment and the system was awarded to the victim of the theft. My chief suspect would be a contractor who needed the cards to finish a job elsewhere.
Those tables they set up with books music , watches videos. They have all sorts of stuff. When I was there, they were selling movies that weren't out in theaters yet. Shrink wrapped and everything.
Could those routers be substituted, at least temporarily, with clusters of cheap PCs running NetBSD? It could be not only cheaper but also faster. The only downside would be the administration and power usage. I think that this is a very interesting idea.
Sincerely,
Pan Tarhei Hosé, PhD.
"Homo sum et cogito ergo odi profanum vulgus et libido."
Ok - this is MY main pop - but im not a Sprint customer.
I mention this because I have some anecdotal evidence that shows that as secure as the mantrapped and biometrically scanned datacenters are, they really arnt.
Location: Exodus, New Jersey (its the datacenter that was in the big new building above the first path station in NJ - I just cannot remember the name of the building or the train station!)
Time: summer 1999
Issue: company needs to move 16 servers, 2 routers, a firewall, some switces and hubs out of the datacenter.
Procedure to enter: get signed in via biometric security and massive checkin procedure at front door. To get to the front door, you have to walk by the freight elevator, as well as a little wooden door with a twist lock on it.
How I got my kit out: I simply "borrowed" a hand cart, and walked out the back door (the little one I passed on the way to the checkin facility) The twist lock was on the inside, so I just un-locked it and walked out without anyone seeing me.
This made me feel REALLY secure.
Sorry for the AC.
My business initally heard stolen equipment but we were later told that it was caused by damaged equipment from a "Verizon union employee".
I was not on the call but that type of information is VERY specific and there is no gray area or room for interpetation there. I assume this is either totally 100% completely false or someone else knows something more.
The datacenters I've had gear located in have had great security. IN YOUR CAGE. They were very upfront about "if you leave your stuff laying around or your cage unlocked there's a good chance it'll be liberated."
Admittedly in these datacenters there were cameras everywhere..
And of course there are stupid people in these places -- like the folks (not from our company) that were wandering around and wondered "What does this Emergency Shutoff Button on this big power unit do?" Shut off a big portion of the datacenter. The result being certain folks permanently banned and better warnings/protection around the EPO buttons. (admittedly that's a tough one -- 'cause when you need to turn it off you REALLY don't want to waste time..)
Invalid Checksum. Retrying.
The physical security is usually pretty good. About on par with a normal Fortune 500 company, where you scan into areas that you have a reason to be in. The switch room is usually a little harder to get in, especially since 9/11. At Nextel, they actually hired armed guards for a short while when we almost hired an alleged Felon. A competitors security guard recognized him and tipped off our security. Turns out he was supposedly part of a crew that carted off entire racks of telecom equipment.
Getting back on topic. The cards sound like they are the DS3 that pop into a larger fiber demark, like an OC12, 48 or 192. The cards are pretty small and just have coax-looking DS3 plugs on the front (in, out, and monitor). These aren't cards you could really ever use anywhere else. It almost sounds like someone accidentally yanked the wrong cards during maintenence. Although, most telecoms are very religious about not doing maint during the day (if the outage started at night, tho, I'd say it was a switch tech who screwed up).
The reason I'd assert this is the theft was too small to be of any other value. Three DS3 cards aren't going to fetch much, and they're tainted goods. If you're malicious, you're not going to just grab 3. If you're damaging a competitor, grabbing 3 cards is somewhat silly. We commonly have a backhaul path in preparation for things such as this. For example, when I worked at Nextel a fiber dig broke a couple DS3s we had going through PacBell. Within 4 or 5 hours, we swung the traffic over to other DS3s that bypassed the carrier and area with the break.
On a side note, it was also an eye opener that the "Protected, Redundant" Ring-topology that we were paying extra for was not being provided by the Telco. Let's just say there were some very colorful conversations going on between companies at the VP level.
I wish the article had indicated how secure the area was where the cards were stolen
If people can steal routers from australian airport customs, this should be easy enough.
"A really cheap bicycle lock can be broken very easily, sometimes with cheap wire-cutters or picked with a hairpin."
Yes indeed.
I had a shop jack that was bicycle locked to an anchored steel pole. I didn't have the combination to the lock, so when I set out to liberate my jack, I prepared for the worst. I was ready to use the torch, a cutoff wheel from my bench grinder, heat/hammer/chisel. As it turned out, all I had to do was smack the chain with my hammer, using the jack base as an anvil. This was not a particularly cheap bicycle chain. But it was still a piece of crap. I don't think a motivated thief with a sawzall or a cutting disc will have much trouble with a U-lock. What's the Rockwell hardness of a Kryptonite lock?
-fb Everything not expressly forbidden is now mandatory.
Not sure about SGI, but most standard UNIX machines can be rooted if you can get a custom boot floppy to boot so you can access the filesystem. NT is more difficult to compromise this way due to its convoluted/security-by-obscurity NTFS. That is not to say that NT is more secure, merely that this particular method is less useful.
I lived in an apartment complex where a small company was offering broadband internet access (circa 1998). Oddly, the day after they installed several grand in upgraded equipment...some jerk off broke into the telephone room and liberated the new routers. As the thief obviously knew the install dates and what to take, they figured it was either an employee or person in the supply chain.
The tiny company went out of business a few monthes later.
It is sad that we can't just put things in locked rooms and call it good. Thievery like this is a major small business killer.
We used to have a rack in a very prominent facility with lots of excellent security measures.
Unfortunately every one of these security measures could be easily bypassed.
The security guards didn't even ask us any questions or look at our ID when we moved our gear out of there. I'm glad we did!
Pop a tile and crawl under the raised flooring. Push up on a tile in the cage of your choice, snag the wics, shove them in an anti-static bag, crawl back to your cage, shove them in your equipment and voila!
Warning: The power runs under the flooring, so don't try this while wet.
They could also keep an eye on this link.
666-607: 6th floor apartment of the beast
Not to sound stupid here, but I believe you can change MAC addresses. My school makes me report my mac address to plug in my computer, so I just swapped MAC's between my little linksys router and my desktop after I reported it, and then put all my equipment behind the router.
That bugs me more than people bringing down the Internet via theft...
sujal
politics, food, music, life: FatMixx
That's the truth. I've worked in places where the management is more concerned with the security of the accounting department (which is paperless btw) than the security of the computer room down the hall. The accounting department had key card access, restricted hours, etc. while the computer room just had a regular old lock and the keys were kept in an office managers drawer.
That harkens back to the days when payroll was paid by cash. Also, there's usually a small amount of petty cash in the accounting department.
The principal risk, however, is physical access to the pre-signed company checks. (Or even the regular unsigned company checks.)
Stealing a $20k server is extremely noticable... a financial thief can rob you blind for weeks/months before getting caught.