Slashdot Mirror
Sprint Routers Stolen; NYC Internet Outage Ensues
cbnet2004 writes "This story on eWEEK reports that late Sunday night a number of Sprint's DS-3 network cards were stolen from a Verizon colocation center at 38th St in Manhattan. Some customers apparently have service back but a number remain down -- it could be a while. The latest rumor on this situation is that some fiber optic cables were cut as well; this could put the affected customers out for days more."
Sure, I could have brought in a stick and poked at lots of other customer's gear, thru the chainlink cubicles, but I mean, I was signed in and on camera.
Check the log - when did the affected net go down and who was there at the time?
It has got to be a short list of visitors and guards or somebody is really stoopid.
This issue is a bit more complicated than you think.
It's an old adage in towns like Boston and New York, where bike theft is extremely prevalent:
:) (Once you factor in the cost of the equipment, the lost productivity and the bad PR, that's a pretty awful thing to have happen to your datacenter.)
"What does a bike thief care about most? It's not the kind of bike-- it's the kind of lock on it."
A really cheap bicycle lock can be broken very easily, sometimes with cheap wire-cutters or picked with a hairpin. As the above statement implies, even a near-worthless bike will be stolen if left unprotected (I've seen it happen) or with a substandard lock (I've had it happen to me).
Conversely, a bike worth forty times as much as an old junker ($800 vs. $20) can be reliably protected by a lock that only costs twice as much as the cheapo version. (The $40 lock I have now vs. the $20 one I owned previously.)
Applied to the datacenter which had three DS-3s stolen: they clearly didn't purchase a strong enough lock, considering how much their bike was worth
Ok - the above was me... As for the 39th Street Verizon CO - GUYS - this is NOT a hosting facility - this is a data CO.
Two cardinal rules of computer security:
1.) If it's plugged into the internet, it can be hacked.
2.) If they get to your hardware, you're fucked.
I'm still voting that it's an inside job.
~Will
sig?
Verizon has more unmanned facilities (at least at night) than you can shake a stick at. As a nocster for a regional ISP, I can tell you - when a circuit goes down at night, if the testing and troubleshooting w/ Vz requires access to a CO, fugetaboutit till daytime - you can escalate to hell and back, but ain't nothing happening (for emergencies, their on-call techs typically don't respond to pages). Compounding the problem, most of our other circuit providers have to use Vz for the last mile 'tail' circuit.
Like most/all Verizon Central Offices, security is via a keycard. If your keycard does not automatically grant you entry to the C.O., you must be manually allowed in by a guard. Each "guest" must sign in.
38th Street C.O. is just about the highest trafficked C.O. in the world, in terms of Frame Terminations and the like. Being in Central Manhattan, near one of the major CoLoc Hotels nearby, only increases the data throughput on all the eqpt therein.
Vandalism is most likely, performed by another company's techs.
Also- when they say it's not considered a "major" failure, it's b/c Verizon is strictly governed by the PSC's guidelines as to what constitutes "major". These guidelines provide the framework that determines how Verizon (and others) are/can be fined each year with respect to how many/long outages.
First off, let me just say that the one thing telcos get right is engineering for uptime and reliability. When companies talk about "dial tone" reliability, there's a reason for it. Think about it, when was the last time your phone stopped working (assuming you're still with a Baby Bell for local calls)? They have engineered triple redundancy for power for the station:
1. Two independent power feeds from separate substations each running at 50% with a crossover switch. If one station goes down, the other flips to 100% draw with no downtime.
2. Failing that, 2 diesel powered generators with enough fuel to run the CO for 3 weeks without interruption.
3. Failing that, enough lead acid batteries to run the entire station for 13 hours. Some of those dated back to the 60s, but were maintained in pristine condition.
Now, the one thing I will say is that co-located equipment was treated like it was coated in anthrax. It was maintained in a separate cage that could not be accessed from the main building. All co-located equipment was accessed from a separate street level entry that only had a single door and no monitoring. So if the stolen equipment was from Sprint in a Verizon CO, odds are that no one from Verizon was even watching it. (This was back when the 94 telecom bill was just coming into effect, so all of these rules were new...)
For the main building, we had to be escorted at all times, and the engineer we were with got antsy if we bumped against any of the equipment (including some great old magnetic physical switches that were still in use for some old lines). But I wasn't too impressed with the overall security. Some locked doors and a security guard but nothing fancy. That said, if any of Verizon's equipment had broken/shut down I'll guarantee that they have an immediate monitoring/notification system.
04/29/04 Washington Post
Patriot Act Suppresses News Of Challenge to Patriot Act
By Dan Eggen
Washington Post Staff Writer
The American Civil Liberties Union disclosed yesterday that it filed a lawsuit three weeks ago challenging the FBI's methods of obtaining many business records, but the group was barred from revealing even the existence of the case until now. The lawsuit was filed April 6 in U.S. District Court in Manhattan, but the case was kept under seal to avoid violating secrecy rules contained in the USA Patriot Act, the ACLU said. The group was allowed to release a redacted version of the lawsuit after weeks of negotiations with the government.
"It is remarkable that a gag provision in the Patriot Act kept the public in the dark about the mere fact that a constitutional challenge had been filed in court," Ann Beeson, the ACLU's associate legal director, said in a statement. "President Bush can talk about extending the life of the Patriot Act, but the ACLU is still gagged from discussing details of our challenge to it."
A Justice Department spokesman declined to comment on the case.
The ACLU alleges that a section of the act is unconstitutional because it allows the FBI to request financial records and other documents from businesses without a warrant or judicial approval. The group also says such requests, known as "national security letters," are being used much more broadly than they were before the Patriot Act. The bureau has issued scores of the letters since late 2001 that require businesses to turn over electronic records about finances, telephone calls, e-mail and other personal information, according to previously released documents. The letters, a type of administrative subpoena, may be issued independently by FBI field offices and are not subject to judicial review unless a case comes to court.
The ACLU's complaint focuses on the use of national security letters to obtain information held by "electronic communication service providers." The group says the letters could force Internet providers to turn over names, screen names, e-mail addresses and other customer information without proper notice to the people involved. The lawsuit names as defendants Attorney General John D. Ashcroft, FBI Director Robert S. Mueller III and FBI Senior Counsel Marion E. "Spike" Bowman. A second plaintiff has joined the ACLU in filing the lawsuit, but that plaintiff's identity has been redacted from the public copy of the complaint.
What I meant is that it is harder to reset the root/admin password and/or install keyboard sniffers etc. And there is no such thing as a NT boot floppy, you mean a set of 4 uber-hacked disks at the minimum to get some sort of command prompt. NT is famous for being a royal PITA to repair from floppies, that is why there are bootable Linux CDs with (partial) NTFS support on them so you can at least try. Most people just pull hard-drives out and stick them into another running NT box in order to access them. In short, it is way more convoluted then a single floppy you can use to achieve that goal on most UNIX machines.
It's hard to imagine anybody would be so stupid, but then, it wouldn't be the first time.
chntpasswd + windows PE = Done & Done. :-)
If you could be told what you can see or read, then it follows that you could be told what to say or think - BoC
Well, depending on the make and model of the switch, the DS-3 interfaces could be rather spendy. We have 4 such Cisco 6500 series modules at my day-job. Together they cost upwards of $100K US 3 years ago.
Someone mentioned tracking by MAC address... an OC-3/DS3 is a channelized TDM line. 28 DS1 channels of serial data. This is below IP layer. The only identification is going to be a serial number. This will only matter if someone attempts to connect it with a service contract with the manufacturer.
Someone at least knew what to take. They could very well be one of the most expensive pieces of WAN hardware for their size and weight (often no larger than an average book).
-apayne
This was back in 1999 or 2000. Back when I worked for Primenet, which later become GlobalCenter, then Frontier... bla bla bla, then Global Crossing, they had a blackout in Michigan. When they sent a tech out to check it, they found that the entire router was gone! It was a 7200VXR with a couple of DS1s and DS3s. Those customers were down for about 60 hours while a new router was purchased and shipped out. The POP was owned by someone else, and they didn't have a camera or recorded check-in, so we have no idea who did it. Insurance paid for the router.
What what I've read we seam to be dealing with an unmanned data center.
What should be done is add nothing more complex than automated net camras.
Program them to automaticly feed all motion into a server in a manned (and guarded) data center.
Put a minnor firewall between the two (just one that says it can only send data to ONE box and NOBODY talks to it..)
Then someone walks into the data center and SMILE!!!
Make sure the camra is dual mode.. night vision and color.
Or if you can only get em in color (in fact maybe this is a better idea) add motion detection flood lamps.
Now it's SMILE while your blind and have no choice but to stand there and let the camra upload your picture to the data center.
The receaving server verifys the repair/service scedual and if nobody should be there a random on staff security guy is given the pritty picture.
From there they can send down police or security staff.
Security staff.. Becouse I know in a few weeks after this is installed SOMEBODY is going to do a service call with out checking in or someone is going to forget to enter someones repair scedual.
I'm also sure service staff are going to stock up on sunglasses and learn to open doors with eyes closed.
I don't actually exist.
Jeepers...
That can be a boot floppy too.
On a NT/2000/2003 box, PDC/AD-PDC etc, I can get complete root access in less than 10 minutes, provided I can boot from CD/floppy. (If the file system is encrypted, then no go, but I suspect that is very rare.)
Sure NT/2000 can be a pain to *fix* if the filesystem trashes a bunch of things. But we're talking about a system that is functioning properly.
And with the shatter exploits, priviledged escalation is trivial on a Win box.
Frankly, for most boxes, local access is game over, but for Win boxes, it's pretty dang easy. Just do a google search for "NT reset password" - the first link is a free-ware utility that will do it easily for you. Ironically, it uses Linux and Linux drivers to do it for you...
Cheers,
Greg