Slashdot Mirror
Massive Fiber Cut Slows Net
netpuppy writes "East coast to west coast connectivity (or the other way around) feeling slow today? Here's why. It appears that the attack of the raging backhoes has hit Ohio today, where an unnamed public utility managed to cut through 4 OC-192 circuits while working on gas lines. 4 OC-192s are roughly equivalent to 40 Gbps traffic, and trunks this size usually carry both voice and data on them. AboveNet, GTE, and Metro Fiber (now part of Worldcom) seem to be the worst hit, according to this Inter@ctive Week article. " OK, I'm not just crazy. It has been slower then molasses today.
Here's a better version of the same thing: "In theory, theory and practice are the same, but in practice they aren't."
---
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
Most line have markers telling you to call before you dig. But most of the time you call and they say they will be out to mark the lines, but it such a low priotiry for them that usually it gets pushed to the bottom of their pile. So most projects can't wait for the line owners to get off thier asses and mark the line, and go ahead and dig any way to stay on schedule. (My wife works for a Gas Company and has had to call for one of their projects.)
But whatever the situation they need to be carefull when they dig.
Actually I was talking about ethnicity as a false indicator of genes, but yes, I agree. Black people really aren't black (genetically) in the US right now -- there is a constant and often high level of non-negroid genes in all but recent African immingrants. Interbreeding in this case it likely to greatly reduce the visible indicators of racial origin which is how black people have been normally identified as black, i.e., dark skin. So, it will be harder to focus on ethnicity as a divisive factor. The fact that mixed-race children show close to no strong interest in mating with anyone in particular (their choices are contrained by background and that generally means higher education and income) means that they are far more likely to mate with someone without any discernable ethnic background (the sort of more-or-less-white people who make up about 80% of the US population), so that becomes something of a genetic black hole for uncommon genes, and with black people making up only 10% of the population as a whole, they are going to get pretty much submerged.
Right now, people have the false impression that ethnicity (who you think you are) is tied strongly to genes (who you really are). This is becoming less and less accurate and stands a good chance of becoming an impossible belief to sustain. I think that this will defuse "racial" tensions to a large degree, which is a good thing. I think that this will make people less snappish with regards to comments like the one on eugenics, which was, as I said, mildly funny, but not a major talking point. The fact that that person who responded couldn't help but to repeat the Party line is irritating to me. It is people like that who keep public schools from being able to track, and that hurts the smart kids most.
We have now put up a press release. Also, we have created a static copy of The MIDS Internet Average front page showing the period of the outage.
if fibre lines were marked the same way gas lines are marked in cities. hmmm....
I'm a loner Dottie, a Rebel.
I think if the day after a nuclear exchange you're worried about whether or not you're going to be able to read your email, then today's society's priorities are more than slightly skewed.
That's odd...I cut my fiber, and it sped things up, as it were. But I guess you Mr. Fancy-Pants electrical engineer types wouldn't care about an old lady like myself.
They should not be fixing it with W2K. It has been proven not to work in thunder storms.
I doubt it... a few years back, Telstra controlled 90% of AU's incoming bandwidth, at approximately 127MBps (don't laugh, we only have a population of 19 million for an area the size of the US - the most sparsely populated country on earth outside Antarctica. Nowadays, with the springing up of Optus (C&W), AOL, and a few other big names, i think our capacity is approaching 400MBps.
Open Source. Closed Minds. We are Slashdot.
I shouldn't mention the name (*sigh*, clue, huge multinational with common two letter initials, who make everything from calculators to computers and medical equipment. Oh, and it isn't TI), but I used to maintain large WANs for our clients - one had a network of approximately 250 ISDN lines around Australia, some very long length. Uptime was superb (99.985%+) and most outages were, as the above said, cable cuts (and powerfailures)
Open Source. Closed Minds. We are Slashdot.
I used to work for a construction company laying new fibre optic cable. I did the calling of all of the under ground utility owners. (very tedious) But, my boss kept on telling me not to screw up, because if I messed up the cost for 1 (one) fiber cut is $100,000. WOW that's too much.
It already can be a major problem with regards to private WAN links. I've seen more than one occaision when a location on a WAN loses connectivity due to a fibre cut. It isn't pretty, and it makes my life interesting at times ;-)
No WONDER I could not get onto slashdot at work today. heh. this reminds me when someone made a ghetto fire under a bridge somewhere in Minneapolis(I think), cutting most connectivity off for the twin cities. I think it was through MR.NET. anyone remember this a couple years ago?
If you are interested in this stuff, look at The history of the Internet for fuller details. But the basic story is that the idea of packet-based networks arose independently in two places. The first was Kleinrock et al at MIT, the second was Baran at RAND. The former group was interested in them as a way of efficiently sharing the same lines between many different computers. The latter was interested in them for creating communication systems that could survive nuclear war. The two groups did not know of each other.
The Internet arose out of ARPANET which was based on the work at MIT. The goal was to allow computer resources at different research institutions with different types of computers to be shared. The fact that surviving nuclear war was not a goal can be seen in the fact that the machinery used to set it up had no protection against the electro-magnetic effects of a nuclear warhead. Furthermore the initial set-up heavily relied upon a single back-bone. With no redundancy in your physical network, what good is a redundant protocol?
In fact the initial proof of concept and then proposal for ARPANET was made before the MIT people even heard of the work at RAND. Indeed the two groups found out about each other at a conference where the ARPANET was being proposed. Don't believe me? Check it out for yourself!
Cheers,
Ben Tilly
My usual seat in the cluetrain is at A HREF="http://pub4.ezboard.com/biwethey.ht
yah, i was upstream from y'all, your uti engineers are really persistent. but it didn't get them fixed any faster:)
...I can endure a lot of hardship. I can also act responsibly. Why can't they?
---
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
Everybody seems to be complaining about the fact that things are slower today, but what about the fact that 40Gbps is now being rerouted through other means. That's pretty damn impressive. Sure the people who snipped the lines are idiots, but the whole net didn't come crashing down.
Anyway just my $.02
>Wouldn't you love to have 40gig of bandwith piped into your home? ;)
:-)
A friend of mine does - he runs an ISP that only resells
bigish pipes (frame relay, T1 etc) out of his basement.
He claims to have more incoming bandwidth to his house
than Australia
Why is it that IP trunks seem to be backhoed at the drop of a hat, while phone lines are somehow immune? Why doesn't the phone line between Boston and San Francisco go down?
:)
answer: phone lines are not immune. they get cut all the time. however, the phone network, like the internet, has redundancy. whe na phone line gets cut calls are rerouted to other trunks, making the problem oblivious to the layman.
so, why do fiber cuts turn out so much worse? because the data network is bursting at the seams. a voice line cut can be directed to ununsed lines. a data line cannot be done in the same manner, because there are no vacant lines, or at least not as many as were lost...
why is this the case? one, because the data network is expanding faster than new fiber can be laid down. we're too big for a britches. also, the phone network, which has grown at a decidedly slower pace, has, at both the users' and phone/data companies' requests, deemed to be more important. let's face it, if a phone line went down to a company hq, they'd be lost for as long as that line was down. just now are people realizing that their data line, is now becoming more important than their voice line.
backhoe?
YeeeeeeeeHAW! Gonna DIG ME A HOLE! Yessireebob!
Well, datacom companies should team up with gas companies and put the fiber right beside the gas lines. That way everyone would be scared to dig near em and thus no busted fiber lines :)
Enjoy your job, make lots of money, work within the law. Choose any two.
Viewed through my Fresnel lens
In the day and age where you can sue your own parents for not making you clean your teeth properly when you were an infant, I'm sure you can sue them for something..
Merlin --- We're an autonomous collective... Help, Help, I'm being oppressed!!
Caching proxies? Get 'em installed at your ISPs!! In situation like this...it can help lower the burden a lot - by caching the pages for the people who browse, you save bandwidth for people who have to telnet, shop, or post at /. I think I'll get mine.
Of course when nuclear war hits it will take at least 3 hours to fix ....
Gates' Law: Every 18 months, the speed of software halves.
My company's T-1 line has been cut at least 3 times in the last 5 years.
This kind of thing could be a MAJOR problem as the internet becomes more important to big companies. I mean right now, sadly, it's still a novelty to many of the people in the world. But can you imagine the hell we'd be in for if everything travelled over the 'net and someone cut a big trunk like this? Yikes.
Werd.
...that would explain why /. has been slow all day long. Good to know it's not our pipe.
On a related topic, aren't lines like this clearly marked? I mean, how tough is it to call the number on the sign and see where the line is?
Or did I miss something?
censorship is a form of noise, which actively seeks to drown out content with silence - Crash Culligan
Thats one hell of an amplifier to draw 9kw. Where the hell are you getting this info?
The first comment was mildly funny, but yours is just wrong. Of IQ is 80%+ genetic in most populations (which is true -- the lowest estimate that anyone reputable, i.e., everyone except rabid Marxists, suggest it 65-70%), then how is it statistically invalid? Methinks that you and industrial psych have not yet met. You will find him interesting, albeit inflexible.
Actually, for crude measures like IQ, you can breed people just like dogs. And because you are almost always dealing with a far wider and (in the US) more diverse pool of genetic material, inbreeding is pretty damned hard, unless you are from a very closed section of society (I was thinking of Boston and parts of Connetticut, not Arkansas, actually). You get people no weirder than before, but, after a few generations, regressing to a far higher mean IQ. One of the more interesting parts of looking at the effects of post-WWII college education and population mobility is that this seems to have spontaneously created a breeding class of people that has actually seperated out a bit from the American population as a whole. Add grad school, the effect becomes even more profound. Add money, and you wind up with similar self-segregation. If you look at population patterns in the US, for the most part you are seeing (outside of some large US cities) a desegregation of neighborhoods and later (about a generation later) marraige patterns and a realignment solely along income lines. The thing that sorts is money and the filter seems to be higher education, and within that the better and worse schools and the amount of work beyond a BA/BS. This is going to have some interesting effects -- black intermarraige exceeds 30% in most urban areas and the children generally show no racial/ethnic preference (yeah -- flame away -- the control is the white mean, so sue me) so within 40 years or so you will see black people essentially start to disappear as a seperate and identifiable ethnic group.
Pop stats and industrial psych are fun. Unless they aren't doing what you want them to, in which case I am sure that it can be hellish when reality refuses to cooperate.
And looking at the Inter@ctive week article, ETIC of **ONE HOUR** ??? To repair fiber ??? Or should I say, to repair that much fiber ???
Somehow humanity got by long before the internet was invented. The net would be about the last thing on my mind after the bomb drops. First would be radiation sickness, then food/water, then protection (my arsenal). Ever see the Omega Man?
Backhoe, Backhoe, Digging Deep
Make the Backbone Go To Sleep!
Muhahaha
Yeah, sort of like how you know that all the guys with Cameros have a third leg, right?
Why is it that IP trunks seem to be backhoed at the drop of a hat, while phone lines are somehow immune? Why doesn't the phone line between Boston and San Francisco go down?
After the big MCI outage, it's time the phone companies put big red signs on OC-* pipes saying "NEVER cut through this pipe". I take it the gas company isn't about to pay for the lost and delayed packets.
hahaha i didn't know that.
i wonder if they have a sign saying
"dont smash into this wall or you will break the internet"
and if they dont, they should.
tyler
Looks like I'll have to give up my online Q3 game tonight. I wonder if I've got any hardcopies... I mean books... around. :-)
What's all that wet stuff outside?!?
-- IANAEG - I am not an elder god.
You are correct, this would not fall under criminal
law, but the next best thing in the civil arena: Torts
Any act, intentional or otherwise, that does "harm" to another.
Oh, and don't forget: if someone kills you, not only can you charge
them with murder, but sue them for wrongful death and get a wad of cash.
"If you kill me, I'm gonna bleed your bank account dry, buster!"
I don't know about y'all, but our connection doesn't seem to be any slower here. That poor bastard with the backhoe is luck he didn't cut that line around these here parts. We'd have his hide! Hell, Billy Bob'd probably string him up! Imagine not being able to get on EBay to see if he won the auction for the entire video tape library of the Dukes of Hazard. The horror!
All opinions expressed with tongue firmly in cheek between the skoal and the ceegar.
That reminds me, the EMP and radiation given out by an atmic blast kills transistors and just about anything silicon. But guess what? Vacuum tubes aren't affected. Thats why the russains are still using them in military aircraft and vehicles. Guess my hifi will survive ok.
Wouldn't you love to have 40gig of bandwith piped into your home? ;)
I know that this has been Godwinated, but don't you have exactly the same thoughts some days? Don't you think how nice it would be if all the HR people could be put into camps and used for medical experiments (i.e., how does one survive without a brain?), preferably really painful ones. I do.
Is there a law against this sort of thing? I mean, can those who did it be arrested? They should be...
And what about lawsuits? I know that if I ran an e-commerce site who's traffic was affected by this outage, I'd be pretty pissed about the possible loss of sales. Is it possible to sue the utility company that did this for something like that? Or do you just have to chalk it up to bad luck and move on?
"I'd like to live in theory, because everything works in theory, in theory." - Can't remember who said this.
Now we know that the "vehicle" in "multiple independently targeted reentry vehicle" (MIRV) is actually a backhoe, rather than a nuclear warhead. I bet the guys who designed the 'Net to be nuke-tolerant are feeling pretty damned embarrassed right about now.
Some people think the cold war was won by USA outspending USSR. But the real truth is that someone finally leaked that we were building bombs rather than just backhoes. Ivan's pants must have gotten pretty soiled at that revelation. Just think: all along, we totally misinterpreted what "We will bury you!" meant.
---
Have a Sloppy day!
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Now that's what I call Slashdotting!
_________________________
Words of Wisdom:
_________________________
Words of Wisdom:
Never pet a burning dog.
I was right in the middle of a kernel compile when I got cut off. Darn!
Correct, but as the other poster mentions, it just gets re-routed. But stop and think for a minute.
... --- ...
Power grids and telephone circuits can be affected the same way, take out one of those big "power towers" that traverse large spans, and a couple of remote stations, a few satellite uplinks, some telco switching stations/relay towers, and havoc *will* ensue. It doesn't matter if it *can* be re-routed, the resultant chaos and downtimes would cost probably millions. Cyberwarfare isn't about information, it's about $$$$$ lost when the infrastructures disappear.
Then we'll be falling back to all the guys with their ham sets.
Blech. Signatures.
Kudos to the inventers of the internet which allows communications between computer networks to be automagically rerouted when a path is severed by a nuclear warhead or a trusty backhoe! :-)
And I aint talking about no Al Gore either.
Hasdi
I work in London (England) and earlier today a colleague told me that he had tried pinging a site in the US. He claimed that the packets were making it across the pond OK and then were being routed via Australia. We didn't believe him. Guess that I'll have to apologise tomorrow.
Yellow tigers crouched in jungles in her dark eyes.
She's just dressing, goodbye windows, tired starlings.
HA! I was WONDERING why things were slow to my ISP today! Those OC192's are probably the ones that Columbia Gas has pipelines running alongside, on I80. (Ha! You thought I wouldn't name the gas company!) IIRC, ICG laid the first OC48, and MCI/MFS as well as Sprint laid more, and more, and more. They're doing a LOT of construction along I80 from what I hear, so it's NOT surprising that a cut occured. Those lines are on a gas-pipeline right-of-way, and there are 5ESS Demarcation points along I80 for 'em.
:)
There've been previous fiber cuts that resulted in me passing a group of 15 MCI/WorldCom vans on my way home. I betcha if it's the link I suspect, they've got the onramp *packed* with all their techs trying to explain to the people staffing the tollbooths that they're with (insert-company) and they're in on an emergency call and they'll be getting right back off! *LAUGH!* Gotta love the Ohio Turnpickle, eh?
-RISCy Business | Rabid unix guy, networking guru
your company here.
shelby != ford
You have no idea what real work is like. Getting a call at 3am to go type on a keyboard. Try running a backhoe or other piece of equipment all day. You have no idea of what real physical labor is.
out of curiosity, why is mae-east, such an important box, located in a parking garage? amd who maintains it?
This is also why many places allow ham radio operators such small perks like a licence plate with their callsign on it...In case of emergency, it is much easier to locate them; and they are able to contact someone that may be able to help......
The conservation goes a long way part is VERY important! moderate this one up! if more people just used their archived pr0n/warez/mp3s, we'd all be a bit better off over the next week(or however long?) that it takes to get it back together....
Dan
The MIDS Internet Average shows the effect of the fibre cut in the context of the Internet as a whole.
Actually, although these cables may follow major access routes, Sprint initially created their fiber network along the right-of-way of the Southern Pacific Railroad, as it was originally formed by the SP. (Hence the SPR in SPRINT as a previous poster noted; INT is probably for internal or something). Considering the SP doesn't run to the east coast, they either use roads or some other railroads tracks. As for how do I get there, there are trucks with rail wheels. But, overlay a rail map on an interstate/US/State highway map some time. You're likely to find that many of the major intercity rail lines correspond quite closely to the major highways, especially those portions of the US highway system which was build, for the most part, alongside the rail lines. Where they separate, at least along Sprints network, I bet the fiber follows the rail and not the highway.
Least your 14.4K can still connect with this slowdown....it connects, but it goes SOOOOOO SLOW....even for a 14.4k to the point where i have to look at .\ in the library. -btw....anyone have a 56K external for sale? buono@erols.com
It's unfortunate this guy didn't slice open some underground power lines and zap himself and his backhoe ... He wouldn't be around to cut it again when it's time to harvest the potatoes.
-Rob
terpmotors.com
Looks like those guys finished digging that hole right about lunchtime, eh? By far the biggest change I've seen on the ITR. Kind of funny how a backhoe could slow down time (internet time that is). Anyone have links (or know offhand :) to a total M/G/T|bs per day index on total Internet Traffic?
+&x
Marge: I really think this is a bad idea.
Homer: Marge, I agree with you -- in theory. In theory, communism works. In theory.
Episode 1F15, "Bart Gets an Elephant"
dopplar effect.
BOFH, ah, how I miss thee.
www.pushove.com new's for jack asss'
when Push Comes to Shove
see, why the hell were these cables running through what was some chicken feed corn field in Ohio? Who is the idiot that didn't mark that there was a HUGE FRIGGIN' CORD underground that probably cost a small fortune? Oh, and how much is the replacement construction and infrastructure going to cost? Geez oh man. See, that's why I like Qwest. All their cords are along train track lines where they're the only ones that are allowed to dig and even if something is cut, they can just pull another cord through the piping.
So lets see, in the past month or so, we've had a problem with MCI WorldCom and UUNet (correct me if I'm wrong) and now some backwater public works moron who was probably driving the backhoe with the blade down on his way to a coffee break killing some serious piping. Hmmm... what if we have a cataclysmic earthquake that splits North America in two by 3 inches... what would happen? Would all of that fiber optic cable stretch or would it pop? hmmm...
Looks like /. is on the other side of the break from my location. This packet travelled all the way around the globe to get here. Anyway, there are only three possible companies responsible for this disaster. Cincinatti Gas and Electricity, Columbia Gas or East Ohio Gas. Each is equally likely, as Ohio has some sort of 'consumer choice' law that allows you to choose your provider (who is in turn responsible for your particular gas feed) My bet is on CG&E, as they seem to have the largest consumer base and the mose infrastructure under their control. When I figure out which one it is (I'm calling each now) /. .
I'll let you all know.. Let them feel the weight of
.sig: Now legally binding!
What is an internet explosion ?
THIS CANNOT BE DONE
If someone does work under the instruction of a corporate identity, the individual CAN NOT be held responsible. That's like the guys charged with murder for the O2 cans on that airplane. The company should be taken to court, but only as a corporation. Sue the company for damages.
BTW: You can't really arrest people for this kinda thing.
My own curiosity, which I'm sure will be my downfall someday, directs me to ask how on earth one actually fixes such an expensive piece of cable. I always fix ethernet wires by just re-stripping and connecting the RX and TX pairs anew; what's it like for fat pipes? Duct tape?
It's probable that someone reading this has fixed a broken line before. Raise up... you know who you are.
My father, brother, and all of my paternal uncles are construction workers. My father is a backhoe operater, and the rest are laborers, of which I was every summer until I graduated from college.
I wouldn't insult them by comparing the job we did to typing on a keyboard. Try standing on new, white concrete in 107-degree heat.
I was lucky enough to get an education and out of the family rut, and I never take that for granted. The minute I even think of complaining about writing code for xx hours a day, I just remember the days I worked the same hours shoveling sand in pouring rain. If I came home complaining of carpal-tunnel, I'd get smacked in the back of the head, and rightfully so.
PLUS I WAS at the edge of a hole when a backhoe cut a gas line. Luckily it didn't ignite, but the burst and hissing scared the shit out of me. Realizing I could have been killed in an instant at 18 because of a shitty job made me want to get out more. These men do risk their lives and often have no way out.
A slowdown or inaccessibility of the Internet is an inconvenience, no one gets killed. We all need to get our priorities straight.
"More organs means more human." - Zim
Well, this probably would not really work on a national scale, but I always have trouble with my phone lines, whenever we have landscaping or any sort of digging done, my phone lines get hit! so we dug down and layed a row of bricks accross them for the duration of the yard.... No more cuts so far ;)
(And yes, I've done a bit of construction work myself...)
No Laughing Allowed!
With all the hundreds of thousands of every kind of fiber and copper lines out there, I am amazed that this doesn't happen more often. Big deal, it was an accident. and I suppose any of YOU have experience operating heavy machinery? Not too tough to cut a line with a two ton bucket. Thank about it !!!
Slighty off topic but a good tale: My father has been working for a national European broadcaster for a long time. Decades ago when they were setting up a new television and radio campus they had to run some data lines out to the techies' new buildings. He and his colleagues indicated the line where they wanted the backhoe to dig and went to lunch in the canteen. About 20 minutes later all the lights went out. Somebody had neglected to put down any markers and the backhoe had cut through the main power cable *after* where it was joined to the back-up generator. As national braodcasters in Europe also double as the goverment's Emergency Broadcast System this caused a lot of people way up the chain to get alarmed. While the cable was being repaired they had to bundle a crew down to the old emergency studio and transmitter in the General Post Office (Think Krusty the Clowns broadcast in The Simpsons).
"Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
Actually, my corporate office is in Texas and this is directly effecting our national WAN traffic. My Texas based IT department is catching shit because the people upsatirs seem to think this is our fault. Can someone with some insite remind me how a gopher can bring down legs of national WAN by chewing on the backbone cable and it can still be my fault . Oh yes, you can also add the list a fiber line in southern Florida that got hit by a construction crew this morning. Those Damn construction crews need to be more careful with my data. Days like today are not worth getting out of bed for.
"Help me Obi-/.-Kenobi,your my only hope!" -$
There is now a thunder storm in that same area that the lines are down. The company that is fixing the lines has stated that a storm is keeping them from fixing the lines. They have no estimation of when it will be fixed.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
On the other hand, at least a cut comm line won't (yet) kill anyone. A backhoe here in Chicago cut a massive gas line this past year -- the flame jet reached up the side of a small skyscraper (public housing) and incinerated several floors . . .
just for all you to know, themes.org is being hosted on abovenet (one of the severely affected isps) and i can't currently reach abovenet's dns servers.
sum: themes.org probably will be down until this is fixed. please be patient.
-- adraken
People fuck up. You can't arrest them all, or everyone on the planet would be in jail.
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
I work and play here at an Ohio university (Case Western Reserve if any of you care). Net access here is terrible. The fastest transfer I've managed has been a meager 120 kbps to anywhere in the world (considering we have OC-3 out, things are usually quite peppy).
I really wish it were possible to see an accurate weather report on how such things affect the rest of the 'net. It's difficult to interpolate results from very few data points. Of course, the feasibility of such a system is quite difficult and prone to error.
In either case, I'm advising everyone here to be nice to the 'net. There's no reason folks need to be downloading their pr0n, mp3z and off-site ftp installs of Linux during something like this. A little conservation can go a long way.
Actually, tcp/ip over ax.25 (ham radio data) is great; the 44.0.0.0/24 range is reserved only for amateur radio networks!
a m_Radio/ and take a look at the listed sites.
As soon as I have enough money, I'll be buying a second tnc (terminal mode controller, more or less a modem, with packet "stuff" added on), to connect to my handheld 2m/70cm radio, and my newton, so that I can do wireless internet from my hand (and much cheaper than a cellphone after a few months).
There are several great places to begin seeing what packet radio is all about, if anyone's interested you might drop by http://www.frostnet.advicom.net/chris/bookmarks/H
I believe the conversation during the cut went something like this. "What the hell, I just got ahold of a whole buncha fishin line, in tha ground. Who'da barried some good ole fishin line." "I dunno, but it sure is purty, I think we needta pull soma it out and take it with us, whaddaya think cleetus?" "Sure, lets pull it out... wait, here comes the boss." "What in the holy hell? What did you 2 dimwits do, do you know what the hell that is?" "Fishin line, we wuz gonna take sum and use it tommorrow." "Oh hell, it's not fishing line, see, these orange marks mean that you don't dig through here. Thats 4 OC-192's, you idiots" "heh, whats that? sounds kinda technical." etc, etc, etc.... Who trains these guys?
AC meet Hitler. Hitler meet AC.
Gee. I don't recall anyone posting this kind of message about sysadmins.. I mean, how many of us have been paged or called at 3 AM to run in and reboot a server, router or something else. (And it's certainly not WORK that my PHB is doing at 3AM... :P )
-Chris
nailfile, a little crazy glue, they'll never notice...
seriously though, isn't there some serious laser light piping through these things? I hope no one got hurt.
as i recall it's very dangerous to be near the open end of high band optics.
I am not sure if it is just me, but why aren't these things marked? I still remember a few months ago when slashdot was shut down for several hours because of the friendly construction worker deciding to have a little too much fun with the tractor. If it isn't power, it the phone line. One solution can be found in the summer issue of 2600. The article is on ground based networks and it does make for interesting reading.
...But we had a redundant line. Too bad it was in the same cable :) Got 'em both with the same backhoe. Oh well...
Suprise! Some of us construction workers already have access. I do. And we try not to rip up lines. Then again the boss wants the job done yesterday. A machine in the field costs about $250 an hour to operate, a worker about $60 on top of that. Not that he's paid $60 but after administrative costs, really for every worker in the field more people make more money in offices shuffling papers related to his activities. Then there's the boss, he's got to get paid too. And he's just the field supervisor, he's got a boss, he's the guy that actually makes money on the job. And he never even shows up!
"Is there any good way to stop this? "
Several ways. Depends on how much you want to spend. You can encase the line in concrete. Electrical cables are done this way and the concrete is dyed red as a warning. That's code by regulation. Concrete is like $60 a yard poured on site. That's just the material cost, of course there's labor involved too. You can surround the line with enough sand and marker ribbons so that someone digging can get warning in advance. Sand costs money too though. And you'd have to dig a fairly wide trench too. I'll bet that these lines are laid in narrow, not too deep trenches cut by trenching machines, not even a backhoe dug trench. Think cheap installation. There's who's fault it is. Not the guy digging it up, but the company laying the line down. These Internet outfits can get away with whatever they want to so they do. A hole's a hole right? Wrong.
In Finland we have made the opposite. In the high-voltage power lines there are two lightning-conductors on the top of these three conductors actually delivering the power.
Now at least one of the lightning-conductors has a fiber line in it, which is then used to build a fast IP-backbone. The cable gets dozens of direct lightning hits every summer, but without any service degradation...
I spent my summers laying water and sewer pie as an undergrad. Good work, really. Some people even worked there rather than have an office job, even given the 30-50% pay cut involved.
As for cutting lines, the charge is well over 1 million dollars per minute of downtime per line. No maximum. So, the company that cut the lines will go out of business. The worst of it is, the phone companies do a terrible job locating their fibre wires, and then sometimes don't even cover them--so it just takes a shovel to clip the whole line. Worse, they tend to be the worst at breaking everyones water and sewer lines, and sewer is much less fun to fix than fibre.
I don't know if this is that reference, but the maps they have there are cool nonetheless. All sorts of stuff, going back to the "original" ARPANET map.
Was this a Y2K dry run for a terrorist?
Just imagine if 20 pipes were cut in 20 states - it would only take 20 backhoes. We would feel the impact for 20 years. 20 billion dollars would be lost by 20 companies, 20 new laws would be passed, 20 thousand law suits would be filed, and it will all happen in 2000.
- daniel
- daniel
Turn off your computer and go outside
Who saw, 'massive fiber cut' and thought that, er, someone was changing providers? And wondering why it was done in the middle of the day?
"But that cut wasn't scheduled for today!"
--- Where's my X.400 protocol decoder?
The severed end of that fiber gives a whole new definition of piping output to /dev/null.
--- A Jesus Fish eating a Darwin Fish only proves Darwin's point.
About 8 years ago while working in Asia in a country to be left unamed (to protect the guilty). We had, had about 20 line cuts in a two week period. All within the same five mile stretch of Highway. It was because local farmers whereout backhoeing their field dikes to prepare for the next seasons planting as well as crews from Highway construction. We kept repairing the cuts in the fiber, putting larger and larger signs and underground tapes all in the local language (none of which helped they dug where they wanted too.) One of our local co-workers came up with a solution to the problem. We place signs along the stretch in ENGLISH only. The locals, and highway crews became so sure that this was some special secret government project that over the course of the next 6 months we had only one cut in our area of responsibility. *sigh* Kinda like my physics prof who said the only thing fibre optics would be good for is "hippy" lamps and nothing more.
I'm sorry, I'm to tired to be witty at the moment so this message will have to do.
Our OC-12 fiber that is the backbone of our WAN got shot clean through. The shooter has yet to be discovered..
.... but whatever did happen to the BOFH? I noticed in about two months ago it wasn't going to be published for a while and it never came back.
As someone who did construction for a couple of years, I can tell you that while you might think there are maps that can tell you where lines are, they are almost always wrong. I've seen clearly marked lines on a map that were more than 10 feet from where they were supposed to be when you get in the ground and start digging. If the original diggers used flags to mark the lines, that makes it a little easier but even then it's like looking for a needle in a haystack but the needle has a string around it. A little better but not much.
About three times in the past few years I have experienced a 'net loss' due to back-hoe operators. How many 'call us before you dig' commercials does there have to be before people stop cutting wires. It is just a silly expense which is naturally passed down to consumers. These people should be wearing wireless net devices which can tell them where all pipes and trunks, etc are running through their work site so this can be avoided. woo. wonder how much this repair will cost and how long it will take. and how long it will be until another company does the same thing.
burn the computers. go back to the abacus.
it was IDIOT proof. Just nuke proof.
DNS, for example, would have a very high priority, and be one of the last services to drop. Without DNS, the network becomes significantly less usable. The services designed for text communication also would have a high priority: smtp and the assorted email services (no attachments), nntp (again, no attachments), finger, time services, gopher, and the like. Even http might be allowed through, but filtered by mime type (text/plain, text/html, x-www-form-unencoded, etc.).
The problem here is that you couldn't realistically do much more than filtering by port, as its not possible for a router to reliably know it's routing enough of a connection that it can filter based off that (and for that matter, it's most definitely not reasonable to have big honkin' overloaded routers take every packet, decode, and analyze everything that's coming through).
It would also be difficult (though conceivably possible) for a local router to be doing this. Trivial example: email attachments are encoded as MIME. But you don't know whether a message is being sent MIME encoded or not until after the SMTP client and server go through their little authentication dance. For a stateful protocol like SMTP, there'd be no way to "ban" MIME unless the router was "listening in" to the whole conversation, and rudely cut it off somewhere in the middle. For stateless protocols like HTTP, the router would need to capture the entire outgoing before it's sent out, to evaluate it, i.e. if the query is broken up into 3 packets, the router would need to capture all 3 packets, reassemble them, and parse HTTP, in order to determine the content type.
What a day to decide to grab the new netscape! I thought everything seemed so slow. But in all seriousness, those must be some big backhoes that cut the connections because I figure those fibers are several feer in diameter.
--
Scott Miga
Something like this seems to happen every 8 months or so. Is there any good way to stop this? Aside from moving to wireless, the only thing I can think of is not put four high-bandwidth cables under the same patch of street - but then laying cable (er, fibre) becomes expensive proportionately to how many baskets you're keeping your eggs in.
Trees can't go dancing
So do them a big favor
Pretend dancing stinks!
Good thing there wasn't anyone around there smoking a cigarette. What with all those loose bits sloshing around, the slightest spark could have set off that 'internet explosion' people keep telling me about.
Trees can't go dancing
So do them a big favor
Pretend dancing stinks!
Sorry guys... my fault. My father once told me that he buried all the valuables in the backyard for "protection." He really thinks Y2K is going to leave us in a Mad Max nightmare. So anyway I tried to dig them out and apparently I cut the lines. Sorry.
--
RumorsDaily
It is similar to something that happened in the town that I live in. Some one in the public works cut a fiber line and basicly took out the whole town. They had to reconnect 1200 extenstions. Some people had access after a few hours but the house I was down was down for over 9 hours. I was resorting to an amateur radio 1200 baud link to do some surfing for home work. 1200 is slow when you comapair it to 28,000.
Hasn't anyone ever seen those signs " Call 24hrs. before you dig " ? I guess its just a thought.
And all I day I thought my ADSL was screwed up... Way to go CG+E...
*shrug* I've operated heavy machinery. (quite efficiently, I might add)
:>
Though I'm not competent enough to pick up a hardboiled egg
with a backhoe... I am competent enough to dig fast holes
and put the dirt where you want it.
IF LINES ARE NOT MARKED, the operator has NO idea what's down there.
Trust me on this. The job is laid out.. you need X amount of dirt moved
over to spot Y. What do you do? You move that dirt, and fast.
(unless it was part of his job description) the operator was more
than likely digging a hole that was laid out for him
by someone else entirely. (only between the blue X's Ray)
Now if the kid/woman/person/guy/whatever was in charge of both? Execute them.
"Call before you dig" Pretty simple, eh?
Seriously though... there should be some tough penalties extracted from the companies that allow
their workers to pull blunders like this.
Remember... it's people like this that take forever to
process your order at your local
burgerdoodle --- Insert favorite fast food resteraunt name here.
Friends don't let friends buy Compaq's. (Dell/Gateway... same same) You want a good computer? Build it yourself.
you know this is when that annoying commercial
"go for the phone and call blue stake" comes in. maybe this commercial isn't everywhere but the midwest people know what im talking about.
makes you wonder what happens if mae-east gets destroyed by some natural (or unnatural) disaster.
oh how much we depend on these plastic boxes!
tyler
I would say that this goes to show the utter bullshit that is the whole cyber terrorism thing. Why spend billions of dollars trying to police imaginary squads of crackers set to destroy our information infrastructure, when a couple of idiots with shovels can create major mayhem like this?
I wonder what an organized group of wire cutters who did a little bit of research on their targets could accomplish. I have a feeling it wouldn't be pretty.
I can't say I noticed anything myself (the net has been dog slow for me as long as I can remember, so), but if a small event like this can cause major problems, then the Internet is definetly closing up on critical mass....
-
Most of the time the telcos themselves doesnt know
where their fibre is. And the biggest accidents of
this kind actually happen when there is telco people there, they just thought the cable was 5 metres to the left.
Now, we can't count on the users cutting down on their bandwidth use conscientiously.
:-)
DNS, for example, would have a very high priority, and be one of the last services to drop
Consider dropping DNS first - loosing all the clueless will certainly free up bandwith!
redundundancy systems should not be alike, so you'll have to cover as many different problems as possible with as many different systems. now it is time for the OT part Closed China might be an example of what happens when everyone is like everyone else. Credit and socialism affect the gene pool. There are arguments as to whether races are able to be subdivided. Forty years could start an effect that would be difficult to reverse because of general growth. Black people just might not be black anymore.
I don't know, but I'm sure he could have made that sentance a little more coherant.
Well, the powering issue being talked about isn't exactly what would be called "common". Most fiber in the ground today has no current-carrying conductors. True, it has a metallic conductor or two in it, and in some cases, a metallic sheath or armour, but carries no exectricity. The metal conductor is used exclusively to locate the cable with inductive/RF location gear. When it comes to re-generation of fiber "signals", byt simple amplification, or a full teardown of the bitstream with drop/add functionality, it's usually done at a special location (in-ground vault, above ground fiber hut, whatever) with external power. That's power supplied by a power company, with backup power. Since fiber traffic routinely travels 50+km without intervention at all, it's a total waste of resources and cable to lay a power cable for equipment. Now, all said, there are exceptions where there is no power readily available, like deep-sea cables, deserts, and such where there may be a power cable present in places. But in general, not present. Now, back to a little more on topic :) As for location, depending on the area, this can be tricky business, big time. In dense metropolitan areas, there can quite literally be cables a foot or less apart "horizontally" as well as several layers deep "vertically". Location of cables isn't always as precise as it ought to be, especially when the phone companies records don't reflect actual construction oddities, like perhaps a four foot fiber "loop" at the base of a pedestal. Well, when the person locating cable gets within a couple feet of a pedestal, you'd just assume it goes right in, right? Wrong :) And another company plans to put a pedestal right next to the existing one, and, well, it's time to break out the Scotch-Lok or quick-fix fiber kit :) Granted, most cable and fiber cuts happen out of pure ignorance or stupidity. Farmer Joe is out in the field fixing a drainage tile with his tractor and doesn't give a single thought to what else may be in the ditch with his tile. Where did those kits go again? :) And as a final note on the buried cable thing, nowadays, it's becoming common, almost required, practice to bury a brightly colored plastic ribbon above the cable being installed. Usually yellow or orange, it's really easy to see against black dirt, and would hopefully be seen before the contractor hits the cable itself. Doesn't do much good in the case of boring or knifing cable in though. All of this is just general information. The cut out east could have been done in one of a dozen or more other common contractor cable goofs that I won't even try to speculate on. Either way, it boils down to this - contractor started digging without requesting a locate, or if the cable was properly located, and the contractor either mis-read the location markings, or ignored them, that contractor had better hope he has a good insurance company. If the cable was not located at all, and a locate request was made and recorded in a timely manner, usually 24 to 48 hours before digging, well it's the telco's tough luck. You know, it's a small wonder that things work as well as they do considering the unimaginable number of perils out there, backhoe's included :) Sorry this is so long! FBG the AC
But now the Internet is a commercial enterprise, and failure is now an option. At worst, some large corporations lose their VPNs and have to prioritize and pick up the phone again. For most of the net it's just a matter of losing porn, IRC, and MP3.
Now if you're a backbone maintainer, do you double your capital costs to achieve more than minimal redundancy just to give the public a warm fuzzy feeling? Or do you maintain the least expensive network you can without losing customers? Market forces will drive the QOS on the net to the lowest tolerable level, and for now people will tolerate a lot of net failure because their lives and livelihoods don't completely depend on it...yet.
Given, they have a tough job. So what, they get paid to work like dogs, they aren't a chain gang for crying out loud.
:-)
Around here, they put commercials on tv all the time saying, "before you dig, call Miss Utility" and flash an 800 number. So you mean all us yokels have to check in with the utility folks but the utility folks don't have to check themselves?
Sure accidents happen, but I can't see a public utility diggin around and not knowing where their own lines are. Sheesh.
The Divine Creatrix in a Mortal Shell that stays Crunchy in Milk
The House Between - Original Sci-Fi Series
Now, we can't count on the users cutting down on their bandwidth use conscientiously. How, then, can we keep the critical services running? For a start, we need to define "critical services". I'll say that the greater the ratio of content to bytes, the higher the priority of a service. The only practical way to filter packets by service is to filter by port. You can run a filtered service on any old port you want, but the goal is not to prohibit services so much as drastically reduce the bandwidth used so that the network remains usable.
DNS, for example, would have a very high priority, and be one of the last services to drop. Without DNS, the network becomes significantly less usable. The services designed for text communication also would have a high priority: smtp and the assorted email services (no attachments), nntp (again, no attachments), finger, time services, gopher, and the like. Even http might be allowed through, but filtered by mime type (text/plain, text/html, x-www-form-unencoded, etc.).
Still, there would be a significant drop in the usefulness of the network. We need more bandwidth than we need to ensure reliability. Make bandwidth, not war! :)
Gates' Law: Every 18 months, the speed of software halves.
What about /dev/digger? Or /dev/JCB? Or even /dev/bonehead?
The net seems to be back to normal speed for me at least... This kind of thing happens from time to time.
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
Except that eugenics is statistically invalid.
Free Mac Mini. Yes, I'm
The Spring has sprung, the grass has riz, I wunder where my packets is?
of course, the most ironic part is that it took a /. user to direct traffic to the Internet Traffic Report, which is also owned by Andover.net Hmmmm.... they still don't have their hats on straight, from a business perspective. ---"Progress is the God of the Machine"
-rt-
** Evil Canadians are taking over the world. Learn about the conspiracy
It amazing how our bandwidth requirements have increased so fast that 99% of all our modern communication is carried through only the newest lines. What would happen if the phone companies stopped increasing bandwidth for just a second?
We never had this problem back in the days of good ol' POTS service. Good old copper wire never cut cut or went dolkjsd;flkh#&&(
NO CARRIER
DrLunch.com The site that tells you what's for lunch!
I work at an internet service provider (UTI) which is being hit by this right now. I hope it gets fixed quick since the tech support calls keep rolling in.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
How much to do you want to be the backhoe operator will be driving the same car 10 years from now?
Well - that would explain this:
:-(
1 198.182.167.17 (198.182.167.17) 0.628 ms 0.564 ms 0.532 ms
2 adsl-63-194-218-254.dsl.snfc21.pacbell.net (63.194.218.254) 26.688 ms 29.433 ms 15.868 ms
3 core1-fe4-1-0.snfc21.pbi.net (206.171.134.209) 13.394 ms 13.701 ms 15.957 ms
4 gsr1-g1-0.snfc21.pbi.net (209.232.130.20) 16.098 ms 13.846 ms 15.215 ms
5 sfra1sr3-so-1-1-1-0.ca.us.prserv.net (165.87.161.74) 16.676 ms 16.208 ms 15.202 ms
6 sfra1sr2-11-0-0.ca.us.prserv.net (165.87.13.17) 16.684 ms 16.286 ms 15.226 ms
7 above-advantis-ds3.sjc.above.net (216.200.0.81) 21.833 ms 24.418 ms 29.271 ms
8 core1-core4-oc3.sjc.above.net (216.200.0.85) 25.864 ms 23.882 ms 17.173 ms
9 core2-core1.sjc.above.net (209.133.31.110) 47.734 ms 39.032 ms 39.107 ms
10 main2-core2.sjc.above.net (207.126.96.138) 44.072 ms 40.088 ms 40.069 ms
11 core2-main2.sjc.above.net (207.126.96.137) 42.369 ms 44.488 ms 46.094 ms
12 * * core3-core2-oc3.iad.above.net (209.249.203.65) 590.011 ms
13 abov-core1-mae-east.netaxs.com (209.249.119.234) 695.855 ms 721.634 ms 848.819 ms
14 dn-netaxs-gw.dc-core.h5-0-45M.netaxs.net (207.106.127.94) 1125.874 ms 1282.923 ms 1408.877 ms
15 h900.ca2.wdc.dn.net (209.207.190.5) 1263.551 ms 1250.176 ms 1252.700 ms
16 209.207.174.23 (209.207.174.23) 1234.975 ms 1256.689 ms 1208.096 ms
and worse yet DNS lookups from here going thru Vienna
But remember folks there's redundancy in the backbone routing but when something big goes down
everyone else gets to suffer as the traffic
gets piled on top of their usual connections.
If there's a lot of traffic going thru Europe I
bet they're getting really steamed over there
How exactly does being able to remember IP addresses make one clueful ?
We recently had fiber installed to out building. We have ISDN as a backup. The fiber travels above ground quite a ways and then goes underground. My boss took the time to check out both the fiber route and the copper one. They only cross at one pole. If that pole goes down . . .
--- If you don't want to know the answer, don't ask the question.
Unless the lines were in some serious sheathing, most backhoes wouldn't even twitch going through it. If the operator wasn't watching the hole close (most tend to be more concerned about where the bucket's going) he'd probably never notice it.
As an aside, I once worked with a gentleman who could knock your hat off with the bucket of his hoe (I held still, most he tried it on wouldn't). I saw him pick up a boiled egg in the bucket without breaking the shell. That kind of trick requires a level of skill not equalled by most airplane pilots. Those machines are *not* that easy to operate, and idiots don't tend be allowed on them.
As opposed to London, WA?
:)
Sheesh
I know that most of the posters here are from the US, but maybe they could have worked this one out for themselves?
The Backhoe
Natural Enemy of the Network Administrator
http://www.23.com/backhoe/
which is probably the downtime cost for this
cable -- someone is going to be sued for
a number with a lot of zeros in it.
Anyone know whose cable this was? Just
from the "who got whacked" it looks like
it belonged to Cable & Wireless, but I haven't
heard anything for sure.
garyr
-- your Web browser is Ronald Reagan
Hell, gas doesn't even slow 'em down. When I was at UNH, they were rebuilding the school library. They tore into a gas pipe no less then three seperate times during the construction. Twice at the same spot, from what I understand. I was only present for one of 'em. It was kinda freaky to have the fire alarms for half the campus going off at once...
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Do you honestly think that if they have the possiblity to destroy something they have access to will make them be more careful? Will giving them free Net access make them understand the data infrastructure? People who pay for the service don't understand it (my Mom still doesn't quite understand why you can't pick up the phone while on the net). Should we give all people who have a job where something might get ruined free use of it? Why don't construction workers get free cable, free phone service, and free electricity along with the free ISP?
If I'm at work, and I'm walking by a UNIX box and accidentally trip over the Ethernet cable and disconnect it, should I not be responsible for that? I'm not the UNIX admin. Perhaps if they gave me root I might not trip over the line.
that you quoted from at. html
:-)
http://www.wired.com/wired/archive/4.12/ffglass
it's 56 pages long. Took me 3 days to read.
Until I read it I didn't know that subarine
cable was interesting. Really good and worth
reading the entire thing. The guy that wrote
it (Neal Stephenson) may just turn out to be
a good writer
garyr
-- your Web browser is Ronald Reagan
I checked out the Internet traffic report to see what kind of effect this harmless mistake had. North American traffic statistics
Keen! Can you spot the time the big bad backhoe operator cut the cord?
Construction/Utility repair guys don't have an easy life. They're the ones out there in the middle of the night when your power line has gone down, or the ones fixing that broken sewer pipe, or the ones who make sure you have water. Near here they're doing major construction over on Grand Avenue, and to avoid traffic they have to do it in the middle of the night. They accidentally cut off a hunk of phone lines and about 1000 people had no phone service, but that happens pretty rarely.. I'd like to see YOU guys doing hard physical labor all night or all day trying to avoid speeding cars and a maze of pipes and cables and never screw up occasionally.
I wonder if this has anything to do with the slow connection rate that I get when I try and telnet into my webserver that sits in the next town over from where I work. After all, when I do a traceroute, I get shipped through machines in Chicago, SanDiego, and other exotic places. And I'm in South Carolina. A month or two ago, there was a slashjot about some guys trying to map the Internet. I think that it would be easier to map a world covered in spaghettti noodles, because that's what the Internet looks like to me!
Brad Johnson
Advisory Editor
Brad Johnson
I do TS for Juno, and thankfully my shift is almost over... no wonder it's been toasty in here since 1-ish. =)
- dom
- gnome
What's up, Mr Jones?
I was under the impression that SONET was designed to help avoid this kind of problem. I haven't seen any problems yet today, I'm on uunet in Pittsburgh tho.
What happened to the idea of redundant links and multiple paths? Sounds like whomever is on this fiber has some explaining to do as to why their OC-x networks were not or do not have fault tolerance as advertised, or even diversified paths. Even in my little town (State College, PA) I know where the OC-3 into my office goes, and there is only one section of about 2000' where all four fibers are on one cable, from my office to the road!
Sounds like the telcos should be held responsible for not building resilience into their networks. As more fiber is laid, and we depend on it increasingly, an outage will be more and more costly. Much cheaper to have the fiber going the other way too.
Just my thoughts.