Slashdot Mirror
Massive Fiber Cut Slows Net
netpuppy writes "East coast to west coast connectivity (or the other way around) feeling slow today? Here's why. It appears that the attack of the raging backhoes has hit Ohio today, where an unnamed public utility managed to cut through 4 OC-192 circuits while working on gas lines. 4 OC-192s are roughly equivalent to 40 Gbps traffic, and trunks this size usually carry both voice and data on them. AboveNet, GTE, and Metro Fiber (now part of Worldcom) seem to be the worst hit, according to this Inter@ctive Week article. " OK, I'm not just crazy. It has been slower then molasses today.
Most line have markers telling you to call before you dig. But most of the time you call and they say they will be out to mark the lines, but it such a low priotiry for them that usually it gets pushed to the bottom of their pile. So most projects can't wait for the line owners to get off thier asses and mark the line, and go ahead and dig any way to stay on schedule. (My wife works for a Gas Company and has had to call for one of their projects.)
But whatever the situation they need to be carefull when they dig.
We have now put up a press release. Also, we have created a static copy of The MIDS Internet Average front page showing the period of the outage.
if fibre lines were marked the same way gas lines are marked in cities. hmmm....
I'm a loner Dottie, a Rebel.
I think if the day after a nuclear exchange you're worried about whether or not you're going to be able to read your email, then today's society's priorities are more than slightly skewed.
I doubt it... a few years back, Telstra controlled 90% of AU's incoming bandwidth, at approximately 127MBps (don't laugh, we only have a population of 19 million for an area the size of the US - the most sparsely populated country on earth outside Antarctica. Nowadays, with the springing up of Optus (C&W), AOL, and a few other big names, i think our capacity is approaching 400MBps.
Open Source. Closed Minds. We are Slashdot.
I shouldn't mention the name (*sigh*, clue, huge multinational with common two letter initials, who make everything from calculators to computers and medical equipment. Oh, and it isn't TI), but I used to maintain large WANs for our clients - one had a network of approximately 250 ISDN lines around Australia, some very long length. Uptime was superb (99.985%+) and most outages were, as the above said, cable cuts (and powerfailures)
Open Source. Closed Minds. We are Slashdot.
I used to work for a construction company laying new fibre optic cable. I did the calling of all of the under ground utility owners. (very tedious) But, my boss kept on telling me not to screw up, because if I messed up the cost for 1 (one) fiber cut is $100,000. WOW that's too much.
It already can be a major problem with regards to private WAN links. I've seen more than one occaision when a location on a WAN loses connectivity due to a fibre cut. It isn't pretty, and it makes my life interesting at times ;-)
No WONDER I could not get onto slashdot at work today. heh. this reminds me when someone made a ghetto fire under a bridge somewhere in Minneapolis(I think), cutting most connectivity off for the twin cities. I think it was through MR.NET. anyone remember this a couple years ago?
If you are interested in this stuff, look at The history of the Internet for fuller details. But the basic story is that the idea of packet-based networks arose independently in two places. The first was Kleinrock et al at MIT, the second was Baran at RAND. The former group was interested in them as a way of efficiently sharing the same lines between many different computers. The latter was interested in them for creating communication systems that could survive nuclear war. The two groups did not know of each other.
The Internet arose out of ARPANET which was based on the work at MIT. The goal was to allow computer resources at different research institutions with different types of computers to be shared. The fact that surviving nuclear war was not a goal can be seen in the fact that the machinery used to set it up had no protection against the electro-magnetic effects of a nuclear warhead. Furthermore the initial set-up heavily relied upon a single back-bone. With no redundancy in your physical network, what good is a redundant protocol?
In fact the initial proof of concept and then proposal for ARPANET was made before the MIT people even heard of the work at RAND. Indeed the two groups found out about each other at a conference where the ARPANET was being proposed. Don't believe me? Check it out for yourself!
Cheers,
Ben Tilly
My usual seat in the cluetrain is at A HREF="http://pub4.ezboard.com/biwethey.ht
yah, i was upstream from y'all, your uti engineers are really persistent. but it didn't get them fixed any faster:)
Everybody seems to be complaining about the fact that things are slower today, but what about the fact that 40Gbps is now being rerouted through other means. That's pretty damn impressive. Sure the people who snipped the lines are idiots, but the whole net didn't come crashing down.
Anyway just my $.02
>Wouldn't you love to have 40gig of bandwith piped into your home? ;)
:-)
A friend of mine does - he runs an ISP that only resells
bigish pipes (frame relay, T1 etc) out of his basement.
He claims to have more incoming bandwidth to his house
than Australia
Why is it that IP trunks seem to be backhoed at the drop of a hat, while phone lines are somehow immune? Why doesn't the phone line between Boston and San Francisco go down?
:)
answer: phone lines are not immune. they get cut all the time. however, the phone network, like the internet, has redundancy. whe na phone line gets cut calls are rerouted to other trunks, making the problem oblivious to the layman.
so, why do fiber cuts turn out so much worse? because the data network is bursting at the seams. a voice line cut can be directed to ununsed lines. a data line cannot be done in the same manner, because there are no vacant lines, or at least not as many as were lost...
why is this the case? one, because the data network is expanding faster than new fiber can be laid down. we're too big for a britches. also, the phone network, which has grown at a decidedly slower pace, has, at both the users' and phone/data companies' requests, deemed to be more important. let's face it, if a phone line went down to a company hq, they'd be lost for as long as that line was down. just now are people realizing that their data line, is now becoming more important than their voice line.
Well, datacom companies should team up with gas companies and put the fiber right beside the gas lines. That way everyone would be scared to dig near em and thus no busted fiber lines :)
Enjoy your job, make lots of money, work within the law. Choose any two.
In the day and age where you can sue your own parents for not making you clean your teeth properly when you were an infant, I'm sure you can sue them for something..
Merlin --- We're an autonomous collective... Help, Help, I'm being oppressed!!
Caching proxies? Get 'em installed at your ISPs!! In situation like this...it can help lower the burden a lot - by caching the pages for the people who browse, you save bandwidth for people who have to telnet, shop, or post at /. I think I'll get mine.
Of course when nuclear war hits it will take at least 3 hours to fix ....
Gates' Law: Every 18 months, the speed of software halves.
My company's T-1 line has been cut at least 3 times in the last 5 years.
This kind of thing could be a MAJOR problem as the internet becomes more important to big companies. I mean right now, sadly, it's still a novelty to many of the people in the world. But can you imagine the hell we'd be in for if everything travelled over the 'net and someone cut a big trunk like this? Yikes.
Werd.
...that would explain why /. has been slow all day long. Good to know it's not our pipe.
On a related topic, aren't lines like this clearly marked? I mean, how tough is it to call the number on the sign and see where the line is?
Or did I miss something?
censorship is a form of noise, which actively seeks to drown out content with silence - Crash Culligan
The first comment was mildly funny, but yours is just wrong. Of IQ is 80%+ genetic in most populations (which is true -- the lowest estimate that anyone reputable, i.e., everyone except rabid Marxists, suggest it 65-70%), then how is it statistically invalid? Methinks that you and industrial psych have not yet met. You will find him interesting, albeit inflexible.
Actually, for crude measures like IQ, you can breed people just like dogs. And because you are almost always dealing with a far wider and (in the US) more diverse pool of genetic material, inbreeding is pretty damned hard, unless you are from a very closed section of society (I was thinking of Boston and parts of Connetticut, not Arkansas, actually). You get people no weirder than before, but, after a few generations, regressing to a far higher mean IQ. One of the more interesting parts of looking at the effects of post-WWII college education and population mobility is that this seems to have spontaneously created a breeding class of people that has actually seperated out a bit from the American population as a whole. Add grad school, the effect becomes even more profound. Add money, and you wind up with similar self-segregation. If you look at population patterns in the US, for the most part you are seeing (outside of some large US cities) a desegregation of neighborhoods and later (about a generation later) marraige patterns and a realignment solely along income lines. The thing that sorts is money and the filter seems to be higher education, and within that the better and worse schools and the amount of work beyond a BA/BS. This is going to have some interesting effects -- black intermarraige exceeds 30% in most urban areas and the children generally show no racial/ethnic preference (yeah -- flame away -- the control is the white mean, so sue me) so within 40 years or so you will see black people essentially start to disappear as a seperate and identifiable ethnic group.
Pop stats and industrial psych are fun. Unless they aren't doing what you want them to, in which case I am sure that it can be hellish when reality refuses to cooperate.
And looking at the Inter@ctive week article, ETIC of **ONE HOUR** ??? To repair fiber ??? Or should I say, to repair that much fiber ???
Backhoe, Backhoe, Digging Deep
Make the Backbone Go To Sleep!
Muhahaha
Why is it that IP trunks seem to be backhoed at the drop of a hat, while phone lines are somehow immune? Why doesn't the phone line between Boston and San Francisco go down?
After the big MCI outage, it's time the phone companies put big red signs on OC-* pipes saying "NEVER cut through this pipe". I take it the gas company isn't about to pay for the lost and delayed packets.
hahaha i didn't know that.
i wonder if they have a sign saying
"dont smash into this wall or you will break the internet"
and if they dont, they should.
tyler
Looks like I'll have to give up my online Q3 game tonight. I wonder if I've got any hardcopies... I mean books... around. :-)
What's all that wet stuff outside?!?
-- IANAEG - I am not an elder god.
You are correct, this would not fall under criminal
law, but the next best thing in the civil arena: Torts
Any act, intentional or otherwise, that does "harm" to another.
Oh, and don't forget: if someone kills you, not only can you charge
them with murder, but sue them for wrongful death and get a wad of cash.
"If you kill me, I'm gonna bleed your bank account dry, buster!"
I don't know about y'all, but our connection doesn't seem to be any slower here. That poor bastard with the backhoe is luck he didn't cut that line around these here parts. We'd have his hide! Hell, Billy Bob'd probably string him up! Imagine not being able to get on EBay to see if he won the auction for the entire video tape library of the Dukes of Hazard. The horror!
All opinions expressed with tongue firmly in cheek between the skoal and the ceegar.
Wouldn't you love to have 40gig of bandwith piped into your home? ;)
Is there a law against this sort of thing? I mean, can those who did it be arrested? They should be...
And what about lawsuits? I know that if I ran an e-commerce site who's traffic was affected by this outage, I'd be pretty pissed about the possible loss of sales. Is it possible to sue the utility company that did this for something like that? Or do you just have to chalk it up to bad luck and move on?
"I'd like to live in theory, because everything works in theory, in theory." - Can't remember who said this.
Now we know that the "vehicle" in "multiple independently targeted reentry vehicle" (MIRV) is actually a backhoe, rather than a nuclear warhead. I bet the guys who designed the 'Net to be nuke-tolerant are feeling pretty damned embarrassed right about now.
Some people think the cold war was won by USA outspending USSR. But the real truth is that someone finally leaked that we were building bombs rather than just backhoes. Ivan's pants must have gotten pretty soiled at that revelation. Just think: all along, we totally misinterpreted what "We will bury you!" meant.
---
Have a Sloppy day!
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Now that's what I call Slashdotting!
_________________________
Words of Wisdom:
_________________________
Words of Wisdom:
Never pet a burning dog.
Correct, but as the other poster mentions, it just gets re-routed. But stop and think for a minute.
... --- ...
Power grids and telephone circuits can be affected the same way, take out one of those big "power towers" that traverse large spans, and a couple of remote stations, a few satellite uplinks, some telco switching stations/relay towers, and havoc *will* ensue. It doesn't matter if it *can* be re-routed, the resultant chaos and downtimes would cost probably millions. Cyberwarfare isn't about information, it's about $$$$$ lost when the infrastructures disappear.
Then we'll be falling back to all the guys with their ham sets.
Blech. Signatures.
Kudos to the inventers of the internet which allows communications between computer networks to be automagically rerouted when a path is severed by a nuclear warhead or a trusty backhoe! :-)
And I aint talking about no Al Gore either.
Hasdi
I work in London (England) and earlier today a colleague told me that he had tried pinging a site in the US. He claimed that the packets were making it across the pond OK and then were being routed via Australia. We didn't believe him. Guess that I'll have to apologise tomorrow.
Yellow tigers crouched in jungles in her dark eyes.
She's just dressing, goodbye windows, tired starlings.
HA! I was WONDERING why things were slow to my ISP today! Those OC192's are probably the ones that Columbia Gas has pipelines running alongside, on I80. (Ha! You thought I wouldn't name the gas company!) IIRC, ICG laid the first OC48, and MCI/MFS as well as Sprint laid more, and more, and more. They're doing a LOT of construction along I80 from what I hear, so it's NOT surprising that a cut occured. Those lines are on a gas-pipeline right-of-way, and there are 5ESS Demarcation points along I80 for 'em.
:)
There've been previous fiber cuts that resulted in me passing a group of 15 MCI/WorldCom vans on my way home. I betcha if it's the link I suspect, they've got the onramp *packed* with all their techs trying to explain to the people staffing the tollbooths that they're with (insert-company) and they're in on an emergency call and they'll be getting right back off! *LAUGH!* Gotta love the Ohio Turnpickle, eh?
-RISCy Business | Rabid unix guy, networking guru
your company here.
shelby != ford
out of curiosity, why is mae-east, such an important box, located in a parking garage? amd who maintains it?
The conservation goes a long way part is VERY important! moderate this one up! if more people just used their archived pr0n/warez/mp3s, we'd all be a bit better off over the next week(or however long?) that it takes to get it back together....
Dan
The MIDS Internet Average shows the effect of the fibre cut in the context of the Internet as a whole.
Actually, although these cables may follow major access routes, Sprint initially created their fiber network along the right-of-way of the Southern Pacific Railroad, as it was originally formed by the SP. (Hence the SPR in SPRINT as a previous poster noted; INT is probably for internal or something). Considering the SP doesn't run to the east coast, they either use roads or some other railroads tracks. As for how do I get there, there are trucks with rail wheels. But, overlay a rail map on an interstate/US/State highway map some time. You're likely to find that many of the major intercity rail lines correspond quite closely to the major highways, especially those portions of the US highway system which was build, for the most part, alongside the rail lines. Where they separate, at least along Sprints network, I bet the fiber follows the rail and not the highway.
Least your 14.4K can still connect with this slowdown....it connects, but it goes SOOOOOO SLOW....even for a 14.4k to the point where i have to look at .\ in the library. -btw....anyone have a 56K external for sale? buono@erols.com
It's unfortunate this guy didn't slice open some underground power lines and zap himself and his backhoe ... He wouldn't be around to cut it again when it's time to harvest the potatoes.
-Rob
terpmotors.com
Looks like those guys finished digging that hole right about lunchtime, eh? By far the biggest change I've seen on the ITR. Kind of funny how a backhoe could slow down time (internet time that is). Anyone have links (or know offhand :) to a total M/G/T|bs per day index on total Internet Traffic?
+&x
Marge: I really think this is a bad idea.
Homer: Marge, I agree with you -- in theory. In theory, communism works. In theory.
Episode 1F15, "Bart Gets an Elephant"
dopplar effect.
BOFH, ah, how I miss thee.
www.pushove.com new's for jack asss'
when Push Comes to Shove
see, why the hell were these cables running through what was some chicken feed corn field in Ohio? Who is the idiot that didn't mark that there was a HUGE FRIGGIN' CORD underground that probably cost a small fortune? Oh, and how much is the replacement construction and infrastructure going to cost? Geez oh man. See, that's why I like Qwest. All their cords are along train track lines where they're the only ones that are allowed to dig and even if something is cut, they can just pull another cord through the piping.
So lets see, in the past month or so, we've had a problem with MCI WorldCom and UUNet (correct me if I'm wrong) and now some backwater public works moron who was probably driving the backhoe with the blade down on his way to a coffee break killing some serious piping. Hmmm... what if we have a cataclysmic earthquake that splits North America in two by 3 inches... what would happen? Would all of that fiber optic cable stretch or would it pop? hmmm...
Looks like /. is on the other side of the break from my location. This packet travelled all the way around the globe to get here. Anyway, there are only three possible companies responsible for this disaster. Cincinatti Gas and Electricity, Columbia Gas or East Ohio Gas. Each is equally likely, as Ohio has some sort of 'consumer choice' law that allows you to choose your provider (who is in turn responsible for your particular gas feed) My bet is on CG&E, as they seem to have the largest consumer base and the mose infrastructure under their control. When I figure out which one it is (I'm calling each now) /. .
I'll let you all know.. Let them feel the weight of
.sig: Now legally binding!
My own curiosity, which I'm sure will be my downfall someday, directs me to ask how on earth one actually fixes such an expensive piece of cable. I always fix ethernet wires by just re-stripping and connecting the RX and TX pairs anew; what's it like for fat pipes? Duct tape?
It's probable that someone reading this has fixed a broken line before. Raise up... you know who you are.
My father, brother, and all of my paternal uncles are construction workers. My father is a backhoe operater, and the rest are laborers, of which I was every summer until I graduated from college.
I wouldn't insult them by comparing the job we did to typing on a keyboard. Try standing on new, white concrete in 107-degree heat.
I was lucky enough to get an education and out of the family rut, and I never take that for granted. The minute I even think of complaining about writing code for xx hours a day, I just remember the days I worked the same hours shoveling sand in pouring rain. If I came home complaining of carpal-tunnel, I'd get smacked in the back of the head, and rightfully so.
PLUS I WAS at the edge of a hole when a backhoe cut a gas line. Luckily it didn't ignite, but the burst and hissing scared the shit out of me. Realizing I could have been killed in an instant at 18 because of a shitty job made me want to get out more. These men do risk their lives and often have no way out.
A slowdown or inaccessibility of the Internet is an inconvenience, no one gets killed. We all need to get our priorities straight.
"More organs means more human." - Zim
Well, this probably would not really work on a national scale, but I always have trouble with my phone lines, whenever we have landscaping or any sort of digging done, my phone lines get hit! so we dug down and layed a row of bricks accross them for the duration of the yard.... No more cuts so far ;)
(And yes, I've done a bit of construction work myself...)
No Laughing Allowed!
Slighty off topic but a good tale: My father has been working for a national European broadcaster for a long time. Decades ago when they were setting up a new television and radio campus they had to run some data lines out to the techies' new buildings. He and his colleagues indicated the line where they wanted the backhoe to dig and went to lunch in the canteen. About 20 minutes later all the lights went out. Somebody had neglected to put down any markers and the backhoe had cut through the main power cable *after* where it was joined to the back-up generator. As national braodcasters in Europe also double as the goverment's Emergency Broadcast System this caused a lot of people way up the chain to get alarmed. While the cable was being repaired they had to bundle a crew down to the old emergency studio and transmitter in the General Post Office (Think Krusty the Clowns broadcast in The Simpsons).
"Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
Actually, my corporate office is in Texas and this is directly effecting our national WAN traffic. My Texas based IT department is catching shit because the people upsatirs seem to think this is our fault. Can someone with some insite remind me how a gopher can bring down legs of national WAN by chewing on the backbone cable and it can still be my fault . Oh yes, you can also add the list a fiber line in southern Florida that got hit by a construction crew this morning. Those Damn construction crews need to be more careful with my data. Days like today are not worth getting out of bed for.
"Help me Obi-/.-Kenobi,your my only hope!" -$
There is now a thunder storm in that same area that the lines are down. The company that is fixing the lines has stated that a storm is keeping them from fixing the lines. They have no estimation of when it will be fixed.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
just for all you to know, themes.org is being hosted on abovenet (one of the severely affected isps) and i can't currently reach abovenet's dns servers.
sum: themes.org probably will be down until this is fixed. please be patient.
-- adraken
People fuck up. You can't arrest them all, or everyone on the planet would be in jail.
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
I work and play here at an Ohio university (Case Western Reserve if any of you care). Net access here is terrible. The fastest transfer I've managed has been a meager 120 kbps to anywhere in the world (considering we have OC-3 out, things are usually quite peppy).
I really wish it were possible to see an accurate weather report on how such things affect the rest of the 'net. It's difficult to interpolate results from very few data points. Of course, the feasibility of such a system is quite difficult and prone to error.
In either case, I'm advising everyone here to be nice to the 'net. There's no reason folks need to be downloading their pr0n, mp3z and off-site ftp installs of Linux during something like this. A little conservation can go a long way.
Actually, tcp/ip over ax.25 (ham radio data) is great; the 44.0.0.0/24 range is reserved only for amateur radio networks!
a m_Radio/ and take a look at the listed sites.
As soon as I have enough money, I'll be buying a second tnc (terminal mode controller, more or less a modem, with packet "stuff" added on), to connect to my handheld 2m/70cm radio, and my newton, so that I can do wireless internet from my hand (and much cheaper than a cellphone after a few months).
There are several great places to begin seeing what packet radio is all about, if anyone's interested you might drop by http://www.frostnet.advicom.net/chris/bookmarks/H
I believe the conversation during the cut went something like this. "What the hell, I just got ahold of a whole buncha fishin line, in tha ground. Who'da barried some good ole fishin line." "I dunno, but it sure is purty, I think we needta pull soma it out and take it with us, whaddaya think cleetus?" "Sure, lets pull it out... wait, here comes the boss." "What in the holy hell? What did you 2 dimwits do, do you know what the hell that is?" "Fishin line, we wuz gonna take sum and use it tommorrow." "Oh hell, it's not fishing line, see, these orange marks mean that you don't dig through here. Thats 4 OC-192's, you idiots" "heh, whats that? sounds kinda technical." etc, etc, etc.... Who trains these guys?
Gee. I don't recall anyone posting this kind of message about sysadmins.. I mean, how many of us have been paged or called at 3 AM to run in and reboot a server, router or something else. (And it's certainly not WORK that my PHB is doing at 3AM... :P )
-Chris
I am not sure if it is just me, but why aren't these things marked? I still remember a few months ago when slashdot was shut down for several hours because of the friendly construction worker deciding to have a little too much fun with the tractor. If it isn't power, it the phone line. One solution can be found in the summer issue of 2600. The article is on ground based networks and it does make for interesting reading.
That's 9 kW total power to run multiple amplifiers on a very long cable. They insert an amplifier in the cable every N kilometers.
Mea navis aericumbens anguillis abundat
In Finland we have made the opposite. In the high-voltage power lines there are two lightning-conductors on the top of these three conductors actually delivering the power.
Now at least one of the lightning-conductors has a fiber line in it, which is then used to build a fast IP-backbone. The cable gets dozens of direct lightning hits every summer, but without any service degradation...
I spent my summers laying water and sewer pie as an undergrad. Good work, really. Some people even worked there rather than have an office job, even given the 30-50% pay cut involved.
As for cutting lines, the charge is well over 1 million dollars per minute of downtime per line. No maximum. So, the company that cut the lines will go out of business. The worst of it is, the phone companies do a terrible job locating their fibre wires, and then sometimes don't even cover them--so it just takes a shovel to clip the whole line. Worse, they tend to be the worst at breaking everyones water and sewer lines, and sewer is much less fun to fix than fibre.
I don't know if this is that reference, but the maps they have there are cool nonetheless. All sorts of stuff, going back to the "original" ARPANET map.
You'll have to pay them that before you can find out, I guess.
Besides.. you can get more money from the company than you can an individual.
Who saw, 'massive fiber cut' and thought that, er, someone was changing providers? And wondering why it was done in the middle of the day?
"But that cut wasn't scheduled for today!"
--- Where's my X.400 protocol decoder?
The severed end of that fiber gives a whole new definition of piping output to /dev/null.
--- A Jesus Fish eating a Darwin Fish only proves Darwin's point.
About 8 years ago while working in Asia in a country to be left unamed (to protect the guilty). We had, had about 20 line cuts in a two week period. All within the same five mile stretch of Highway. It was because local farmers whereout backhoeing their field dikes to prepare for the next seasons planting as well as crews from Highway construction. We kept repairing the cuts in the fiber, putting larger and larger signs and underground tapes all in the local language (none of which helped they dug where they wanted too.) One of our local co-workers came up with a solution to the problem. We place signs along the stretch in ENGLISH only. The locals, and highway crews became so sure that this was some special secret government project that over the course of the next 6 months we had only one cut in our area of responsibility. *sigh* Kinda like my physics prof who said the only thing fibre optics would be good for is "hippy" lamps and nothing more.
I'm sorry, I'm to tired to be witty at the moment so this message will have to do.
.... but whatever did happen to the BOFH? I noticed in about two months ago it wasn't going to be published for a while and it never came back.
As someone who did construction for a couple of years, I can tell you that while you might think there are maps that can tell you where lines are, they are almost always wrong. I've seen clearly marked lines on a map that were more than 10 feet from where they were supposed to be when you get in the ground and start digging. If the original diggers used flags to mark the lines, that makes it a little easier but even then it's like looking for a needle in a haystack but the needle has a string around it. A little better but not much.
About three times in the past few years I have experienced a 'net loss' due to back-hoe operators. How many 'call us before you dig' commercials does there have to be before people stop cutting wires. It is just a silly expense which is naturally passed down to consumers. These people should be wearing wireless net devices which can tell them where all pipes and trunks, etc are running through their work site so this can be avoided. woo. wonder how much this repair will cost and how long it will take. and how long it will be until another company does the same thing.
burn the computers. go back to the abacus.
it was IDIOT proof. Just nuke proof.
DNS, for example, would have a very high priority, and be one of the last services to drop. Without DNS, the network becomes significantly less usable. The services designed for text communication also would have a high priority: smtp and the assorted email services (no attachments), nntp (again, no attachments), finger, time services, gopher, and the like. Even http might be allowed through, but filtered by mime type (text/plain, text/html, x-www-form-unencoded, etc.).
The problem here is that you couldn't realistically do much more than filtering by port, as its not possible for a router to reliably know it's routing enough of a connection that it can filter based off that (and for that matter, it's most definitely not reasonable to have big honkin' overloaded routers take every packet, decode, and analyze everything that's coming through).
It would also be difficult (though conceivably possible) for a local router to be doing this. Trivial example: email attachments are encoded as MIME. But you don't know whether a message is being sent MIME encoded or not until after the SMTP client and server go through their little authentication dance. For a stateful protocol like SMTP, there'd be no way to "ban" MIME unless the router was "listening in" to the whole conversation, and rudely cut it off somewhere in the middle. For stateless protocols like HTTP, the router would need to capture the entire outgoing before it's sent out, to evaluate it, i.e. if the query is broken up into 3 packets, the router would need to capture all 3 packets, reassemble them, and parse HTTP, in order to determine the content type.
What a day to decide to grab the new netscape! I thought everything seemed so slow. But in all seriousness, those must be some big backhoes that cut the connections because I figure those fibers are several feer in diameter.
--
Scott Miga
Something like this seems to happen every 8 months or so. Is there any good way to stop this? Aside from moving to wireless, the only thing I can think of is not put four high-bandwidth cables under the same patch of street - but then laying cable (er, fibre) becomes expensive proportionately to how many baskets you're keeping your eggs in.
Trees can't go dancing
So do them a big favor
Pretend dancing stinks!
Good thing there wasn't anyone around there smoking a cigarette. What with all those loose bits sloshing around, the slightest spark could have set off that 'internet explosion' people keep telling me about.
Trees can't go dancing
So do them a big favor
Pretend dancing stinks!
Sorry guys... my fault. My father once told me that he buried all the valuables in the backyard for "protection." He really thinks Y2K is going to leave us in a Mad Max nightmare. So anyway I tried to dig them out and apparently I cut the lines. Sorry.
--
RumorsDaily
*shrug* I've operated heavy machinery. (quite efficiently, I might add)
:>
Though I'm not competent enough to pick up a hardboiled egg
with a backhoe... I am competent enough to dig fast holes
and put the dirt where you want it.
IF LINES ARE NOT MARKED, the operator has NO idea what's down there.
Trust me on this. The job is laid out.. you need X amount of dirt moved
over to spot Y. What do you do? You move that dirt, and fast.
(unless it was part of his job description) the operator was more
than likely digging a hole that was laid out for him
by someone else entirely. (only between the blue X's Ray)
Now if the kid/woman/person/guy/whatever was in charge of both? Execute them.
"Call before you dig" Pretty simple, eh?
Seriously though... there should be some tough penalties extracted from the companies that allow
their workers to pull blunders like this.
Remember... it's people like this that take forever to
process your order at your local
burgerdoodle --- Insert favorite fast food resteraunt name here.
Friends don't let friends buy Compaq's. (Dell/Gateway... same same) You want a good computer? Build it yourself.
you know this is when that annoying commercial
"go for the phone and call blue stake" comes in. maybe this commercial isn't everywhere but the midwest people know what im talking about.
makes you wonder what happens if mae-east gets destroyed by some natural (or unnatural) disaster.
oh how much we depend on these plastic boxes!
tyler
I would say that this goes to show the utter bullshit that is the whole cyber terrorism thing. Why spend billions of dollars trying to police imaginary squads of crackers set to destroy our information infrastructure, when a couple of idiots with shovels can create major mayhem like this?
I wonder what an organized group of wire cutters who did a little bit of research on their targets could accomplish. I have a feeling it wouldn't be pretty.
I can't say I noticed anything myself (the net has been dog slow for me as long as I can remember, so), but if a small event like this can cause major problems, then the Internet is definetly closing up on critical mass....
-
Most of the time the telcos themselves doesnt know
where their fibre is. And the biggest accidents of
this kind actually happen when there is telco people there, they just thought the cable was 5 metres to the left.
Now, we can't count on the users cutting down on their bandwidth use conscientiously.
:-)
DNS, for example, would have a very high priority, and be one of the last services to drop
Consider dropping DNS first - loosing all the clueless will certainly free up bandwith!
Well, the powering issue being talked about isn't exactly what would be called "common". Most fiber in the ground today has no current-carrying conductors. True, it has a metallic conductor or two in it, and in some cases, a metallic sheath or armour, but carries no exectricity. The metal conductor is used exclusively to locate the cable with inductive/RF location gear. When it comes to re-generation of fiber "signals", byt simple amplification, or a full teardown of the bitstream with drop/add functionality, it's usually done at a special location (in-ground vault, above ground fiber hut, whatever) with external power. That's power supplied by a power company, with backup power. Since fiber traffic routinely travels 50+km without intervention at all, it's a total waste of resources and cable to lay a power cable for equipment. Now, all said, there are exceptions where there is no power readily available, like deep-sea cables, deserts, and such where there may be a power cable present in places. But in general, not present. Now, back to a little more on topic :) As for location, depending on the area, this can be tricky business, big time. In dense metropolitan areas, there can quite literally be cables a foot or less apart "horizontally" as well as several layers deep "vertically". Location of cables isn't always as precise as it ought to be, especially when the phone companies records don't reflect actual construction oddities, like perhaps a four foot fiber "loop" at the base of a pedestal. Well, when the person locating cable gets within a couple feet of a pedestal, you'd just assume it goes right in, right? Wrong :) And another company plans to put a pedestal right next to the existing one, and, well, it's time to break out the Scotch-Lok or quick-fix fiber kit :) Granted, most cable and fiber cuts happen out of pure ignorance or stupidity. Farmer Joe is out in the field fixing a drainage tile with his tractor and doesn't give a single thought to what else may be in the ditch with his tile. Where did those kits go again? :) And as a final note on the buried cable thing, nowadays, it's becoming common, almost required, practice to bury a brightly colored plastic ribbon above the cable being installed. Usually yellow or orange, it's really easy to see against black dirt, and would hopefully be seen before the contractor hits the cable itself. Doesn't do much good in the case of boring or knifing cable in though. All of this is just general information. The cut out east could have been done in one of a dozen or more other common contractor cable goofs that I won't even try to speculate on. Either way, it boils down to this - contractor started digging without requesting a locate, or if the cable was properly located, and the contractor either mis-read the location markings, or ignored them, that contractor had better hope he has a good insurance company. If the cable was not located at all, and a locate request was made and recorded in a timely manner, usually 24 to 48 hours before digging, well it's the telco's tough luck. You know, it's a small wonder that things work as well as they do considering the unimaginable number of perils out there, backhoe's included :) Sorry this is so long! FBG the AC
But now the Internet is a commercial enterprise, and failure is now an option. At worst, some large corporations lose their VPNs and have to prioritize and pick up the phone again. For most of the net it's just a matter of losing porn, IRC, and MP3.
Now if you're a backbone maintainer, do you double your capital costs to achieve more than minimal redundancy just to give the public a warm fuzzy feeling? Or do you maintain the least expensive network you can without losing customers? Market forces will drive the QOS on the net to the lowest tolerable level, and for now people will tolerate a lot of net failure because their lives and livelihoods don't completely depend on it...yet.
Given, they have a tough job. So what, they get paid to work like dogs, they aren't a chain gang for crying out loud.
:-)
Around here, they put commercials on tv all the time saying, "before you dig, call Miss Utility" and flash an 800 number. So you mean all us yokels have to check in with the utility folks but the utility folks don't have to check themselves?
Sure accidents happen, but I can't see a public utility diggin around and not knowing where their own lines are. Sheesh.
The Divine Creatrix in a Mortal Shell that stays Crunchy in Milk
The House Between - Original Sci-Fi Series
Now, we can't count on the users cutting down on their bandwidth use conscientiously. How, then, can we keep the critical services running? For a start, we need to define "critical services". I'll say that the greater the ratio of content to bytes, the higher the priority of a service. The only practical way to filter packets by service is to filter by port. You can run a filtered service on any old port you want, but the goal is not to prohibit services so much as drastically reduce the bandwidth used so that the network remains usable.
DNS, for example, would have a very high priority, and be one of the last services to drop. Without DNS, the network becomes significantly less usable. The services designed for text communication also would have a high priority: smtp and the assorted email services (no attachments), nntp (again, no attachments), finger, time services, gopher, and the like. Even http might be allowed through, but filtered by mime type (text/plain, text/html, x-www-form-unencoded, etc.).
Still, there would be a significant drop in the usefulness of the network. We need more bandwidth than we need to ensure reliability. Make bandwidth, not war! :)
Gates' Law: Every 18 months, the speed of software halves.
The net seems to be back to normal speed for me at least... This kind of thing happens from time to time.
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
Except that eugenics is statistically invalid.
Free Mac Mini. Yes, I'm
It amazing how our bandwidth requirements have increased so fast that 99% of all our modern communication is carried through only the newest lines. What would happen if the phone companies stopped increasing bandwidth for just a second?
We never had this problem back in the days of good ol' POTS service. Good old copper wire never cut cut or went dolkjsd;flkh#&&(
NO CARRIER
DrLunch.com The site that tells you what's for lunch!
I work at an internet service provider (UTI) which is being hit by this right now. I hope it gets fixed quick since the tech support calls keep rolling in.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
How much to do you want to be the backhoe operator will be driving the same car 10 years from now?
Well - that would explain this:
:-(
1 198.182.167.17 (198.182.167.17) 0.628 ms 0.564 ms 0.532 ms
2 adsl-63-194-218-254.dsl.snfc21.pacbell.net (63.194.218.254) 26.688 ms 29.433 ms 15.868 ms
3 core1-fe4-1-0.snfc21.pbi.net (206.171.134.209) 13.394 ms 13.701 ms 15.957 ms
4 gsr1-g1-0.snfc21.pbi.net (209.232.130.20) 16.098 ms 13.846 ms 15.215 ms
5 sfra1sr3-so-1-1-1-0.ca.us.prserv.net (165.87.161.74) 16.676 ms 16.208 ms 15.202 ms
6 sfra1sr2-11-0-0.ca.us.prserv.net (165.87.13.17) 16.684 ms 16.286 ms 15.226 ms
7 above-advantis-ds3.sjc.above.net (216.200.0.81) 21.833 ms 24.418 ms 29.271 ms
8 core1-core4-oc3.sjc.above.net (216.200.0.85) 25.864 ms 23.882 ms 17.173 ms
9 core2-core1.sjc.above.net (209.133.31.110) 47.734 ms 39.032 ms 39.107 ms
10 main2-core2.sjc.above.net (207.126.96.138) 44.072 ms 40.088 ms 40.069 ms
11 core2-main2.sjc.above.net (207.126.96.137) 42.369 ms 44.488 ms 46.094 ms
12 * * core3-core2-oc3.iad.above.net (209.249.203.65) 590.011 ms
13 abov-core1-mae-east.netaxs.com (209.249.119.234) 695.855 ms 721.634 ms 848.819 ms
14 dn-netaxs-gw.dc-core.h5-0-45M.netaxs.net (207.106.127.94) 1125.874 ms 1282.923 ms 1408.877 ms
15 h900.ca2.wdc.dn.net (209.207.190.5) 1263.551 ms 1250.176 ms 1252.700 ms
16 209.207.174.23 (209.207.174.23) 1234.975 ms 1256.689 ms 1208.096 ms
and worse yet DNS lookups from here going thru Vienna
But remember folks there's redundancy in the backbone routing but when something big goes down
everyone else gets to suffer as the traffic
gets piled on top of their usual connections.
If there's a lot of traffic going thru Europe I
bet they're getting really steamed over there
Hmm. Isn't a prerequisite of a redundant connection that it is not affected to ailments of the primary connection? Whoever sold you on a redundant connection on the same cable I think ripped you off. :>
- Darchmare
- Axis Mutatis, http://www.axismutatis.net
- Jeff
How exactly does being able to remember IP addresses make one clueful ?
We recently had fiber installed to out building. We have ISDN as a backup. The fiber travels above ground quite a ways and then goes underground. My boss took the time to check out both the fiber route and the copper one. They only cross at one pole. If that pole goes down . . .
--- If you don't want to know the answer, don't ask the question.
Unless the lines were in some serious sheathing, most backhoes wouldn't even twitch going through it. If the operator wasn't watching the hole close (most tend to be more concerned about where the bucket's going) he'd probably never notice it.
As an aside, I once worked with a gentleman who could knock your hat off with the bucket of his hoe (I held still, most he tried it on wouldn't). I saw him pick up a boiled egg in the bucket without breaking the shell. That kind of trick requires a level of skill not equalled by most airplane pilots. Those machines are *not* that easy to operate, and idiots don't tend be allowed on them.
The Backhoe
Natural Enemy of the Network Administrator
http://www.23.com/backhoe/
which is probably the downtime cost for this
cable -- someone is going to be sued for
a number with a lot of zeros in it.
Anyone know whose cable this was? Just
from the "who got whacked" it looks like
it belonged to Cable & Wireless, but I haven't
heard anything for sure.
garyr
-- your Web browser is Ronald Reagan
Hell, gas doesn't even slow 'em down. When I was at UNH, they were rebuilding the school library. They tore into a gas pipe no less then three seperate times during the construction. Twice at the same spot, from what I understand. I was only present for one of 'em. It was kinda freaky to have the fire alarms for half the campus going off at once...
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Do you honestly think that if they have the possiblity to destroy something they have access to will make them be more careful? Will giving them free Net access make them understand the data infrastructure? People who pay for the service don't understand it (my Mom still doesn't quite understand why you can't pick up the phone while on the net). Should we give all people who have a job where something might get ruined free use of it? Why don't construction workers get free cable, free phone service, and free electricity along with the free ISP?
If I'm at work, and I'm walking by a UNIX box and accidentally trip over the Ethernet cable and disconnect it, should I not be responsible for that? I'm not the UNIX admin. Perhaps if they gave me root I might not trip over the line.
that you quoted from at. html
:-)
http://www.wired.com/wired/archive/4.12/ffglass
it's 56 pages long. Took me 3 days to read.
Until I read it I didn't know that subarine
cable was interesting. Really good and worth
reading the entire thing. The guy that wrote
it (Neal Stephenson) may just turn out to be
a good writer
garyr
-- your Web browser is Ronald Reagan
I checked out the Internet traffic report to see what kind of effect this harmless mistake had. North American traffic statistics
Keen! Can you spot the time the big bad backhoe operator cut the cord?
Construction/Utility repair guys don't have an easy life. They're the ones out there in the middle of the night when your power line has gone down, or the ones fixing that broken sewer pipe, or the ones who make sure you have water. Near here they're doing major construction over on Grand Avenue, and to avoid traffic they have to do it in the middle of the night. They accidentally cut off a hunk of phone lines and about 1000 people had no phone service, but that happens pretty rarely.. I'd like to see YOU guys doing hard physical labor all night or all day trying to avoid speeding cars and a maze of pipes and cables and never screw up occasionally.
I wonder if this has anything to do with the slow connection rate that I get when I try and telnet into my webserver that sits in the next town over from where I work. After all, when I do a traceroute, I get shipped through machines in Chicago, SanDiego, and other exotic places. And I'm in South Carolina. A month or two ago, there was a slashjot about some guys trying to map the Internet. I think that it would be easier to map a world covered in spaghettti noodles, because that's what the Internet looks like to me!
Brad Johnson
Advisory Editor
Brad Johnson
I do TS for Juno, and thankfully my shift is almost over... no wonder it's been toasty in here since 1-ish. =)
- dom
- gnome
What's up, Mr Jones?
I was under the impression that SONET was designed to help avoid this kind of problem. I haven't seen any problems yet today, I'm on uunet in Pittsburgh tho.
What happened to the idea of redundant links and multiple paths? Sounds like whomever is on this fiber has some explaining to do as to why their OC-x networks were not or do not have fault tolerance as advertised, or even diversified paths. Even in my little town (State College, PA) I know where the OC-3 into my office goes, and there is only one section of about 2000' where all four fibers are on one cable, from my office to the road!
Sounds like the telcos should be held responsible for not building resilience into their networks. As more fiber is laid, and we depend on it increasingly, an outage will be more and more costly. Much cheaper to have the fiber going the other way too.
Just my thoughts.