Slashdot Mirror

← Back to Stories (view on slashdot.org)

Massive Fiber Cut Slows Net

Posted by ryuzaki0 on from the i'm-moovvvinngg-vveerrryy-sloowwlllyyy dept.

netpuppy writes "East coast to west coast connectivity (or the other way around) feeling slow today? Here's why. It appears that the attack of the raging backhoes has hit Ohio today, where an unnamed public utility managed to cut through 4 OC-192 circuits while working on gas lines. 4 OC-192s are roughly equivalent to 40 Gbps traffic, and trunks this size usually carry both voice and data on them. AboveNet, GTE, and Metro Fiber (now part of Worldcom) seem to be the worst hit, according to this Inter@ctive Week article. " OK, I'm not just crazy. It has been slower then molasses today.

24 of 204 comments (clear)

  1. Re:IP vs Phone lines by Anonymous Coward · · Score: 4

    Why is it that IP trunks seem to be backhoed at the drop of a hat, while phone lines are somehow immune? Why doesn't the phone line between Boston and San Francisco go down?

    answer: phone lines are not immune. they get cut all the time. however, the phone network, like the internet, has redundancy. whe na phone line gets cut calls are rerouted to other trunks, making the problem oblivious to the layman.

    so, why do fiber cuts turn out so much worse? because the data network is bursting at the seams. a voice line cut can be directed to ununsed lines. a data line cannot be done in the same manner, because there are no vacant lines, or at least not as many as were lost...

    why is this the case? one, because the data network is expanding faster than new fiber can be laid down. we're too big for a britches. also, the phone network, which has grown at a decidedly slower pace, has, at both the users' and phone/data companies' requests, deemed to be more important. let's face it, if a phone line went down to a company hq, they'd be lost for as long as that line was down. just now are people realizing that their data line, is now becoming more important than their voice line. :)

  2. Top X Things To Do The Day After Nuclear War by Imperator · · Score: 4
    1. Contribute to K-rad, a KDE frontend to the GNU/rad utility to calculate radiation exposure.
    2. Check /. to see if the major backbone providers are doing ok.
    3. Run traceroutes to guess which places weren't hit. (Whoa, the Maldives are doing well. Doh! DC! It was all in vain!.)
    4. Try to avoid the BSoD (Brown Smoke of Death).
    5. Assist local disaster-relief organizations by migrating their proprietary birth/taxes/death-certificate database to an open one running Linux and MySQL, with a Perl/Tk frontend on X clients.
    6. Don't miss the opportunity to plug FS/OSS: the cathedral is centralized/vaporized, but the bazaar keeps moving forward.
    7. Reload /. incessantly in your hope to get a First Roast.
    8. Help out any hurt penguins you find while on the beach.
    9. Never forget the importance of an off-site backup!
    10. Yay! Finally, there are plenty of IPs available!
    11. Engage in a productive vi/Emacs flamewar.

    --

    Gates' Law: Every 18 months, the speed of software halves.
  3. dern it by Suydam · · Score: 3
    Why is it that this happens so frequently in the midwest (i live in Michigan).

    My company's T-1 line has been cut at least 3 times in the last 5 years.

    This kind of thing could be a MAJOR problem as the internet becomes more important to big companies. I mean right now, sadly, it's still a novelty to many of the people in the world. But can you imagine the hell we'd be in for if everything travelled over the 'net and someone cut a big trunk like this? Yikes.

    --


    Werd.
  4. Texas by scumdamn · · Score: 3

    I don't know about y'all, but our connection doesn't seem to be any slower here. That poor bastard with the backhoe is luck he didn't cut that line around these here parts. We'd have his hide! Hell, Billy Bob'd probably string him up! Imagine not being able to get on EBay to see if he won the auction for the entire video tape library of the Dukes of Hazard. The horror!

    All opinions expressed with tongue firmly in cheek between the skoal and the ceegar.

  5. Sudden realizations about the cold war by Sloppy · · Score: 4

    Now we know that the "vehicle" in "multiple independently targeted reentry vehicle" (MIRV) is actually a backhoe, rather than a nuclear warhead. I bet the guys who designed the 'Net to be nuke-tolerant are feeling pretty damned embarrassed right about now.

    Some people think the cold war was won by USA outspending USSR. But the real truth is that someone finally leaked that we were building bombs rather than just backhoes. Ivan's pants must have gotten pretty soiled at that revelation. Just think: all along, we totally misinterpreted what "We will bury you!" meant.


    ---
    Have a Sloppy day!
    --
    As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
  6. Re:Nuke safe huh? by Ripp · · Score: 3

    Correct, but as the other poster mentions, it just gets re-routed. But stop and think for a minute.

    Power grids and telephone circuits can be affected the same way, take out one of those big "power towers" that traverse large spans, and a couple of remote stations, a few satellite uplinks, some telco switching stations/relay towers, and havoc *will* ensue. It doesn't matter if it *can* be re-routed, the resultant chaos and downtimes would cost probably millions. Cyberwarfare isn't about information, it's about $$$$$ lost when the infrastructures disappear.

    Then we'll be falling back to all the guys with their ham sets. ... --- ...

    --
    Blech. Signatures.
  7. That explains a mystery by hedgehog_uk · · Score: 4

    I work in London (England) and earlier today a colleague told me that he had tried pinging a site in the US. He claimed that the packets were making it across the pond OK and then were being routed via Australia. We didn't believe him. Guess that I'll have to apologise tomorrow.

    --
    Yellow tigers crouched in jungles in her dark eyes.
    She's just dressing, goodbye windows, tired starlings.
  8. Name that CircuitID! by RISCy+Business · · Score: 3

    HA! I was WONDERING why things were slow to my ISP today! Those OC192's are probably the ones that Columbia Gas has pipelines running alongside, on I80. (Ha! You thought I wouldn't name the gas company!) IIRC, ICG laid the first OC48, and MCI/MFS as well as Sprint laid more, and more, and more. They're doing a LOT of construction along I80 from what I hear, so it's NOT surprising that a cut occured. Those lines are on a gas-pipeline right-of-way, and there are 5ESS Demarcation points along I80 for 'em.

    There've been previous fiber cuts that resulted in me passing a group of 15 MCI/WorldCom vans on my way home. I betcha if it's the link I suspect, they've got the onramp *packed* with all their techs trying to explain to the people staffing the tollbooths that they're with (insert-company) and they're in on an emergency call and they'll be getting right back off! *LAUGH!* Gotta love the Ohio Turnpickle, eh? :)

    -RISCy Business | Rabid unix guy, networking guru

    --
    your company here.
    shelby != ford
  9. Re:IP vs Phone lines by Xenu · · Score: 3
    Ma Bell put a lot of effort into designing a system that could route calls around congestion and equipment failure. Their routing system was very flexible and they had a NOC (Network Operations Center) to stay on top of problems. I'm not sure how much of that system has survived the breakup and deregulation of the Bell System.

    The data networking companies don't seem to be as concerned about reliability and availability. There are too many single points of failure. I've heard stories about the lack of excess/spare capacity in some big IP networks. The recent MCI Worldcom frame relay network failure was unforgivable. Some people were cut off for a week.

    You have to assume that lines will go out, equipment will fail and that software may not work properly.

  10. Knocking out a national broadcast network by orac2 · · Score: 3

    Slighty off topic but a good tale: My father has been working for a national European broadcaster for a long time. Decades ago when they were setting up a new television and radio campus they had to run some data lines out to the techies' new buildings. He and his colleagues indicated the line where they wanted the backhoe to dig and went to lunch in the canteen. About 20 minutes later all the lights went out. Somebody had neglected to put down any markers and the backhoe had cut through the main power cable *after* where it was joined to the back-up generator. As national braodcasters in Europe also double as the goverment's Emergency Broadcast System this caused a lot of people way up the chain to get alarmed. While the cable was being repaired they had to bundle a crew down to the old emergency studio and transmitter in the General Post Office (Think Krusty the Clowns broadcast in The Simpsons).

    --
    "Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
  11. themes.org by adraken · · Score: 3

    just for all you to know, themes.org is being hosted on abovenet (one of the severely affected isps) and i can't currently reach abovenet's dns servers.

    sum: themes.org probably will be down until this is fixed. please be patient.

    --
    -- adraken
  12. To inject a bit of Humor if I may. by Allnighterking · · Score: 4

    About 8 years ago while working in Asia in a country to be left unamed (to protect the guilty). We had, had about 20 line cuts in a two week period. All within the same five mile stretch of Highway. It was because local farmers whereout backhoeing their field dikes to prepare for the next seasons planting as well as crews from Highway construction. We kept repairing the cuts in the fiber, putting larger and larger signs and underground tapes all in the local language (none of which helped they dug where they wanted too.) One of our local co-workers came up with a solution to the problem. We place signs along the stretch in ENGLISH only. The locals, and highway crews became so sure that this was some special secret government project that over the course of the next 6 months we had only one cut in our area of responsibility. *sigh* Kinda like my physics prof who said the only thing fibre optics would be good for is "hippy" lamps and nothing more.

    --

    I'm sorry, I'm to tired to be witty at the moment so this message will have to do.

  13. They never said... by neuroid · · Score: 3

    it was IDIOT proof. Just nuke proof.

  14. Re:So... by netpuppy · · Score: 4

    ok, I worked at an isp for a while, and here's what I saw.

    Apparently, people digging tend to call blue stakes or whoever when they are digging in areas that might have natural gas lines, 'cuz they'll explode if they screw up. Fiber lines, on the other hand, pose no such problem unless you're standing in water when you hit one (there's a lot of power going through to feed all the repeaters on a long-haul circuit). So utilities and construction workers don't tend to worry about calling any utilities to find out if fiber is buried, because they really never see the effects. I believe they are legally liable, though. Anyone know if someone has ever been brought to court for cutting fiber??

    --
    good. fast. cheap. (pick any two, you can't have all three)
  15. Whew! by Cuthalion · · Score: 5

    Good thing there wasn't anyone around there smoking a cigarette. What with all those loose bits sloshing around, the slightest spark could have set off that 'internet explosion' people keep telling me about.

    --
    Trees can't go dancing
    So do them a big favor
    Pretend dancing stinks!
  16. Nuke safe huh? by Hobbex · · Score: 4


    I would say that this goes to show the utter bullshit that is the whole cyber terrorism thing. Why spend billions of dollars trying to police imaginary squads of crackers set to destroy our information infrastructure, when a couple of idiots with shovels can create major mayhem like this?

    I wonder what an organized group of wire cutters who did a little bit of research on their targets could accomplish. I have a feeling it wouldn't be pretty.

    I can't say I noticed anything myself (the net has been dog slow for me as long as I can remember, so), but if a small event like this can cause major problems, then the Internet is definetly closing up on critical mass....


    -
    /. is like a steer's horns, a point here, a point there and a lot of bull in between.

  17. Fiber Route Engineering, Cable Location, et al. by Anonymous Coward · · Score: 5

    Well, the powering issue being talked about isn't exactly what would be called "common". Most fiber in the ground today has no current-carrying conductors. True, it has a metallic conductor or two in it, and in some cases, a metallic sheath or armour, but carries no exectricity. The metal conductor is used exclusively to locate the cable with inductive/RF location gear. When it comes to re-generation of fiber "signals", byt simple amplification, or a full teardown of the bitstream with drop/add functionality, it's usually done at a special location (in-ground vault, above ground fiber hut, whatever) with external power. That's power supplied by a power company, with backup power. Since fiber traffic routinely travels 50+km without intervention at all, it's a total waste of resources and cable to lay a power cable for equipment. Now, all said, there are exceptions where there is no power readily available, like deep-sea cables, deserts, and such where there may be a power cable present in places. But in general, not present. Now, back to a little more on topic :) As for location, depending on the area, this can be tricky business, big time. In dense metropolitan areas, there can quite literally be cables a foot or less apart "horizontally" as well as several layers deep "vertically". Location of cables isn't always as precise as it ought to be, especially when the phone companies records don't reflect actual construction oddities, like perhaps a four foot fiber "loop" at the base of a pedestal. Well, when the person locating cable gets within a couple feet of a pedestal, you'd just assume it goes right in, right? Wrong :) And another company plans to put a pedestal right next to the existing one, and, well, it's time to break out the Scotch-Lok or quick-fix fiber kit :) Granted, most cable and fiber cuts happen out of pure ignorance or stupidity. Farmer Joe is out in the field fixing a drainage tile with his tractor and doesn't give a single thought to what else may be in the ditch with his tile. Where did those kits go again? :) And as a final note on the buried cable thing, nowadays, it's becoming common, almost required, practice to bury a brightly colored plastic ribbon above the cable being installed. Usually yellow or orange, it's really easy to see against black dirt, and would hopefully be seen before the contractor hits the cable itself. Doesn't do much good in the case of boring or knifing cable in though. All of this is just general information. The cut out east could have been done in one of a dozen or more other common contractor cable goofs that I won't even try to speculate on. Either way, it boils down to this - contractor started digging without requesting a locate, or if the cable was properly located, and the contractor either mis-read the location markings, or ignored them, that contractor had better hope he has a good insurance company. If the cable was not located at all, and a locate request was made and recorded in a timely manner, usually 24 to 48 hours before digging, well it's the telco's tough luck. You know, it's a small wonder that things work as well as they do considering the unimaginable number of perils out there, backhoe's included :) Sorry this is so long! FBG the AC

  18. Interesting issue: packet priorities the day after by Imperator · · Score: 3
    Let's suppose that a nuclear exchange destroys the US. (We can also suppose it destroys Russia or China, but they're insignificant because they don't have much bandwidth to begin with. :)) If you're going to nitpick the nuclear war example, let's just suppose that for some generic reason, the available bandwidth decreased significantly and rapidly.

    Now, we can't count on the users cutting down on their bandwidth use conscientiously. How, then, can we keep the critical services running? For a start, we need to define "critical services". I'll say that the greater the ratio of content to bytes, the higher the priority of a service. The only practical way to filter packets by service is to filter by port. You can run a filtered service on any old port you want, but the goal is not to prohibit services so much as drastically reduce the bandwidth used so that the network remains usable.

    DNS, for example, would have a very high priority, and be one of the last services to drop. Without DNS, the network becomes significantly less usable. The services designed for text communication also would have a high priority: smtp and the assorted email services (no attachments), nntp (again, no attachments), finger, time services, gopher, and the like. Even http might be allowed through, but filtered by mime type (text/plain, text/html, x-www-form-unencoded, etc.).

    Still, there would be a significant drop in the usefulness of the network. We need more bandwidth than we need to ensure reliability. Make bandwidth, not war! :)

    --

    Gates' Law: Every 18 months, the speed of software halves.
  19. Re:another reason to favor Qwest by Allnighterking · · Score: 3

    Actually these cables usually follow major access roads whenever possible. This allows for access to the lines regardless of weather conditions. (the mud in an Ohio cornfield can get 2 feet deep after a really good rain) Next 4 to 6 feet above the cable they bury a bright orange tape that warns about cable below. Hopefully this can be seen and Hopefully it doesn't settle out of alignment with the cable below. Third most cable cuts I've seen were not the result of direct contanct with the cable by the backhoe but rather the result of mud and rock slides/shifts due to digging near them. Cables aren't and can't be that strong. More so pulling a cable isn't as easy as you make it sound. This isn't a piece of thread It can weigh several tons (tonnes) and requires heavy equipment to pull it through a pipe. Finally one of the problems with using train tracks is how in the hell do I get a truck back where the cable is? Finally glass doesn't stretch.

    --

    I'm sorry, I'm to tired to be witty at the moment so this message will have to do.

  20. Bah, the new technology... by Rombuu · · Score: 3

    We never had this problem back in the days of good ol' POTS service. Good old copper wire never cut cut or went dolkjsd;flkh#&&(

    NO CARRIER

    --

    DrLunch.com The site that tells you what's for lunch!
  21. Packets are getting rerouted .... everyone suffers by taniwha · · Score: 3

    Well - that would explain this:

    1 198.182.167.17 (198.182.167.17) 0.628 ms 0.564 ms 0.532 ms
    2 adsl-63-194-218-254.dsl.snfc21.pacbell.net (63.194.218.254) 26.688 ms 29.433 ms 15.868 ms
    3 core1-fe4-1-0.snfc21.pbi.net (206.171.134.209) 13.394 ms 13.701 ms 15.957 ms
    4 gsr1-g1-0.snfc21.pbi.net (209.232.130.20) 16.098 ms 13.846 ms 15.215 ms
    5 sfra1sr3-so-1-1-1-0.ca.us.prserv.net (165.87.161.74) 16.676 ms 16.208 ms 15.202 ms
    6 sfra1sr2-11-0-0.ca.us.prserv.net (165.87.13.17) 16.684 ms 16.286 ms 15.226 ms
    7 above-advantis-ds3.sjc.above.net (216.200.0.81) 21.833 ms 24.418 ms 29.271 ms
    8 core1-core4-oc3.sjc.above.net (216.200.0.85) 25.864 ms 23.882 ms 17.173 ms
    9 core2-core1.sjc.above.net (209.133.31.110) 47.734 ms 39.032 ms 39.107 ms
    10 main2-core2.sjc.above.net (207.126.96.138) 44.072 ms 40.088 ms 40.069 ms
    11 core2-main2.sjc.above.net (207.126.96.137) 42.369 ms 44.488 ms 46.094 ms
    12 * * core3-core2-oc3.iad.above.net (209.249.203.65) 590.011 ms
    13 abov-core1-mae-east.netaxs.com (209.249.119.234) 695.855 ms 721.634 ms 848.819 ms
    14 dn-netaxs-gw.dc-core.h5-0-45M.netaxs.net (207.106.127.94) 1125.874 ms 1282.923 ms 1408.877 ms
    15 h900.ca2.wdc.dn.net (209.207.190.5) 1263.551 ms 1250.176 ms 1252.700 ms
    16 209.207.174.23 (209.207.174.23) 1234.975 ms 1256.689 ms 1208.096 ms

    and worse yet DNS lookups from here going thru Vienna :-(

    But remember folks there's redundancy in the backbone routing but when something big goes down
    everyone else gets to suffer as the traffic
    gets piled on top of their usual connections.

    If there's a lot of traffic going thru Europe I
    bet they're getting really steamed over there

  22. Re:So... by netpuppy · · Score: 5

    From Wired 4.12 ... article called Mother Earth Mother Board ... url at http://www.wired.com/wired/archive/4.12/ffglass.ht ml

    The signal coming down the FLAG cable passes through the doped fiber and causes it to lase, i.e., the excited electrons drop back down to a lower energy level, emitting light that is coherent with the incoming signal - which is to say that it is an exact copy of the incoming signal, except more powerful.

    The amplifiers need power - up to 10,000 volts DC, at 0.9 amperes. Since public 10,000-volt outlets are few and far between on the bottom of the ocean, this power must be delivered down the same cable that carries the fibers. The cable, therefore, consists of an inner core of four optical fibers, coated with plastic jackets of different colors so that the people at opposite ends can tell which is which, plus a thin copper wire that is used for test purposes. The total thickness of these elements taken together is comparable to a pencil lead; they are contained within a transparent plastic tube. Surrounding this tube is a sheath consisting of three steel segments designed so that they interlock and form a circular jacket. Around that is a layer of about 20 steel "strength wires" - each perhaps 2 mm in diameter - that wrap around the core in a steep helix. Around the strength wires goes a copper tube that serves as the conductor for the 10,000-volt power feed. Only one conductor is needed because the ocean serves as the ground wire. This tube also is watertight and so performs the additional function of protecting the cable's innards. It then is surrounded by polyethylene insulation to a total thickness of about an inch. To protect it from the rigors of shipment and laying, the entire cable is clothed in good old-fashioned tarred jute, although jute nowadays is made from plastic, not hemp.

    --
    good. fast. cheap. (pick any two, you can't have all three)
  23. Check out the Internet traffic report by Oates · · Score: 5

    I checked out the Internet traffic report to see what kind of effect this harmless mistake had. North American traffic statistics

    Keen! Can you spot the time the big bad backhoe operator cut the cord?

  24. Hey, cut them some slack.. by Pyr · · Score: 4

    Construction/Utility repair guys don't have an easy life. They're the ones out there in the middle of the night when your power line has gone down, or the ones fixing that broken sewer pipe, or the ones who make sure you have water. Near here they're doing major construction over on Grand Avenue, and to avoid traffic they have to do it in the middle of the night. They accidentally cut off a hunk of phone lines and about 1000 people had no phone service, but that happens pretty rarely.. I'd like to see YOU guys doing hard physical labor all night or all day trying to avoid speeding cars and a maze of pipes and cables and never screw up occasionally.