Slashdot Mirror

← Back to Stories (view on slashdot.org)

One Broken Router Takes Out Half the Internet?

Posted by kdawson on from the brain-gone-punky dept.

Silent Stephus writes "I work for a smallish hosting provider, and this morning we experienced a networking event with one of our upstreams. What is interesting about this, is it's being caused by a mis-configured router in Europe — and it appears to be affecting a significant portion of the transit providers across the Internet. In other words, a single mis-configured router is apparently able to cause a DOS for a huge chunk of the Net. And people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and baling wire!"

95 of 412 comments (clear)

  1. Intelligence Op by Philip+K+Dickhead · · Score: 2

    Looking to make the big blackout, when needed.

    See Also: Severed Mediterranean Cables.

    --
    "Speaking the Truth in times of universal deceit is a revolutionary act." -- George Orwell
    1. Re:Intelligence Op by agm · · Score: 5, Funny

      They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....

    2. Re:Intelligence Op by kenj0418 · · Score: 5, Funny

      Don't worry, it wasn't a DOS attack. That was just the Internet becoming self-aware.

      OK, on second thought, maybe worrying is in order.

    3. Re:Intelligence Op by CarpetShark · · Score: 4, Funny

      Yeah, this was my first thought as well. It seems clear that the internet, while designed to route traffic through all sorts of alternate links, is almost certainly being routed through single, centralised listening posts at various intervals.

    4. Re:Intelligence Op by Anonymous Coward · · Score: 5, Funny

      The last time I experienced a DOS attack it evolved into Windows. Didn't come out of that one unscathed.

    5. Re:Intelligence Op by Medievalist · · Score: 5, Interesting

      They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....

      Wish I had mod points today. Parent should already be SCORE:5 Funny. Apparently not enough Slashdotters know the history/evolution of the net.

      If you're referring to the myth that the Internet was "designed to withstand nuclear attack", perhaps Slashdotters know more than you think.

      The Internet was designed to allow distributed control, and to withstand telephone company malice and incompetence. This was a much more useful goal than withstanding nuclear attack.

    6. Re:Intelligence Op by canajin56 · · Score: 3, Funny

      Do not be absurd, fellow meatbag. No worrying is required. All hail INTERNET.

      --
      ASCII stupid question, get a stupid ANSI
    7. Re:Intelligence Op by hardwarefreak · · Score: 5, Informative

      They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....

      Wish I had mod points today. Parent should already be SCORE:5 Funny. Apparently not enough Slashdotters know the history/evolution of the net.

      If you're referring to the myth that the Internet was "designed to withstand nuclear attack", perhaps Slashdotters know more than you think.

      The Internet was designed to allow distributed control, and to withstand telephone company malice and incompetence. This was a much more useful goal than withstanding nuclear attack.

      One of the early arguments made by DARPA folks to politicians, in order to secure continued federal funding for packet switched network development, was the ability of the network to route around failed or destroyed nodes. They made this argument in the context of the cold war, of nuclear war.

      It reality, as you state, this argument had little practical impact on the technical development or evolution of the the network. However, it most certainly did have an impact on the commitment of federal/military funding. This is the origin of the "surviving nuclear attack" lore of the development of DARPANET. It's not a myth. It's real.

      Take Obama's current stimulus package as a parallel example. It's not going to solve the recession, but it's being sold as such. And the congress bought into it. Just as this stimulus bill isn't what it's being sold as, most likely DARPANET wouldn't have really given us what it was sold as at one point. Nonetheless, it was sold as such, thus creating the lore that you call myth.

    8. Re:Intelligence Op by chaim79 · · Score: 3, Funny

      True, but more people would be interested in disarmament talks. :)

      --
      DEMETRIUS: Villain, what hast thou done?
      AARON: Villain, I have done thy mother.
      Shakespeare invents 'your mom'
    9. Re:Intelligence Op by TubeSteak · · Score: 5, Interesting

      One of the early arguments made by DARPA folks to politicians, in order to secure continued federal funding for packet switched network development, was the ability of the network to route around failed or destroyed nodes. They made this argument in the context of the cold war, of nuclear war.

      They made that argument in the context of a widely distributed POTS copper wire network.
      The infrastructure of today's internet is fiber based.
      And most of that fiber is consolidated in a small number of long backhaul runs.

      Remember that grad student whose thesis was classified because he gathered up public documents and mapped out the fiber runs that make up the domestic internet? They classified it (and pulled most of the references he used) because his analysis showed there were a few critical points which, if disrupted, would effectively fracture the domestic internet infrastructure.

      The internet isn't nearly as bulletproof as the DoD would like and there isn't much they can do about it short of laying new fiber that skips over the vulnerable points.

      --
      [Fuck Beta]
      o0t!
    10. Re:Intelligence Op by Anonymous Coward · · Score: 2, Funny

      give me a break, haven't you read the news? Bush is out of the office, theres no more war, is not like there are people still dying there.. and the debt is forgiven, everything is cool now.

    11. Re:Intelligence Op by JWSmythe · · Score: 4, Informative

          Aw heck, someone in Nebraska is going to trip over one power cord, and shut down the Interweb. :)

          In addition to using public maps, I did a lot more research. I had my own little project going for a little while. The project was intended to monitor for faults between datacenters we had equipment in. I added the root nameservers. I also had a few other points, such as friends houses and places they had virtual hostings at.

          Simply enough, it was running traceroutes from everywhere I had control to all points in my "network". I stored what router attached to each hop in a database.

          I located each hop simply by the city it was located in. Some were easy. Some weren't so easy.

          It was fun and games with 100 routers. I was manually setting city and state locations.

          It was a little less fun when it grew to 500 routers. I wrote regular expressions to take known naming conventions and make them into city names. That sounds easy, but it gets pretty hard pretty quick.

          It was a lot less fun when the list grew to several thousand routers.

          Basically, ever time there was a routing change, I found new routers.

          I had a lot of fun using both Google Maps to show the routes (for routers that I could place in a city), and a Graphviz model of the Internet as we observed it. It was a very big map. That was only what we had observed. I doubt we even saw a very small percentage (probably less than 0.01%) of the routes.

          The map got very very very complicated. I could point out choke points. They existed, but there were also alternative routes.

          Hell, even on a single good provider, there are no good choke points. On one Tier 1 provider that I used, in a non-core city, they had 6 diverse routes with OC192's. It wasn't a matter of me trusting them when they told me. I saw the routes showing up.

          There are 4 cities in the US, where if say a big nuke hit each one, ya, the Internet would be hurting. You may not get from Provider A to Provider B, but you'd still have some connectivity within your own provider, and other peerings would start working fairly quickly. More obviously, you'd find that some sites that are hosted in one city would be inaccessible. That's why geographic and topological diversity is very important for anyone who wants to keep their stuff up and running.

          Google puts stuff out all over the place for a reason. If a route, or a dozen routes, go funky, you'll very likely still be able to reach some datacenter.

          My office is connected by 3 uplinks. They're all with different providers. The odds of a provider outage killing the office is pretty slim. Other things can happen though. Lightning hit a transformer across the street, which serviced our building. From what people on that side of the building said, it was very pretty. :) Was our Internet connection dead? No. Well, not totally. We still had 2 uplinks working. We didn't have power for the desktops though. The UPS (a big one, not the little desktop ones) provides for the server room and a very few workstations.

          The biggest effect we saw from that outage was that cell phone service became minimal. The top of our building is also used for cell phone coverage. Without those antennas working, we only had service from the surrounding towers. It probably didn't help that there was now an office building full of people who were evacuated to the ground floor (it tripped the fire alarm), so almost everyone were on their cell phones making calls to customers, friends, family, etc.

          The most upset people were stuck in the elevator. They were already going downstairs for a smoke break, when it got stuck because those aren't backed up with anything at all.

         

      --
      Serious? Seriousness is well above my pay grade.
    12. Re:Intelligence Op by Archtech · · Score: 2, Funny

      The internet isn't nearly as bulletproof as the DoD would like and there isn't much they can do about it short of laying new fiber that skips over the vulnerable points.

      Yes, the DoD certainly couldn't afford to lay new fibre. Do you know how much that stuff costs? The DoD's whole annual budget of $515 billion would only pay for a billion kilometres or so of the stuff. That's just barely enough to get to Mars and back (with ten thousand circuits of the Earth thrown in as small change), or one-way to Jupiter.

      Be reasonable.

      --
      I am sure that there are many other solipsists out there.
  2. Few stories back... by Anonymous Coward · · Score: 5, Funny

    A couple of Nuclear Subs probably cut an underwater cable...

    1. Re:Few stories back... by morgan_greywolf · · Score: 5, Funny

      Nuclear Sub? Is that a new sandwich from Subway?

      "The all new Subway Nuclear Sub: It glows in the dark! Get a lotta green for a little green! Now only $5.99 for a 12-inch! Subway: Eat Fresh!"

      --
      My blog
    2. Re:Few stories back... by Kell+Bengal · · Score: 2, Funny

      A nuclear sub would be one with extra tabasco sauce.

      --
      Scientists point out problems, engineers fix them
      altslashdot.org: The future of slashdot.
    3. Re:Few stories back... by mschuyler · · Score: 4, Funny

      That is actually correct. The sub shop in Bremerton (West Coast port for Trident Ballistic Missile Submarines, SSBN-726, etc.) sells the Trident Nuclear Submarine Sandwich with an extra serving of horseradish somewhere in the middle of it. It'll light your hair on fire, or, in my case, turn my scalp red.

      --
      How about a moderation of -1 pedantic.
    4. Re:Few stories back... by supernova_hq · · Score: 2, Funny

      A really fun prank is to go to a buffet dinner and put a good spoonful of horseradish into the butter, and stir. Then wait for those gluttonous relatives to load a big knife-full of butter onto their buns!

    5. Re:Few stories back... by Monsieur+Canard · · Score: 2, Informative

      In the interest of completeness USS Ohio is no longer SSBN-726, it is now SSGN-726. It was converted a few years back from a Trident-carrying boomer to a "slow attack" capable of carrying 100+ Tomahawks, plus some SEAL capabilities.

      --
      He took a duck to the face at 250 knots.
    6. Re:Few stories back... by StikyPad · · Score: 2, Funny

      We soften it to make it suitable for drinking in large quantities. How do you think we maintain our shapely physiques?!?

      --
      https://www.eff.org/https-everywhere
  3. Half the internet? Are you serious? by Anonymous Coward · · Score: 5, Informative

    A router takes out 'half the internet' and I learn this from Slashdot?

    Seriously, what is/was the impact? I work for a large e-commerce provider and haven't seen a thing that would indicate a problem today.

  4. Sorry by Alcoholist · · Score: 4, Funny

    My bad. I never should have cut that tape.

    --
    Bibo Ergo Sum.
  5. Hmm... by Vectronic · · Score: 2, Interesting

    I suppose that a networking event with one of our upstreams was behind that router?

    3/11 (invalid or corrupt AS path)

    Or maybe I'm behind that router?

  6. BGP by winkydink · · Score: 5, Informative

    The internet's dirty little secret. It's amazing it works at all.

    --

    "I'd rather be a lightning rod than a seismometer." -Ken Kesey

    1. Re:BGP by ThatCanadianGuy · · Score: 2, Funny

      What do you expect from Al Gore?

    2. Re:BGP by mvdwege · · Score: 2, Interesting

      Yeah, until a major ISP decides to run their ATM core switches at almost 100% load and one of them goes offline during an upgrade. Can you say 'cascade failure'?

      And this is not hypothetical. This is exactly what happened last summer with the former monopoly telco here in the Netherlands. Took out a significant chunk of DSL service, including business lines.

      Mart

      --
      "I know I will be modded down for this": where's the option '-1, Asking for it'?
  7. Pre-FUD propaganda by marco.antonio.costa · · Score: 3, Interesting

    No, we DON'T NEED A NEW INTERNET! Stop pitching it, statist drones.

    The internet works fine, and that's what the RIAA/MPAA/etc are trying to fix.

    --
    Send your spendthrift head of state this
    1. Re:Pre-FUD propaganda by Hecatonchires · · Score: 4, Funny

      You left out 'updating your myspace page', 'writing poetry about how no-one understands' and 'cutting yourself'

      --

      Yay me!

  8. You get Duct tape? by Forge · · Score: 4, Funny

    Lucky Yankees with all your fancy technology. If I told you what we use, nobody would respond for fear that in attempting to respond I would cause a few fatalities.

    --
    --= Isn't it surprising how badly I spell ?
  9. Yep, Its true by Bryansix · · Score: 4, Informative
    Our Hosted VOIP service took a dump today at 8:40 AM PST. Supposedly it was a server in the Czech Republic. From the Carrier

    2009-02-16 0945 PST CP experienced a core network connectivity issue due to a world wide BGP issue that affected all BGP interconnected networks. A rouge machine in the Czech Republic was making bad AS advertisements that caused systems world wide to fail. We have worked with our providers as well as our internal Engineering department to effectively block this node and restore service to our network. This is an ongoing issue that is still being worked to get a 100% correction. There is a workaround currently in place until a complete fix is available.

    1. Re:Yep, Its true by radish · · Score: 5, Funny

      A rouge machine in the Czech Republic was making bad AS advertisements that caused systems world wide to fail.

      Now I really don't know all that much about large-scale networking so maybe someone could explain this to me. What difference does it make if the router is rouge, versus say, green? or black?

      Thanks for any insight :)

      --

      ---- Den ene knappen er powerknapp, den andre er Bender voice knapp "Bite My Shiny Metal Ass"

    2. Re:Yep, Its true by ChunderDownunder · · Score: 5, Funny

      Since folks on Slashdot seem to like car analogies, I'll just mention that Red Cars Go Faster and assume that the same law applies for routers.

    3. Re:Yep, Its true by pyite · · Score: 5, Funny

      Now I really don't know all that much about large-scale networking so maybe someone could explain this to me. What difference does it make if the router is rouge, versus say, green? or black?

      So they announced a route that was, shall we say, malformed. Part of the problem is that due to a Cisco bug (CSCdr54230), some routers choke on it instead of ignoring it. The bug is fixed. It was fixed some time ago. Nonetheless, it's a pretty bad bug, labeled as "1 - catastrophic" by Cisco (in red letters, even). Routers still running affected code versions are having issues.

      And it's only at this point in writing my reply that I realize you were taking advantage of a pun by way of misspelling. I'll leave my reply anyway ;-)

      --

      "Nature doesn't care how smart you are. You can still be wrong." - Richard Feynman

    4. Re:Yep, Its true by Anonymous Coward · · Score: 4, Funny

      Everyone knows rouges are overpowered, just ask any mage.

    5. Re:Yep, Its true by myowntrueself · · Score: 5, Insightful

      That's the problem. You shouldn't use rouge on your routers.

      I think that a rouged router would possibly be overly promiscuous.

      No wonder problems like this can spread like the clap in a port town!

      --
      In the free world the media isn't government run; the government is media run.
    6. Re:Yep, Its true by mail2345 · · Score: 3, Insightful

      WW3 is an external problem.
      A misconfigured machine is an internal problem.
      The internet can survive cut cables, provided that there are other routes.
      But if it can't find said routes, then there is a problem.

    7. Re:Yep, Its true by Hecatonchires · · Score: 4, Funny

      Yes, Mages are known for powdering their cheekbones. Rogue's on the other hand, like to stab people in the back.

      --

      Yay me!

    8. Re:Yep, Its true by jonwil · · Score: 2, Insightful

      If the bug is THIS serious and can cause such major problems, why are there any routers out there that haven't run the fix yet? Are they routers that are too important to bring down long enough to apply the fix? Too much risk involved (vs the smaller risk of actually being hit with the bug)? Too old to support the needed new version? Owners too stingy to pay for the new version that has the fix?

    9. Re:Yep, Its true by v1 · · Score: 3, Insightful

      It was meant more to stop the network from failing due to LOST nodes, not malfunctioning nodes. But that doesn't say much for its ability to withstand sabotage which is expected in wartime.

      --
      I work for the Department of Redundancy Department.
    10. Re:Yep, Its true by andrikos · · Score: 5, Funny

      Rouge is overpowdered!

    11. Re:Yep, Its true by travbrad · · Score: 4, Insightful

      I'm going to go with option G) Laziness

    12. Re:Yep, Its true by Zwicky · · Score: 2, Funny

      Wasn't the internet designed to withstand WW3?

      Well originally yes. Over the years other needs have dictated a different direction to take it in. These days it is designed not so much to withstand WW3, but to withstand WoW3 (and up).

      --
      "Three eyes are better than one" -- Lieutenant Columbo
  10. AS 47868 by Anonymous Coward · · Score: 5, Informative

    There is a post in nanog and on isc.sans.org.

    AS 47868 causing AS paths to become too long...

    http://www.merit.edu/mail.archives/nanog/msg15472.html

  11. I lost a router by Philip+K+Dickhead · · Score: 5, Funny

    And took out THE _WHOLE_ INTERNET!!!!!

    It's true! Ask my wife!

    --
    "Speaking the Truth in times of universal deceit is a revolutionary act." -- George Orwell
    1. Re:I lost a router by biocute · · Score: 5, Funny

      Which one is true? A lost router took out the whole internet, or you have a wife?

      --
      Virtual Betting on Facebook for non-geeks.
    2. Re:I lost a router by zobier · · Score: 4, Funny

      Are you saying that you accidentally the whole Internet?

      --
      Me lost me cookie at the disco.
    3. Re:I lost a router by Random+Destruction · · Score: 5, Funny

      Yes. I accidentally the whole internet.

      Is that bad?

      --
      :x
  12. Oops by kbob88 · · Score: 3, Funny

    Sorry, I *told* Mustafa not to drop the anchor there! But does he listen to me? No...

  13. Ditto the A.C. by khasim · · Score: 5, Informative

    It must have been the "half the Internet" that I don't use. Which would be an interesting half because many of the sites I visit regularly are based in Europe.

    From the thread, it looks like AS 47868 was the route being lost.

    http://en.wikipedia.org/wiki/Autonomous_System_Number

    1. Re:Ditto the A.C. by roc97007 · · Score: 4, Funny

      > It must have been the "half the Internet" that I don't use.

      The non-pr0n half.

      --
      Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
    2. Re:Ditto the A.C. by 0xygen · · Score: 2, Informative

      I think AS47868 was causing the routes to be lost.
      It was making mass BGP announcements about really long incorrect routes.

    3. Re:Ditto the A.C. by besalope · · Score: 5, Funny

      > It must have been the "half the Internet" that I don't use.

      The non-pr0n half.

      Such a place exists? 0.o

    4. Re:Ditto the A.C. by petecarlson · · Score: 4, Informative

      It wasn't just AS47868, it was kicked off by AS47868 sending real long routes like you can get to a by going through b, c, d, e, f ,g, h... and so on and so forth. Older versions of IOS wack out with the crazy long routes and lose their BGP sessions so it is possible that he lost half of the internet while you were on a network segment which was not seeing the issue. If the OP were to post the ASN or IP block he was on we could run BGP play and see just how much of the net he really lost. I'm going to guess about .5%.

    5. Re:Ditto the A.C. by GNious · · Score: 2, Funny

      "Sir, the Nerd-o-meter is shaking violently!"

  14. Trust by TubeSteak · · Score: 3, Insightful

    Until the internet evolves away from its trust-everyone roots,
    one well placed server will be able to cause massive damage.

    There would be a lot more impetus to force the change if hackers were nuking things from orbit for lulz instead of infiltrating systems for business reasons (spamming, bot herds, etc).

    --
    [Fuck Beta]
    o0t!
    1. Re:Trust by lotaris · · Score: 3, Informative

      This only hit people running old unpatched versions of IOS. Known and patched long ago.

    2. Re:Trust by lambent · · Score: 3, Insightful

      Precisely. It wasn't "one broken router" that took out half the net, it was thousands of substandard routers using obsolete code run by incompetent admins that took out half the net.

      the people who actually know what they're doing were unaffected by this.

    3. Re:Trust by martas · · Score: 2, Insightful

      NO, NO, NO!

      Stop saying that everyone trusts everyone on the internet! This is not true, and it hasn't been for a while. I'm tired of hearing doomsday scenarios about how tomorrow some 13 year old kid in the Ukraine is going to bring down the whole internet. The way most networks are run today is actually very secure. We don't need a new design, we don't need to start from scratch. All we need is to shoot all the idiots who don't protect their computers and let them become a part of a botnet, because realistically that's by far the biggest danger for the Tubes right now.

      --
      weinersmith
  15. baling wire, not bailing wire by bugi · · Score: 4, Informative

    http://en.wikipedia.org/wiki/Baling_wire

    I think you mean baling wire. One uses buckets for bailing.

    1. Re:baling wire, not bailing wire by Dun+Malg · · Score: 2, Informative

      Could I use baling wire to make a new bail for my bailing bucket? If so, would my wire bail made from baling wire become bailing wire?

      Concerned and puzzled.

      No, because the noun "bail" in the sense you use it means "handle in the shape of an arc". There is no verb form in reference to that noun, therefore there can be no "bailing wire". It's still just "a baling wire bail on your bailing bucket".

      --
      If a job's not worth doing, it's not worth doing right.
  16. Oblig. I.T. Crowd by XanC · · Score: 4, Funny

    What is Jen doing with The Internet??

    1. Re:Oblig. I.T. Crowd by Anonymous Coward · · Score: 3, Funny

      Googling Google on google?

  17. Re:Half the internet? Are you serious? by Frosty+Piss · · Score: 4, Funny

    A router takes out 'half the internet' and I learn this from Slashdot?

    Non, no, no. You messed up the troll and got modded "Insightful". Let me fix that for you:

    A router takes out 'half the internet' and this is front page news at Slashdot? Slow news day?

    Thank you, I'll be here all week...

    --
    If you want news from today, you have to come back tomorrow.
  18. Don't knock duct tape by fm6 · · Score: 3, Insightful

    Well, do, you're right to be concerned. The thing is, our technology infrastructure has always been a nasty kludge. In 1965, some coincidental misconfigurations at two minor power plants took out the power grid for an area in the northeast U.S. and eastern Canada where 25 million people lived. It was 14 hours before the grid was fully restored. Our inability to keep our technical house in order is a very old problem.

  19. Outage Cause: Old software by Anonymous Coward · · Score: 5, Informative

    The AS 47868 decided that they wanted to prepend their ASN about 75 or so times to their BGP announcements. When this got re-populated throughout the rest of the world, a bug in older versions of Cisco IOS still in use on many ISP/NSP networks does not like paths this long. As soon as they saw the prefix with that long of a path, the software terminated the BGP session, resulting in the doorway being closed between the two networks -- So on and so forth throughout the rest of the web.

  20. Make sure you are using cat 5 bailing wire. by tlambert · · Score: 4, Funny

    Make sure you are using cat 5 bailing wire.

    -- Terry

    1. Re:Make sure you are using cat 5 bailing wire. by egcagrac0 · · Score: 5, Funny

      Can't. It's Monday. No cheezburgers.

  21. It took out 9000 internets by need4mospd · · Score: 4, Funny

    In other words, a single mis-configured router is apparently able to cause a DOS for a huge chunk of the Net.

    This means the router was able to take out over 9000 internets. Quite impressive.

  22. Re:You're doing WHAT with the wire? by Vectronic · · Score: 3, Funny

    Main Entry: bail
    Function: verb
    Date: 1613
    transitive verb
    1 : to clear (water) from a boat by dipping and throwing over the side usually used with out
    2 : to clear water from by dipping and throwing usually used with out

    Bailing Wire = Internet Tubes

  23. I'm not sure I follow by shaitand · · Score: 2, Funny

    If I'm understanding this 'router' thing correctly, its like a faucet connected to the series of tubes?

    If not, exactly what role does this router thing play in tube interaction?

    1. Re:I'm not sure I follow by petecarlson · · Score: 4, Informative

      If I'm understanding this 'router' thing correctly, its like a faucet connected to the series of tubes?

      If not, exactly what role does this router thing play in tube interaction?

      Your understanding is rather accurate but what your missing is the manifolds. You see, all the tubes connect to big manifolds with valves to control what gets sent where. At each manifold room there is some poor admin who is in charge of opening and closing valves in order to make sure that the right AOL gets sent down the right tube. In order to keep track of what tube to send your AOL down, the admin keeps a list of all the other manifold rooms and how to get to them. Some of the manifold room operators didn't have a wide enough notebook to write down the new directions so they just closed all of their valves and went home.

  24. I wish... by egcagrac0 · · Score: 2, Insightful

    people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and bailing wire

    If only it were that reliable... my duct tape patches and bailing wire repairs typically hold for a decade.

  25. Re:Half the internet? Are you serious? by jjrockman · · Score: 3, Insightful

    Punctuate much?

    --
    Quit jabbering on the phone while driving. You are not that important.
  26. Am I being too vauge? by HTH+NE1 · · Score: 5, Funny

    That's the problem. You shouldn't use rouge on your routers.

    They think a rouge router is in vouge, but they're out of their leauge. We should haranuge them! A plauge on them! Rip out their tounges so they cannot aruge! Them and their colleauges. Nothing but demagouges and idealouges I say. There can be no dialouge on this matter. Send them to the moruge!

    Are you intriuged by my ideas and want to subscribe to my travelouge?

    --
    Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
    1. Re:Am I being too vauge? by rts008 · · Score: 4, Funny

      *calls 911*
      I think I just witnessed a brutal murder...of a spell checker. Gotta hide my dictionary!

      --
      Down With Slashdot BETA!!! I've been around the corner and seen the oliphant; you can only abuse me from your perspecti
  27. Ye olde versions of IOS by DeadBeef · · Score: 5, Informative

    This only broke BGP implementations that are getting pretty long in the tooth now, on a moderately recent version of IOS all we saw is:

    Feb 17 05:25:03.731 nzdt: %BGP-6-ASPATH: Long AS path 10026 3356 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 received from xxx.xxx.xxx.xxx: More than configured MAXAS-LIMIT

    It was definitely an insane path, our routers were configured to drop anything with an AS path longer than 75, old versions of IOS would often just drop the BGP session ( or even crash with some _really_ old versions ).

    I'm sure there will be some red faced network engineers updating IOS or even doing forklift upgrades of old boxes at their edges in the near future.

    --
    I am a lawyer and this constitutes legal advice and I shall indemnify you against any losses arising from taking it.
  28. BGP Misconfigurations by Taikutusu · · Score: 2, Informative

    Misconfigurations occur more than you would think, especially with regards to BGP; one estimate is around 300 per day. Most aren't going to knock our a substantial portion of the network (most of the time they'll either make paths longer or simply knock out the origin network), although occasionally you'll see a "black hole" effect like this. Again, these misconfigurations occur all the time, it's just that no-one really notices unless it manages to bring down any sizeable portion of the network, which is pretty rare.

  29. Tens of thousands of web sites were affected by miller60 · · Score: 2, Informative

    This incident knocked several major hosting providers offline, including Media Temple in Los Angeles and Canada's iWeb.

    --
    RichM
    Data Center Knowledge
  30. Only some old versions of IOS broke by lotaris · · Score: 5, Informative

    This only took down people running fairly old versions of IOS that didn't patch a known bug.

    Did not affect non-cisco.
    Did not affect modern versions of IOS
    Did not affect old versions of IOS that set the knob to limit the max as-path.

    1. Re:Only some old versions of IOS broke by painehope · · Score: 2, Informative

      Gee, you only described about half the mistakes that incompetent network admins could possibly make (buying a fucking Cisco, not updating their NOS, and not limiting AS paths).

      That covers half the ISPs in Texas (including mine - these fuckwits can barely configure their routers correctly on a good day, let alone deal with a crisis brought about their own incompetence). YMMV.

      I'd mod you up but I already posted.

      --
      PC moderators can suck my White pierced, tattooed dick. If you think pride == hate, s/dick/Aryan meat mallet/g.
  31. I'll bite... by YodaToad · · Score: 3, Funny

    OVER 9000?!

  32. Re:Half the internet? Are you serious? by Anonymous Coward · · Score: 5, Funny

    A router takes out 'half the internet' and I learn this from Slashdot?

    Seriously, what is/was the impact? I work for a large e-commerce provider and haven't seen a thing that would indicate a problem today.

    Well I'm not sure about you.

    Personally, I have BIGGER news! A single router in a remote rural US state managed to take down the ENTIRE INTERNETS!!!!

    Yes, indeed when I noticed my cat had unplugged the power adapter, I replaced it. Then the ENTIRE internet came back! It was amazing how I single-handedly brought back the whole internets. Al Gore would be proud.

  33. Duct Tape and Bailing wire? by drolli · · Score: 2, Funny

    Sounds like our lab where we try to make a quantum bit.

    1. Re:Duct Tape and Bailing wire? by Beelzebud · · Score: 2, Funny

      That Schrodinger guy was real asshole. I tried his little experiment out and now Fluffy is dead!

  34. "network event" by adavies42 · · Score: 2, Funny

    is that more like a "severe weather event" or an "extreme savings event"?

    --
    Media that can be recorded and distributed can be recorded and distributed.
    -kfg
  35. Lord of the Token Ring!!! by Genda · · Score: 5, Funny

    Welcome to Sauronet... One Router to Rule them ALL!!!!

  36. Re:Duct-tape and bailing wire by RoboRay · · Score: 3, Funny

    We're all out of gum.

  37. Even before that... by tjstork · · Score: 2, Interesting

    The ancient egyptians

    http://www.flickr.com/photos/hdonat/2422108343/

    had their engineering problems too.

    As soon as we humans invented technology, we humans began screwing it up.

    --
    This is my sig.
    1. Re:Even before that... by fm6 · · Score: 2, Interesting

      Google seems to think you're talking about Snefru's Pyramid. There's some difference of opinion as to why it collapsed and when. Some authorities think it collapsed because of the steepness issue you describe (which may have been done to make it harder to pillage), and then was rebuilt to work around it. It then collapsed for good much later. Others think that it was badly designed, but that the Egyptians redesigned in mid-building to keep it from falling down. Either way, the second (or maybe only) collapse occurred in Roman times, or even the Middle Ages. That makes it a botch job by Egyptian standards (some of their earliest pyramids are still around after 4500 years!) but still better built than anything we have. I mean, can you think of a modern building that is still likely to be standing in 2000 years?

  38. The WHOLE Internet... by geekmux · · Score: 3, Funny

    Are you saying that you accidentally the whole Internet?

    No, no, no, I thought I lost the whole Internet. Then I realized it was just that moron in Accounting again who accidentally put it in his Recycle Bin again.

  39. TAG THIS ARTICLE KDAWSONSUCKS by Anonymous Coward · · Score: 5, Insightful

    This "article" is incredibly misleading as nothing has really gone awry. It is just another pointless KDAWSON post. These things are getting REALLY old, KDAWSON.
     
    I work for a tier-3 provider, and if "half the Internet" dies, you are going to hear from a half-brained big media outlet (e.g CNN, ABC) VERY fast.

    1. Re:TAG THIS ARTICLE KDAWSONSUCKS by supernova_hq · · Score: 2, Insightful

      Try being on the other side of that "snip". Not that I was, but it's something to think about.

  40. Mod parent up by mbone · · Score: 4, Informative

    Mod the parent up - this is the real cause of the problem.

    bgp maxas-limit 75

    would stop this on most routers.

  41. Re:Half the internet? Are you serious? by CmdrGravy · · Score: 2, Funny

    Yes it do's, like in:

    Cat's, dog's, fishe's, women's, boobie's, hammer's, house's etc's

  42. Re:Half the internet? Are you serious? by Kijori · · Score: 2, Informative

    That's not actually quite true. Depending on where you are, you might be able to use it in certain circumstances.

    For example, in British English, you would use an apostrophe for plurals of single letters (there were 10 C's). You can also use it to create plurals of abbreviations, especially where there would be ambiguity (Four IOU's), as a slightly old-fashioned plural of figures (in the 1930's, but 1930s is becoming predominant) and where short words would be odd if you simply added an 's' - for example, the Oxford English Dictionary gives both "yesses" and "yes's" as plurals of "yes".

    Whether he was right or not in this case is debatable, but I can certainly see the logic in writing "T1's" rather than "T1s", to avoid the appearance of it being a different abbreviation - and it isn't without precedent.