Slashdot Mirror

← Back to Stories (view on slashdot.org)

Akamai DNS Outage Messes up Net

Posted by ryuzaki0 on from the point-of-failure dept.

katre writes "Checking all my favorite sites this morning, I saw that about half a dozen seem to be offline. Trying to figure out why, I found an interesting article on the front page at http://isc.incidents.org/. Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others. Whatever happened to my decentralized net with no single point of failure?"

11 of 522 comments (clear)

  1. I'm definitely not a technical guru... by Dagny+Taggert · · Score: 5, Interesting

    but I believe the centralized concept of the 'net is something that is coming to an end, much to our loss. I'm pretty bothered by the fragility of this system. How many of you can't work without web access?

    --
    Don't be a looter...and yes, I know that it's spelled with an "A" instead of an "E".
    1. Re:I'm definitely not a technical guru... by Malc · · Score: 4, Interesting

      I'm not being condescending, but unless you're very junior or just a small pawn in a very large company, even server mainenance positions require some offline work. There's always some planning that needs to be done for tasks during today, the coming week, or even long term. Phone calls can be made, documentation updated, etc. It really depends on how long the outage is for and how quickly you're able or willing to switch gears and tasks.

      I realise that some jobs are much more inpractical when there is downtime, but not everybody even here on /. requires 100% connectivity. I myself live over 4,000 km from work (I haven't even met them face-to-face for more than four years) and I would be pretty pissed off and delayed on one of those days I have to do all my work via Terminal Services on machines at the main office or colocation facility... but let's be honest, a day's outage wouldn't mean I couldn't do any productive work. I once had a 10 day outage when the local telco switched my DSL line from interleaved channelisation to fast-path, but as I was in the middle of a long stretch of software development, I really didn't need a lot of internet. Batches of dialup and patience all-around sufficed ;)

    2. Re:I'm definitely not a technical guru... by jdray · · Score: 4, Interesting

      I work for an electric utility and, with the current state of web-based scheduling of electricity in the U.S. (a mandatory requirement by regulatory agencies), loss of proper traffic routing on the Internet can have difficult-to-overcome effects.

      --
      The Spoon
      Updated 6/28/2011
  2. points of failure by rlthomps-1 · · Score: 4, Interesting

    DNS dying on you? Just throw it on the pile of other connection problems

    I think everyone has several "single" points of failure -- my cable modem dies at least twice a month and my wireless router conks out at least twice a day ;)

  3. releted to linux kernel DoS exploit? by Anonymous Coward · · Score: 4, Interesting

    Do we know if this at all related to the Linux kernel 2.4.2x/2.6 DoS exploit discovered yesterday?

  4. Lack of notification by sphealey · · Score: 5, Interesting

    What ticks me off about this incidents (and I suspect that there have been several in the last 6 months) is that there is absolutely no notification given, either during or after the event. During this outage, some news outlets were still reachable (including Slashdot), and a simple notification would have saved hours (* 10s of thousands of network dudes worldwide) of time and much grief from the big bosses who couldn't reach Yahoo Finance, I mean critical business web sites.

    Are these guys so convinced of their omnipotence and indispensibility that they don't feel the need to communcate with the world about what is going on?

    sPh

  5. Akamai's DNS black magic by frankie · · Score: 4, Interesting
    Akamai uses (some would say ABuses) DNS in ways the rest of us (even global megacorps) wouldn't dare. Half of Akamai's magic is their 10000+ carefully-scattered servers, but the other half is their routing. Those servers are listed differently depending on where you ask from.

    It's not like a092156fg.akamai.net is in Seattle and k1039665.akamai.net is in Saskatoon. Instead, all of *.akamai.net goes to whatever cluster is "closest" to the requesting IP (based on BGP, Colonel's Secret Recipe, etc)

    So if Akamai's DNS gets screwed up, I would expect major weirdness. And as more sites join EdgeSuite (where you host your entire domain on Akamai's servers & DNS) the effect must magnify.

    Of course, I could be completely wrong. I'm not a routing god, just a guy who thinks Akamai is a cool hack.

  6. Re:Root servers not decentralized? by Syberghost · · Score: 4, Interesting

    The fact is that there are 13 of them, in widely scattered locations across the globe, and it's not decentralized?

    Damn man, what exactly would you consider "decentralized" then?

    Akamai has 13, in widely scattered locations, as well. That in itself doesn't make them sufficiently decentralized.

    The reason the root servers don't have this problem is that they don't all run the same software (anymore) and aren't all administrated by the same people.

    I'm making an assumption here, of course, but I will not be a bit surprised if it turns out that Akamai loaded something that hit all their routers at once.

  7. Reminds me of a story by Venner · · Score: 5, Interesting

    Not too long after 9/11, I was surfing the net and needed to look up something at the Library of Congress for one of my classes. It wouldn't connect. At first I thought we'd just lost DNS (not so uncommon an occurance at my university in those days), but found I could still connect to slashdot.org and some other sites.

    Being a geek, I thought up a list of about 30 sites to ping, scattered across the US. (.govs and .edus mostly.) The ones that replied, I plotted on a US map based on their DNS LOC. (A project I wrote for a previous class.)

    I freaked out a bit when the mid-atlantic seaboard came up missing. I crossed my fingers hoping that it was just some idiot who'd accidently cut one of the main fibers (which it what it ended up being) and not that Washington DC was now a big hole in the ground.

    --
    A preposition is a terrible thing to end a sentence with.
  8. Re:Root servers not decentralized? by Omnifarious · · Score: 4, Interesting

    The root nameservers are not under decentralized political control, which still makes them a single point of failure, albeit a different kind of failure.

    --
    Need a Python, C++, Unix, Linux develop
  9. Created SPoF by Todd+Knarr · · Score: 4, Interesting

    The problem is that those sites created their own single point of failure by all using Akamai for DNS. When Akamai DNS fails, sites that depend on it for their own DNS fail.

    It used to be nearly impossible for this to happen. The original rules for DNS were that you had to have at least 2 nameservers for your domain, preferrably 3 or more, and they couldn't be on the same physical networks. With that rule having a single network go down rarely made any domain unresolvable (backbone networks whose outages could render dozens or hundreds of other networks unreachable being the exception). Maybe we should put the old nameserver-diversity rules back into place.