Slashdot Mirror
Akamai DNS Outage Messes up Net
katre writes "Checking all my favorite sites this morning, I saw that about half a dozen seem to be offline. Trying to figure out why, I found an interesting article on the front page at http://isc.incidents.org/. Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others. Whatever happened to my decentralized net with no single point of failure?"
but I believe the centralized concept of the 'net is something that is coming to an end, much to our loss. I'm pretty bothered by the fragility of this system. How many of you can't work without web access?
Don't be a looter...and yes, I know that it's spelled with an "A" instead of an "E".
DNS dying on you? Just throw it on the pile of other connection problems
;)
I think everyone has several "single" points of failure -- my cable modem dies at least twice a month and my wireless router conks out at least twice a day
Do we know if this at all related to the Linux kernel 2.4.2x/2.6 DoS exploit discovered yesterday?
When Akamai's system was first announced, most people thought this was a great idea. It made sure that the sites that used this technology would always have the bandwidth they needed, when they needed it. Like with everything else in life, there's always a trade-off between preformance and reliability...
------------------
"Never Attribute to malice what is adequately explained by stupidity..."
What ticks me off about this incidents (and I suspect that there have been several in the last 6 months) is that there is absolutely no notification given, either during or after the event. During this outage, some news outlets were still reachable (including Slashdot), and a simple notification would have saved hours (* 10s of thousands of network dudes worldwide) of time and much grief from the big bosses who couldn't reach Yahoo Finance, I mean critical business web sites.
Are these guys so convinced of their omnipotence and indispensibility that they don't feel the need to communcate with the world about what is going on?
sPh
Akamai is a distrubuted server platform they are all over a hard target, but they are prone to software updates and virus. =)
War isn't about who's right. It's about who's left.
It's not like a092156fg.akamai.net is in Seattle and k1039665.akamai.net is in Saskatoon. Instead, all of *.akamai.net goes to whatever cluster is "closest" to the requesting IP (based on BGP, Colonel's Secret Recipe, etc)
So if Akamai's DNS gets screwed up, I would expect major weirdness. And as more sites join EdgeSuite (where you host your entire domain on Akamai's servers & DNS) the effect must magnify.Of course, I could be completely wrong. I'm not a routing god, just a guy who thinks Akamai is a cool hack.
I wonder why these companies wholly switched their nameservers over? Why not have #1 and #2 be Akami, and #3 & #4 be your own nameservers? Preferably on different coasts or in different countries.
This would seem an obvious solution. You are allowed to have many nameservers you know...
Natural != (nontoxic || beneficial)
The fact is that there are 13 of them, in widely scattered locations across the globe, and it's not decentralized?
Damn man, what exactly would you consider "decentralized" then?
Akamai has 13, in widely scattered locations, as well. That in itself doesn't make them sufficiently decentralized.
The reason the root servers don't have this problem is that they don't all run the same software (anymore) and aren't all administrated by the same people.
I'm making an assumption here, of course, but I will not be a bit surprised if it turns out that Akamai loaded something that hit all their routers at once.
So I wasn't the only one who couldn't get to Google the Great. Fortunately, Dogpile still worked. I used that meta search engine until Google started getting big and beating all the others in turning up relevant search results.
I wonder if Google will now turn to fully manage all their assets themselves...
Please correct me if I got my facts wrong.
Now now. I'm sure most of these people don't actually mean "is the Internet down"; they really mean "is something wrong on your end?", they just lack the technical experience and vocabulary to really understand things.
When a number of sites stop working, it can be for several reasons. The last time it happened on my ISP, part of their backbone was down.
LOAD "SIG",8,1
Not too long after 9/11, I was surfing the net and needed to look up something at the Library of Congress for one of my classes. It wouldn't connect. At first I thought we'd just lost DNS (not so uncommon an occurance at my university in those days), but found I could still connect to slashdot.org and some other sites.
.edus mostly.) The ones that replied, I plotted on a US map based on their DNS LOC. (A project I wrote for a previous class.)
Being a geek, I thought up a list of about 30 sites to ping, scattered across the US. (.govs and
I freaked out a bit when the mid-atlantic seaboard came up missing. I crossed my fingers hoping that it was just some idiot who'd accidently cut one of the main fibers (which it what it ended up being) and not that Washington DC was now a big hole in the ground.
A preposition is a terrible thing to end a sentence with.
I remember when people were bashing Microsoft for using Akamai caching to avoid Windows Update getting hit by the first RPC worm (the one that was patched two months beforehand), since Akamai used Linux and it was somehow amusing that Microsoft chose that caching service.
If Akamai was running on Windows servers, I guarantee it would have been mentioned in both the headline and in the article summary today. But instead it's just mysterious "DNS issues." It's kind of like how when that Windows source code was stolen, Slashdot reported on it yet neglected to mention that the code was stolen from a hacked Linux computer at a company called Mainsoft.
Just little slants in reporting I can't help but notice.
Take a look at what internic.net gave me on some of these domains....
F REAK.ORG.RULEZ.AND.DIOX YTECH.NET.DELETED.GANDI.NET. SIMPLECODES.COMA USE.LINUXISGOD.CO M
M ICROSOFT.COM.OHMYGODITBURNS.COMV ES.JU1C3.COMO MM O FT.COM.IS.GOD.BECOUSE.UNIXSUCKS.COMM .IS.A.STEAMING.HEAP.OF.FUCKING-BULLSH IT.NETC ROSOFT.COM.HAS.ITS.OWN.CRACKLAB.COMM .HAS.A.PRESENT.COMING.FROM.HUGHESMISS ILES.COMC OM
MICROSOFT.COM.CAN.GO.FUCK.ITSELF.AT.SECZY.COMC ROSOFT.COM.ARE.GODDAMN.PIGFUCKERS.NETC OM.AND.MINDSUCK.BOTH.SUCK.HUGE.ONES.AT. EXEGETE.NET
. COM.TWIXTEARS.COMB CENTER.COMM S .1337.AS.SEARCH.GULLI.COMM . COM
7 .AS.SEARCH.GULLI.COM
T H.SEARCH.GULLI.COM I NE .THAN.SECZY.COM
Microsoft.com
----
MICROSOFT.COM.SUX.BUT.PYRO
MICROSOFT.COM.SMELLS
MICROSOFT.COM.SHOULD.GIVE.UP.BEC
MICROSOFT.COM.RAWKZ.MUH.WERLD.MENTALFLOSS.CA
MICROSOFT.COM.LO
MICROSOFT.COM.LIVES.AT.SHAUNEWING.C
MICROSOFT.COM.IS.NOT.AS.COOL.AS.SIMPLECODES.CO
MICROSOFT.COM.IS.IN.BED.WITH.CURTYV.COM
MICROS
MICROSOFT.CO
MICROSOFT.COM.HAS.TEH.GAY.OMFGLOL.COM
MI
MICROSOFT.CO
MICROSOFT.COM.FLINGS.POO.AT.MONKEYCORE.
MICROSOFT.COM.FILLS.ME.WITH.BELLIGERENCE.NET
MI
MICROSOFT.
MICROSOFT.COM
Yahoo.com
---
YAHOO.COM.WANADOODOO.COM
YAHOO
YAHOO.COM.TW
YAHOO.COM.SUPERC
YAHOO.COM.SG
YAHOO.COM.PURRFURRED.CO
YAHOO.COM.OPTIONSCORNER.COM
YAHOO.COM.IS.N0T.A
YAHOO.COM.DALLARIVA.CO
YAHOO.COM.BR
YAHOO.COM.BERKELEYNATURALBEAUTIES
YAHOO.COM.AU
YAHOO.COM
Altavista.com
---
ALTAVISTA.COM.IS.N0T.AS.133
ALTAVISTA.COM
Apple.com
---
GOOGLE.COM.SUCKS.FIND.CRACKZ.WI
GOOGLE.COM.HAS.LESS.FREE.PORN.IN.ITS.SEARCH.ENG
GOOGLE.COM
The root nameservers are not under decentralized political control, which still makes them a single point of failure, albeit a different kind of failure.
Need a Python, C++, Unix, Linux develop
Hmmm, corporate whore much? Slashdot, Debian and my own two sites seem to be working just fine. Maybe the sites you choose to visit just don't get the 'net and it's decentralized nature.
Nathan's blog
For 10 years I was a net junkie. If I didn't get my email, news, laugh, or enough time on my fav mmorpg then I was twitchy and grouchy.
:)
:) But while I'm here in the states, I *need* to be connected. I think because everybody else is.
Then, two years ago my wife and I decided to take a year off and go tour SE Asia, mainly Viet Nam.
Yes, they have Internet there but it is mainly in Internet cafes, which are hot, crowded, and quite slow. There are dialups but once you've lived on broadband for such a long time the dialup becomes something you use only when you have to. And so that was what happened. Internet became something that was used when needed. I still checked my email regularly but instead of every hour it was every 2 or 3 days, same with Slashdot.
I had a few personal (programming) projects I was working on which fit nicely onto the laptop, along with a good 20gig of mp3s. I was amazed at how fast I detached from the net. My productivity shot thru the roof, namely because my concentration was focused.
Even here in the states I have yet to reach that state of Zen again primarily because, even though I try, I know the net is right there. The little net thoughts nag at you.
But, back to the topic. You would be amazed at how much technical work you can accomplish without the net being there.
Would I give up what I have now and go back? You bet. Would I miss it? Nope. Broadband is used for P2P or games. That's all I use broadband for anyway.
On a global scope, 99% of all the really cool groundbreaking stuff in the last 100 years, computer or not, was done detached from the net.
and folks often do... witness the onerous "personal contracts" you have to sign to get into the music business, where you are essentiall a creative wage slave and don't own your stuff. non-compete and discoveries-belong clauses in your work contract also sign your rights away to The Man. similarly, if you register your DNS information independently and run your own servers, your ISP and its uplines do the same, and so on including all the sites you visit, you theoretically should not be captive to any of the commercial DNS services.
as I understand it, akamai is a distributed content hosting/caching service that also does DNS server services. they put a blade in your local ISP under contract, and popular pages from their customers serve off the local akamai server cache. they handle the DNS for those sites as I understand. if their blade caches get fed evil data, you get evil data, and www.fartblossom.org may disappear.
you can kill DNS by screwing up your own router, too. lots of ways to kill a distributed service that requires everybody to cooperate on a common set of standards and parameters.
if this is supposed to be a new economy, how come they still want my old fashioned money?
The problem is that those sites created their own single point of failure by all using Akamai for DNS. When Akamai DNS fails, sites that depend on it for their own DNS fail.
It used to be nearly impossible for this to happen. The original rules for DNS were that you had to have at least 2 nameservers for your domain, preferrably 3 or more, and they couldn't be on the same physical networks. With that rule having a single network go down rarely made any domain unresolvable (backbone networks whose outages could render dozens or hundreds of other networks unreachable being the exception). Maybe we should put the old nameserver-diversity rules back into place.
I guess seeing things like "PWD=/usr/ms/win2k_sp1/private/security/msv_sspi" isn't enough to convince this troll. So do a Google search, like I said. The code was taken from one of Mainsoft's hacked Linux machines. This was already reported in the past on other sites.
im sure the admin made a mistake -- huge corporations do not put everything on a thin wire, and hope they stay up. there are multiple connections and multiple servers. if the stuff goes down, its usually because of a mis-deployment of some new code, or, mis-deployment of some zones
whatever...i couldnt read the article either -- it wouldnt resolve. oh, the irony.
We're like rats, in some experiment! -- George Costanza
I've been on-line a lot today and didn't even know those sites were down. Didn't effect me in the least. The internet, by it's nature, will always be plagued by the occasional downtime of various services here and there. But in the end, the Internet keeps moving right along.
Think about the worst thing that's ever happened to the Internet and how much that really impacted your daily activity. I don't know about you, but it's always been local connectivity failures that have caused me the most trouble. The occasional site being down really doesn't make a big difference.
This sig has been temporarily disconnected or is no longer in service
They are telling me that it was indeed an attack, but an attack aimed not only at them but other companies as well.
I wonder what really happened and who else was attacked..
The Register must be wrong about this. I used to work at Akamai, and I feel pretty damn sure that no one crashed those servers by getting *on* them to run the 20-line snippet of code that locks the kernel (assuming we're talking about the kernel lock exploit that was being widely discussed recently; it requires shell access).
What is much more likely is that somebody found a way to DDOS the Akamai top-level name servers, or that configuration files containing incorrect/conflicting/nefarious information were pushed out to the top-levels.
Knowing how many stages and checks there are in the Akamai deployment procedures, and how much monitoring there is of the network health, I would be astonished if someone managed to foobar the top-levels with a bad configuration. A co-wortker of mine did it once, a long time ago, so I guess it *could* happen, but it was one of those perfect-storm sorts of things. And even then, it just slowed things down a little - certainly not enough to make the news like this.