Slashdot Mirror
Akamai DNS Outage Messes up Net
katre writes "Checking all my favorite sites this morning, I saw that about half a dozen seem to be offline. Trying to figure out why, I found an interesting article on the front page at http://isc.incidents.org/. Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others. Whatever happened to my decentralized net with no single point of failure?"
Yahoo is already resolving through scd instead of akamai. I didn't check any of the others.
If you clear your cache, you will probably get the new entries, unless your ISP hasn't caught onto the problem yet.
...I can't even get to http://isc.incidents.org/
This should cause some problems for akami, they had an outage may 24th. Once can be overlooked twice? these are some big companies they are going to be calling them. I bet there is some sweating techs in the cool noc right now
War isn't about who's right. It's about who's left.
I don't think this had anything whatsoever to do with any of the root servers. This has to do with Akamai's DNS servers, and the companies (domains) that are using them.
Pwned by CNAME to Akamai?
(You can't have CNAME records for the base domain, hence google.com would have had an A record instead, whilst www.google.com would have been a CNAME to akamai)
It would be hard to do most of my work (Server Maint.) without the net. I might have to actually go to the servers instead of ssh. Wait, what am I talkin about, without the net I wouldn't HAVE a job.
How to Speak Leet
Uhm, the root servers are not overloaded... this has nothing to do with the root servers, this has to do with Akamai having problems.
They have a private cached network they sell access to. It's like taking a service road around crowded highways to get closer to the final destination.
One of the companies I used to work for used Akamai, nice network... not so great customer service unless you are a really big customer.
This sig is the express property of someone.
Solution to akamai problems:
go to <a href="http://www.dnsstuff.com/">your favorite DNS lookup page</a> and lookup the akamai hosted site. (getting the real address rather than the akamized version) Now open your hosts file and add that in.
Now you will always get the non-akamized version of that site. Akamai problem solved.
I keep google in my hosts just so I can be sure that DNS issues like this won't cut me off from my favorite search engine.
I run a small ISP and we happen to have 3 of their linux boxes on our network. I've never experienced a problem with them before today. For the hack of it we decided to just reboot their servers and now things are working correctly.
For those that were wondering why it would affect DNS; Akamai somehow tinkers with DNS and BGP to redirect content to their edge servers.
As for Akamai being outdated, it still seems to me that its a good idea for Yahoo and some of the high traffic sites on the net. Akamai has thousands of distributed servers colocated with ISPs and NAPs. And they do seem to absorb nasty bursts in traffic (ie Star Report) better than a centralized server farm. But for their own sake, they better hope to not have another repeat of todays events.
From NANOG:
From here neither www.google.com, nor www.apple.com work. Both seem to return CNAMES to akadns.net addresses (eg, www.google.akadns.net, www.apple.com.akadns.net), and from here all of the akadns.net servers listed in whois are failing to respond.
The real cost of a web site dropping is a lot more difficult to figure out than you might imagine. Say Amazon goes down for a couple of hours. Are all those potential sales lost forever? I doubt it. Some people will just come back and order later. The firm is unlikely to see any long term impact unless the outage becomes habitual. Non-retail sites probably have even more flexability. About the only area in which an outage could have a real, long term adverse impact would likely be in financial services. If Schwab goes down for half a day they will suffer big time for a long time. If you're talking "the economy" as in the big picture economy" suffering - forget it. Web based commerace isn't that important yet.
It appears that, at around 8:30 AM EDT (US Eastern Daylight Time), Akamai's DNS network experiened some kind of major failure. All of their DNS servers (that anybody could find) were not responding to DNS queries. It appears that Akamai started to come back online at around 10:00 AM EDT.
Since a great many big name sites use Akamai, this effectively made large parts of the Internet unreachable. The destination servers themselves were up, but clients were unable to turn names (like www.example.com) into network addresses (like 192.0.2.42).
As Akamai maintains dozens, if not hundreds, of DNS servers across the globe, it is extremely unlikely that this was due to a normal equipment failure or DoS attack. Some kind of internal system trouble is much more likely. Whether a deliberate attack, or an accident, is unknown to me at this time. It could just be an internal configuration change blew up in a really bad way. Sh*t happens.
I do not know if this was just an Akamai DNS problem, or if other Akamai services were also affected.
Due to the way Akamai is usually implemented, it happened that, in many cases, the second-level domain names (like example.com) worked, but subdomains (like www.example.com and mail.example.com) did not. This is because most organizations put in CNAME records (pointing to names in *.akadns.net) for the subdomains. You cannot use a CNAME record for a domain that has other records, though, so most domains still had traditional A records, on their own nameservers, at the second-level.
The following sites/organizations are known to use Akamai: Yahoo, Google, Microsoft, Altavista, FedEx, Xerox, Apple
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Newsfollow.com
From NANOG mailing list again:
Google pulled references for akamais dns servers a short period ago. they are presently serving their own dns requests.
Also:
People seem to be getting around this by changing their DNS entries.
E.g. www.yahoo.com always used to be a CNAME for www.yahoo.akadns.net. But
now:
# host www.yahoo.com
www.yahoo.com is an alias for www.dcn.yahoo.com.
www.dcn.yahoo.com has address 216.109.118.64
www.dcn.yahoo.com has address 216.109.118.65
www.dcn.yahoo.com has address 216.109.118.66
www.dcn.yahoo.com has address 216.109.118.67
www.dcn.yahoo.com has address 216.109.118.68
www.dcn.yahoo.com has address 216.109.118.69
www.dcn.yahoo.com has address 216.109.118.70
www.dcn.yahoo.com has address 216.109.118.71
www.dcn.yahoo.com has address 216.109.118.72
www.dcn.yahoo.com has address 216.109.118.73
www.dcn.yahoo.com has address 216.109.118.74
www.dcn.yahoo.com has address 216.109.118.75
Which is owned by Yahoo! (via HotJobs.com).
Handlers Diary June 15th 2004
m sg05267. html
Updated June 15th 2004 14:31 UTC (Handler: Lenny Zeltser)
Akamai DNS outage
Akamai DNS problem
Starting at around 8:30 am EDT (12:30 UTC), a number of sources started to report a widespread Akamai DNS issue. Large web sites, which use Akamai for its DNS service, did no longer resolve. Effected sites are Yahoo, Google, Microsoft, Fedex, Xerox, Apple and likely many others.
At this time (10:30 am EDT), some effected domains removed the Akamai DNS servers and are reachable again using their own DNS servers.
Typically, the domain itself (e.g. 'google.com') still resolves, but popular hostnames, like 'www.google.com' will not resolve. As a result, the web site is no longer reachable.
The effect appears to be world wide. Some of the Akamai servers do respond to pings, but do not respond to DNS queries.
posts to the NANOG mailing list regarding this issue:
http://www.merit.edu/mail.archives/nanog/
It's never too late to have a happy childhood.
Robert
The reason why it's a mysterious "DNS issues" is because we don't know what the problem is. It'd be the same if it was a Windows DNS server (not that anybody uses those for major networks like Akamai). Seeing as Akamai uses more than one DNS server it's more likely a administrator error than a Linux crash. Nobody would be blaming Windows if an administrator screwed up.
You are also confusing their cache servers with their DNS servers. They're completely different.
The global economy is a great thing until you feel it locally.
I wouldn't presume they use any for their dns funtionality, but fact of the matter is Akamai does have a small proportion of windows servers in their distributed clusters. Seen 'em with my own eyes.
...according to this story at washingtonpost.com The story says it was a distributed denial of service attack against Akamai, among others.
...because you never know who you're dealing with.