Slashdot Mirror
Akamai DNS Outage Messes up Net
katre writes "Checking all my favorite sites this morning, I saw that about half a dozen seem to be offline. Trying to figure out why, I found an interesting article on the front page at http://isc.incidents.org/. Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others. Whatever happened to my decentralized net with no single point of failure?"
Yahoo is already resolving through scd instead of akamai. I didn't check any of the others.
If you clear your cache, you will probably get the new entries, unless your ISP hasn't caught onto the problem yet.
...I can't even get to http://isc.incidents.org/
This should cause some problems for akami, they had an outage may 24th. Once can be overlooked twice? these are some big companies they are going to be calling them. I bet there is some sweating techs in the cool noc right now
War isn't about who's right. It's about who's left.
I don't think this had anything whatsoever to do with any of the root servers. This has to do with Akamai's DNS servers, and the companies (domains) that are using them.
Pwned by CNAME to Akamai?
(You can't have CNAME records for the base domain, hence google.com would have had an A record instead, whilst www.google.com would have been a CNAME to akamai)
It would be hard to do most of my work (Server Maint.) without the net. I might have to actually go to the servers instead of ssh. Wait, what am I talkin about, without the net I wouldn't HAVE a job.
How to Speak Leet
The Root DNS servers are kept up-to-date. .com part, for IP of the responsible DNS server. This server you query regarding the IP the google.com server (and the DNS server responsible for google.com). .com adresses) points to a working google server and you can use that one instead.
But they don't supply subdomain DNS services (www.google.com), only TLD DNS services (google.com)
Otherwise the rootservers would be overwhelmed with the amount of data they would have to handle.
If you look for e.g. www.google.com, first you lookup the
Then you ask the google.com DNS server for the www.google.com IP adress.
Unfortunately that last server is not responding, so you get stuck. But luckily, in this case, the google.com IP adress (recieved from the DNS server for
+++ MELON MELON MELON +++ Out of Cheese Error +++ redo from start +++
#define CLUE 0
Uhm, the root servers are not overloaded... this has nothing to do with the root servers, this has to do with Akamai having problems.
They have a private cached network they sell access to. It's like taking a service road around crowded highways to get closer to the final destination.
One of the companies I used to work for used Akamai, nice network... not so great customer service unless you are a really big customer.
This sig is the express property of someone.
Solution to akamai problems:
go to <a href="http://www.dnsstuff.com/">your favorite DNS lookup page</a> and lookup the akamai hosted site. (getting the real address rather than the akamized version) Now open your hosts file and add that in.
Now you will always get the non-akamized version of that site. Akamai problem solved.
I keep google in my hosts just so I can be sure that DNS issues like this won't cut me off from my favorite search engine.
I run a small ISP and we happen to have 3 of their linux boxes on our network. I've never experienced a problem with them before today. For the hack of it we decided to just reboot their servers and now things are working correctly.
For those that were wondering why it would affect DNS; Akamai somehow tinkers with DNS and BGP to redirect content to their edge servers.
As for Akamai being outdated, it still seems to me that its a good idea for Yahoo and some of the high traffic sites on the net. Akamai has thousands of distributed servers colocated with ISPs and NAPs. And they do seem to absorb nasty bursts in traffic (ie Star Report) better than a centralized server farm. But for their own sake, they better hope to not have another repeat of todays events.
From NANOG:
From here neither www.google.com, nor www.apple.com work. Both seem to return CNAMES to akadns.net addresses (eg, www.google.akadns.net, www.apple.com.akadns.net), and from here all of the akadns.net servers listed in whois are failing to respond.
The real cost of a web site dropping is a lot more difficult to figure out than you might imagine. Say Amazon goes down for a couple of hours. Are all those potential sales lost forever? I doubt it. Some people will just come back and order later. The firm is unlikely to see any long term impact unless the outage becomes habitual. Non-retail sites probably have even more flexability. About the only area in which an outage could have a real, long term adverse impact would likely be in financial services. If Schwab goes down for half a day they will suffer big time for a long time. If you're talking "the economy" as in the big picture economy" suffering - forget it. Web based commerace isn't that important yet.
most big sites have changed their DNS CNAMEs to point directly to one of their datacenters rather than relying on Akamai to route users to the "nearest" datacenter.
SIGUSR1
It appears that, at around 8:30 AM EDT (US Eastern Daylight Time), Akamai's DNS network experiened some kind of major failure. All of their DNS servers (that anybody could find) were not responding to DNS queries. It appears that Akamai started to come back online at around 10:00 AM EDT.
Since a great many big name sites use Akamai, this effectively made large parts of the Internet unreachable. The destination servers themselves were up, but clients were unable to turn names (like www.example.com) into network addresses (like 192.0.2.42).
As Akamai maintains dozens, if not hundreds, of DNS servers across the globe, it is extremely unlikely that this was due to a normal equipment failure or DoS attack. Some kind of internal system trouble is much more likely. Whether a deliberate attack, or an accident, is unknown to me at this time. It could just be an internal configuration change blew up in a really bad way. Sh*t happens.
I do not know if this was just an Akamai DNS problem, or if other Akamai services were also affected.
Due to the way Akamai is usually implemented, it happened that, in many cases, the second-level domain names (like example.com) worked, but subdomains (like www.example.com and mail.example.com) did not. This is because most organizations put in CNAME records (pointing to names in *.akadns.net) for the subdomains. You cannot use a CNAME record for a domain that has other records, though, so most domains still had traditional A records, on their own nameservers, at the second-level.
The following sites/organizations are known to use Akamai: Yahoo, Google, Microsoft, Altavista, FedEx, Xerox, Apple
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Newsfollow.com
From NANOG mailing list again:
Google pulled references for akamais dns servers a short period ago. they are presently serving their own dns requests.
Also:
People seem to be getting around this by changing their DNS entries.
E.g. www.yahoo.com always used to be a CNAME for www.yahoo.akadns.net. But
now:
# host www.yahoo.com
www.yahoo.com is an alias for www.dcn.yahoo.com.
www.dcn.yahoo.com has address 216.109.118.64
www.dcn.yahoo.com has address 216.109.118.65
www.dcn.yahoo.com has address 216.109.118.66
www.dcn.yahoo.com has address 216.109.118.67
www.dcn.yahoo.com has address 216.109.118.68
www.dcn.yahoo.com has address 216.109.118.69
www.dcn.yahoo.com has address 216.109.118.70
www.dcn.yahoo.com has address 216.109.118.71
www.dcn.yahoo.com has address 216.109.118.72
www.dcn.yahoo.com has address 216.109.118.73
www.dcn.yahoo.com has address 216.109.118.74
www.dcn.yahoo.com has address 216.109.118.75
Which is owned by Yahoo! (via HotJobs.com).
Handlers Diary June 15th 2004
m sg05267. html
Updated June 15th 2004 14:31 UTC (Handler: Lenny Zeltser)
Akamai DNS outage
Akamai DNS problem
Starting at around 8:30 am EDT (12:30 UTC), a number of sources started to report a widespread Akamai DNS issue. Large web sites, which use Akamai for its DNS service, did no longer resolve. Effected sites are Yahoo, Google, Microsoft, Fedex, Xerox, Apple and likely many others.
At this time (10:30 am EDT), some effected domains removed the Akamai DNS servers and are reachable again using their own DNS servers.
Typically, the domain itself (e.g. 'google.com') still resolves, but popular hostnames, like 'www.google.com' will not resolve. As a result, the web site is no longer reachable.
The effect appears to be world wide. Some of the Akamai servers do respond to pings, but do not respond to DNS queries.
posts to the NANOG mailing list regarding this issue:
http://www.merit.edu/mail.archives/nanog/
It's never too late to have a happy childhood.
yoink
You know, in hawaiian, "akamai" means smart...
Robert
How much do you want to bet someone was fiddling with the database and accidentally dropped a table, or tried to delete a record in SQL but forgot the "where" clause.
For you non-database people out there, that's the SQL equivilent of "rm -rf", except that it's easier to do because SQL defaults to dealing with all record unless you tell it otherwise.
"Learning is not compulsory... neither is survival."
--Dr.W.Edwards Deming
The most obvious example? The fact is that there are 13 of them, in widely scattered locations across the globe, and it's not decentralized?
Even more to your point, there are many more than 13 root name servers. There are 13 root name server IP addresses, but some of those belong to many different servers.
For example, the "f" root server is really 22 servers, themselves distributed around the world. Check out ISC F-Root Information.
I don't know how many root servers there really are, though. Anyone?
The reason why it's a mysterious "DNS issues" is because we don't know what the problem is. It'd be the same if it was a Windows DNS server (not that anybody uses those for major networks like Akamai). Seeing as Akamai uses more than one DNS server it's more likely a administrator error than a Linux crash. Nobody would be blaming Windows if an administrator screwed up.
You are also confusing their cache servers with their DNS servers. They're completely different.
The global economy is a great thing until you feel it locally.
I wouldn't presume they use any for their dns funtionality, but fact of the matter is Akamai does have a small proportion of windows servers in their distributed clusters. Seen 'em with my own eyes.
...according to this story at washingtonpost.com The story says it was a distributed denial of service attack against Akamai, among others.
...because you never know who you're dealing with.
Summary:
Between approximately 8:30 AM ET and 10:45 AM ET (GMT +4 hours) on Tuesday, June 15, 2004, some Akamai customers using Global Traffic Manager (FirstPoint), NetStorage (Akamai Content Storage), and Akamai services that utilize Global Traffic Manager and NetStorage experienced performance and availability issues.
This incident resulted from a sophisticated, large-scale attack on Internet infrastructure. This attack impacted Akamai's Internet naming functionality (Domain Name Service or DNS), and resulted in delays in DNS name resolution and, in some cases, timed-out DNS requests. Some end users trying to reach affected sites would have experienced slow responses from the Akamai name servers, potentially resulting in page time-outs. The attack did not cause an outage in Akamai services, as Akamai continued to serve DNS requests. However, the amount and nature of attack traffic created degradation in performance.
The problem was quickly detected by Akamai's automated monitoring systems, and Akamai personnel identified the root cause as a large Internet attack. The attack was mitigated by a combination of actions by Akamai to adjust our infrastructure in response to the attack, along with working with network partners to shut down the source of the attack.
As result of these actions, all Akamai services had returned to normal operating performance by 10:45 AM ET.
Akamai is continuing to work closely with several network partners and legal authorities around the world to identify both the nature of the attack and its intended targets.
We regret any inconvenience this may have caused you or your users. Please contact your Akamai Customer Care representative at 1-877-4-AKATEC (1-877-425-2832) if you have any questions.
Service Note: One of the actions taken during the attack was to temporarily increase the DNS TTL (time to live) on responses being returned from Akamai. This action is helping end-users cache successful responses for longer, thus improving service.
Pretty cool stuff, to be sure.
But all of the proprietary stuff means that there's only one implementation. There's no RFC describing what they do. There's no alternate implementations that might show flaws. There's no cross-checks that outsiders might provide.
Like others have said, it's a mono-culture. And they've done it so well, there's been no interest in creating a set of standards or IETF working group to try and create the multiple, compatible offerings that might guard against mono-culture (and give customers a chance to avoid vendor lock-in.)
(Score:5, Insightful, right...) Actually, it was. If Google et al were all using a single Akamai backbone TCP/IP routers and they went down, they would be affected as well.
Google was using some DNS servers as their DNS servers (NSs for their domain zone). Their servers went down and then Google was unreachable because their DNS was down, nothing more. Nothing magical about DNS per se. TCP/IP routing was working but this hardly means DNS is any more "centeral point of failure" than TCP/IP. Google should not rely on a single network of DNS servers and it would be fine, because DNS is designed in such a way and has been for over twenty years.
The problem here is the bastardization of DNS standard by Akamai. DNS records should be cached on recursive name servers. Google is used everywhere. If Google had sane TTL and expiration times set for their zone, their zone would be cached by every ISP in the world and their DNS servers could be down for a week and no one would even notice.
This is how DNS should work, can work, and have been working for literally decades. Please read RFC 882: DOMAIN NAMES - CONCEPTS and FACILITIES (P. Mockapetris, November 1983), RFC 883: DOMAIN NAMES - IMPLEMENTATION and SPECIFICATION (P. Mockapetris, November 1983), RFC 1034: DOMAIN NAMES - CONCEPTS AND FACILITIES (P. Mockapetris, November 1987) and RFC 1035: DOMAIN NAMES - IMPLEMENTATION AND SPECIFICATION (November 1987).
Sincerely,
Pan Tarhei Hosé, PhD.
"Homo sum et cogito ergo odi profanum vulgus et libido."