Slashdot Mirror
Akamai Having Problems?
A reader writes:"It appears that sometime during the night, Akamai had some problems causing some connectivitly issues with many hosts thoughout the night. Akamai provides a DNS load balancing solution to many major internet companies/sites including (but notlimited to) Google, Yahoo, etc. Is it a bad idea to rely so heavily upon one service for our major internet needs? " Not much details - but I can confirm having problems this morning. Thanks to alert readers for pointing that they were having "DoS related issues" and that service was restored as of 1400 GMT.
Perhaps this is related to the SBC strike?
Akamai also hosts files (images, binaries) for many major websites. Seems like they have some pretty insane bandwidth too...
- here
- here
- here
- here
Nice bit of bandwidth theft, there.Their system is supposed to be distributed in such a way that any major outage in a section of the internet would not affect their overall ability to deliver the content, so presuambly any outage an ISP would not hit their too hard.
w ww.peacefire.o rg/bypass/Proxy/akamai.html
BTW something interesting:
http://a1.g.akamaitech.net/6/6/6/6/
**FREE** Track and view your phone's via CellID and/or WIFI and/or GPS
I love how the first reaction when something goes wrong is to replace it, or introduce competiton, or whatever. Yes, there are plenty of times when a service needs competition to encourage it to suck less. But go find me another company that is even remotely prepared to do DNS load-balancing. Verisign? Oh, that's a great idea. Going to start one yourself? Let us know when you have the infrastructure.
The fact is, we have NO idea what caused this. There's no link to any story anywhere - just one reader report. It could be Akamai's fault. It could be their upstream providers. It could be failures elsewhere in the Internet. Could be someone uploaded a bad zone file. Or maybe some over-zealous backhoe operator slashed some fiber somewhere.
It's probably best to reserve judgement until you have all the facts. (And if you're about to hit the reply button, yes, I'd say the exact same thing if MSFT lost their DNS service).
There is no sig, there is only Zuul.
People say that the Internet can't be knocked out. That may be true in the infra-structure sense, but if you're able to knock down Akamai or any other major solution provider, think of the sites that would go down (Google, Yahoo et al), and the repercution on the global economy. So yes, the domino theory doesn't apply to the Internet, but it becomes exponentially more dangerous when we rely on one domino for a significant share of of communications.
Our Akamai rep tells us that it was an issue with a software version rollout. They flushed all their image caches, and effectively caused a DOS on themselves.
A guy I spoke with this morning at Akamai said this morning that the problem was NOT the result of any outside attack on the company's servers. Rather, he said, the problem stemmed from a bug within a tool that allows customers to purge old content and update their cache with new content. Akamai said the problem lasted about 90 minutes, and affected numerous Akamai customers. No response, though, as to why this bug suddenly reared its head.
...because you never know who you're dealing with.
This is nicely commented on in a recent story over at CFO where it says "Broadly speaking, Akamai needs servers near the consumers of content..[] Akamai, on the other hand, has servers pretty much everywhere."
To trim the facts down a bit: Akamai has servers near by most users these days, and the distributed DNS gives you returning DNS to the closest contentserver. If I, who live in Norway, try to access fbi.gov from any computer from a ISP connected to the NIX (Norwegian Internet eXchange) I get a DNS response that leads me to Akamais servers in Oslo, Norway. I've tried this for some time, just to see what happens, with cnn.com, apple.com and fbi.gov. While on a trip to Sweden I tried this while connecting through a local DSL-provider and I got a response from a server located in Sweden, hence even the swedes have their own Akamai mirror these days.
The problems with a DDOS from someone in Norway would, if directed towards a domain or webpage and not an IP-address lead to downtime on that specific local mirror, not Akamais entire network. We can from this conclude that only such events as a major blackout in Akamais core network or like this time, DOS'ing their own network would take out their service.
"-Who said sit down?!"
-- S. Ballmer @ MSDC 2003.
An isolated issue occurred this morning (roughly during the period of 8:00 a.m. - 9:30 a.m. ET), where multiple Akamai customers experienced intermittent performance and availability degradation.
This degradation was the result of a bug within one of Akamai's backend content control management tools, which allows the expiration of content on the Akamai network. The degradation was not a result of any outside interference with Akamai's network (such as Denial of Service or hacking).
Upon identification of the bug, Akamai quickly took corrective action which returned customers to normal service levels. Akamai is currently putting measures in place to return the content management tool to its normal working order and is adding safeguards such that the issue will not occur in the future. In the meantime, Akamai customers are able to serve their content through the Akamai Network normally.
As part of Akamai's normal proactive customer communication policy, Akamai customers will be kept informed of the latest developments through the Akamai portal, the EdgeControl Management Center, https://control.akamai.com. Any further inquiries may be directed at Akamai Customer Care at 1-877-4-AKATEC.