Slashdot Mirror
Akamai Having Problems?
A reader writes:"It appears that sometime during the night, Akamai had some problems causing some connectivitly issues with many hosts thoughout the night. Akamai provides a DNS load balancing solution to many major internet companies/sites including (but notlimited to) Google, Yahoo, etc. Is it a bad idea to rely so heavily upon one service for our major internet needs? " Not much details - but I can confirm having problems this morning. Thanks to alert readers for pointing that they were having "DoS related issues" and that service was restored as of 1400 GMT.
No, they don't need to. Akamai's model is to install a bunch of their own machines (a PoP) in each and every middle-to-large ISP. They then use source-based DNS to direct requests to the nearest PoP (with some luck, it'll be within your ISP's network). They basically work as a smart reverse-proxy. You make your request to their PoP, and the PoP serves the content from cache. If you happen to be the first person requesting said content, the PoP will fetch it from the originating server (Apple, MS, CNN, whatever) and cache it to serve following request.
A guy I spoke with this morning at Akamai said this morning that the problem was NOT the result of any outside attack on the company's servers. Rather, he said, the problem stemmed from a bug within a tool that allows customers to purge old content and update their cache with new content. Akamai said the problem lasted about 90 minutes, and affected numerous Akamai customers. No response, though, as to why this bug suddenly reared its head.
...because you never know who you're dealing with.
This is nicely commented on in a recent story over at CFO where it says "Broadly speaking, Akamai needs servers near the consumers of content..[] Akamai, on the other hand, has servers pretty much everywhere."
To trim the facts down a bit: Akamai has servers near by most users these days, and the distributed DNS gives you returning DNS to the closest contentserver. If I, who live in Norway, try to access fbi.gov from any computer from a ISP connected to the NIX (Norwegian Internet eXchange) I get a DNS response that leads me to Akamais servers in Oslo, Norway. I've tried this for some time, just to see what happens, with cnn.com, apple.com and fbi.gov. While on a trip to Sweden I tried this while connecting through a local DSL-provider and I got a response from a server located in Sweden, hence even the swedes have their own Akamai mirror these days.
The problems with a DDOS from someone in Norway would, if directed towards a domain or webpage and not an IP-address lead to downtime on that specific local mirror, not Akamais entire network. We can from this conclude that only such events as a major blackout in Akamais core network or like this time, DOS'ing their own network would take out their service.
"-Who said sit down?!"
-- S. Ballmer @ MSDC 2003.
An isolated issue occurred this morning (roughly during the period of 8:00 a.m. - 9:30 a.m. ET), where multiple Akamai customers experienced intermittent performance and availability degradation.
This degradation was the result of a bug within one of Akamai's backend content control management tools, which allows the expiration of content on the Akamai network. The degradation was not a result of any outside interference with Akamai's network (such as Denial of Service or hacking).
Upon identification of the bug, Akamai quickly took corrective action which returned customers to normal service levels. Akamai is currently putting measures in place to return the content management tool to its normal working order and is adding safeguards such that the issue will not occur in the future. In the meantime, Akamai customers are able to serve their content through the Akamai Network normally.
As part of Akamai's normal proactive customer communication policy, Akamai customers will be kept informed of the latest developments through the Akamai portal, the EdgeControl Management Center, https://control.akamai.com. Any further inquiries may be directed at Akamai Customer Care at 1-877-4-AKATEC.