Akamai Having Problems?

A reader writes:"It appears that sometime during the night, Akamai had some problems causing some connectivitly issues with many hosts thoughout the night. Akamai provides a DNS load balancing solution to many major internet companies/sites including (but notlimited to) Google, Yahoo, etc. Is it a bad idea to rely so heavily upon one service for our major internet needs? " Not much details - but I can confirm having problems this morning. Thanks to alert readers for pointing that they were having "DoS related issues" and that service was restored as of 1400 GMT.

Comments

Apple down, Microsoft up

[G�tz]

I can confirm problems accessing the apple.com trailers, but microsoft.com has no problems. I thought they were using Akamai's services as well?

Re:Apple down, Microsoft up

[the frizz]

See the Speedrank index for the affects this has had on 100 popular web sites.

Disclaimer. I work for Speedera, an Akamai competitor.

apple trailers

[pinky99]

yes, i noted also it, when i wanted to watch new movie trailers at apple's qt site, which is appearantly and unfortunately hosted by akamai.

Internet Storm Centre has a little

[Zocalo]

"Akamai problems. Quiet, well kinda quiet, day on the Internet Update (Mon. May 24th 9 am EST, 13:00 UTC, 15:00 CEST)

It appears that websites that use Akamai's distribution system are currently not reachable. Security related web sites effected are symantec.com and trendmicro.com. Virus updates may fail as a result. Further details are currently not available and updates will be posted here as they become available. Thanks to Vidar Wilkens for alerting us of this problem.

According to a post to NANOG, the outage may be the result of a DDOS attack. At this point, Akamai has not ETA for a resolution.

Update 09:45 EST: Looks like some of the Akamai hosted sites start to come back."

You gotta love that "Quiet, well kinda quiet". ;)

NOC Says:

[j0keralpha]

Akamai's NOC says service restored approx 1400GMT. Earlier NOC quotes include: It is a system-wide problem that "looks like it may be a DOS attack".

Re:NOC Says:

[MoonBuggy]

While collectively Akamai is near impervious, there's probably a 'weak link' in there somewhere. I would guess that the servers which direct you to the local cache were the target - they deal only with requests and routing so they wouldn't need anything like the bandwidth that the actual media caching servers have, and if the media servers are up but the routing servers are down then the system is essentailly dead.

Kinda like the time they DDoSed some of the DNS roots - if they'd got a few more of them it could've pretty much taken out the entire web without actually needing to attempt the near impossible task of offlining all of the millions(?) of normal site servers out there.

Yahoo had trouble for at least an hour or so.

[Anonymous Coward]

Yahoo had trouble for at least an hour or so.

Apple.com Slow down

[koniosis]

Me and a lot of people I know have been having issues with apple.com specifically the quick time trailers section. Download speeds hit rock bottom, at about 200bytes/second on a 3MB cable connection. As I said, this was a number of people experiencing the same speeds.

Blueyonder UK

Discussed on Nanog...

[rf0]

NANOG Archieve

Rus

Answer

[Mr_Silver]

Is it a bad idea to rely so heavily upon one service for our major internet needs?

Of course it is a bad idea.

However, blame that on the other competing services who haven't become cheaper, faster or better at whatever it is that makes Akamai so popular.

eBay affected also

[jelevy01]

I couldn't get to eBay this morning either. It seems to be resolved now though.

Re:I thought they do file hosting also

[r_cerq]

No, they don't need to. Akamai's model is to install a bunch of their own machines (a PoP) in each and every middle-to-large ISP. They then use source-based DNS to direct requests to the nearest PoP (with some luck, it'll be within your ISP's network). They basically work as a smart reverse-proxy. You make your request to their PoP, and the PoP serves the content from cache. If you happen to be the first person requesting said content, the PoP will fetch it from the originating server (Apple, MS, CNN, whatever) and cache it to serve following request.

Notice on Akamai Control site

[Reckless Visionary]

Akamai has posted a notice on the website customers use to get reporting and manage content.

Due to a peering problem between ATT and UUNet, a subset of UUNet users may have experienced problems accessing Akamai delivered sites between 8-10pm EDT on Saturday May 22, 2004. The problem has been fully resolved.

Re:Notice on Akamai Control site

[Zocalo]

8-10pm EDT on Saturday May 22, 2004

Well, unless you have a *really* bad latency problem, I don't think that's going to be an issue with a problem on May 24th...

from their support website

[john_uy]

Advisories

Due to a peering problem between ATT and UUNet, a subset of UUNet users may have experienced problems accessing Akamai delivered sites between 8-10pm EDT on Saturday May 22, 2004. The problem has been fully resolved.

Maybe the problem has recurred.

Re:from their support website

[jea6]

This was a different issue altogether. Saturday's issue only affected incoming traffic from any UUNet network. Today's issue was much more widespread.

Akamai says it's a bug in the software, not DDoS

[tsu doh nimh]

A guy I spoke with this morning at Akamai said this morning that the problem was NOT the result of any outside attack on the company's servers. Rather, he said, the problem stemmed from a bug within a tool that allows customers to purge old content and update their cache with new content. Akamai said the problem lasted about 90 minutes, and affected numerous Akamai customers. No response, though, as to why this bug suddenly reared its head.

Akamais distributed DNS & content solutions

[akaiONE]

Akamai may have problems from time to time over in the US, while not in Europe. The fact that Akamai uses a distributed network of both DNS and content servers helps them deliver content to most users in other regions even if some servers are down in the US.

This is nicely commented on in a recent story over at CFO where it says "Broadly speaking, Akamai needs servers near the consumers of content..[] Akamai, on the other hand, has servers pretty much everywhere."

To trim the facts down a bit: Akamai has servers near by most users these days, and the distributed DNS gives you returning DNS to the closest contentserver. If I, who live in Norway, try to access fbi.gov from any computer from a ISP connected to the NIX (Norwegian Internet eXchange) I get a DNS response that leads me to Akamais servers in Oslo, Norway. I've tried this for some time, just to see what happens, with cnn.com, apple.com and fbi.gov. While on a trip to Sweden I tried this while connecting through a local DSL-provider and I got a response from a server located in Sweden, hence even the swedes have their own Akamai mirror these days.

The problems with a DDOS from someone in Norway would, if directed towards a domain or webpage and not an IP-address lead to downtime on that specific local mirror, not Akamais entire network. We can from this conclude that only such events as a major blackout in Akamais core network or like this time, DOS'ing their own network would take out their service.

Nothing to see here...move along.

[Hiawatha]

Akamai just told me it was a 90-minute glitch (between 8 and 9:30 Eastern time) caused by a software bug. The company says everything's back to normal.

Akamai was down from 8:00am to 9:15am

[dloyer]

We are an Akamai customer. All of our content cached through Akamai was offline for a little over an hour as measured by keynote, a site testing tool.

I spoke with Akamai support. They indicated that it was a far reaching problem, but I have not heard the reason yet.

The customer login to the admin portal was down as well. It was almost like someone dump the customer account database.

Akamai has a QOS commitment of 100% uptime based on the idea that not all of the 1,000's of servers could go down at the same time. But... There you go.

Re:i've always wondered...

[ps]

ak a my

Simple. Just like it looks.

Explaination from Akamai

[slashusrslashbin]

An isolated issue occurred this morning (roughly during the period of 8:00 a.m. - 9:30 a.m. ET), where multiple Akamai customers experienced intermittent performance and availability degradation.

This degradation was the result of a bug within one of Akamai's backend content control management tools, which allows the expiration of content on the Akamai network. The degradation was not a result of any outside interference with Akamai's network (such as Denial of Service or hacking).

Upon identification of the bug, Akamai quickly took corrective action which returned customers to normal service levels. Akamai is currently putting measures in place to return the content management tool to its normal working order and is adding safeguards such that the issue will not occur in the future. In the meantime, Akamai customers are able to serve their content through the Akamai Network normally.

As part of Akamai's normal proactive customer communication policy, Akamai customers will be kept informed of the latest developments through the Akamai portal, the EdgeControl Management Center, https://control.akamai.com. Any further inquiries may be directed at Akamai Customer Care at 1-877-4-AKATEC.

Re:latest advisory

[Anonymous Coward]

"Akamai Customer Response - May 24rd 2004 11:41am ET Degradation Issue
An isolated issue occurred this morning (roughly during the period of 8:00 a.m. - 9:30 a.m. ET), where multiple Akamai customers experienced intermittent performance and availability degradation.

This degradation was the result of a bug within one of Akamai's backend content control management tools, which allows the expiration of content on the Akamai network. The degradation was not a result of any outside interference with Akamai's network (such as Denial of Service or hacking).

Upon identification of the bug, Akamai quickly took corrective action which returned customers to normal service levels. Akamai is currently putting measures in place to return the content management tool to its normal working order and is adding safeguards such that the issue will not occur in the future. In the meantime, Akamai customers are able to serve their content through the Akamai Network normally."

We were affected too, this is the RCA.

Scalability and bandwidth

[billstewart]

No, Akamai as a whole really does have humongous amount of bandwidth, it's just distributed among 14000+ small machines. Their web site says they crank out "40 GPS", which is probably gigabits per second rather than gigabytes per second, so that's about 3 Mbps per machine, and that's probably aggregate peak delivered bandwidth, but most of their machines probably have a lot more capacity than that (10 Mbps would seem to be obvious for the smaller Ethernet-connected ones), because different machines will be busy at different times. It's not the kind of job that needs lots of CPU, but it does need lots of memory (at least by the standards of when the initial machines were deployed), because you don't want to wait 10ms for a disk drive to fetch your data when the reason the content provided chose you was to speed up their delivery and cut out latency (though you could get some performance wins by locking the first 10-20ms of each file in RAM and paging the rest.)

Akamai's competitors have different scaling tradeoffs. The last time I knew numbers was a couple of years ago, and it may have changed, but Akamai had a very large number of mostly small servers located on many carriers networks, AT&T had a couple hundred very large servers (mostly at peering points, which takes advantage of being a carrier, though they also bought some transit for content distribution), and Speedera was somewhere in between. AT&T's directions included lots of streaming media, and Akamai was doing fancy database things.