Slashdot Mirror
Akamai Having Problems?
A reader writes:"It appears that sometime during the night, Akamai had some problems causing some connectivitly issues with many hosts thoughout the night. Akamai provides a DNS load balancing solution to many major internet companies/sites including (but notlimited to) Google, Yahoo, etc. Is it a bad idea to rely so heavily upon one service for our major internet needs? " Not much details - but I can confirm having problems this morning. Thanks to alert readers for pointing that they were having "DoS related issues" and that service was restored as of 1400 GMT.
Perhaps this is related to the SBC strike?
I can confirm problems accessing the apple.com trailers, but microsoft.com has no problems. I thought they were using Akamai's services as well?
yes, i noted also it, when i wanted to watch new movie trailers at apple's qt site, which is appearantly and unfortunately hosted by akamai.
Akamai also hosts files (images, binaries) for many major websites. Seems like they have some pretty insane bandwidth too...
Posting a link to their website on Slashdot should help them out.
Beauty is in the eye of the beerholder.
And this is *news*???
I don't think it's a problem to go with one company. As long as that company has a distributed solution with many uplink providers. So, basically redundancy when something happens because no matter how good you are there will always be hickups.
Evolution or ID?
It appears that websites that use Akamai's distribution system are currently not reachable. Security related web sites effected are symantec.com and trendmicro.com. Virus updates may fail as a result. Further details are currently not available and updates will be posted here as they become available. Thanks to Vidar Wilkens for alerting us of this problem.
According to a post to NANOG, the outage may be the result of a DDOS attack. At this point, Akamai has not ETA for a resolution.
Update 09:45 EST: Looks like some of the Akamai hosted sites start to come back."
You gotta love that "Quiet, well kinda quiet". ;)
UNIX? They're not even circumcised! Savages!
- here
- here
- here
- here
Nice bit of bandwidth theft, there.Is it a bad idea to rely so heavily upon one service for our major internet needs?
We do that already. Remember when verisign introduced Sitefinder, thus effectively making various services (like spam filters etc) unusable because non-existing domains all of a sudden replied with a valid IP.
Underholdning.info
Akamai's NOC says service restored approx 1400GMT. Earlier NOC quotes include: It is a system-wide problem that "looks like it may be a DOS attack".
Their system is supposed to be distributed in such a way that any major outage in a section of the internet would not affect their overall ability to deliver the content, so presuambly any outage an ISP would not hit their too hard.
w ww.peacefire.o rg/bypass/Proxy/akamai.html
BTW something interesting:
http://a1.g.akamaitech.net/6/6/6/6/
**FREE** Track and view your phone's via CellID and/or WIFI and/or GPS
The cleaning lady needed electricity to her vacuum cleaner.
Poor sysadmins.
I demand the Cone of Silence!
NANOG Archieve
Rus
Cheap UK and US VPS
Of course it is a bad idea.
However, blame that on the other competing services who haven't become cheaper, faster or better at whatever it is that makes Akamai so popular.
Avantslash - View Slashdot cleanly on your mobile phone.
I love how the first reaction when something goes wrong is to replace it, or introduce competiton, or whatever. Yes, there are plenty of times when a service needs competition to encourage it to suck less. But go find me another company that is even remotely prepared to do DNS load-balancing. Verisign? Oh, that's a great idea. Going to start one yourself? Let us know when you have the infrastructure.
The fact is, we have NO idea what caused this. There's no link to any story anywhere - just one reader report. It could be Akamai's fault. It could be their upstream providers. It could be failures elsewhere in the Internet. Could be someone uploaded a bad zone file. Or maybe some over-zealous backhoe operator slashed some fiber somewhere.
It's probably best to reserve judgement until you have all the facts. (And if you're about to hit the reply button, yes, I'd say the exact same thing if MSFT lost their DNS service).
There is no sig, there is only Zuul.
People say that the Internet can't be knocked out. That may be true in the infra-structure sense, but if you're able to knock down Akamai or any other major solution provider, think of the sites that would go down (Google, Yahoo et al), and the repercution on the global economy. So yes, the domino theory doesn't apply to the Internet, but it becomes exponentially more dangerous when we rely on one domino for a significant share of of communications.
I guess this throws a wrench in their claim of 24/7 uptime on their main page. Nice how their marketing team says 100% availability, when people get PhD's by adding more 9's to their 99.99..%'s
I hope its not my fault. I knocked out 3 of akamai servers with a router problem =(
War isn't about who's right. It's about who's left.
The outage was apparently related to a DDOS attack against project Gutenberg that started this morning.
At work we lost connectivity to a handful of remote sites located in the Northeast, Midwest, and Southeast. Other sites in the same region but different cities were not affected. I was told it was a fiber cut on AT&T's backbone.. wonder if it has anything to do with this.
A slip of the foot you may soon recover, but a slip of the tongue you may never get over. -Benjamin Franklin
Our Akamai rep tells us that it was an issue with a software version rollout. They flushed all their image caches, and effectively caused a DOS on themselves.
I've had so many problems with akamai as of late...it seemingly has a monopoly over just about any commercial website I'm interested in. I don't see images very often while I'm at work...they just idle. Maybe it's a sign that I should stop shopping when I'm supposed to be working :(
Is it a bad idea to rely so heavily upon one service for our major internet needs?
Yeah. Duh. But, where else can I get a /. fix?
I couldn't get to eBay this morning either. It seems to be resolved now though.
I was unable to get to the sites for the major AV Vendors this morning. I chalk it up to Agobot as it
f au lt5.asp?VName=WORM_AGOBOT.GN&VSect=T
DDOS's their sites. See the following link:
http://www.trendmicro.com/vinfo/virusencyclo/de
Due to a peering problem between ATT and UUNet, a subset of UUNet users may have experienced problems accessing Akamai delivered sites between 8-10pm EDT on Saturday May 22, 2004. The problem has been fully resolved.
I think I'll stop here.
As a small company we have a limited view of the Internet, but it seems to us that there have been DNS and connectivity problems thoughout the Internet for the last 90 days or so. I was guessing that there was a DDoS attack against the root DNS servers that wasn't being reported. This would seem to be along the same lines.
sPh
Mail? Put "slashdot" in the subject to pass the spam filters.
Bittorrent reduces the load on the central server by having everyone who downloads content upload content to other users. Couldn't a similar system be designed for HTTP connections? Obviously it would be designed with much smaller files in mind and with less overhead.
I realise no one give a shit about some large company's bandwith but for small community sites it could really make a difference. They wouldn't have to pay for a company to mirror their site and would save on bandwith costs.
This wouldn't work for server side scripts (as the HTML output would be different for every user) but for static HTML and images it would be perfect.
Due to a peering problem between ATT and UUNet, a subset of UUNet users may have experienced problems accessing Akamai delivered sites between 8-10pm EDT on Saturday May 22, 2004. The problem has been fully resolved.
Maybe the problem has recurred.
Live your life each day as if it was your last.
Since I've had problems like this with my ISP, I figured it was something local. I guess not.
OK, moderate me redundant because now I see a million other people saw the same thing...
Am I part of the core demographic for Swedish Fish?
.. with akamai-hosted sites that has an odd effect in Mozilla Firefox 0.8 on Linux. A combination of Firefox doing an unnecessary reverse lookup on the IP that's being connected to (this is in addition to the regular forward lookup to get the IP, and waits until timeout, usually 30 seconds) and akamai's lack of any reverse zones configured for their boxes.
A buddy of mine worked through further diagnosis to reveal this problem and registered a bug report with the MozDev team, however, after he contacted Google to inform them of the problem, they put in a blank in-addr.arpa zone file for their IP's, which resulted in an immediate negative result on that reverse zone lookup. If the rest of akamai would get on the stick and do the same, the problem would be history.
seems also to be down. I was trying to access it after the Cannes result, and thought the US government had censored it...
Google passes Turing test : see my journal
A guy I spoke with this morning at Akamai said this morning that the problem was NOT the result of any outside attack on the company's servers. Rather, he said, the problem stemmed from a bug within a tool that allows customers to purge old content and update their cache with new content. Akamai said the problem lasted about 90 minutes, and affected numerous Akamai customers. No response, though, as to why this bug suddenly reared its head.
...because you never know who you're dealing with.
Akamai provides a DNS load balancing solution to many major internet companies/sites including (but notlimited to) Google, Yahoo, etc. Is it a bad idea to rely so heavily upon one service for our major internet needs?
Don't you see the irony? How much of the internet populace depends on Google for their searching needs?
I suspect the problem here, as there, is that there aren't many who can compete at a service level.
tasks(723) drafts(105) languages(484) examples(29106)
Currently akamai is configured as an open proxy. This is obviously bad (for them), since anyone can steal the service they're selling for big bucks.
i mages.sl ashdot.org/topics/topiclinux.gif
Try:
http://a40.g.akamaitech.net/7/40/1601/1d/
(it works with ANY URL)
Obviously, they noticed it, and tried to fix it. Their fix turned out to block valid customers (like Apple, as has been mentioned), so now they have rolled it back to the free-for-all setup.
They're probably working on a better fix right now.
This is nicely commented on in a recent story over at CFO where it says "Broadly speaking, Akamai needs servers near the consumers of content..[] Akamai, on the other hand, has servers pretty much everywhere."
To trim the facts down a bit: Akamai has servers near by most users these days, and the distributed DNS gives you returning DNS to the closest contentserver. If I, who live in Norway, try to access fbi.gov from any computer from a ISP connected to the NIX (Norwegian Internet eXchange) I get a DNS response that leads me to Akamais servers in Oslo, Norway. I've tried this for some time, just to see what happens, with cnn.com, apple.com and fbi.gov. While on a trip to Sweden I tried this while connecting through a local DSL-provider and I got a response from a server located in Sweden, hence even the swedes have their own Akamai mirror these days.
The problems with a DDOS from someone in Norway would, if directed towards a domain or webpage and not an IP-address lead to downtime on that specific local mirror, not Akamais entire network. We can from this conclude that only such events as a major blackout in Akamais core network or like this time, DOS'ing their own network would take out their service.
"-Who said sit down?!"
-- S. Ballmer @ MSDC 2003.
Akamai just told me it was a 90-minute glitch (between 8 and 9:30 Eastern time) caused by a software bug. The company says everything's back to normal.
Hiawatha Bray
Tech Reporter
Boston Globe
...for those that don't know, a market where it is unprofitable to be the 2nd company around (usually, you can sell cheap because the major company wants to reap profits). A small "Akamai" competitor is no competitor at all, really. You need to have a similar huge network in order to compete. They would undoubtably clash and one would come out as the winner.
So well, if it hadn't been Akamai it'd probably be someone else. Of course, one company can still build a helluva redundant network, if they want to... it's just usually not cost-efficient.
Kjella
Live today, because you never know what tomorrow brings
I spoke with Akamai support. They indicated that it was a far reaching problem, but I have not heard the reason yet.
The customer login to the admin portal was down as well. It was almost like someone dump the customer account database.
Akamai has a QOS commitment of 100% uptime based on the idea that not all of the 1,000's of servers could go down at the same time. But... There you go.
Akamai is aware of a service interuption earlier today affecting content delivery.
We have identified the root cause and have implemented the fix. Issues retrieving content should be decreasing or resolved. Updates will continue to be posted on the Akamai Edge Control Management Center.
so there is something wrong with their cdn. so much for 100% availability. my guess, all the edge servers were ok but there may be a problem with their noc or software.
Live your life each day as if it was your last.
Do you mean "grammatical mistakes"?
It took Microsoft down for DAYS.
All due to a router config bug introduced by Microsoft.. So it was really Microsoft DoSing themselves via Akamai.
And it would be unfair to blame the router config for more than a few hours of outage. The big problem was the complete and utter paralysis of management on the conference calls.
I don't think the details of that outage have been leaked much. It was quite a hoot talking to those involved during the outage. And it wasn't hard, given the duration.
It is my recollection that the problem related to Microsoft's filtering DNS requests from Akamai.
ak a my
Simple. Just like it looks.
An isolated issue occurred this morning (roughly during the period of 8:00 a.m. - 9:30 a.m. ET), where multiple Akamai customers experienced intermittent performance and availability degradation.
This degradation was the result of a bug within one of Akamai's backend content control management tools, which allows the expiration of content on the Akamai network. The degradation was not a result of any outside interference with Akamai's network (such as Denial of Service or hacking).
Upon identification of the bug, Akamai quickly took corrective action which returned customers to normal service levels. Akamai is currently putting measures in place to return the content management tool to its normal working order and is adding safeguards such that the issue will not occur in the future. In the meantime, Akamai customers are able to serve their content through the Akamai Network normally.
As part of Akamai's normal proactive customer communication policy, Akamai customers will be kept informed of the latest developments through the Akamai portal, the EdgeControl Management Center, https://control.akamai.com. Any further inquiries may be directed at Akamai Customer Care at 1-877-4-AKATEC.
Akamai's competitors have different scaling tradeoffs. The last time I knew numbers was a couple of years ago, and it may have changed, but Akamai had a very large number of mostly small servers located on many carriers networks, AT&T had a couple hundred very large servers (mostly at peering points, which takes advantage of being a carrier, though they also bought some transit for content distribution), and Speedera was somewhere in between. AT&T's directions included lots of streaming media, and Akamai was doing fancy database things.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks