Slashdot Mirror
Akamai Having Problems?
A reader writes:"It appears that sometime during the night, Akamai had some problems causing some connectivitly issues with many hosts thoughout the night. Akamai provides a DNS load balancing solution to many major internet companies/sites including (but notlimited to) Google, Yahoo, etc. Is it a bad idea to rely so heavily upon one service for our major internet needs? " Not much details - but I can confirm having problems this morning. Thanks to alert readers for pointing that they were having "DoS related issues" and that service was restored as of 1400 GMT.
Perhaps this is related to the SBC strike?
I can confirm problems accessing the apple.com trailers, but microsoft.com has no problems. I thought they were using Akamai's services as well?
Akamai also hosts files (images, binaries) for many major websites. Seems like they have some pretty insane bandwidth too...
Posting a link to their website on Slashdot should help them out.
Beauty is in the eye of the beerholder.
And this is *news*???
I don't think it's a problem to go with one company. As long as that company has a distributed solution with many uplink providers. So, basically redundancy when something happens because no matter how good you are there will always be hickups.
Evolution or ID?
It appears that websites that use Akamai's distribution system are currently not reachable. Security related web sites effected are symantec.com and trendmicro.com. Virus updates may fail as a result. Further details are currently not available and updates will be posted here as they become available. Thanks to Vidar Wilkens for alerting us of this problem.
According to a post to NANOG, the outage may be the result of a DDOS attack. At this point, Akamai has not ETA for a resolution.
Update 09:45 EST: Looks like some of the Akamai hosted sites start to come back."
You gotta love that "Quiet, well kinda quiet". ;)
UNIX? They're not even circumcised! Savages!
- here
- here
- here
- here
Nice bit of bandwidth theft, there.Is it a bad idea to rely so heavily upon one service for our major internet needs?
We do that already. Remember when verisign introduced Sitefinder, thus effectively making various services (like spam filters etc) unusable because non-existing domains all of a sudden replied with a valid IP.
Underholdning.info
Akamai's NOC says service restored approx 1400GMT. Earlier NOC quotes include: It is a system-wide problem that "looks like it may be a DOS attack".
Their system is supposed to be distributed in such a way that any major outage in a section of the internet would not affect their overall ability to deliver the content, so presuambly any outage an ISP would not hit their too hard.
w ww.peacefire.o rg/bypass/Proxy/akamai.html
BTW something interesting:
http://a1.g.akamaitech.net/6/6/6/6/
**FREE** Track and view your phone's via CellID and/or WIFI and/or GPS
The cleaning lady needed electricity to her vacuum cleaner.
Poor sysadmins.
I demand the Cone of Silence!
NANOG Archieve
Rus
Cheap UK and US VPS
Of course it is a bad idea.
However, blame that on the other competing services who haven't become cheaper, faster or better at whatever it is that makes Akamai so popular.
Avantslash - View Slashdot cleanly on your mobile phone.
I love how the first reaction when something goes wrong is to replace it, or introduce competiton, or whatever. Yes, there are plenty of times when a service needs competition to encourage it to suck less. But go find me another company that is even remotely prepared to do DNS load-balancing. Verisign? Oh, that's a great idea. Going to start one yourself? Let us know when you have the infrastructure.
The fact is, we have NO idea what caused this. There's no link to any story anywhere - just one reader report. It could be Akamai's fault. It could be their upstream providers. It could be failures elsewhere in the Internet. Could be someone uploaded a bad zone file. Or maybe some over-zealous backhoe operator slashed some fiber somewhere.
It's probably best to reserve judgement until you have all the facts. (And if you're about to hit the reply button, yes, I'd say the exact same thing if MSFT lost their DNS service).
There is no sig, there is only Zuul.
People say that the Internet can't be knocked out. That may be true in the infra-structure sense, but if you're able to knock down Akamai or any other major solution provider, think of the sites that would go down (Google, Yahoo et al), and the repercution on the global economy. So yes, the domino theory doesn't apply to the Internet, but it becomes exponentially more dangerous when we rely on one domino for a significant share of of communications.
I guess this throws a wrench in their claim of 24/7 uptime on their main page. Nice how their marketing team says 100% availability, when people get PhD's by adding more 9's to their 99.99..%'s
Our Akamai rep tells us that it was an issue with a software version rollout. They flushed all their image caches, and effectively caused a DOS on themselves.
Due to a peering problem between ATT and UUNet, a subset of UUNet users may have experienced problems accessing Akamai delivered sites between 8-10pm EDT on Saturday May 22, 2004. The problem has been fully resolved.
I think I'll stop here.
As a small company we have a limited view of the Internet, but it seems to us that there have been DNS and connectivity problems thoughout the Internet for the last 90 days or so. I was guessing that there was a DDoS attack against the root DNS servers that wasn't being reported. This would seem to be along the same lines.
sPh
Bittorrent reduces the load on the central server by having everyone who downloads content upload content to other users. Couldn't a similar system be designed for HTTP connections? Obviously it would be designed with much smaller files in mind and with less overhead.
I realise no one give a shit about some large company's bandwith but for small community sites it could really make a difference. They wouldn't have to pay for a company to mirror their site and would save on bandwith costs.
This wouldn't work for server side scripts (as the HTML output would be different for every user) but for static HTML and images it would be perfect.
Due to a peering problem between ATT and UUNet, a subset of UUNet users may have experienced problems accessing Akamai delivered sites between 8-10pm EDT on Saturday May 22, 2004. The problem has been fully resolved.
Maybe the problem has recurred.
Live your life each day as if it was your last.
.. with akamai-hosted sites that has an odd effect in Mozilla Firefox 0.8 on Linux. A combination of Firefox doing an unnecessary reverse lookup on the IP that's being connected to (this is in addition to the regular forward lookup to get the IP, and waits until timeout, usually 30 seconds) and akamai's lack of any reverse zones configured for their boxes.
A buddy of mine worked through further diagnosis to reveal this problem and registered a bug report with the MozDev team, however, after he contacted Google to inform them of the problem, they put in a blank in-addr.arpa zone file for their IP's, which resulted in an immediate negative result on that reverse zone lookup. If the rest of akamai would get on the stick and do the same, the problem would be history.
A guy I spoke with this morning at Akamai said this morning that the problem was NOT the result of any outside attack on the company's servers. Rather, he said, the problem stemmed from a bug within a tool that allows customers to purge old content and update their cache with new content. Akamai said the problem lasted about 90 minutes, and affected numerous Akamai customers. No response, though, as to why this bug suddenly reared its head.
...because you never know who you're dealing with.
This is nicely commented on in a recent story over at CFO where it says "Broadly speaking, Akamai needs servers near the consumers of content..[] Akamai, on the other hand, has servers pretty much everywhere."
To trim the facts down a bit: Akamai has servers near by most users these days, and the distributed DNS gives you returning DNS to the closest contentserver. If I, who live in Norway, try to access fbi.gov from any computer from a ISP connected to the NIX (Norwegian Internet eXchange) I get a DNS response that leads me to Akamais servers in Oslo, Norway. I've tried this for some time, just to see what happens, with cnn.com, apple.com and fbi.gov. While on a trip to Sweden I tried this while connecting through a local DSL-provider and I got a response from a server located in Sweden, hence even the swedes have their own Akamai mirror these days.
The problems with a DDOS from someone in Norway would, if directed towards a domain or webpage and not an IP-address lead to downtime on that specific local mirror, not Akamais entire network. We can from this conclude that only such events as a major blackout in Akamais core network or like this time, DOS'ing their own network would take out their service.
"-Who said sit down?!"
-- S. Ballmer @ MSDC 2003.
...for those that don't know, a market where it is unprofitable to be the 2nd company around (usually, you can sell cheap because the major company wants to reap profits). A small "Akamai" competitor is no competitor at all, really. You need to have a similar huge network in order to compete. They would undoubtably clash and one would come out as the winner.
So well, if it hadn't been Akamai it'd probably be someone else. Of course, one company can still build a helluva redundant network, if they want to... it's just usually not cost-efficient.
Kjella
Live today, because you never know what tomorrow brings
An isolated issue occurred this morning (roughly during the period of 8:00 a.m. - 9:30 a.m. ET), where multiple Akamai customers experienced intermittent performance and availability degradation.
This degradation was the result of a bug within one of Akamai's backend content control management tools, which allows the expiration of content on the Akamai network. The degradation was not a result of any outside interference with Akamai's network (such as Denial of Service or hacking).
Upon identification of the bug, Akamai quickly took corrective action which returned customers to normal service levels. Akamai is currently putting measures in place to return the content management tool to its normal working order and is adding safeguards such that the issue will not occur in the future. In the meantime, Akamai customers are able to serve their content through the Akamai Network normally.
As part of Akamai's normal proactive customer communication policy, Akamai customers will be kept informed of the latest developments through the Akamai portal, the EdgeControl Management Center, https://control.akamai.com. Any further inquiries may be directed at Akamai Customer Care at 1-877-4-AKATEC.