Slashdot Mirror
Akamai DNS Outage Messes up Net
katre writes "Checking all my favorite sites this morning, I saw that about half a dozen seem to be offline. Trying to figure out why, I found an interesting article on the front page at http://isc.incidents.org/. Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others. Whatever happened to my decentralized net with no single point of failure?"
but I believe the centralized concept of the 'net is something that is coming to an end, much to our loss. I'm pretty bothered by the fragility of this system. How many of you can't work without web access?
Don't be a looter...and yes, I know that it's spelled with an "A" instead of an "E".
provider of real time market data...
hope the al quedas aren't taking notes on this..
Its still there, and you're using it. The only organizations affected by this are those who chose to use a service that acts as a single point of failure.
trustedworlds.net - gaming, security, and the gunk that lives in between
Whatever happened to my decentralized net with no single point of failure?
Its there. Get out your old Usenet reader. See, you still have your porn.
Know what I like about atheists? I've yet to meet one that believes God is on their side.
Whatever happened to my decentralized net with no single point of failure?
Never existed. Internet myth. The robustness is only for routing around damage.
DNS dying on you? Just throw it on the pile of other connection problems
;)
I think everyone has several "single" points of failure -- my cable modem dies at least twice a month and my wireless router conks out at least twice a day
Yahoo is already resolving through scd instead of akamai. I didn't check any of the others.
If you clear your cache, you will probably get the new entries, unless your ISP hasn't caught onto the problem yet.
vague explanation, just a link to the ISC's Incidents website and not the article, and now that site is inaccessible courtesy the slashdot effect. Nice job, now we cant even find out what's going on!
How ya doin', Al?
...I can't even get to http://isc.incidents.org/
You could still access Slashdot, couldnt you?
Be very, very careful what you put into that head, because you will never, ever get it out. - Cardinal Wolsey
The internet is completely vulnerable to virus attacks, terrorist attacks because of the single point of failures that still exist - despite everyone preaching to the contrary.
Hmmm.
Akamai, this is what happened? Akamai and the rest of the big bussiness that offer servers. Decentralized net seems to be comming to an end. Big server businesses claim 24/7 and no outages; this crap happens, and countless sites go down.
The article said the problem is worldwide.
I tried the specific URL they said didn't work - http://www.google.com/ - and it works for me.
So do Microsoft, Yahoo and Xerox - all with the www.
Can someone explain WTF is going on?
It's never too late to have a happy childhood.
The web happened my dear friend, and it was based on the predominant distributed computing model at the time: client/server. Even DNS, with its highly distributed spread of processing and data, has a set of (overloaded) root servers with the commensurate single points of failure. The solution? Peer-to-peer.
:)
Too bad even the term P2P raises so many red flags with certain Associations of America.
This should cause some problems for akami, they had an outage may 24th. Once can be overlooked twice? these are some big companies they are going to be calling them. I bet there is some sweating techs in the cool noc right now
War isn't about who's right. It's about who's left.
Whatever happened to my decentralized net with no single point of failure?"
You are completely right. Besides technical failures (which can happen), 'we' are also very prone to terrorist attacks on such facilities. Taking down major websites like the ones from the article cost real big $$$ and are really a pain for the economics. Especially in times like these.
In need of reliable and affordable server monitoring?
we might as well crash isc.incidents.org
Casual Games/Downloads
And now http://isc.incidents.org/ isn't working either :-P
I see two problems with the internet today: Akamai and the Slashdot effect.
AccountKiller
Either they got slashdotted, or they are affected by the very problem they are reporting.
thisnukes4u.net
You would think that the root DNS servers would be kept up to date with critical information. Just what happened, and how did Akamai get knocked around this? Did they screw with their DNS information and change their nameserver addresses or something?
Striking fear in the authors of godawful fanfiction, I am here, appearing in darkness, Tuxedo Jack!
Do we know if this at all related to the Linux kernel 2.4.2x/2.6 DoS exploit discovered yesterday?
Turn on WAN ping, this way your provider's DHCP server sees that the CPE device is still using the IP address, and doesn't assign it elsewhere.
Why worry? Each of us is wearing an unlicensed "nucular" accelerator on his back.
Sig changed for readability by G.W.
My Yahoo Email is down this morning, first time I can remember this happening. At least gotapex, techbargains and dealmac still work, otherwise I'd have to actually start working!
I can't reach the isc incidents site, so don't know the real reason, but any system where a human has access has a point of failure...
My primary point of failure is my router, the damn clip that keeps the cat6 cable plugged in the router always falls out.
:(
My central point of failure...
When Akamai's system was first announced, most people thought this was a great idea. It made sure that the sites that used this technology would always have the bandwidth they needed, when they needed it. Like with everything else in life, there's always a trade-off between preformance and reliability...
------------------
"Never Attribute to malice what is adequately explained by stupidity..."
The problem, as I understand it, is that Yahoo, Google & co. "outsourced" their DNS service.
I could have accepted that medium-big sized IT companies don't want to run their own DNS servers, but giants Google & co. should have enough money to do so instead of relying on servers located somewhere else.
Funnily enough www.google.com still works for me (thanks to DNS caching I guess)
+++ MELON MELON MELON +++ Out of Cheese Error +++ redo from start +++
Our company relies on net access, so these flaky problems always really worry me, since I have a hard time figuring out what's going on, and I'm not sure how to get info on this sort of thing? Of course, the fragility of my company's system has a lot to do with things like previously relying on a firewall that rebooted when you bump into it.
At least I knew what was wrong that time.
how they can screw up there entire DNS, and it's still down. It started as far as I can tell right after 8:30 or so, the last outage was due to a software update on there own site. It's now nearly 11am and it's still not working.. Man, I would think you could restore from backup at least in that time frame, and have something up for people.. Wonder if there will be an credit on the account this month...
What ticks me off about this incidents (and I suspect that there have been several in the last 6 months) is that there is absolutely no notification given, either during or after the event. During this outage, some news outlets were still reachable (including Slashdot), and a simple notification would have saved hours (* 10s of thousands of network dudes worldwide) of time and much grief from the big bosses who couldn't reach Yahoo Finance, I mean critical business web sites.
Are these guys so convinced of their omnipotence and indispensibility that they don't feel the need to communcate with the world about what is going on?
sPh
The net itself is still very much descentralized. If Chicago got nuked we should be able to reroute traffic through the rest of the net. This was the point after all, back before we even had DNS. Google and Yahoo like many sites opt to use Akamai to fill out there infrastructure rather than take that problem on themselves, so when Akamai fails is it really a suprise that it makes Google fail? Key point here is that Akamai critical to the net, but rather something companies elect to use.
...thinking Yahoo Mail had been slashdotted.
that the /.'ers aren't trying to take credit for slashdotting the entire WWW.
"Facts are meaningless. You could use facts to prove anything that's even remotely true!" -- Homer Simpson
Pwned by CNAME to Akamai?
(You can't have CNAME records for the base domain, hence google.com would have had an A record instead, whilst www.google.com would have been a CNAME to akamai)
... a way to blame the outage on Microsoft instead of (or in addition to) Akamai?
(come on, it's funny. at least I didn't suggest blaming SCO...)
They are windows users. They like the blue screen of death.
When I was in grad school at Cornell, my O/S professor went on a rant about the evils of Akamai. No one believed him. Now we know he was right.
Wer mit Ungeheuern kämpft, mag zusehn, dass er nicht dabei zum Ungeheuer wird. --Nietzsche
It's not truely decentralized...
The root nameservers are the most obvious example...
The most obvious example? The fact is that there are 13 of them, in widely scattered locations across the globe, and it's not decentralized?
Damn man, what exactly would you consider "decentralized" then?
Root servers go down all the time. It's not particularly unusual. There's THIRTEEN of the things. Up to 8 have been down at once with no major effects on the network, IIRC.
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.
What I'm concerned with is the fact that mozilla.org doesn't provide Firefox MD5 or SHA1 checksums. Ftp.mozilla.org resolves to eight different sites. Who's to say a trojaned copy of Firefox won't pop up at one of them? With the breakins at the GNU Savannah and Debian servers still fresh in the mind, it seems irresponsible of the Mozilla foundation to not provide this protection.
I can see the logic that went into this plan:
"Well, Akamai has a few million DNS boxes, if we put everything there we'll be fine! That's not a single point of failure!"
Yeah, about that... multiple vendors may have been a good idea in retrospect instead of just one monolithic provider.
Time to re-examine the definition of Single Point of Failure.
Let's see so far today.. We had a report on Yahoo... They're down. A report to a virus linked to Symantec.. they are up and down. We always link to Google, they are having problems... wooo. Now we just need another patent from Microsoft to bring them down... which by my records shouldn't be too long.
Hmmm.
It's not a problem for me, from where I'm at, so therefore, ergo, etc. SOMETHING must be working right!
Handlers Diary June 15th 2004 Updated June 15th 2004 14:31 UTC (Handler: Lenny Zeltser) Akamai DNS outage Akamai DNS problem Starting at around 8:30 am EDT (12:30 UTC), a number of sources started to report a widespread Akamai DNS issue. Large web sites, which use Akamai for its DNS service, did no longer resolve. Effected sites are Yahoo, Google, Microsoft, Fedex, Xerox, Apple and likely many others. At this time (10:30 am EDT), some effected domains removed the Akamai DNS servers and are reachable again using their own DNS servers. Typically, the domain itself (e.g. 'google.com') still resolves, but popular hostnames, like 'www.google.com' will not resolve. As a result, the web site is no longer reachable. The effect appears to be world wide. Some of the Akamai servers do respond to pings, but do not respond to DNS queries.
It's never too late to have a happy childhood.
Hey at least all the major search engines don't use Akamai..
If you run your own DNS, just clear the cache and everything will be ok...will probably need to flush the cache on your client as well...
What are they doing over at Akami, switching to .net or something? 1/2 :-)
Didn't Akamai go down a few weeks ago?
The way to solve it is get more companies out there who provide the same sevices, something not easy after the dot bust era when people dont want to take such risks.
"Slashdot, where telling the truth is overrated but lying is insightful."
Good job... you managed to slashdot the last remaining page on the net.
Yahoo, Google, Microsoft, Fedex, Xerox, Apple
.com tld. I enjoyed he internet much more before it was a corporate advertising scheme anyway.
If thats your idea of what the web is I feel sorry for you. As far as I am concerned the big corps can all use akami and create a single point of failure for the entire
Why are you checking Xerox.com first thing in the morning?
My understanding was the Internet was fault tolerant b/c even if points along it were destroyed, the network would still survive. Not the contents on those destroyed machines.
And I thought Akamai accelerated delivery of content and relieved the stress on servers. I can see how problems with Akamai would really mess things up.
The Net is decentralized... however, if several *LARGE* sites happen to be resolved through one DNS server and it crashes, people think that the 'net is down'... IIRC, Helldesk people bitch about this - people calling up and saying 'I can't get to www.mytimewastingbullshitpage.com, is the net down?' Not realizing that just becuase one or two or thirty sites are down, the net is still up....
FWIW, I missed google for all of 10 minutes, and figured it was my work ISP....
Checking all my favorite sites this morning...
Microsoft, Xerox and FedEx are some of my favorite sites too! But due to the outage I'm stuck slumming it here on Slashdot...
It's only a sinlge point of failure if you can't get to *ALL* of yout websites, instead of some.
I was asked to come up with a list of sites that were not responding, or repsonding really slow, so the top 5 were all job hunting related sites ie: hotjobs.com, monster.com, careerbuilder.com.
He didnt find it as humerous as i did though.
Semi-official response is DDOS on their DNS service.
The problem has been mitigated by working with their ISP's, and service should be returning to normal.
.signature not found
Yeah, google didn't work and we didn't know what to do. We tested and determined the problem was akamai within a minute. So I used AIM to ask a friend who could still resolve google what the ip was. he passed it to me over aim using gaim encryption no less. We then created an alias for google on our dns server. google.ourdomain.com.
We also developed a new DNS protocol in the process. ESEDOIM: Extremely slow encrypted DNS over instant messenger. Who wants to write an RFC?
The GeekNights podcast is going strong. Listen!
I run a small ISP and we happen to have 3 of their linux boxes on our network. I've never experienced a problem with them before today. For the hack of it we decided to just reboot their servers and now things are working correctly.
For those that were wondering why it would affect DNS; Akamai somehow tinkers with DNS and BGP to redirect content to their edge servers.
As for Akamai being outdated, it still seems to me that its a good idea for Yahoo and some of the high traffic sites on the net. Akamai has thousands of distributed servers colocated with ISPs and NAPs. And they do seem to absorb nasty bursts in traffic (ie Star Report) better than a centralized server farm. But for their own sake, they better hope to not have another repeat of todays events.
It's not like a092156fg.akamai.net is in Seattle and k1039665.akamai.net is in Saskatoon. Instead, all of *.akamai.net goes to whatever cluster is "closest" to the requesting IP (based on BGP, Colonel's Secret Recipe, etc)
So if Akamai's DNS gets screwed up, I would expect major weirdness. And as more sites join EdgeSuite (where you host your entire domain on Akamai's servers & DNS) the effect must magnify.Of course, I could be completely wrong. I'm not a routing god, just a guy who thinks Akamai is a cool hack.
From NANOG:
From here neither www.google.com, nor www.apple.com work. Both seem to return CNAMES to akadns.net addresses (eg, www.google.akadns.net, www.apple.com.akadns.net), and from here all of the akadns.net servers listed in whois are failing to respond.
Fools, things like this only ever happen in the morning. Start work at noon and go till late into the night. More productive that way.
I wonder why these companies wholly switched their nameservers over? Why not have #1 and #2 be Akami, and #3 & #4 be your own nameservers? Preferably on different coasts or in different countries.
This would seem an obvious solution. You are allowed to have many nameservers you know...
Natural != (nontoxic || beneficial)
I was thinking about this while scrambling to answer the phone, check outage reports, and generally calm down customers.
:) )
If a product or service, such as Akamai, does their job very well, everybody will want to use them. If everybody uses them, you create a single point-of-failure. Any design flaw in that product or service becomes a disaster, simply through volume. Does this mean a successful product or service can actually be a bad thing for people?
Other examples include just about anything from Microsoft, older versions of Sendmail and BIND (worm-of-the-week problem), and Firestone tires.
(I'm not trying to advocate communism, excessive government regulation, or anything like that. So fanatical libertarians, conspiracy theorists, etc., can put down the rant-o-matic flamethrowers.
Comments?
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
- Whatever happened to my decentralized net with no single point of failure?"
[Homer] Welcome to the internet my friend, how may I help you?CB
free ipod and free gmail!
Akamai didn't mess up the net. Akamai messed up some web sites that are akamai customers. Remember kids, www is only a subset of the internet, and akamai customers a small fraction of the www.
I heard from them, that it was a DOS attack against there DNS infrastructure.. Not sure if I believe that yet, but...
Judging by the response time of isc.incidents.org, I'd say slashdot is the single point of failure.
Quantum materiae materietur marmota monax si marmota monax materiam possit materiari?
Whatever happened to my decentralized net with no single point of failure?
You didn't pay the rent.
Comparing it to Windows will be a moot point, since El Dorado is going to have a 40% larger code base than XP.
--Soon-to-be-fired Akamai employee
Later they can post an 'incident report' on the slashdotting they're experiencing right now!
As a pragmatic move, wouldn't it be wise to use the IP address of any critical web-based resource you might need like Google?
http://216.239.37.99/ works for me.
I noticed this problem this morning when I was hunting for an updated version of YahooPOPs. I wasnt getting replies from Google. I opened another FirePanda window and my homepage, slashdot, was working fine (Hey look at that on the homepage, Yahoo changed their mail service today, no luck for YahooPOPs). I tried yahoo, altavista, even msn in different tabs but I wasnt getting anywhere.
I tried pinging google and I was getting a reply so my first thought was, there is something terribly wrong at verizon DSL. I must make the most of what fragmented connection I have now before its down all day and I'm stranded actually doing work.
Thats when I started opening every story on slashdot's homepage in different tabs and setting them all to threshold 3, threaded... Just incase.
Come to think of it, I'm going to change my slashdot bookmark from slashdot.org to 66.35.250.151 just incase of DNS failure.
Need my SlashCrack
Im dreaming ofa big bndwdth, That can resist the
Seriously we need a *.sht domain.
That must include a lot of users on the internet.
When google goes down, that may, cosmically speaking, be simply one site on the www, but it certainly doesn't have a small fraction of users.
To many, google IS the internet.
So I wasn't the only one who couldn't get to Google the Great. Fortunately, Dogpile still worked. I used that meta search engine until Google started getting big and beating all the others in turning up relevant search results.
I wonder if Google will now turn to fully manage all their assets themselves...
Please correct me if I got my facts wrong.
It appears that, at around 8:30 AM EDT (US Eastern Daylight Time), Akamai's DNS network experiened some kind of major failure. All of their DNS servers (that anybody could find) were not responding to DNS queries. It appears that Akamai started to come back online at around 10:00 AM EDT.
Since a great many big name sites use Akamai, this effectively made large parts of the Internet unreachable. The destination servers themselves were up, but clients were unable to turn names (like www.example.com) into network addresses (like 192.0.2.42).
As Akamai maintains dozens, if not hundreds, of DNS servers across the globe, it is extremely unlikely that this was due to a normal equipment failure or DoS attack. Some kind of internal system trouble is much more likely. Whether a deliberate attack, or an accident, is unknown to me at this time. It could just be an internal configuration change blew up in a really bad way. Sh*t happens.
I do not know if this was just an Akamai DNS problem, or if other Akamai services were also affected.
Due to the way Akamai is usually implemented, it happened that, in many cases, the second-level domain names (like example.com) worked, but subdomains (like www.example.com and mail.example.com) did not. This is because most organizations put in CNAME records (pointing to names in *.akadns.net) for the subdomains. You cannot use a CNAME record for a domain that has other records, though, so most domains still had traditional A records, on their own nameservers, at the second-level.
The following sites/organizations are known to use Akamai: Yahoo, Google, Microsoft, Altavista, FedEx, Xerox, Apple
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
It seems that paypal is affected too. At least I'm not getting any mails from them. On any of the four mail addresses I use.
Any technology distinguishable from magic, is insufficiently advanced.
Not too long after 9/11, I was surfing the net and needed to look up something at the Library of Congress for one of my classes. It wouldn't connect. At first I thought we'd just lost DNS (not so uncommon an occurance at my university in those days), but found I could still connect to slashdot.org and some other sites.
.edus mostly.) The ones that replied, I plotted on a US map based on their DNS LOC. (A project I wrote for a previous class.)
Being a geek, I thought up a list of about 30 sites to ping, scattered across the US. (.govs and
I freaked out a bit when the mid-atlantic seaboard came up missing. I crossed my fingers hoping that it was just some idiot who'd accidently cut one of the main fibers (which it what it ended up being) and not that Washington DC was now a big hole in the ground.
A preposition is a terrible thing to end a sentence with.
I'm waiting for the "Fix it by installing Linux" cries.
That seems to be the answer for every other virus/worm around here.
I remember when people were bashing Microsoft for using Akamai caching to avoid Windows Update getting hit by the first RPC worm (the one that was patched two months beforehand), since Akamai used Linux and it was somehow amusing that Microsoft chose that caching service.
If Akamai was running on Windows servers, I guarantee it would have been mentioned in both the headline and in the article summary today. But instead it's just mysterious "DNS issues." It's kind of like how when that Windows source code was stolen, Slashdot reported on it yet neglected to mention that the code was stolen from a hacked Linux computer at a company called Mainsoft.
Just little slants in reporting I can't help but notice.
Take a look at what internic.net gave me on some of these domains....
F REAK.ORG.RULEZ.AND.DIOX YTECH.NET.DELETED.GANDI.NET. SIMPLECODES.COMA USE.LINUXISGOD.CO M
M ICROSOFT.COM.OHMYGODITBURNS.COMV ES.JU1C3.COMO MM O FT.COM.IS.GOD.BECOUSE.UNIXSUCKS.COMM .IS.A.STEAMING.HEAP.OF.FUCKING-BULLSH IT.NETC ROSOFT.COM.HAS.ITS.OWN.CRACKLAB.COMM .HAS.A.PRESENT.COMING.FROM.HUGHESMISS ILES.COMC OM
MICROSOFT.COM.CAN.GO.FUCK.ITSELF.AT.SECZY.COMC ROSOFT.COM.ARE.GODDAMN.PIGFUCKERS.NETC OM.AND.MINDSUCK.BOTH.SUCK.HUGE.ONES.AT. EXEGETE.NET
. COM.TWIXTEARS.COMB CENTER.COMM S .1337.AS.SEARCH.GULLI.COMM . COM
7 .AS.SEARCH.GULLI.COM
T H.SEARCH.GULLI.COM I NE .THAN.SECZY.COM
Microsoft.com
----
MICROSOFT.COM.SUX.BUT.PYRO
MICROSOFT.COM.SMELLS
MICROSOFT.COM.SHOULD.GIVE.UP.BEC
MICROSOFT.COM.RAWKZ.MUH.WERLD.MENTALFLOSS.CA
MICROSOFT.COM.LO
MICROSOFT.COM.LIVES.AT.SHAUNEWING.C
MICROSOFT.COM.IS.NOT.AS.COOL.AS.SIMPLECODES.CO
MICROSOFT.COM.IS.IN.BED.WITH.CURTYV.COM
MICROS
MICROSOFT.CO
MICROSOFT.COM.HAS.TEH.GAY.OMFGLOL.COM
MI
MICROSOFT.CO
MICROSOFT.COM.FLINGS.POO.AT.MONKEYCORE.
MICROSOFT.COM.FILLS.ME.WITH.BELLIGERENCE.NET
MI
MICROSOFT.
MICROSOFT.COM
Yahoo.com
---
YAHOO.COM.WANADOODOO.COM
YAHOO
YAHOO.COM.TW
YAHOO.COM.SUPERC
YAHOO.COM.SG
YAHOO.COM.PURRFURRED.CO
YAHOO.COM.OPTIONSCORNER.COM
YAHOO.COM.IS.N0T.A
YAHOO.COM.DALLARIVA.CO
YAHOO.COM.BR
YAHOO.COM.BERKELEYNATURALBEAUTIES
YAHOO.COM.AU
YAHOO.COM
Altavista.com
---
ALTAVISTA.COM.IS.N0T.AS.133
ALTAVISTA.COM
Apple.com
---
GOOGLE.COM.SUCKS.FIND.CRACKZ.WI
GOOGLE.COM.HAS.LESS.FREE.PORN.IN.ITS.SEARCH.ENG
GOOGLE.COM
and there's nothing black magic about it :)
and global megacorps have certainly been doing it for a *long* time...
I browse at +5 Flamebait- moderation for all or moderation for none.
DNS was designed to be robust enough. Not one root server but many (ok, that's the weak point, we've all seen many DDoS against them, but it's not THAT bad). All zones are handled by their own servers, and (in theory) multiple servers for each zone. All in all, it's not a bad design.
If what happened was that someone put all the servers behind one link, it's not DNS' fault, the BOFH there screwed up (and considering it's akamai, they should not have done that).
(If that's not what happened, sorry, I couldn't RTFA, it's slashdotted or there's some sort of DNS problem there too).
GPG 0x1B479C78
If you want to have a true dialogue instead of fingerpointing with "nah-nah" gibes, you'll have to actually state which films you're talking about and what were the quotes that are "out-of-context".
Not to worry, I'll be hiding behind my bogon.
A computer once beat me at chess, but it was no match for me at kick boxing. Emo Philips
Newsfollow.com
From NANOG mailing list again:
Google pulled references for akamais dns servers a short period ago. they are presently serving their own dns requests.
Also:
People seem to be getting around this by changing their DNS entries.
E.g. www.yahoo.com always used to be a CNAME for www.yahoo.akadns.net. But
now:
# host www.yahoo.com
www.yahoo.com is an alias for www.dcn.yahoo.com.
www.dcn.yahoo.com has address 216.109.118.64
www.dcn.yahoo.com has address 216.109.118.65
www.dcn.yahoo.com has address 216.109.118.66
www.dcn.yahoo.com has address 216.109.118.67
www.dcn.yahoo.com has address 216.109.118.68
www.dcn.yahoo.com has address 216.109.118.69
www.dcn.yahoo.com has address 216.109.118.70
www.dcn.yahoo.com has address 216.109.118.71
www.dcn.yahoo.com has address 216.109.118.72
www.dcn.yahoo.com has address 216.109.118.73
www.dcn.yahoo.com has address 216.109.118.74
www.dcn.yahoo.com has address 216.109.118.75
Which is owned by Yahoo! (via HotJobs.com).
Whatever happened to my decentralized net with no single point of failure?
Outsourcing and consolidation.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
Handlers Diary June 15th 2004
m sg05267. html
Updated June 15th 2004 14:31 UTC (Handler: Lenny Zeltser)
Akamai DNS outage
Akamai DNS problem
Starting at around 8:30 am EDT (12:30 UTC), a number of sources started to report a widespread Akamai DNS issue. Large web sites, which use Akamai for its DNS service, did no longer resolve. Effected sites are Yahoo, Google, Microsoft, Fedex, Xerox, Apple and likely many others.
At this time (10:30 am EDT), some effected domains removed the Akamai DNS servers and are reachable again using their own DNS servers.
Typically, the domain itself (e.g. 'google.com') still resolves, but popular hostnames, like 'www.google.com' will not resolve. As a result, the web site is no longer reachable.
The effect appears to be world wide. Some of the Akamai servers do respond to pings, but do not respond to DNS queries.
posts to the NANOG mailing list regarding this issue:
http://www.merit.edu/mail.archives/nanog/
It's never too late to have a happy childhood.
Hmmm, corporate whore much? Slashdot, Debian and my own two sites seem to be working just fine. Maybe the sites you choose to visit just don't get the 'net and it's decentralized nature.
Nathan's blog
Comment removed based on user account deletion
For 10 years I was a net junkie. If I didn't get my email, news, laugh, or enough time on my fav mmorpg then I was twitchy and grouchy.
:)
:) But while I'm here in the states, I *need* to be connected. I think because everybody else is.
Then, two years ago my wife and I decided to take a year off and go tour SE Asia, mainly Viet Nam.
Yes, they have Internet there but it is mainly in Internet cafes, which are hot, crowded, and quite slow. There are dialups but once you've lived on broadband for such a long time the dialup becomes something you use only when you have to. And so that was what happened. Internet became something that was used when needed. I still checked my email regularly but instead of every hour it was every 2 or 3 days, same with Slashdot.
I had a few personal (programming) projects I was working on which fit nicely onto the laptop, along with a good 20gig of mp3s. I was amazed at how fast I detached from the net. My productivity shot thru the roof, namely because my concentration was focused.
Even here in the states I have yet to reach that state of Zen again primarily because, even though I try, I know the net is right there. The little net thoughts nag at you.
But, back to the topic. You would be amazed at how much technical work you can accomplish without the net being there.
Would I give up what I have now and go back? You bet. Would I miss it? Nope. Broadband is used for P2P or games. That's all I use broadband for anyway.
On a global scope, 99% of all the really cool groundbreaking stuff in the last 100 years, computer or not, was done detached from the net.
Unfortunately there is no such thing as no single point of failure with technology.
Even the best clusters have problems failing over and back sometimes.
Why do you start crying when somebody writes something negative about /. ?
Denial is bliss?
"Whatever happened to my decentralized net with no single point of failure?"
How ignorant can you be? The internet still worked even though some sites that are cached by Akamai, went down. I had no problem getting to slashdot, or google, or any other site today. I admit there were a couple that were off line, but that is just because the service at Akamai failed.
I am really getting tired of these article posters that want to comment on something but really know nothing about what they are commenting on. I guess he doesn't really understand that the decentrailized network that is failure proof only applies to the middle parts. Not the begining or ending point.
I was wondering why my pager went off at 5:30am today. Check it, it complained of DNS being down. Digging further it appeared that only the two sites I was checking were down: www.yahoo.com and www.google.com.
Everything else seemed to be working just fine so I let it be. Good to know I wasn't the only one having problems.
Handlers Diary June 15th 2004
m sg05267. html
Updated June 15th 2004 14:31 UTC (Handler: Lenny Zeltser)
Akamai DNS outage
Akamai DNS problem
Starting at around 8:30 am EDT (12:30 UTC), a number of sources started to report a widespread Akamai DNS issue. Large web sites, which use Akamai for its DNS service, did no longer resolve. Effected sites are Yahoo, Google, Microsoft, Fedex, Xerox, Apple and likely many others.
At this time (10:30 am EDT), some effected domains removed the Akamai DNS servers and are reachable again using their own DNS servers.
Typically, the domain itself (e.g. 'google.com') still resolves, but popular hostnames, like 'www.google.com' will not resolve. As a result, the web site is no longer reachable.
The effect appears to be world wide. Some of the Akamai servers do respond to pings, but do not respond to DNS queries.
posts to the NANOG mailing list regarding this issue:
http://www.merit.edu/mail.archives/nanog/
Interested in meeting handlers in person? Discuss this diary over a beer? Visit us at SANSFIRE, the Internet Storm Center Conferences. Monterey, CA, July 6-11th. @Night talks picked by the ISC and the best security training you can get.
I am VERY suspicous of Akamai. I think they are an extension of carnivore, double-click and others unnamed to date. I tend to block them in my IDS/firewall.
Admittedly, I have not dug into them, but ANYbody can have a company front for an intelligence/security activity.
DAVID the suspicious...
Previously: "Linux... Toward the Sunrise..." Now: "Linux... Toward the-- No, now, part of Every Sunrise"
and folks often do... witness the onerous "personal contracts" you have to sign to get into the music business, where you are essentiall a creative wage slave and don't own your stuff. non-compete and discoveries-belong clauses in your work contract also sign your rights away to The Man. similarly, if you register your DNS information independently and run your own servers, your ISP and its uplines do the same, and so on including all the sites you visit, you theoretically should not be captive to any of the commercial DNS services.
as I understand it, akamai is a distributed content hosting/caching service that also does DNS server services. they put a blade in your local ISP under contract, and popular pages from their customers serve off the local akamai server cache. they handle the DNS for those sites as I understand. if their blade caches get fed evil data, you get evil data, and www.fartblossom.org may disappear.
you can kill DNS by screwing up your own router, too. lots of ways to kill a distributed service that requires everybody to cooperate on a common set of standards and parameters.
if this is supposed to be a new economy, how come they still want my old fashioned money?
I work with a customer of Akamai, and they have problems like this on a very regular basis.
All of them are caused internally. You see, they have this really slick centralized admin facility that lets them deploy metadata and ESI code changes from the NOC by clicking a button on a web-based GUI. The changes then get propagated through their entire network in 15 - 30 minutes.
I know of at least three complete outages (to one customer!) caused by a tech sending out an untested metadata change. (Don't ever tell me "It should work!")
The key difference between DNS and Akamai-DNS is that the real DNS roots are deliberately designed so that a single bad admin cannot corrupt the entire system: different hardware platforms, running different operating systems, with different admins.
Akamai has automated their systems so they can be run with a skeleton crew. As a result, a single boob pushing a single button can take down their customers -- one at a time or in groups.
Damn that was funny 4 years ago. Do you have any good "hanging chad" material?
Oh, so you guys are over that? Good ... I don't think some of you got the memo ...
Al Gore was talking about creating *legislation* that helped foster the Internet.
But that's not what he said. And yes, I've seen the "full quote", and it doesn't change anything. The stuff around it is just vapor.
He made a grandiose claim. It was stupid, and funny to those who knew better. Get over it! I mean I know it's not as bad as using a regional spelling for "potato", but come on ... no need to be so sensitive about it ;)
Isn't slashdot the most obvious example of a single point of failure? Just post a URL to the site and watch it fail.
The code was taken from a hacked Linux computer at Mainsoft. It was reported elsewere (you might even find the old articles via Google)--but it wasn't reported on Slashdot. One poster even investigated things for himself:
6 501
http://slashdot.org/comments.pl?sid=96614&cid=826
Do a Google search, it was widely reported on most of the other tech sites.
The problem is that those sites created their own single point of failure by all using Akamai for DNS. When Akamai DNS fails, sites that depend on it for their own DNS fail.
It used to be nearly impossible for this to happen. The original rules for DNS were that you had to have at least 2 nameservers for your domain, preferrably 3 or more, and they couldn't be on the same physical networks. With that rule having a single network go down rarely made any domain unresolvable (backbone networks whose outages could render dozens or hundreds of other networks unreachable being the exception). Maybe we should put the old nameserver-diversity rules back into place.
This was years ago (3? 4)... I set up a novell server and setup dns on it as a forwarder and pointed workstations to my novell server for dns.
One of the neat things was the log screen that showed dns actions and you could follow the trail of dns requests to see how they were resolved. what makes this not O/T is that i beleive that this went into a log.
The reason that I think about that is, if DNS stopped working, i'm not sure that i have cached numbers that i could easily get to....
eric
You know, in hawaiian, "akamai" means smart...
damn, i didnt know what the problem was. though my ISP was busted.
There isn't a single post in that thread that proves that it was a Linux machine that leaked it (there is one post that implies it, but that isn't the same thing as proof now, is it?).
Nice try, though.
I was only pointing out that his example was bad.
.com traffic goes to Verisign's control, etc, etc.
In this case, Akamai had some sort of major issue. Okay, fine. Fair enough.
But the root servers themselves are a bad example to point to for a "single point of failure". They're not. The root servers, by themselves, are very robust, widely scattered, and any one of them can, in theory, handle the whole load. Admittedly, for the root, that load ain't a heck of a lot by comparison.
Now, the DNS system itself has several thousand single points of failure, depending on how you define failure. Like you said, all
The root servers, however, are not one of these points of failure. They do what they were meant to do.. to be the root DNS servers. Several can fail and the root lives on.
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.
Did you bother to read the parent post? Where is he/she crying about Slashdot? The parent post is point out that the OP is spouting off about something without any proof.
Idiot.
True, but that's beside the point. There's only 13 visible entities to the world. If those entities fail, then they fail. Consider it as a black box thing. Each root entity has it's own level of robustness, but if it fails, for whatever reason, then it fails. We don't care about the internal workings of each one, because we only have 13 black boxes to talk to. a.root-servers.org, b.root-servers.org, etc, etc.
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.
I'd have loved to have read the article, only you didn't include the IP address and... ;-)
I mean I know it's not as bad as using a regional spelling for "potato"
What region is that where they spell it potatoe ? Do any print examples of that spelling exist?
Or is it just in the "region" where an "s" appears in close proximity to the end of the word?
Copsville?
trailer park, no-shirt wearin' mulletville?
Was that idiot even from the region you are claiming the spelling was from?
BTW, Mr. Bush please tell the FERC to start prosecuting Enron for manipulating the energy market as all the "tin foil hat crowd" claimed when it was happening - if that Washington State utility keeps releasing tapes it is going to get embarassing for you and Dick.
the net itself wasn't down, just happened that some centralized services for largish, geographically spread hosts were compromised all at the same time. That many of those are used mostly by end users makes it look like the net itself is damaged, but it's only edges of the network that are affected, not the core.
Robert
Unless the server that lives at IPaddress W.X.Y.Z only hosts 1 server, and that server has it's documents in the server root folder. Most webservers any more use virtual name services to map HTTP requests to the right "web server" and set of documents.
My personal server runs 7 domains with 12 or 13 sites. Some have real docroot folders, some use the default "you aren't looking in the right place" set of docs. But using an IP address to access a web site probably won't work in these days of many servers per machine.
This space for rent. Call 1-800-STEAK4U
I've always advocated the .cum domain for all porn. It would solve a lot of problems.
Celebrate the finer things in life
just because these guys use akamai hosted dns, and it broke, doesn't mean the rest of the world cares, or is even affected.
Can anyone suggest that these guys build in some redundancy into their architecture? Using dns zone servers from only one provider is begging for trouble, since if that provider goes down, your servers no longer resolve.
This is an architectural problem created by poor planning. Anyone who has a single point of failure in their architecture will eventually go down. Doesn't matter if this SPoF is Akamai, UUNET, or ATT. Regardless of how redundant any one provider is internally, a single provider is a SPoF from the architectural perspective of the website owner.
That's why we host at UUNET and have a second shop and dns zone servers at a local ISP who is connected via a provider who is not UUNET.
If UUNET wrecks their network in some massive outage, our backup site (webservers and ternary dns) kicks in.
"The ISC site gets slashdotted
If you are encountering intermittent problems connecting to our site, it is because we got slashdotted. These connectivity problems are not directly related to the Akamai outage, but are the result of a large number of visitors accessing our site today. Thanks for being patient while waiting for the ISC site to load."
My god... with google down my effective IQ is 12!
Okay ... it is still a decentralized network with no single point of failure. DNS is NOT the Internet. It simply makes it easy for people. When I set up an application to use a server on the Internet I typically use the IP address in case DNS problems happen.
Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others.
Well, as strange as this may seem, these aren't the only sites on the net. I hardly see how these sites failing constitutes a "the sky is falling due to uncentralized DNS servers!" mentality.
-- MrMud
Who promised you THAT?
... is dependent on the net now. everything. You can't just work all by yourself, it's all interconnected. "Work" implies society around you is functioning adequately. We live daily with a certain small amount of the web/computers, etc borked,say a few small percent, but there's a critical mass there that if poofed would collapse the "system". That was what the whole y2k deal was about,and why it needed to be fixed, if most or all of the infrastructure collapses, we are en-screwed, and very few people have the skills,resources or wherewithal to exist totally independent of the rest of society "working" and that most definetly includes the net working. For a SHORT period of time you could keep bangin away on your computer, eventually there would be no electric of note, no telco, no food production, no shipping, no energy production, etc, because any big failure any place in a chain destroys the chain, it no longer exists then, and our economy is all chained together..
Did you ever read, "I, pencil"? It explains it pretty well, you could probably still find it on the net with a search, it's quite good.
When I was younger I lived almost totally wild-literally feral- for several years, about as far as you can get away with "no technology" and living completely independent of the rest of society, it was a hoot, I learned a lot and glad I did it, but HARD and in a lot of cases DANGEROUS, not "sport" type, temporary dangerous and hard, but eeek, you could starve or whatnot. I'm somewhat of an expert at it, and I tell you, in any massive technological collapse the actual mortality rate would be high, let alone just inconvenient, from a variety of factors. It depends on how much borks and how fast and how long it lasts obviously. We no longer have a non computerised infrastructure like we had in the 40's, there is no backup non-computerised non net enabled civilisation to fall back on. Either the net works, or WE DON'T.
Civilization works as long as it is all working, if a big piece isn't, it rapidly de-evolves. Like for instance after a tornado or hurricane hits, or blizzard or big fire, etc, all normal induistrial type life comes to a halt, and what replaces it is not normal, not regular work, and you wouldn't have the luxury of ignoring it in most cases.
It's a nmatter of time/duration and initial severity. I can't say how exactly much total failure it would take to reach a tip over point, but I would 100% guarantee it's a much lower figure than most people imagine it would take.
This is a great post. I get the same results.
I wouldn't presume they use any for their dns funtionality, but fact of the matter is Akamai does have a small proportion of windows servers in their distributed clusters. Seen 'em with my own eyes.
Notice the blurb says "One theory." Besides, the Register is not exactly authoritative.
Second DNS flop? Let's see how long business stays with Akamai. There are plenty of others
From the article...
If you are encountering intermittent problems connecting to our site, it is because we got slashdotted. These connectivity problems are not directly related to the Akamai outage, but are the result of a large number of visitors accessing our site today. Thanks for being patient while waiting for the ISC site to load.
just thought I'd mention that ;)
I'M NOT ANGRY!
5.77% increase for 100% failure... must be the *new* math.
I've been on-line a lot today and didn't even know those sites were down. Didn't effect me in the least. The internet, by it's nature, will always be plagued by the occasional downtime of various services here and there. But in the end, the Internet keeps moving right along.
Think about the worst thing that's ever happened to the Internet and how much that really impacted your daily activity. I don't know about you, but it's always been local connectivity failures that have caused me the most trouble. The occasional site being down really doesn't make a big difference.
This sig has been temporarily disconnected or is no longer in service
I remember a while ago were I had this problem where I couldn't connect to ten percent of the internet. I first noticed I could not connect to Kuro5hin. But I also couldn't connect to some other websites. People on IRC told me they could connect to these sites fine.
I play Magic Online I could get into magic online but 30 percent of the time if I tried to get into a game it would say it could not find the server. Guess the normal server and the servers you play games are different, one or more of those servers was in my "blind spot". I ran a traceroute to kuro5hin and noticed my trace seemed to always get stuck so it never reach kuro5hin.
This problem seems to cleared up a week later
And what do you have to back up your assertions besides what you just pulled out of your ass? Oh, that's right, nothing. This is just another lame attempt to whore some karma.
Sad thing is, it'll probably work because the MODERATORS here as just about as STUPID as YOU ARE.
Even if I lost the phone line I still have some other options.
I could go to my office.
Go to a friend's house.
Connect using my mobile phone; very slooooooooooooooooooooooow
Or as a last resort I guess I could drive to another part of the country and use Airsnort to hack into a WIFI hotspot.
So there really isn't a single point of failure in the net.
http://itworldcanada.com/Pages/Docbase/ViewArtic le.aspx?id=idgml-c624fd4e-b7be-4eaa&Portal=Informa tion%20Architecture&s=393631
Except when it bashes Microsoft, of course.
Simple: it got too slow. Without Akamai, day-to-day operations would positively suck for most people. When Akamai is working (most of the time), the 'net just screams. Quitcherbitchin! Things are better today than before content distribution networks...
There exists no way of exchanging information without making judgments. --Bene Gesserit Axiom
"Several major Web sites - including Yahoo!, Microsoft and Google - were inaccessible at times early Tuesday due to what the company that distributes them online called an attack."
In my job I telecomute 80% of the time - from a San Francisco based home office to the central office in Vancouver.
Without decent connectivity I would be 80% less productive.
Explain how "Offline working can be surprisingly productive" for that career choice.
And I have long thought that Google was several big rooms of white boxes running Linux, with occasional dead nodes in there that aren't worth the trouble to locate, disconnect, and repair, that Google ran themselves.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
...according to this story at washingtonpost.com The story says it was a distributed denial of service attack against Akamai, among others.
...because you never know who you're dealing with.
They are telling me that it was indeed an attack, but an attack aimed not only at them but other companies as well.
I wonder what really happened and who else was attacked..
Every time I try to go to a SourceForge.net site, I get redirected to a 404 page on "portland.co.uk", and their page even shows up in the ad-boxes on Slashdot; is there any connection here? Or has someone been messing with the DNS locally (at Wayne State University, Detroit, MI, USA)?
umm...have you forgotten what article thread you're posting in? :-P
...and that's the way the cookie crumbles.
Summary:
Between approximately 8:30 AM ET and 10:45 AM ET (GMT +4 hours) on Tuesday, June 15, 2004, some Akamai customers using Global Traffic Manager (FirstPoint), NetStorage (Akamai Content Storage), and Akamai services that utilize Global Traffic Manager and NetStorage experienced performance and availability issues.
This incident resulted from a sophisticated, large-scale attack on Internet infrastructure. This attack impacted Akamai's Internet naming functionality (Domain Name Service or DNS), and resulted in delays in DNS name resolution and, in some cases, timed-out DNS requests. Some end users trying to reach affected sites would have experienced slow responses from the Akamai name servers, potentially resulting in page time-outs. The attack did not cause an outage in Akamai services, as Akamai continued to serve DNS requests. However, the amount and nature of attack traffic created degradation in performance.
The problem was quickly detected by Akamai's automated monitoring systems, and Akamai personnel identified the root cause as a large Internet attack. The attack was mitigated by a combination of actions by Akamai to adjust our infrastructure in response to the attack, along with working with network partners to shut down the source of the attack.
As result of these actions, all Akamai services had returned to normal operating performance by 10:45 AM ET.
Akamai is continuing to work closely with several network partners and legal authorities around the world to identify both the nature of the attack and its intended targets.
We regret any inconvenience this may have caused you or your users. Please contact your Akamai Customer Care representative at 1-877-4-AKATEC (1-877-425-2832) if you have any questions.
Service Note: One of the actions taken during the attack was to temporarily increase the DNS TTL (time to live) on responses being returned from Akamai. This action is helping end-users cache successful responses for longer, thus improving service.
Pretty cool stuff, to be sure.
But all of the proprietary stuff means that there's only one implementation. There's no RFC describing what they do. There's no alternate implementations that might show flaws. There's no cross-checks that outsiders might provide.
Like others have said, it's a mono-culture. And they've done it so well, there's been no interest in creating a set of standards or IETF working group to try and create the multiple, compatible offerings that might guard against mono-culture (and give customers a chance to avoid vendor lock-in.)
This may be related to the new voice over public DNS research done by Dan Kaminsky and presented on Sunday. For those of you who missed Kaminsky's talk at Level 1, it was all about DNS hacking -- including voice over DNS radio broadcasts using 32,000 public DNS servers for bandwidth, and more bizare stuff I don't pretend to understand.
If you use a service for important work, you should know at least one of it's IPs. I have a slew inside /etc/hosts (and \WINNT\HOSTS). www.google.com and mail.yahoo.com for sure.
Don't make me break out the Trillian logs... --LordPixie
Akamai has their DNS servers located on plenty of different networks. (assuming you don't count 'The Internet' as one big physical network)
The problem here is twofold. First, a sites' DNS services were provided by the same company. This is going to happen no matter what, and even the rule you mentioned doesn't prvent it. However, and here's the kicker, lots of sites use the same company. So instead of one company's network outage affecting that one company's sites, we have it toasting half the internet.
--LordPixie
Erm, in my Windoze boxen it's in
WINNT\system32\drivers\etc\HOSTS
Actually, I have been having resolution issues into yahoo via akamai on and off for a couple weeks.
But I want to know WHY akamai was having a problem. What's the scoop?
This is ridiculous. You're basically arguing that you can't call a computer running Linux a Linux box. I merely pointed out how different things would be reported if Akamai ran on Windows boxes. Several major websites were wiped out for a while today.
It is misleading to refer to the box as a "Linux" box. Was it really the kernel that was at fault for the machine being cracked, or was it a bug in one of the daemons that the machine was running? There are differences between a Linux box that runs BIND and another that runs EZ-DNS (or whatever).
What you're effectively saying is, "Uh, you can't call a machine running Linux a Linux box, because that would sound, like, detrimental to Linux!" I'm sorry, a Linux box is a Linux box. There's no judgment going on there. And this website has no problem with calling a user-ran executable trojan a "Microsoft hole" whenever they can.
I'm not going to call a Linux box a "BIND machine" just because it's running BIND. You're splitting hairs here. I'm sorry, most people will refer to a computer running Linux as a Linux box. It's just a generic term and not an indictment! I made no such judgment other than on Slashdot journalism. For the record, I happily run Gentoo Linux.
Masturbation is better than nothing.
Nothing is better than sex.
Therefore, masturbation is better than sex.
Whatever happened to my decentralized net with no single point of failure?
Oddly enough I've just read the part of Weaving the Web that points out how, for all the Internet's and web's decentralised methods, they still used DNS which is essentially a heirarchy pointing to very few computers, which can cause problems later, being the Internet's Achille's Heel. It mentions the biggest fear not being technical failure but human maliciousness.
DNS is so core to a company that outsourcing it is absolutely ludicrous, IMO. Even third party "secondaries" can be disasterous.
Considering that DNS is one of the easier things to replicate internally (djbdns can do it securely, quickly, automatically and atomically with cdb/ssh/rsync/cron), it makes little sense to hand it off to a third party. On the flip side, this ease-of-replication is probably why DNS outsourcing is so common (despite being a bad idea).
The Register must be wrong about this. I used to work at Akamai, and I feel pretty damn sure that no one crashed those servers by getting *on* them to run the 20-line snippet of code that locks the kernel (assuming we're talking about the kernel lock exploit that was being widely discussed recently; it requires shell access).
What is much more likely is that somebody found a way to DDOS the Akamai top-level name servers, or that configuration files containing incorrect/conflicting/nefarious information were pushed out to the top-levels.
Knowing how many stages and checks there are in the Akamai deployment procedures, and how much monitoring there is of the network health, I would be astonished if someone managed to foobar the top-levels with a bad configuration. A co-wortker of mine did it once, a long time ago, so I guess it *could* happen, but it was one of those perfect-storm sorts of things. And even then, it just slowed things down a little - certainly not enough to make the news like this.
Correction: Dead-tree versions are HARDER to work with.
i am a soviet space shuttle
Visitors to our site may have experienced intermittent problems today because we got Slashdotted. These connectivity problems are not directly related to the Akamai outage, but are the result of a large number of visitors accessing our site today. Thanks for being patient while waiting for the ISC site to load. (from their website)
The post states: "Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others" It seems as if it is being implied that these companies represent the 'Internet'.
http://news.yahoo.com/news?tmpl=story&u=/washpost/ 20040615/tc_washpost/a43635_2004jun15
man rtfm
"Akamai is confirming that network outages this morning were caused by a distributed denial of service (DDoS) attack that affected its DNS management system.
The performance problems affected Microsoft, Google, Yahoo and antivirus update services from Symantec and TrendMicro, which are among Akamai's 1,100 customers. Some of the largest affected sites were able to switch their DNS settings to their internal network, rather than akadns.net, which handles domain name service management for Akamai customers. The akadns.net system routs requests for high-volume customer web pages to content stored on its network of distributed servers, easing traffic to the client's main server and speeding delivery to the end user. Akamai performs similar function for downloads of audio and video files, software patches and antivirus definitions.
The outages mark the second disruption of Akamai's network in less than a month, following a similar incident May 24. "
-
Netcraft
"Yahoo and Google have both been hit by a DoS attack. The attack has been hitting Google, Yahoo, and other sites that include Microsoft for the past couple of hours. The attacks started this morning and it was detected by Keynote Systems, a web tracking company that is able to track the load and bandwidth on the Internet. According to Keynote they saw an "Internet performance issue" this morning.
"The availability issues were limited to several large sites, all of whom outsource their domain name server (DNS) services to Akamai. These sites dropped to near-zero availability," a spokesman for Keynote said. They have tracked the attacker back to person that is at the Akamai Technologies ISP. No other information has been given to us at this time. We do not know if the FBI is working on this issue right now, but we expect them to do so. "
- OverclockersClub
(Score:5, Insightful, right...) Actually, it was. If Google et al were all using a single Akamai backbone TCP/IP routers and they went down, they would be affected as well.
Google was using some DNS servers as their DNS servers (NSs for their domain zone). Their servers went down and then Google was unreachable because their DNS was down, nothing more. Nothing magical about DNS per se. TCP/IP routing was working but this hardly means DNS is any more "centeral point of failure" than TCP/IP. Google should not rely on a single network of DNS servers and it would be fine, because DNS is designed in such a way and has been for over twenty years.
The problem here is the bastardization of DNS standard by Akamai. DNS records should be cached on recursive name servers. Google is used everywhere. If Google had sane TTL and expiration times set for their zone, their zone would be cached by every ISP in the world and their DNS servers could be down for a week and no one would even notice.
This is how DNS should work, can work, and have been working for literally decades. Please read RFC 882: DOMAIN NAMES - CONCEPTS and FACILITIES (P. Mockapetris, November 1983), RFC 883: DOMAIN NAMES - IMPLEMENTATION and SPECIFICATION (P. Mockapetris, November 1983), RFC 1034: DOMAIN NAMES - CONCEPTS AND FACILITIES (P. Mockapetris, November 1987) and RFC 1035: DOMAIN NAMES - IMPLEMENTATION AND SPECIFICATION (November 1987).
Sincerely,
Pan Tarhei Hosé, PhD.
"Homo sum et cogito ergo odi profanum vulgus et libido."
From isc.incidents.org:
Not directly related to the Akamai outage? And they think why on Earth have we bloody slashdotted them in the first place if not because of the very Akamai outage and their coverage therof?! This is related as directly as it gets:
Don't they know Slashdot?! Kids...
Sincerely,
Pan Tarhei Hosé, PhD.
"Homo sum et cogito ergo odi profanum vulgus et libido."
Fuzzy bunny warned last year akamai was compromised!
The intrusion on the web security site was in good taste as well, a banner with a pink bunny revolving on the top of the web page! (they only compromised the ad server)
We outsourced it... the beancounters say its cheaper and more reliable!
Someone please explain the concept of "secondary DNS" to these folks. Backup DNS, folks, backup DNS. Never put all your eggs in one basket and all that.
Note the fact that during the outage for example google.com got you through to the Google home page because google.com had and still has (due to DNS standards) an A record in a Google DNS server (as opposed to host names like www.google.com which can have CNAMEs to outside domains). And there were reportedly no problems getting there (for example I didn't have any, but this arguably could be because so many others were failing this simple "try the domain name only nerd test").
The great thing about DSL is that you can use any provider you want... From your post, it sounds like you experience numerous outages, so you most likely do need to change.
In my personal experience, I've been using my DSL service for about 2 years now, and I've experienced only one outage, and even it only lasted 5 minutes. Just for the sake of filling in the details, my local telco is Verizon (unless they've changed their name once again this week) and my DSL ISP is Earthlink.
Well, that solution is riddled with problems. The real solution to the DNS issue is in MaraDNS now... As per my suggestion, the DNS server will try to update it's expired records with the upstream server, and if it is unable to do so, it will serve the expired record. So, for all users served by a MaraDNS server, any DNS records it has stored it it's cache will be served no matter what the state of all other DNS servers in the world.
I'm considering modifying it to write all records ever served into a file on disk, which is updated when a newer record is served, and can be used as a hosts file in the event of global DNS collapse. Just a small modification to MaraDNS, and large ISPs will be able to keep all internet access running perfectly normally, for several months on end, without a single DNS server being online. But I'm just ranting now...
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Back in the early days of the Space Shuttle, there was some problems with the plumbing. It was reported by NASA that the "GE Space Toilet" was having issues. However the "NASA Space Kitchen" was working out just fine, thank you. You'd never know they were both built in the same building in King of Prussia, by the same contractor, GE.
Sorta like the claims that "Linux is ready for the desktop" when really they mean KDE/GNOME looks cool and doesn't crash (much). But have a DNS problem, and suddenly you want to make sure that every one knows BIND is independent of Linux, which is just a kernel after all.
That said, I agree with the point not to throw out the baby with the bath water.
Your assumption is too simple, and I can think of two exceptions right off the bat.
First of all, non-essential services lose money when their business is closed. A restaraunt does not get the night's customers back another night if they have a power failure, and a ski resort doesn't get the weekend's customers back the next weekend if it rains.
Second, you aren't factoring in competition. Amazon is down? I guess I'll check eBay for that book, or Barnes and Noble, etc.
Economics is never a simple linear relationship in the real world.
Long live the Speaker Bracelet
Rolo D. Monkey
I'm considering modifying it to write all records ever served into a file on disk, which is updated when a newer record is served, and can be used as a hosts file in the event of global DNS collapse.
;)
Yikes! Better get your bomb shelter ready too.
All jokes aside, your right, that makes much more sense. As for the DSL provider, this is the first hiccup so I wont leave Bell Atlantic for it.
Who replies to a 4 day old post anyway? Sounds like some bad net lag
If I ever get my hands on the little teenage script kiddy bastards who created these bots I will personally strangle their damned windpipes!
I was in the process yesterday of trying to upgrade 300 OSX clients and this little shithead wasted several hours of my time. and also time for my customers.
If I catch one of these little teenage assholes responsible I will rip their testicles off with a pair of wire cutters and feed what was left to the damned possums and racoons in this area.
I just want to see these assholes killed! I'm tired of these little bastards creating hours of work for IT dealing with their childish pranks. You hear me script kiddies reading this? No jury for you! I will provide judicial punshment myself if I find you!
The key points are:
Where Akamai tries to sidestep the issue that some of the nets most accessed sites were inaccessible for millions of users (sure, those that were not spesifically targetted had no impact).
Also:
Still no mention that the only effective solution to the attack was dropping Akamai DNS completely, which was employed in the customer DNS, not in Akamai. Also, it talks about a single source of attack. By definition, that's DoS, not DDoS. Which should be child's play to filter. Something is missing here.
I think the most important piece of information in that press release is the announcement that FBI is involved in the investigation. Apparently, however the attack was done, Akamais is now firmly committed to it being a deliberate attack and not a problem caused by their own operations.
The key points are:
Where Akamai tries to sidestep the issue that some of the nets most accessed sites were inaccessible for millions of users (sure, those that were not spesifically targetted had no impact). Also later they bash Keynote for not accurately portraying site availability due to different DNS caching than the end-users (which I don't believe without details).
Also:
Still no mention that the only effective solution to the attack was dropping Akamai DNS completely, which was employed in the customer DNS, not in Akamai. Also, it talks about a single source of attack.
I think the most important piece of information in that press release is the announcement that FBI is involved in the investigation. Apparently, however the attack was done, Akamais is now firmly committed to it being a deliberate attack and not a problem caused by their own operations.
An article also reveals that the attack involved a bot net:
'Zombie' PCs caused Web outage, Akamai says
99.999% of the living that has been done in the last ten thousand years has been done detached from the net.
What we call folk wisdom is often no more than a kind of expedient stupidity.-Edward Abbey
Tahya al-Moqawama al-Iraqiya!
Soon the Americans will have 11 September all over again. Our brothers in Iraq, and in Palestine, and in Afghanistan will be avenged by our brothers already in America. Soon we will show the American pigs what it is like to live in burning cities with their women and children dying around them. Ten thousand died in Iraq, and thousands of Mujaheddin in Afghanistan, and we will make the Americans suffer ten million deaths!
Tahya al-Moqawama al-Iraqiya!
Hear us now, America! Your days of easy existence will soon be over! We will set your cities ablaze and make you regret the day you decided to invade our lands!
Tahya al-Moqawama al-Iraqiya!
Tahya al-Moqawama al-Iraqiya!
TAHYA AL-MOQAWAMA AL-IRAQIYA!