Slashdot Mirror
Akamai DNS Outage Messes up Net
katre writes "Checking all my favorite sites this morning, I saw that about half a dozen seem to be offline. Trying to figure out why, I found an interesting article on the front page at http://isc.incidents.org/. Seems that the problems at Akamai are screwing over Yahoo, Google, Microsoft, Fedex, Xerox, Apple, and others. Whatever happened to my decentralized net with no single point of failure?"
provider of real time market data...
hope the al quedas aren't taking notes on this..
Its still there, and you're using it. The only organizations affected by this are those who chose to use a service that acts as a single point of failure.
trustedworlds.net - gaming, security, and the gunk that lives in between
How many *think* they can't live without web access? Offline working can be surprisingly productive, and as it often forces more thinking and planning (e.g. in preparation for being back online, and just thinking through what would happen of you could be online) the results end up being better.
You mean decentralized?
Anyways butting both DNS records on the same point of failure breaks standards. These companies deserve to be hit hard (PR wise) for not building a roburst network.
I actually would probably get work done without web access!
The web happened my dear friend, and it was based on the predominant distributed computing model at the time: client/server. Even DNS, with its highly distributed spread of processing and data, has a set of (overloaded) root servers with the commensurate single points of failure. The solution? Peer-to-peer.
:)
Too bad even the term P2P raises so many red flags with certain Associations of America.
You would think that the root DNS servers would be kept up to date with critical information. Just what happened, and how did Akamai get knocked around this? Did they screw with their DNS information and change their nameserver addresses or something?
Striking fear in the authors of godawful fanfiction, I am here, appearing in darkness, Tuxedo Jack!
I am unable to access the server listed above from various server locations spread across the country & using different ISP's.
That's not the DNS outage problem -- the site is simply slashdotted.
Rule #1 -- Politics always trumps technology.
The problem, as I understand it, is that Yahoo, Google & co. "outsourced" their DNS service.
I could have accepted that medium-big sized IT companies don't want to run their own DNS servers, but giants Google & co. should have enough money to do so instead of relying on servers located somewhere else.
Funnily enough www.google.com still works for me (thanks to DNS caching I guess)
+++ MELON MELON MELON +++ Out of Cheese Error +++ redo from start +++
how they can screw up there entire DNS, and it's still down. It started as far as I can tell right after 8:30 or so, the last outage was due to a software update on there own site. It's now nearly 11am and it's still not working.. Man, I would think you could restore from backup at least in that time frame, and have something up for people.. Wonder if there will be an credit on the account this month...
When I was in grad school at Cornell, my O/S professor went on a rant about the evils of Akamai. No one believed him. Now we know he was right.
Wer mit Ungeheuern kämpft, mag zusehn, dass er nicht dabei zum Ungeheuer wird. --Nietzsche
It's not truely decentralized...
The root nameservers are the most obvious example...
The most obvious example? The fact is that there are 13 of them, in widely scattered locations across the globe, and it's not decentralized?
Damn man, what exactly would you consider "decentralized" then?
Root servers go down all the time. It's not particularly unusual. There's THIRTEEN of the things. Up to 8 have been down at once with no major effects on the network, IIRC.
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.
I can see the logic that went into this plan:
"Well, Akamai has a few million DNS boxes, if we put everything there we'll be fine! That's not a single point of failure!"
Yeah, about that... multiple vendors may have been a good idea in retrospect instead of just one monolithic provider.
Time to re-examine the definition of Single Point of Failure.
you can still get to all those sites. You just have to REMEMBER the ip instead of depending on the computer to look it up for you ;). TCP/IP was designed to have not centeral point of failure and still does it's job well. DNS was not quite designed in such a way.
âoeTolerance applies only to persons, but never to truth. Intolerance applies only to truth, but never to persons.
Damn that was funny 4 years ago. Do you have any good "hanging chad" material?
Al Gore was talking about creating *legislation* that helped foster the Internet.
Why do Conservatives bitch to high hell when anything they say it taken out of context, but repeat dumb quotes by Liberals out of context for years and years?
Maybe they should stop worrying so much about people who havn't had a political job in 4 years and worry about the people who do have important jobs now and are doing them so amazingly badly.
-B
The way to solve it is get more companies out there who provide the same sevices, something not easy after the dot bust era when people dont want to take such risks.
"Slashdot, where telling the truth is overrated but lying is insightful."
Err.. What are they supposed to do? Spam everyone who ever registered a domain and say, "oops our bad, but by the time you get this, it'll all be over?"
If it's really that critical, then set up Nagios to monitor those ips or something.
I had one person call this morning because they couldn't reach Google. And what was she trying to use it for? She broke a window this weekend and was looking for a dealer who sells her type of window.
I have a much bigger issue with spams clogging my incoming mail folders than I do with transient DNS issues.
*Live* and *work* are too entirely different things. I could not get any of my work done with network access.
...how many *think* they can't live without web access? Offline working can be surprisingly productive, and as it often forces more thinking and planning (e.g. in preparation for being back online, and just thinking through what would happen of you could be online) the results end up being better.
F'real. To think, they did all that even before the Altair was a twinkle in Ed Roberts' jockey shorts!
Quod scripsi, scripsi.
The Net is decentralized... however, if several *LARGE* sites happen to be resolved through one DNS server and it crashes, people think that the 'net is down'... IIRC, Helldesk people bitch about this - people calling up and saying 'I can't get to www.mytimewastingbullshitpage.com, is the net down?' Not realizing that just becuase one or two or thirty sites are down, the net is still up....
FWIW, I missed google for all of 10 minutes, and figured it was my work ISP....
The difference is that we know we're joking and just being mean in a school yard sort of way. We don't take it seriously and only keep doing it in the 'little brother poking at big brother because it gets a rise out of him and there's nothing he can do about it' way. It's childish amusement.
...".
When liberals do it, they're telling The Big Lie and with the help of your liberal dominated media, turn those Big Lies into Pravda-like Truth and then use their own lies as political weapons.
Your media boosts the left while hurting the right at every opportunity.
How many times have you read "So-n-so, ultra conservative Congressman from xyz"? When it comes to someone like Kerry who is a top 5 ultra liberal, they never tell you that. They sure as hell never refer to him as "Senetar Kerry, ultra liberal Senator from ultra liberal Mass. Junior Senator to Ted Kennedy.
See the difference now? Probably not, but it was worth a shot.
It's only a sinlge point of failure if you can't get to *ALL* of yout websites, instead of some.
I was thinking about this while scrambling to answer the phone, check outage reports, and generally calm down customers.
:) )
If a product or service, such as Akamai, does their job very well, everybody will want to use them. If everybody uses them, you create a single point-of-failure. Any design flaw in that product or service becomes a disaster, simply through volume. Does this mean a successful product or service can actually be a bad thing for people?
Other examples include just about anything from Microsoft, older versions of Sendmail and BIND (worm-of-the-week problem), and Firestone tires.
(I'm not trying to advocate communism, excessive government regulation, or anything like that. So fanatical libertarians, conspiracy theorists, etc., can put down the rant-o-matic flamethrowers.
Comments?
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Akamai didn't mess up the net. Akamai messed up some web sites that are akamai customers. Remember kids, www is only a subset of the internet, and akamai customers a small fraction of the www.
You're also relying on these random companies to not violate your privacy and equally importantly to keep your data safe from destruction.
Do you have any idea how poor the data safety & recovery policies are at most of these places?
You're much better off having your own PC, putting VNC on it behind a firewall with an SSL VPN or even just ssh, and copying your precious data to a CD once a week or so. That's far better than most places are doing for you.
You know how liable they are when they lose your data? Not at all. Just poof, gone. They might say they're sorry but it is unlikely they'll even admit anything happened. User error, you know?
If they would do their jobs, there would not be an issue.
If the users, who think they don't need to worry about the net, would stop surfing porn with IE, stop clicking on every goddamned attachment that says "A fun game to play" or "Thought you'd like to see this", would stop signing up for every privacy-violating list on the planet then maybe the network guys would actually have a POSSIBILITY of keeping the network online!
Oh yeah, and yo momma wears combat boots!
I noticed this problem this morning when I was hunting for an updated version of YahooPOPs. I wasnt getting replies from Google. I opened another FirePanda window and my homepage, slashdot, was working fine (Hey look at that on the homepage, Yahoo changed their mail service today, no luck for YahooPOPs). I tried yahoo, altavista, even msn in different tabs but I wasnt getting anywhere.
I tried pinging google and I was getting a reply so my first thought was, there is something terribly wrong at verizon DSL. I must make the most of what fragmented connection I have now before its down all day and I'm stranded actually doing work.
Thats when I started opening every story on slashdot's homepage in different tabs and setting them all to threshold 3, threaded... Just incase.
Come to think of it, I'm going to change my slashdot bookmark from slashdot.org to 66.35.250.151 just incase of DNS failure.
Need my SlashCrack
Im dreaming ofa big bndwdth, That can resist the
DNS was designed to be robust enough. Not one root server but many (ok, that's the weak point, we've all seen many DDoS against them, but it's not THAT bad). All zones are handled by their own servers, and (in theory) multiple servers for each zone. All in all, it's not a bad design.
If what happened was that someone put all the servers behind one link, it's not DNS' fault, the BOFH there screwed up (and considering it's akamai, they should not have done that).
(If that's not what happened, sorry, I couldn't RTFA, it's slashdotted or there's some sort of DNS problem there too).
GPG 0x1B479C78
If you want to have a true dialogue instead of fingerpointing with "nah-nah" gibes, you'll have to actually state which films you're talking about and what were the quotes that are "out-of-context".
Whatever happened to my decentralized net with no single point of failure?
Outsourcing and consolidation.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
This was years ago (3? 4)... I set up a novell server and setup dns on it as a forwarder and pointed workstations to my novell server for dns.
One of the neat things was the log screen that showed dns actions and you could follow the trail of dns requests to see how they were resolved. what makes this not O/T is that i beleive that this went into a log.
The reason that I think about that is, if DNS stopped working, i'm not sure that i have cached numbers that i could easily get to....
eric
I was only pointing out that his example was bad.
.com traffic goes to Verisign's control, etc, etc.
In this case, Akamai had some sort of major issue. Okay, fine. Fair enough.
But the root servers themselves are a bad example to point to for a "single point of failure". They're not. The root servers, by themselves, are very robust, widely scattered, and any one of them can, in theory, handle the whole load. Admittedly, for the root, that load ain't a heck of a lot by comparison.
Now, the DNS system itself has several thousand single points of failure, depending on how you define failure. Like you said, all
The root servers, however, are not one of these points of failure. They do what they were meant to do.. to be the root DNS servers. Several can fail and the root lives on.
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.
A reasonable idea... I however doubt that any service would issue anything alert wise unless it was caused by some sensational event. New nasty worm, terrorism... A simple outage, even on this scale just isn't exciting enough for the newschannels.
Shame that. Might warrant a blurb tonight on the news, but it certainly won't dislodge the scroller that has the most recent body count in it, and probably no "this just in" by the talking heads.
Unless the server that lives at IPaddress W.X.Y.Z only hosts 1 server, and that server has it's documents in the server root folder. Most webservers any more use virtual name services to map HTTP requests to the right "web server" and set of documents.
My personal server runs 7 domains with 12 or 13 sites. Some have real docroot folders, some use the default "you aren't looking in the right place" set of docs. But using an IP address to access a web site probably won't work in these days of many servers per machine.
This space for rent. Call 1-800-STEAK4U
just because these guys use akamai hosted dns, and it broke, doesn't mean the rest of the world cares, or is even affected.
Can anyone suggest that these guys build in some redundancy into their architecture? Using dns zone servers from only one provider is begging for trouble, since if that provider goes down, your servers no longer resolve.
This is an architectural problem created by poor planning. Anyone who has a single point of failure in their architecture will eventually go down. Doesn't matter if this SPoF is Akamai, UUNET, or ATT. Regardless of how redundant any one provider is internally, a single provider is a SPoF from the architectural perspective of the website owner.
That's why we host at UUNET and have a second shop and dns zone servers at a local ISP who is connected via a provider who is not UUNET.
If UUNET wrecks their network in some massive outage, our backup site (webservers and ternary dns) kicks in.
My god... with google down my effective IQ is 12!
Life without irony would be quite dull!
It is misleading to refer to the box as a "Linux" box. Was it really the kernel that was at fault for the machine being cracked, or was it a bug in one of the daemons that the machine was running? There are differences between a Linux box that runs BIND and another that runs EZ-DNS (or whatever).
How about this: Instead of labelling the Akamai boxes that have problems as "Linux" boxes, label them as "BIND" boxes, or whatever DNS server it is that it runs. Perhaps there's a FreeBSD machine in there that is having similar problems.
It is allowable, though, to refer to a Windows box as just that. MS ships an all-in-one product, and seldomly do admins use Windows to run BIND, Apache or other OSS servers.
All of this hand-ringing in an effort to paint "Linux" as bad, or as "just as bad" is dopey. One might as well point a finger at the administrator of the machine that was hacked, the services that were running on it, etc. Most Windows problems are caused by the same thing too. It is wiser to point at the admin (and the services one chooses to run) than to point at the OS, or the kernel.
You need to restart your computer. Hold down the Power button for several seconds or press the Restart button.