Slashdot Mirror
Akamai: How They Fought Recent DDoS Attacks
yootje writes "Infoworld is running an interesting article about Akamai and the DDoS attack that hit the network of Akamai Tuesday. According to this article one of the defenses of Akamai is the big diversity of their hardware: 'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.' So says Paul Vixie, architect of BIND and president of the ITC." Yootje points to another article on this subject as well, this one at Internetnews.com. Update: 07/07 19:38 GMT by T : Note that Vixie's quote here is actually presented out of context; he was commenting by way of contrast on the diversity of the root DNS servers, not Akamai's content-serving system.
"We wired a million dollars into the attackers' Swiss account."
That's shocking!
The diversity of hardware and software may be an IT nightmare but I think this shows how effective it really is. Now all we need is a concise cost/benefit analysis.
'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.'
Wow, your sys admins and help desk must LOVE supporting that!
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
They've achieved deliberately what happens naturally in a lot of other companies.
It says the root servers use different stuff, not akamai. RTFA.
nobody knows what they run, so nobody can make a decent attack ..
Unfortunately, the ""We deliberately use different operating systems, different name server implementations..." quote is from Paul Vixie, president of the Internet Systems Consortium, and it's about the root name servers, not about Akamai.
Human/Ranger/Zangband
When you say "It didn7 w0rk" are you talking about the "Post Anonymously" checkbox?
Just askin you big hacker, you.
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
I have to wonder if the diversity of systems was an intentional choice of theirs way back to face these kinds of attacks or if it just grew that way from rapid growth and having their systems spread all over.
They survived the attack and "Oh yea, we MEANT for it to happen that way".
I think it's spin.
Akamai claims over 1,100 customers and indicated that only 2 percent of them were noticeably impacted by the attack, such as not being available for about an hour.
Theo only statistic they ofer is the percentage of customers that were impacted. To me this hints of trying to play down the severity of the situation. When only 2 percent of your customers comprise (following is is a made up statistic since they didn't give me one) 80 percent of your traffic, you're lying by omission by only giving customer statistics.
The submitter's description of the article was completely incorrect and backwards.
Diversity of hardware makes ROOT DNS SERVERS more defensible. Akamai is NOT diverse, and they do not want to be.
The quote on diversity is by Vixie wrt the roots servers--it's a criticism of Akamai! Jesus H. Christ, it's in the first paragraph!
Media that can be recorded and distributed can be recorded and distributed.
-kfg
Also, Paul Vixie is the founder of ISC, not ITC. What a shoddy article write-up -- two blatantly obvious mistakes I caught by skimming the articles got front-paged.
"No one likes working in a hamster wheel, and your shop smells of cedar shavings from here." - TaleSpinner
This article has nothing to do with Akamai, other than pointing out that Akamai DNS is vulnerable to DOS.
Most of this "article" is a puff-piece (or paid advert) for one "CloudShield Technologies," pimping their (vaporware) "server for applications that do deep packet processing at gigabit-per-second rates."
-Isaac
I am not a lawyer, and this is not legal advice. For Entertainment Purposes Only.
Actually, according to the article the diversity approach is part of what's used to defend the DNS root servers, not Akamai. Vixie specifically mentions that this approach is not practical for an ordinary content provider like Akamai because, 'the cost would "drive their accountants crazy."' I'm dubious about just how helpful diversity would be against a DDoS attack in the first place. Diversity won't solve the problem of requests coming in faster than they can be processed.
There's no point in questioning authority if you aren't going to listen to the answers.
Just ask MS, they will tell you.
This is not the sig line you are looking for... -- Old Jedi Sig Line Trick
SYN cookies are for TCP connections (because TCP uses a three-way
handshake to set up a connection). DNS uses (primarily) UDP traffic,
which is connectionless (there is no "stateful" connection with UDP).
SYN cookies do no good when your DNS servers are under attack.
a Mac running Classic on a beefy box
You mean like a Quadra 950 (~35lbs.) or a pallet of hamburger helper?
Rampant Ninja related crimes these days...Whitehouse is not the exception
couldn't you just link to them on slash dot
that's been proven to be an effective, system independent DoS attack (even if the attack was unintentional or brought about by the owner)
The Neo-Bohemian Techno-Socialist
Is that like using Windows 98 and Windows ME?
It sounds like a recipe for success!
A valid tactic... it mitigates the problems with a unified vendor, but it costs lots more...
I wish the net was headed in the right direction, but it's not. No single site or company will ever "win". The resilience of the web lies in it's redundancy and distribution. What I see is continued centralization and creation of points of failure. As "Broadband" internet access is more monopolized and treated as a platform for mindless browsing, and smaller ISPs are destroyed, the net is being squeezed into fewer and fewer hands. This invites attacks that can not be protected against. The real solution is to let everyone run everthing they want. That's the only way to route around damage.
Friends don't help friends install M$ junk.
...is like trying to wipe out swarm of gnats with a shotgun.
dinner: it's what's for beer
Boss: "Why did nearly half our service go down Friday?"
CTO: "Actually, sir, the real question is why did we lose less than half of our service. The answer is that I've, uh, been strategically using different systems and components throughout the enterprise on purpose to prevent drastic losses. No one else could have even kept 10% of their machines up under that DDOS."
Boss: "I knew I could count on you for the right PR spin job. Go back and think up some other good excuses."
-Adam
Correct me if I'm wrong.
I remember reading an article about the US Army using classic Mac for their webservers for just that reason. Hey, an URL: http://www.wired.com/news/politics/0,1283,21725,00 .html
In the case of the Akamai incident, the vulnerable service was DNS. Paul Vixie, architect of BIND (Berkeley Internet Name Domain) and president of the Internet Systems Consortium, charged that Akamai's proprietary approach to DNS makes it a single point of failure. He added that the 13 DNS root servers, which weathered a vicious DDoS attack in 2002, are even more defensible today than they were back then. The root servers are resilient, Vixie said, because their operators embrace diversity. "We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures," Vixie told Internetnews.com.
He's not talking about how great Akamai is. He's talking about how great everyone else is.
On another note: What the heck does this story have to do with Akamai operators fighting DDoS attacks? They more than likely sat with their thumbs up their rears contemplating how having such a structured and inflexible DNS system could possibly be in err.
tm
Support TBI Research: http://www.raisinhope.org
I thought we were disorganized here where I work, but it turns out we were just throwing up a good defense.
'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.'
I remember Several, init/cdev A, init B, etc.
The article summary is incorrect. Diversity was not a defense for Akamai, it is a defense for the 13 DNS root servers. In fact, in the article, Paul Vixie "charged that Akamai's proprietary approach to DNS makes it a single point of failure." The diversity approach is what is used to help prevent these kinds of failures in the global DNS system.
lick the cancle button (at least thats what our Chinese QA says)
"Hey I'm in!"
ver^M
MS DOS 6.22
"wtf?"
Two roads diverged in a wood, and I - I took the one the bus load of girls just went down.
What the heck are those? Are they like bad karma points for articles that have overlapping information with other articles?
By the way, which one of the articles is it that says Akamai did anything right to fight attacks?
Just because you can, does not mean you should.
'We deliberately use different operating systems . . . .'
They called me crazy for using Windows 95, 98, 2000, CE and ME . . . I'm invincible! Bwahahaha!
'It's about CloudShield Technologies ... recently announced CS-2000', and nothing but a fluff peice meant to sell some hardware. Sure, Akami's DDOS is discussed ("DDOSs are ba-ad, mmkay."), but then it just goes on to talk about the CS-2000.
RandomAndInteresting.comdefending the world from stupidity since 1979
not only did the submitter not rtfa
the editors did not rtfa
and after the first five posts pointing this out, it was obvious that nobody was reading the responses either.
nobody was reading anything, and now we have a 1000 responses saying the same thing, it wasn't akamai, it was the root servers, blah blah blah.
Akamai is, at best, being disingenous when they say only 2 percent of their customers were affected by the outage. Maybe 2 percent of their customers, but how many of their customers customers were affected?
2 percent may not sound like much on the surface, but if that percentage includes companies like Microsoft, MSNBC, Amazon, Yahoo, CNN, Lycos and other big-shot content providers then the relative number of "customers" affected by the outage is a lot more notable.
I'm not tense. I'm just terribly, terribly, alert.
I'm sorry, next time I will read the article ten times before I post...
My photo's.
No, no, no... it's just pronounced "Paul Vixie" but the correct spelling is V-I-N-T C-E-R-F.
I have a feeling it was more like,
(BOFH types RETURN, followed by)
"Oh Shit!"
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
Dude, calm down. I'm sorry, I admit I wanted to have it fast on Slashdot, but not for my ego, but I like it to have it on Slashdot quick. You are talking to real persons, it was a mistake. Come on, it's not like your life depends on Slashdot.
My photo's.
First, the root servers have different dns server software and OSes, not because Vixie thought of it, but because it is policy codified in the BCP RFC for root servers best practices. In fact, I think he was unhappy about other root servers using non-BIND software in the beginning.
Second, he is being disingenuous about his comments about patents, his company owns at least one patent related to the Verisign "Site Finder" service methodology. Nominum Patent I didn't see any statements by him disparaging his company when they applied for that patent. So it isn't that he doesn't like patents, it is that he doesn't like that Akamai is making money doing third party DNS without paying him money or homage. Note: His commercial, for profit dns server software company has a white paper enumerating the scalability and other problems with BIND, and they use an architecture more similar to DJBDNS than to BIND 9 - separate auth and resolving dns server packages, most modern dns server software uses this architecture to reduce code complexity and improve security and performance.
Third, if he wanted to be the pillar of dns server software that he supposedly is, he could have sent a few goons from Nominum over to Akamai and set up some boxes with his commercial, for profit, "scalable" dns server software and Akamai would have been able to see if his software was able to stand up to the ddos attack better than what they have. If it did, he probably could have gotten a sweet, lucrative contract out of it and been a hero for helping thwart the attack, rather than a hypocritical, self serving competitor hiding behind Open Source to appear credible.
Fourth, Akamai is a single point of failure because that is what they do - offload dns and content load from the biggest companies on the net life MS, google and ebay. No, I don't work there, but I would venture a guess that they carry more traffic than (maybe) any other company. So I am sure it is easy to armchair quarterback and say they should do this and that, but when the attacks are probably at 10's or 100's of GiB/s I am not sure what I would do.
Nominum is also involved in RFID stuff, so I will be interested to see what happens with him and his companies as that ramps up. And who knows what deals have already been made - "the future of DNS is right."
Some DNS software links:
nsd - high performance, uses BIND style files and authoritative only
They have an interesting testing procedure where they run nsd and BIND, have them build responses to the same queries and then analyze any differences: diff analysis
maradns
Powerdns, mysql and a pretty website
djbdns he's grouchy and the no license license thing freaks people out and pisses them off, but people become attached to the quirky but rock solid software.
nstx, ip over dns, yeah...
(Yes, reliable. It only crashes when you are doing something, so it gets rebooted often enough ;)
That's it. My reading comprehension is gone. I'm going to bed now and hopefully not dream of anything remotely related to this Daliesque image.