Slashdot Mirror
Akamai: How They Fought Recent DDoS Attacks
yootje writes "Infoworld is running an interesting article about Akamai and the DDoS attack that hit the network of Akamai Tuesday. According to this article one of the defenses of Akamai is the big diversity of their hardware: 'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.' So says Paul Vixie, architect of BIND and president of the ITC." Yootje points to another article on this subject as well, this one at Internetnews.com. Update: 07/07 19:38 GMT by T : Note that Vixie's quote here is actually presented out of context; he was commenting by way of contrast on the diversity of the root DNS servers, not Akamai's content-serving system.
"We wired a million dollars into the attackers' Swiss account."
That's shocking!
The diversity of hardware and software may be an IT nightmare but I think this shows how effective it really is. Now all we need is a concise cost/benefit analysis.
'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.'
Wow, your sys admins and help desk must LOVE supporting that!
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
They've achieved deliberately what happens naturally in a lot of other companies.
It says the root servers use different stuff, not akamai. RTFA.
nobody knows what they run, so nobody can make a decent attack ..
Unfortunately, the ""We deliberately use different operating systems, different name server implementations..." quote is from Paul Vixie, president of the Internet Systems Consortium, and it's about the root name servers, not about Akamai.
Human/Ranger/Zangband
When you say "It didn7 w0rk" are you talking about the "Post Anonymously" checkbox?
Just askin you big hacker, you.
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
If I read it right, one of their problems was their lack of diversity -- they all use Akamai's proprietary DNS.
"No one likes working in a hamster wheel, and your shop smells of cedar shavings from here." - TaleSpinner
I have to wonder if the diversity of systems was an intentional choice of theirs way back to face these kinds of attacks or if it just grew that way from rapid growth and having their systems spread all over.
They survived the attack and "Oh yea, we MEANT for it to happen that way".
I think it's spin.
I don't know how related these two things are, but the AfterNET IRC network has been ^H^H^H^H^H^H^H is being flooded with SYN packets and is -down-.
Is this related to these DDoS attacks?
Akamai claims over 1,100 customers and indicated that only 2 percent of them were noticeably impacted by the attack, such as not being available for about an hour.
Theo only statistic they ofer is the percentage of customers that were impacted. To me this hints of trying to play down the severity of the situation. When only 2 percent of your customers comprise (following is is a made up statistic since they didn't give me one) 80 percent of your traffic, you're lying by omission by only giving customer statistics.
The submitter's description of the article was completely incorrect and backwards.
Diversity of hardware makes ROOT DNS SERVERS more defensible. Akamai is NOT diverse, and they do not want to be.
Some people aren't afraid to make a crappy joke using their own name, just as some are not afraid to run it into the ground.
The quote on diversity is by Vixie wrt the roots servers--it's a criticism of Akamai! Jesus H. Christ, it's in the first paragraph!
Media that can be recorded and distributed can be recorded and distributed.
-kfg
I've often wondered how a Mac running Classic on a beefy box as a server would stand up to an attemp to h4x0r it. To really get at it, seems to me you would have to get to the base underpinnings of the OS on some level. Which are arcane and hard to master, even (I'm told) to seasoned Mac programmers.
Not that I'm implying that it would be invulnerable to some attacks (like DDOS) but surely it seems that many of your other bases would be covered.
Quod scripsi, scripsi.
This article has nothing to do with Akamai, other than pointing out that Akamai DNS is vulnerable to DOS.
Most of this "article" is a puff-piece (or paid advert) for one "CloudShield Technologies," pimping their (vaporware) "server for applications that do deep packet processing at gigabit-per-second rates."
-Isaac
I am not a lawyer, and this is not legal advice. For Entertainment Purposes Only.
maybe that is what afternet should do..
they are totally hosed right now due to a huge ddos.
see http://www.afternet.org/ for all the details
sucks
anime+manga together at last.. in real time.
Actually, according to the article the diversity approach is part of what's used to defend the DNS root servers, not Akamai. Vixie specifically mentions that this approach is not practical for an ordinary content provider like Akamai because, 'the cost would "drive their accountants crazy."' I'm dubious about just how helpful diversity would be against a DDoS attack in the first place. Diversity won't solve the problem of requests coming in faster than they can be processed.
There's no point in questioning authority if you aren't going to listen to the answers.
Just ask MS, they will tell you.
This is not the sig line you are looking for... -- Old Jedi Sig Line Trick
SYN cookies are for TCP connections (because TCP uses a three-way
handshake to set up a connection). DNS uses (primarily) UDP traffic,
which is connectionless (there is no "stateful" connection with UDP).
SYN cookies do no good when your DNS servers are under attack.
I love how the "ITC" links to www.isc.org
Moo.
Erm, I think the poster made a mistake here. This diversity is attributed to the 13 root servers. Akamai's services do not employ such techniques due to the unsupportable cost. Based on the problems we saw during the DDoS, I can't say Akamai had much to offer in its arsenal.
Or am I the one who misread?
GreyPoopon
--
Why is it I can write insightful comments but can't come up with a clever signature?
IIRC, TinyDNS can integrate with LDAP, then you can SSH in and use an ldap browser/client to modify and add records..
It's a better solution, on paper, since LDAP is optimized for the fastest retrieval, at the expense of write time. RDBMS's are generally the other way around, or at least balanced.
Of course, you can have OpenLDAP use mysql as a backend if you really want to bring that abomination into the equation.
I don't need no instructions to know how to rock!!!!
Bind9 in Woody never dies.
Friends don't help friends install M$ junk.
I ran BIND9 on Red Hat 7.2 for about 2 years.. its still running now in fact. No random crashes on BIND ever in that time. It was rock solid...
"Hey! Unless this is a nude love-in, get the hell off my property!!"
couldn't you just link to them on slash dot
that's been proven to be an effective, system independent DoS attack (even if the attack was unintentional or brought about by the owner)
The Neo-Bohemian Techno-Socialist
Is that like using Windows 98 and Windows ME?
Moderators, please correct the lead-in... BIND and the global DNS system is what has the diversity. The problem with Akamai was their lack of diversity on top of their proprietary hacks to DNS.
It sounds like a recipe for success!
A valid tactic... it mitigates the problems with a unified vendor, but it costs lots more...
I wish the net was headed in the right direction, but it's not. No single site or company will ever "win". The resilience of the web lies in it's redundancy and distribution. What I see is continued centralization and creation of points of failure. As "Broadband" internet access is more monopolized and treated as a platform for mindless browsing, and smaller ISPs are destroyed, the net is being squeezed into fewer and fewer hands. This invites attacks that can not be protected against. The real solution is to let everyone run everthing they want. That's the only way to route around damage.
Friends don't help friends install M$ junk.
...is like trying to wipe out swarm of gnats with a shotgun.
dinner: it's what's for beer
Not only that, Vint Cerf's name is Paul Vixie!
/. editors learn.
When will
I don't need no instructions to know how to rock!!!!
Boss: "Why did nearly half our service go down Friday?"
CTO: "Actually, sir, the real question is why did we lose less than half of our service. The answer is that I've, uh, been strategically using different systems and components throughout the enterprise on purpose to prevent drastic losses. No one else could have even kept 10% of their machines up under that DDOS."
Boss: "I knew I could count on you for the right PR spin job. Go back and think up some other good excuses."
-Adam
Correct me if I'm wrong.
In the case of the Akamai incident, the vulnerable service was DNS. Paul Vixie, architect of BIND (Berkeley Internet Name Domain) and president of the Internet Systems Consortium, charged that Akamai's proprietary approach to DNS makes it a single point of failure. He added that the 13 DNS root servers, which weathered a vicious DDoS attack in 2002, are even more defensible today than they were back then. The root servers are resilient, Vixie said, because their operators embrace diversity. "We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures," Vixie told Internetnews.com.
He's not talking about how great Akamai is. He's talking about how great everyone else is.
On another note: What the heck does this story have to do with Akamai operators fighting DDoS attacks? They more than likely sat with their thumbs up their rears contemplating how having such a structured and inflexible DNS system could possibly be in err.
tm
Support TBI Research: http://www.raisinhope.org
I've never had a problem with any version of bind on any operating system.
I have seen the bind 9 problem - if you are using a single CPU system you may never see a problem but a heavily loaded bind 9 on an SMP redhat system does die fairly often, and leaves a suicide note about a failed assertion. We use a cron job to check bind for signs of life every 3 minutes and restart if need be.
I don't think I've ever seen the problem on SuSE linux though...
Do a search on freshmeat.net. MyDNS also runs straight off MySQL.
I'm sorry if I haven't offended anyone
but why didn't it work? Or is this a case of "it could have been worse?" And if it is, then why does it even matter?
Backing up OpenBSD to Windows 95 is not as stupid as it looks.
;)
lol, you are correct! One of our backup solutions is having a win98 box with ActivePerl installed go grab a copy of the datafiles every night. In the event of data corruption, THAT is usually the copy I restore from, purely because it is fast to restore from and highly reliable. (Yes, reliable. It only crashes when you are doing something, so it gets rebooted often enough
Not sure why, but that reminded me of the Simpson's episode where Burns and Smithers are going to the power plant's main switching system, and have to go through more locked doors and devices than the intro to a "Get Smart" episode, and once they get there, it has a broken screen door that is open to the outdoors and Burns runs off a dog that wandered in...
Tequila: It's not just for breakfast anymore!
With all this diversity in system, one would think that setting up decoy Windows boxes would serve as good bait for hackers as well.
I thought we were disorganized here where I work, but it turns out we were just throwing up a good defense.
'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.'
they showed how they survived DDOS attack by using various platform and causing themselves trouble, but they failed to demonstrate that they can't do that without various platform, actually, that's doable.
Make everything so ass backwards and broken that even if someone did get in, they couldn't do anything useful anyway. :D
The article summary is incorrect. Diversity was not a defense for Akamai, it is a defense for the 13 DNS root servers. In fact, in the article, Paul Vixie "charged that Akamai's proprietary approach to DNS makes it a single point of failure." The diversity approach is what is used to help prevent these kinds of failures in the global DNS system.
lick the cancle button (at least thats what our Chinese QA says)
What the heck are those? Are they like bad karma points for articles that have overlapping information with other articles?
By the way, which one of the articles is it that says Akamai did anything right to fight attacks?
Just because you can, does not mean you should.
'We deliberately use different operating systems . . . .'
They called me crazy for using Windows 95, 98, 2000, CE and ME . . . I'm invincible! Bwahahaha!
'It's about CloudShield Technologies ... recently announced CS-2000', and nothing but a fluff peice meant to sell some hardware. Sure, Akami's DDOS is discussed ("DDOSs are ba-ad, mmkay."), but then it just goes on to talk about the CS-2000.
RandomAndInteresting.comdefending the world from stupidity since 1979
not only did the submitter not rtfa
the editors did not rtfa
and after the first five posts pointing this out, it was obvious that nobody was reading the responses either.
nobody was reading anything, and now we have a 1000 responses saying the same thing, it wasn't akamai, it was the root servers, blah blah blah.
Akamai is, at best, being disingenous when they say only 2 percent of their customers were affected by the outage. Maybe 2 percent of their customers, but how many of their customers customers were affected?
2 percent may not sound like much on the surface, but if that percentage includes companies like Microsoft, MSNBC, Amazon, Yahoo, CNN, Lycos and other big-shot content providers then the relative number of "customers" affected by the outage is a lot more notable.
I'm not tense. I'm just terribly, terribly, alert.
I'm sorry, next time I will read the article ten times before I post...
My photo's.
I think it's when you do logic using reads from data beyond the packet header.
Packets are composed in layers. The lower ones have to do with transmission over the network. The higher ones have to do with the interpretation of the packet (like which application session etc it belongs to). And of course you have the payload, the data being sent, like the letter in the envelope.
Mu!
"A witty saying proves nothing." ~Voltaire
"d'Oh!" ~Homer
No, no, no... it's just pronounced "Paul Vixie" but the correct spelling is V-I-N-T C-E-R-F.
Now if the attack is a new type with no easy fix I don't neccessarily want them to publish it far and wide, but I'd at least like to know who are the relevant authorities Akamai says they've given the details to. Is there an FBI agent or case number I can make enquires to? Akamai is not very forthcoming.
Personally I think this attack was just a known exploit or a huge volume of spoofed DNS traffic and Akamai is hiding behind a veil here to protect its image.
I have a feeling it was more like,
(BOFH types RETURN, followed by)
"Oh Shit!"
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
Dude, calm down. I'm sorry, I admit I wanted to have it fast on Slashdot, but not for my ego, but I like it to have it on Slashdot quick. You are talking to real persons, it was a mistake. Come on, it's not like your life depends on Slashdot.
My photo's.
... penetration or a simple DDoS attack. No one is immune from DDoS if the other guy has more bandwith than you do to use, doesn't matter the OS I would imagine.. Although with akami you would think that would be hard to pull off. Perhaps it was a big state sponsored attack? The details have been meagre.
I've seen in in the dumb mechanical world too. Machines are alive, I just know it! I had a little italian sportscar once,a raggedy old fiat, but I liked it. Anyway, I SWEAR it was alive and female and jealous. It worked great all the time until "date night" for zogger. Go to pick up my date, it would break down. Never missed.
In un-configures terminals you'll get a ^H when you hit backspace. So it's a way of saying that you went back, erased, and continued typing.
First, the root servers have different dns server software and OSes, not because Vixie thought of it, but because it is policy codified in the BCP RFC for root servers best practices. In fact, I think he was unhappy about other root servers using non-BIND software in the beginning.
Second, he is being disingenuous about his comments about patents, his company owns at least one patent related to the Verisign "Site Finder" service methodology. Nominum Patent I didn't see any statements by him disparaging his company when they applied for that patent. So it isn't that he doesn't like patents, it is that he doesn't like that Akamai is making money doing third party DNS without paying him money or homage. Note: His commercial, for profit dns server software company has a white paper enumerating the scalability and other problems with BIND, and they use an architecture more similar to DJBDNS than to BIND 9 - separate auth and resolving dns server packages, most modern dns server software uses this architecture to reduce code complexity and improve security and performance.
Third, if he wanted to be the pillar of dns server software that he supposedly is, he could have sent a few goons from Nominum over to Akamai and set up some boxes with his commercial, for profit, "scalable" dns server software and Akamai would have been able to see if his software was able to stand up to the ddos attack better than what they have. If it did, he probably could have gotten a sweet, lucrative contract out of it and been a hero for helping thwart the attack, rather than a hypocritical, self serving competitor hiding behind Open Source to appear credible.
Fourth, Akamai is a single point of failure because that is what they do - offload dns and content load from the biggest companies on the net life MS, google and ebay. No, I don't work there, but I would venture a guess that they carry more traffic than (maybe) any other company. So I am sure it is easy to armchair quarterback and say they should do this and that, but when the attacks are probably at 10's or 100's of GiB/s I am not sure what I would do.
Nominum is also involved in RFID stuff, so I will be interested to see what happens with him and his companies as that ramps up. And who knows what deals have already been made - "the future of DNS is right."
Some DNS software links:
nsd - high performance, uses BIND style files and authoritative only
They have an interesting testing procedure where they run nsd and BIND, have them build responses to the same queries and then analyze any differences: diff analysis
maradns
Powerdns, mysql and a pretty website
djbdns he's grouchy and the no license license thing freaks people out and pisses them off, but people become attached to the quirky but rock solid software.
nstx, ip over dns, yeah...
Windows 95 and 98 were still DOS based (even ME, though they ripped out the command shell so the end user couldn't use it). Only the NT/2000/XP family isn't DOS based, strictly speaking.
----- Question authority, but not ours. Hate the man, but we're not him.
By the time BIND 9 came out, I knew better than to mess with redhate.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
new security company, fraud and ict sec.., I need help! wanna be my accomplice? Only open for the ict section, need folks who know nip, tcp/IP, ipSec, des/PGP, fourthfloor, nsk, an most urgent html security and programming.. check this out..
Helt seriost trenger jeg folk som snakker norsk ihvertfall, som kan deler av overnevnte, og/eller som har annen sikkerhets relatert bakgrunn innen hacker/cracker miljo.. dette er viktig for aa faa edge paa de andre etablerte firmaene som finnes allerede..
Give me a pip in tnys@start.no
Og folkens, vaer seriose da.. trenger ikke crapmail!!!!!!