Rackspace Restored After DDOS Takes Out DNS

An anonymous reader sends word that Rackspace has recovered from a severe distributed denial of service attack. "Over on the company's Google+ page Rackspace warned of 'intermittent periods of latency, packet loss, or connectivity failures when attempting to reach rackspace.com or subdomains within rackspace.com.' The company's status report later confirmed it had '... identified a UDP DDoS attack targeting the DNS servers in our IAD, ORD, and LON data centers [North Virigina, Chicago and London]. As a result of this issue, authoritative DNS resolution for any new request to the DNS servers began to fail in the affected data centers. In order to stabilize the issue, our teams placed the impacted DNS infrastructure behind mitigation services. This service is designed to protect our infrastructure, however, due to the nature of the event, a portion of legitimate traffic to our DNS infrastructure may be inadvertently blocked. Our teams are actively working to mitigate the attack and provide service stability.'"

Comments

"designed to protect our infrastructure"

[mexsudo]

what about your customers?

Re:"designed to protect our infrastructure"

[OhPlz]

BYOB.

Bring your own body-guard.

Re: "designed to protect our infrastructure"

[Anonymous Coward]

If the infrastructure goes down so do their customers

Re:"designed to protect our infrastructure"

[petermgreen]

Typically you have peices of infrastructure which is required by many service instances belonging to many customers.

It's nearly always better to have one service instance drop offline than to have the whole peice of shared infrastructure become unusable.

What, no blaming haxxorz?

[Anonymous Coward]

Who else could possibly have done this? Only cyberbogeymen could have been quite this evil, obviously.

Re:What, no blaming haxxorz?

[Frosty Piss]

Who else could possibly have done this? Only cyberbogeymen could have been quite this evil, obviously.

It was those pesky North Koreans, of course...

The story is 3 days old

[Anonymous Coward]

Seriously.. if you actually read the horse's mouth, you would know that this all transpired back 3 days ago.

Re: The story is 3 days old

[Anonymous Coward]

Seriously if everyone read from every horse's mouth, there wouldn't be a need for a new aggregator service like Slashdot. Also slashdot doesn't even race primary news services, let alone the horse's mouths. If you are here for sure, you are here for the wrong reason.

Re:Merry Christmas & a present

[Anonymous Coward]

There are over 300 million active domains, with more coming and going every day. Does your hosts file account for this? Where do I get the version with every active domain in it? Otherwise I find it highly impractical to "avoid [DNS] totally".

Astroturf

[Anonymous Coward]

Can you feel it?

Surely the controlling powers of /. aren't so dumb that they can't hide their statistically recurring concentration of themed stories to push their agenda, every 2 months like clockwork - come on guys!

Re: Merry Christmas & a present

[Anonymous Coward]

Why does that matter though? So you can't connect to the latest malware serving domains? Sounds like a good thing to me. Most sites' addresses never change, the ones that do are crap and viruses.

Alex Jones attacks!

[Anonymous Coward]

Why? Because of Alex Jones!

Should have gone for an F5 GTM balancer

[Anonymous Coward]

i heard they protect against such attacks

Re:Merry Christmas & a present

[Anonymous Coward]

Jesus, I never thought I'd end up defending He Who Must Not Be Named, but he's right that most people spend most of their online time visiting a small number of sites. If you analyzed the frequency of your web browsing habits, you'll probably find that at least 70% of your browsing is to a small set of sites (for me, slashdot, stackexchange, google news, a few porn sites). Putting just those in /etc/hosts will save you a lot of lookups. It won't let you avoid DNS totally if you wander off your usual paths, but it will do some good.

How to mitigate similar UDP port DDOS attack

[Anonymous Coward]

TFA mentioned a UDP port DDPS attack at 3 rackspace's datacenter

TFA also mentioned that rackspace people are working to mitigate the DDOS attack and the ensuing effect, but never tell us how and what they do

Can anybody kindly share with us what small flies like us, can do, if we are under similar kind of attack?

Re: How to mitigate similar UDP port DDOS attack

[Anonymous Coward]

There are numerous affordable ddos mitigation providers (e.g x4b, staminus, etc).

Re: How to mitigate similar UDP port DDOS attack

[Anonymous Coward]

Yeah it's called start dropping packets from the attackers.

There is only so much you can do depending on how bad you are being DDoS'd but good firewalls running linux or one of the BSDs can handle a ton of DDoS traffic if you completely drop packets from the source hosts and don't let the traffic reach a server at all.

Once packets start reaching a server (like a DNS server) and the server needs to respond or figure out what to do based on it's config, that's when you're dead in the water.

The tough part is how the attack is carried out, is most of the traffic coming from the same subnet or area? Is it someone running a botnet with 5000+ hosts all around the world? It really depends. Sometimes it's best to go offline completely or shift to a secondary method of serving legitimate customer traffic, etc.

Re:How to mitigate similar UDP port DDOS attack

[Gumbercules!!]

People generally use UDP because it doesn't require a handshake and the amplification attacks are generally UDP (time server or DNS server amplification attacks can go as high as 200x - i.e. you can send 1mb outbound and get 200mb back; so with address spoofing, it's easy to overwhelm someone with such an attack).

What can you do about it to protect yourself? Stuff all, I am afraid. At the end of the day, if you cop a 100gbps attack on a 100mbps pipe, it's game over, no matter what you try to pull. All you can do is beg for help upstream, where someone can handle that traffic.

If you're talking about websites, I guess CloudFlare would help - and it's basically free (and no, I don't work for them or have any association with them) but that pretty much only works for websites, I think - not other services.

Re:How to mitigate similar UDP port DDOS attack

[RabidReindeer]

UDP reflection attacks are one case when "economies of scale" work in reverse.

I can throttle such attacks on my DNS servers, since I'm only serving for a few domains and there's not much urgency.

For large ISPs, however, there's going to be a lot more legitimate traffic for a lot more domains and if you bounce a request, you may be turning away the one legitimate customer in the flood.

I hate UDP reflection with a passion. Ordinary attacks are annoying enough, but if I ever got my hands on the people behind this, I could cheerfully do things involving crudely-sharpened objects, live coals and molten plastic. The CIA could take notes and learn new techniques.

Re:How to mitigate similar UDP port DDOS attack

[Gumbercules!!]

I don't see how throttling works in a UDP reflection attack, from the perspective of the intended target? Sure you can throttle the number of requests per minute you answer from your DNS server - but if someone is requesting DNS packets from you, you're not likely the target (so it works for you, the DNS server owner but doesn't help the attack victim, in short, unless every DNS server does it - and there's a hell of a lot of IPs in the open resolver project).

What's far more likely is that they'd be using one of the multitudes of locations that allows spoofed IP addresses, and then requesting a 50x amplified DNS dump from you back to a spoofed address - and that address it the real target. Plus they'd be hitting up 100 other DNS servers at the same time.

Collectively, that spoofed IP address can be made to cop a 100gbps attack with virtually no effort and then those poor bastards basically can't do a thing about it. They can throttle or firewall anything they like but unless their router and pipe can handle 100gbps - and chances are it can't - they're screwed.

Re:How to mitigate similar UDP port DDOS attack

[RabidReindeer]

That's why I hate it. The "mirror" can throttle, but that's just a drop in the bucket. Only if all the reflection mirrors are throttling can it help, and the larger the mirror, the larger the number of apparently legitimate requests would be, so it's harder to make them good throttles.

Plus, not only the target system is getting blasted. The mirroring systems are getting a pretty heavy load. They can throttle this, but then they risk choking off the legitimate requests, since a legitimate request and a reflection attack request are indistinguishable.

I think the only real cure would be to drop UDP. You can't do a reflection attack from TCP, since you cannot spoof the sender. At least without some major mucking around with the Internet as a whole.

Learn to read (for your favorite sites)

[Anonymous Coward]

See my subject-line above, & the post you replied to - LEARN TO READ!

* In case you hadn't noticed, I note that in my original post...

APK

Re:I submitted it back then too (was rejected)

[Anonymous Coward]

A simpler and easier solution is to just manually set your primary, secondary, and tertiary DNS resolvers to point to different companies' servers. And you can still go and tinker with your hosts files on top of that for sites which have fairly static/stable DNS entries.

The reason you always get modded down is because a) you're spamming your own personal 'service' and b) you're a pompous dick about it. Your posts are also a disjointed, rambling series of poorly connected sentences which consist mainly of misused and/or unrelated buzzwords and jargon.

Re:Merry Christmas (& a present)

[Anonymous Coward]

Would you please fuck off?

Thanks!

Re: You *CAN* do the following things

[Anonymous Coward]

for fuck sakes man. why would someone set all those registry keys to protect one fucking personal computer. oh yea and it's running windows. people rarely ddos personal computers inside homes.

just another post so APK can troll his host file solution. same posts over and over and over again. I would like to meet this guy in real life. I bet he has a mangina. fucking faggot troll.

Re: Merry Christmas (& a present)

[Anonymous Coward]

LOL, who the fuck needs a host file program. I bet not many n*x people use this tool. probably a windows only program. you are a bigger tool than your host file tool ;)

Re: You *CAN* do the following things

[Anonymous Coward]

Apk didn't note hosts in his post and why? Stupid question, answer is to protect themselves is why and they aren't just registry hacks, there's far more listed there with practical examples from articles that actually work on large scale against DDOS!

Re:Merry Christmas (& a present)

[Anonymous Coward]

Dear offtopic troll, would you take your own advice please? Thanks. Apk's solution works. Where's yours? It ain't.

Re: Merry Christmas (& a present)

[Anonymous Coward]

Who needs it? Users of the affected networks. Apk's solution works against this and other DNS issues. What a stupid question on your end. Have you made a better solution yourself? Obviously not.

Re:I offer a solution for end users

[Anonymous Coward]

Even if you made the worlds best bagels, and gave them away for free, people at the coffee shop, bus station, and laundromat would probably get sick to death of hearing you talk about them at every f'n opportunity. You'd actually get fewer people trying them than if you were polite and relevant about your advertising. Maybe try as a personal challenge to only throw in host file ads on 1/3 of the threads you comment on for a while?

-Someone else, not the original poster, and not a mod in this conversation

Re:I offer a solution for end users

[Anonymous Coward]

If a solution works (apk's does) shout it from the rooftops, troll. Where's yours? It isn't. Just off topic troll crap from you.

I thought we solved this already.

[The New Guy 2.0]

The key to this is the ability to send NAK packets back upstream so that the DDoSers' ping requests get returned to sender instead of making it to their intended target. Seems like we need a better roll-out of this idea if RackSpace is still falling victim to this.

Remember...
ACK means acknowledged, I've got that and it sticks.
RST means reset, I didn't get that right, we've got something that doesn't add up to the checksum, let's go back to a previous numbered packet.
NAK means, I got that and I don't like that. Go away, and would a router upstream please add a firewall rule blocking whoever sent me that.

Re:TCP vs. UDP for DNS *should* work

[Anonymous Coward]

Let's play a simple game, shall we?

You give us the IP address of a box you have modified the HOSTS file and registry on and we'll see if registry hacks are any use against a volumetric attack which is simply larger than the pipe leading to the registry key in question....

Try do it to AMAZON instead - ok?

[Anonymous Coward]

Especially when you have trouble shutting down your own PC, lol!

* Per my subject-line above: Good luck "taking 'em down", ok?

APK

P.S.=> I only put out valid information with backing evidences thereof from reputable sources - you don't & haven't! Thus, you're full of shit & mere "hot-air" on your end, lmao... apk

Re:I offer a solution for end users

[Anonymous Coward]

With all the bolding, quoting and capitalization, you need to stop drinking coffee. Makes it hard to follow you!

Re:I offer a solution for end users

[Anonymous Coward]

You need to stop drinking alcohol so you can read.

Merry Christmas (& a present vs. this)

[Anonymous Coward]

Article submitter here: This-> http://start64.com/index.php?o... solves THIS & other DNS issues by avoiding it FOR FAVORITE SITES (emphasizing that for the moron who downmddded me earlier on this same post since HE's ILLITERATE-> http://it.slashdot.org/comment... & LATER HERE AGAIN http://it.slashdot.org/comment... w/ BOTH POSTS pointing that out to him)...

ALL operating locally from RAM + your IP stack (hosts file) & diskcaching kernelmode subsystems (less messagepassing overheads & GREATER EFFICIENCY by far being ALL in ring 0/kernelmode native parts you already have that are proven + refined - For going faster, safer, & more reliably online).

* Enjoy...

---

I.E./E.G.-> I do for 24 of my fav. sites I spend 95++% of my time online, placing them @ the TOP of my hosts file to avoid DNS redirect poisoning (kaminsky bug of which 99.999% of ISP DNS are *NOT PATCHED* against mind you) & downed DNS too (or exploited ones per this article), & it ends up resolving sites FASTER locally from RAM once cached.

Equating to approximately 2-3 MILLION indexed REMOTE DNS lookups worth saved (wasting time querying remote DNS which is exploitable as hell & insecure mostly) & works for me locally faster & more reliably by far vs. such exploits this article notes + more 95++% of the time (per my router logs analyzed on where I spend my time online MOSTLY).

---

Now - Sub 4% of the time when I DO use remote DNS, I use OpenDNS (secured, filtered vs. threats, patched vs. the Kaminsky flaw & DNSSEC secured to its upstream updaters) BOTH in my router/firewall + OS IP Stack settings.

APK

P.S.=> It's 100% free, & works doing MORE with LESS, no less (especially vs. other "so-called 'solutions'", it's "Stronger than steel & a 3rd the weight" ala Howard Stark) & my program is recommended + hosted by MalwareBytes' hpHosts (reputable + reliable as it gets) -> http://hosts-file.net/?s=Downl...

... apk