Slashdot Mirror
Rackspace Restored After DDOS Takes Out DNS
An anonymous reader sends word that Rackspace has recovered from a severe distributed denial of service attack. "Over on the company's Google+ page Rackspace warned of 'intermittent periods of latency, packet loss, or connectivity failures when attempting to reach rackspace.com or subdomains within rackspace.com.' The company's status report later confirmed it had '... identified a UDP DDoS attack targeting the DNS servers in our IAD, ORD, and LON data centers [North Virigina, Chicago and London]. As a result of this issue, authoritative DNS resolution for any new request to the DNS servers began to fail in the affected data centers. In order to stabilize the issue, our teams placed the impacted DNS infrastructure behind mitigation services. This service is designed to protect our infrastructure, however, due to the nature of the event, a portion of legitimate traffic to our DNS infrastructure may be inadvertently blocked. Our teams are actively working to mitigate the attack and provide service stability.'"
Seriously.. if you actually read the horse's mouth, you would know that this all transpired back 3 days ago.
Who else could possibly have done this? Only cyberbogeymen could have been quite this evil, obviously.
It was those pesky North Koreans, of course...
If you want news from today, you have to come back tomorrow.
BYOB.
Bring your own body-guard.
There are numerous affordable ddos mitigation providers (e.g x4b, staminus, etc).
Typically you have peices of infrastructure which is required by many service instances belonging to many customers.
It's nearly always better to have one service instance drop offline than to have the whole peice of shared infrastructure become unusable.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
People generally use UDP because it doesn't require a handshake and the amplification attacks are generally UDP (time server or DNS server amplification attacks can go as high as 200x - i.e. you can send 1mb outbound and get 200mb back; so with address spoofing, it's easy to overwhelm someone with such an attack).
What can you do about it to protect yourself? Stuff all, I am afraid. At the end of the day, if you cop a 100gbps attack on a 100mbps pipe, it's game over, no matter what you try to pull. All you can do is beg for help upstream, where someone can handle that traffic.
If you're talking about websites, I guess CloudFlare would help - and it's basically free (and no, I don't work for them or have any association with them) but that pretty much only works for websites, I think - not other services.
UDP reflection attacks are one case when "economies of scale" work in reverse.
I can throttle such attacks on my DNS servers, since I'm only serving for a few domains and there's not much urgency.
For large ISPs, however, there's going to be a lot more legitimate traffic for a lot more domains and if you bounce a request, you may be turning away the one legitimate customer in the flood.
I hate UDP reflection with a passion. Ordinary attacks are annoying enough, but if I ever got my hands on the people behind this, I could cheerfully do things involving crudely-sharpened objects, live coals and molten plastic. The CIA could take notes and learn new techniques.
The key to this is the ability to send NAK packets back upstream so that the DDoSers' ping requests get returned to sender instead of making it to their intended target. Seems like we need a better roll-out of this idea if RackSpace is still falling victim to this.
Remember...
ACK means acknowledged, I've got that and it sticks.
RST means reset, I didn't get that right, we've got something that doesn't add up to the checksum, let's go back to a previous numbered packet.
NAK means, I got that and I don't like that. Go away, and would a router upstream please add a firewall rule blocking whoever sent me that.
I don't see how throttling works in a UDP reflection attack, from the perspective of the intended target? Sure you can throttle the number of requests per minute you answer from your DNS server - but if someone is requesting DNS packets from you, you're not likely the target (so it works for you, the DNS server owner but doesn't help the attack victim, in short, unless every DNS server does it - and there's a hell of a lot of IPs in the open resolver project).
What's far more likely is that they'd be using one of the multitudes of locations that allows spoofed IP addresses, and then requesting a 50x amplified DNS dump from you back to a spoofed address - and that address it the real target. Plus they'd be hitting up 100 other DNS servers at the same time.
Collectively, that spoofed IP address can be made to cop a 100gbps attack with virtually no effort and then those poor bastards basically can't do a thing about it. They can throttle or firewall anything they like but unless their router and pipe can handle 100gbps - and chances are it can't - they're screwed.
That's why I hate it. The "mirror" can throttle, but that's just a drop in the bucket. Only if all the reflection mirrors are throttling can it help, and the larger the mirror, the larger the number of apparently legitimate requests would be, so it's harder to make them good throttles.
Plus, not only the target system is getting blasted. The mirroring systems are getting a pretty heavy load. They can throttle this, but then they risk choking off the legitimate requests, since a legitimate request and a reflection attack request are indistinguishable.
I think the only real cure would be to drop UDP. You can't do a reflection attack from TCP, since you cannot spoof the sender. At least without some major mucking around with the Internet as a whole.