Slashdot Mirror
Ask Slashdot: Experience Handling DDoS Attacks On a Mid-Tier Site?
New submitter caboosesw writes "A customer of mine recently was hit by a quick and massive DDoS attack. As we were in the middle of things, we learned that there are proxy services of varying maturity to deal with these kinds of outbreaks from the small and mysterious (DOSArrest, ServerOrigin, BlackLotus, DDOSProtection, CloudFlare, etc.) to the large and mature (Prolexic, Verisign, etc.) Have you guys used any of these services? Especially on the lower price point that a small e-commerce (not pr0n or gambling) company could afford? Is a DDoS service really mandatory as Gartner now puts this type of service in the same tier as SEIM, firewalls, IPS, etc?"
There's two key strategies to avoid being DDoSed... first, have more processor, network speed, and disk I/O resources than you need for normal load so that the attacker can't fill one of your computers pipes. Then, host your server or servers at multi-connected datacenters which can cut off large users of your server before it reaches your NIC card. Firewalls at the server can't get back the bandwidth lost to needless connections, but firewalls at the datacenter entry points can. Basically, make sure none of your time-sensitive loads reach 100% and you're fine.
Remember this really cool slashdot story about a sysadmin on the receiving end of a DDOS?
http://slashdot.org/story/01/05/31/1330202/post-mortem-of-a-dos-attack
The original writeup link is dead but I found it here (warning: PDF). This was a really cool story.
http://www.stanford.edu/class/msande91si/www-spr04/readings/week1/grcdos.pdf
"You cannot find out which view is the right one by science in the ordinary sense." - C.S. Lewis on Intelligent Design
In most cases, your customers are going to exist in one or a few countries. It would be valuable ahead of time to add redirect rules to your iptables for entire ranges of IP addresses located in countries that don't host your customers. Redirect these IP ranges to a sacrificial server on a different pipe to the backbone. That way, when some of your customers are abroad and need access to your services, they can still get some amount of response.
Additionally, you can proactively parse your user accounts for IP addresses and build a whitelist ruleset for your iptables to implement in a defcon 0 situation. Don't use this as a normal operations mode, just when the shit has really hit the fan and you need to block everyone except your known-good account holders.
Seth
$5 / month hosted VPS on linux = awesome!
Having been the target of an HTTP-DDOS attack, I can tell you that manually blacklisting IP ranges is really ineffective. A DDOS botnet is comprised of thousands of machines that have been randomly infected by whatever vector the botnet operator used: Emails, web drive-by, etc. The result is that the source addresses are scattered widely with little relation between most participating addresses.
To defend against the attack, I wrote up an automatic firewall blacklisting program that detected and blocked each participating IP address individually in near-realtime. I was blocking more than 31,000 separate addresses before the DDOSers finally gave up trying to down the attacked website. Wierdly, there appears to have been no motive at all for the attack, yet they spent weeks attacking the target machine and actively trying to tune their attack to get past my filtering.
Even if you knew with 100% certainty which packets were "bad" packets and which were "good" packets, if your uplink is saturated, dropping them on your edge router/firewall/whatever is 100% ineffective.
Your "best strategy" advice is very good, but it is not the "only strategy."
As others have said, you can also have multiple entry points all sharing the same back end. Each of these entry points can be on their own hosting provider. In principle, you can arrange for the front-end/back-end connection at your front-end provider to NOT share a physical wire with the "public" side of your front-end, so if it gets hit hard it crowd out traffic going to/from the back end.
Here's an example:
I run poormeddosvictim.com. I have servers at 3 sites around the country, 1.666.3.4 1.2.666.4, and 1.2.3.666.
For some reason, some mining company on Mars thinks I am evil so they keep DDOSing me.
Hosting provider A is widely connected. I advertise 1.666.3.4 so all but one of A's pipes see direct connections. I use A's remaining pipe to connect back to my back-end. I work with A so the traffic to the back-end never shares a wire or router with incoming traffic. Bang on A's incoming pipes all you want, I'll still be able to talk to my back end unless you crash me entirely.
I have similar arrangements with hosting providers B and C.
I put my back end at hosting provider D and, just for grins, have a backup back end on hosting provider E that syncs up regularly with the back-end on D.
From our experience packet flooding attacks are rare, most are application layer attacks because they're cheaper:
- If your landing page is dynamic chances are a small site can be choked at the database from a few hundred zombies, and it's much harder to detect the zombies from the genuine clients in a safe automated fashion
- If you don't have a lot of CPU at your firewall layer you can't create long enough rule tables to stop the bad traffic as you detect it
- Often you can simplify your rules but just starting by blocking China, Russia, Korea, then smaller countries that are hosting bots.
If they are genuine flood attacks:
The idea that your ISP will block a "list of addresses" is comical, it's not nearly responsive enough, and if you're lucky your ISP will agree to block countries and only if you have a business account which you're paying over the odds access fees for. Some will even null route YOUR IP instead of the attackers to save their own infrastructure: http://www.abc.net.au/4corners/content/2009/s2658405.htm
ANDREW FOWLER: The Russian cyber attack was so sustained it backed up through Telstra's network, knocking out the whole of Alice Springs, part of Adelaide, and Telstra central in Sydney.
DAN CRANE, FORMER TECHNOLOGY MANAGER, MULTIBET: And that's when they sort of started to panic a bit I think because all of a sudden it wasn't just a, you know run of the mill attack, this was a pretty hardcore attack because that's when it started, that's when it took out Alice Springs, that's when it degraded Adelaide and that's when it melted their routers in Sydney so that's when they said that's it, we don't want a bar of it.
ANDREW FOWLER: According to Dan Crane, Telstra stopped accepting any of Multibet's internet traffic from entering Australia.
Not to mention even creating this list is a continual task. Botnets rent out "so many connections", but the computers that are active at any time rotate in and out of the pool. We saw probably around 1000 computers at a time hitting the firewall, but from a pool of more like 100,000 addresses we discovered over the course of a week. We initially took a strategy of programmatically blocking individual IPs as they came in at a response rate of about 5 seconds with some scripting, but soon moved to blocking entire countries that we didn't do business with and doing some daily post processing of the IP list as well to consolidate IPs into /27s and sometimes as far as /24s
Our last client to have this issue used Black Lotus and they seemed to do a good job for the price and be quite responsive, though they were still learning their trade at that time... I don't think they were terribly cheap. And botnets are much much cheaper, so unless you're lucky and it's someone that loses interest and not a competitor attacking you it can end up making your web hosting very expensive.
This is a bit of a naive explanation.
Let me explain how a DDoS mitigation strategy works for many of the companies listed in the summary. They setup datacenters in 10, 15, or more places all hosting a proxy. Some of these solutions use DNS to route traffic around problems (GSLB) while others like CloudFlare use Anycast which is awesome and super hard to get right. Each of these services are typically setup with tons of bandwidth capacity, well over 10Gb, but often times into the 100Gb range. They also often have deals with upstream providers that can filter traffic at the edges meaning it never makes it onto the internet in the first place.
Since you servers are not exposed to the internet, and the ones are are have far, far more horsepower to deal with this than a DDoS will even manage from the client side they can easily just churn through the attack, discarding connections and never letting them hit your limited servers. This is how they can easily survive Anonymous style DDoS attacks.
The other thing is to ensure you have turned of every "feature" your load balancer is giving you. SSL termination at the LB, full session management, etc. All of these cost load balancer CPU which is easy to take advantage of, even if there is a DDoS mitigation system in front of your site. You can't just add a few more servers either. Adding capacity to a load balancer is nearly impossible to do mid-attack.
Even more interesting is that you can often times trick the crappy ddos software by doing things like excessively slow responses (tarpitting) making its loop take ages to try again. This is pretty much using the tactics of a DDoS directly against the attackers.
Another common tactic is to add attackers to a view in your bind config that resolves your hostname to 127.0.0.1 just for them. This works if you do not have long TTLs and they are using hostnames. If they are using direct IPs then you simply move your traffic to a second IP and drop the one they are attacking. Best case is if you can do this via BGP announcements so the traffic simply will fail to route and everybody wins.
And yes, I do this professionally but not for any commercial product.
The essence comes down to two things. Neither is particularly complicated in principle, although getting it right can be a bit fiddly.
1) Detect attacking IPs.
HTTP Flood DDOS bots aren't (at least not yet) smart enough to look and behave EXACTLY like people using web browsers. They do wierd things like load web pages repeatedly while never loading images/running javascript/loading CSS stylesheets. They make sequential requests from the same IP address - but with different user agents. They might load a web page that uses cookies - but never return the cookies that are set. Or they might return a cookie - but from a different source address or with a different user agent. They might send user agents that haven't been in widespread use in half a decade. They might not set the 'referer' header, or some other header that a browser DOES set correctly. They probably don't follow HTTP redirects. What you are looking for is any behavior that distinguishes the good traffic and the bad traffic.
So I 'tailed' the web server log and analyzed it in one to ten minute chunks to detect abnormal accesses. All detected addresses were added to a persistent database of blacklisted addresses.
2) Add the detected attacking addresses to an efficient firewall.
A naive firewall blacklist might try to just put each addresses in one big long list. This doesn't scale well beyond a couple of hundred attacking addresses. On the older machine I had, I used a 'divide and conquer' approach: I created a few hundred filter chains based on a /n subnet division of the attacking ip addresses. I then wrote a set of rules that divided incoming traffic into those chains based on the /n they were a member of. That made the number of rules required to filter n attacking IP addresses scale as about O(log n). If I had had a more recent kernel I could have used a hashed map of addresses to take that down to O(1).
After that it became a slow game of cat and mouse. The attacker would alter his attack to try and slip by the detection, I would update the detection software to detect something else he wasn't getting perfect if he managed to by-pass the filters. After about two weeks they quit attacking the web server.
The largest issue I had really was that I was starting my defense from a 'standing start': I had to write all the needed scripts from scratch while the attack was still on going.