Ask Slashdot: Experience Handling DDoS Attacks On a Mid-Tier Site?

Best defense: Overprovisioning and cutoffs by LostCluster · 2012-04-09 12:02 · Score: 5, Insightful

There's two key strategies to avoid being DDoSed... first, have more processor, network speed, and disk I/O resources than you need for normal load so that the attacker can't fill one of your computers pipes. Then, host your server or servers at multi-connected datacenters which can cut off large users of your server before it reaches your NIC card. Firewalls at the server can't get back the bandwidth lost to needless connections, but firewalls at the datacenter entry points can. Basically, make sure none of your time-sensitive loads reach 100% and you're fine.

Re:Best defense: Overprovisioning and cutoffs by ScentCone · 2012-04-09 12:10 · Score: 2

Basically, make sure none of your time-sensitive loads reach 100% and you're fine
As long as the people running the datacenter aren't metering the noise that hits the interface set up to catch your traffic, whether or not they pass it along to your server. If you're burning up their resources, somebody's got to pay the pipe(r).

--
Don't disappoint your bird dog. Go to the range.
Re:Best defense: Overprovisioning and cutoffs by TWX · 2012-04-09 12:13 · Score: 4, Funny

You forgot a key third strategy- not posting one's dilemma and site to a popular geek news and discussion forum. At least this submitter was smart enough to not post his URL, so he hasn't gotten slashdotted...

--
Do not look into laser with remaining eye.
Re:Best defense: Overprovisioning and cutoffs by Anonymous Coward · 2012-04-09 12:13 · Score: 1

Adding a reverse cache (such as https://www.varnish-cache.org/ ) in front of your smart web servers for static and mostly static content really helps until they overload your pipe or start hitting dynamic URL's.
Re:Best defense: Overprovisioning and cutoffs by shentino · 2012-04-09 14:10 · Score: 1

So basically, surviving a DDoS is nothing more than a brute force battle of attrition where the goal is to have more resources than the other guy can take down.
Re:Best defense: Overprovisioning and cutoffs by Minupla · 2012-04-09 15:16 · Score: 3, Interesting

Typically, yes (assuming your OS platform of choice doesn't have some other resource that can be remotely exhausted more cheaply then bandwidth). The problem is one of the standard defender delimas: The attacker needs bandwidth for a short period of time (typically), as their goal is to make you say "Uncle" weather that means paying their ransom, capitulating to some demand or whatever. You as a defender have to incur a cost for your defensive strategy that is either (relatively) low, non-scalable, and continuing (trying to out provision the attacker) or a high cost outsourcing solution. The attacker on the other hand rents 10,000 nodes for 200$/day. Figure that's about 5gigs conservatively (we'll say .5mbit upload as an average per node). Now assuming your data center will handle a sudden 5gig burst without cutting you off (good ones will, cheap ones will just cut you off) your hosting bill just went up by 54TB (5*3600*24/8) per day. That's not going to be sustainable for long.
That's why the outsourcing solution tends to be the way to go if you're being targeted by anyone willing to spend halfway decent money on attacking you. The ROI from the attacker POV looks pretty good. Say they ransom you for 50K (an average number for such things). If they have to keep you under DDOS for even a week till you cave, (378 TB worth of data) that nets them 48600. That's a pretty good business case from their point of view.
It's one of those moments when it sucks to be the good guys.
Min

--
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before
Re:Best defense: Overprovisioning and cutoffs by FoolishOwl · 2012-04-09 19:02 · Score: 1

This is the first time I've heard of demands for ransom. I suppose it shouldn't be a surprise that's the motive, and come to think of it, it shouldn't be a surprise that corporations aren't eager to admit to having been extorted.
Re:Best defense: Overprovisioning and cutoffs by dropadrop · 2012-04-09 19:14 · Score: 1

Basically, make sure none of your time-sensitive loads reach 100% and you're fine.
We did that, and then attackers started to flood us with over 10gbit/s of traffic that just flooded the network link and basically forces us to null route the service.
Re:Best defense: Overprovisioning and cutoffs by outsider007 · 2012-04-09 22:35 · Score: 1

I doubt that it happens like that. Where does the 50k go? Swiss bank accounts? That's seems like spy movie BS to me. You can't receive a 50k ransom without getting nailed for it. Not even a nigerian prince.

--
If you mod me down the terrorists will have won
Re:Best defense: Overprovisioning and cutoffs by Minupla · 2012-04-09 23:17 · Score: 3, Interesting

In the case I was involved with it was wired via Western Union to a place in Moscow where (according to the PI we hired) it was picked up by call girls and taken back to the culprits. They did eventually get nailed but it took years due to the complexities of law enforcement in an international environment.
We eventually signed with Prolexic to stop them coming back.
Min

--
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before
Re:Best defense: Overprovisioning and cutoffs by Minupla · 2012-04-09 23:19 · Score: 2

[citation]http://www.cbc.ca/news/technology/story/2006/10/28/online-gambling.html[/citation]

--
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before
Re:Best defense: Overprovisioning and cutoffs by outsider007 · 2012-04-10 00:01 · Score: 1

Ok, so these organized russian criminals are investing all this money into committing crimes that they ultimately don't get away with? I'm sorry I just don't buy it. I'm calling bs.

--
If you mod me down the terrorists will have won
Re:Best defense: Overprovisioning and cutoffs by Minupla · 2012-04-10 09:51 · Score: 1

You are welcome to, but if the story I quoted before doesn't convince you feel free to search google news. I would suggest "Gambling site" "Random" and "DoS" as search criteria.
Min

--
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before

Load balancing and an experienced sysadmin by FireballX301 · 2012-04-09 12:10 · Score: 4, Insightful

The load balancer to take the brunt of the attack and distribute traffic to multiple mirrors, and the sysadmin to watch the attack and start blacklisting IP ranges. Your service provider should have some kind of service in place unless you got the cheapest of cheap hosting solutions.

With that being said, hiring a third party ddos mitigator is entirely a cost benefit analysis that should be done on your end. Can whoever's providing your hosting now provision some extra servers and some harried sysadmins to keep you floating? See if you can ask for additional service support from your current provider.

Re:Load balancing and an experienced sysadmin by SethJohnson · 2012-04-09 12:23 · Score: 5, Informative

watch the attack and start blacklisting IP ranges.
In most cases, your customers are going to exist in one or a few countries. It would be valuable ahead of time to add redirect rules to your iptables for entire ranges of IP addresses located in countries that don't host your customers. Redirect these IP ranges to a sacrificial server on a different pipe to the backbone. That way, when some of your customers are abroad and need access to your services, they can still get some amount of response.

Additionally, you can proactively parse your user accounts for IP addresses and build a whitelist ruleset for your iptables to implement in a defcon 0 situation. Don't use this as a normal operations mode, just when the shit has really hit the fan and you need to block everyone except your known-good account holders.

Seth

--
$5 / month hosted VPS on linux = awesome!
Re:Load balancing and an experienced sysadmin by Snowhare · 2012-04-09 12:40 · Score: 5, Insightful

Having been the target of an HTTP-DDOS attack, I can tell you that manually blacklisting IP ranges is really ineffective. A DDOS botnet is comprised of thousands of machines that have been randomly infected by whatever vector the botnet operator used: Emails, web drive-by, etc. The result is that the source addresses are scattered widely with little relation between most participating addresses.
To defend against the attack, I wrote up an automatic firewall blacklisting program that detected and blocked each participating IP address individually in near-realtime. I was blocking more than 31,000 separate addresses before the DDOSers finally gave up trying to down the attacked website. Wierdly, there appears to have been no motive at all for the attack, yet they spent weeks attacking the target machine and actively trying to tune their attack to get past my filtering.
Re:Load balancing and an experienced sysadmin by Anonymous Coward · 2012-04-09 13:15 · Score: 1

Could you post a link to your program's source code? Or, if you don't want to, could you give us some pseudocode?
Re:Load balancing and an experienced sysadmin by Taco+Cowboy · 2012-04-09 13:40 · Score: 1

Could you post a link to your program's source code? Or, if you don't want to, could you give us some pseudocode?
Yeah, I'd second that

--
Muchas Gracias, Señor Edward Snowden !
Re:Load balancing and an experienced sysadmin by Zaiff+Urgulbunger · 2012-04-09 13:55 · Score: 2, Funny

And a link to your web-server too, so we can test it! :D
Re:Load balancing and an experienced sysadmin by Snowhare · 2012-04-09 14:42 · Score: 5, Interesting

The essence comes down to two things. Neither is particularly complicated in principle, although getting it right can be a bit fiddly.
1) Detect attacking IPs.
HTTP Flood DDOS bots aren't (at least not yet) smart enough to look and behave EXACTLY like people using web browsers. They do wierd things like load web pages repeatedly while never loading images/running javascript/loading CSS stylesheets. They make sequential requests from the same IP address - but with different user agents. They might load a web page that uses cookies - but never return the cookies that are set. Or they might return a cookie - but from a different source address or with a different user agent. They might send user agents that haven't been in widespread use in half a decade. They might not set the 'referer' header, or some other header that a browser DOES set correctly. They probably don't follow HTTP redirects. What you are looking for is any behavior that distinguishes the good traffic and the bad traffic.
So I 'tailed' the web server log and analyzed it in one to ten minute chunks to detect abnormal accesses. All detected addresses were added to a persistent database of blacklisted addresses.
2) Add the detected attacking addresses to an efficient firewall.
A naive firewall blacklist might try to just put each addresses in one big long list. This doesn't scale well beyond a couple of hundred attacking addresses. On the older machine I had, I used a 'divide and conquer' approach: I created a few hundred filter chains based on a /n subnet division of the attacking ip addresses. I then wrote a set of rules that divided incoming traffic into those chains based on the /n they were a member of. That made the number of rules required to filter n attacking IP addresses scale as about O(log n). If I had had a more recent kernel I could have used a hashed map of addresses to take that down to O(1).
After that it became a slow game of cat and mouse. The attacker would alter his attack to try and slip by the detection, I would update the detection software to detect something else he wasn't getting perfect if he managed to by-pass the filters. After about two weeks they quit attacking the web server.
The largest issue I had really was that I was starting my defense from a 'standing start': I had to write all the needed scripts from scratch while the attack was still on going.
Re:Load balancing and an experienced sysadmin by Nesa2 · 2012-04-09 15:17 · Score: 1

I've used WatchGuard appliance and it did just that under DDOS attack. It also has a timer you can set for how long the IPs that are found performing an attack will be blocked for... worked really well with minimal setup effort, but it can be costly appliance depending on company size.

I've used some other appliances as well that did not have this feature, and it has caused issues under DDOS. You could also setup scripts with iptables under any UNIX flavor you pick to do just that as well.
Re:Load balancing and an experienced sysadmin by Anonymous Coward · 2012-04-09 15:28 · Score: 1

Damn, I'd pay good money to watch this movie!
Re:Load balancing and an experienced sysadmin by jaymemaurice · 2012-04-09 15:58 · Score: 1

I've done something similar to this with my router - it would BGP null route the zombies by distributing the /32 route with a community tag and the upstream router would use policy routing to null route the routes in that community... unfortunately, trying to find an upstream ISP that supports such null routing is now hard to find.

--
120 characters ought to be enough for anyone
Re:Load balancing and an experienced sysadmin by TooMuchToDo · 2012-04-09 16:11 · Score: 1

Redirect these IP ranges to a sacrificial server on a different pipe to the backbone.

More appropriately, have your upstream provider(s) null route these address blocks before they get to your uplink.
Re:Load balancing and an experienced sysadmin by Alex+Belits · 2012-04-09 18:03 · Score: 1

Why do you think that it worked? Load decreased?
The goal of DoS is to make your server unusable to the clients. More likely than not, your "appliance" blocked most of legitimate requests, thus fulfilling the purpose of the attack.

--
Contrary to the popular belief, there indeed is no God.
Re:Load balancing and an experienced sysadmin by Intropy · 2012-04-09 19:26 · Score: 1

Load balancing is a given. Even to handle normal traffic you have to distribute it to your front ends somehow. But a sysadmin blacklisting IP address ranges? Oh my. You do not want this task to be done manually. A human, or a team of them, isn't going to be able to keep up with with a distributed attack, and they're going to make mistakes when determining which addresses to block. Remember that in a distributed attack they're not all coming from the same address and there's not much reason to expect them to come in some sufficiently narrow band for the idea of ranges to be useful.
Re:Load balancing and an experienced sysadmin by Intropy · 2012-04-09 19:45 · Score: 1

Instead of trying to "detect" a DDOS, you can simply replace 1) with rate limiting. Say you're not willing to let any single IP address hit you more than 10 times a second, for example. You can even bucket rates. Say 10 is an attack.
For 2) instead of putting everything in a firewall you can have whatever is your closest-to-the-client server throw away a request, send an empty reply, send a captcha page or whatever else you think is appropriate given the rate being exceeded.
I also think you left out an important part 3). You need some way for your various entry points to talk to one another about the traffic they're seeing. Otherwise you're relying on the individual (in the rate limit method) or the DDOS network (in the detection method) to come in through the same path or you're risking them separately filling up each entrance before determining to block.
Re:Load balancing and an experienced sysadmin by Ginger+Unicorn · 2012-04-09 22:50 · Score: 1

that sounds like the flavour text from a NetRunner card

--
(1.21 gigawatts) / (88 miles per hour) = 30 757 874 newtons
Re:Load balancing and an experienced sysadmin by Bert64 · 2012-04-10 00:03 · Score: 1

Get multiple transit links...
Advertise your range over the links to the locations where the vast majority of your customers come from, and then advertise everything else over a very slow backup link.
Most sites cater to local customers, so have 10gb local peering and maybe 10mb international transit... Chances are most of the ddos drones won't be local, and so the attack will quickly destroy the 10mb link while your customers are happily using the 10gb peering.
If there are ddos nodes local to you, the isps hosting them are more likely to be responsive and shut them down than trying to contact a hosting company in a different timezone speaking a different language etc.

--
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
Re:Load balancing and an experienced sysadmin by Nesa2 · 2012-04-10 00:07 · Score: 1

No, it just blocked IPs from DoS traffic because it was easy to identify... customers ware unaffected. If you can block anything that sends you more than 200 requests within a minute you are off to a good start.

It depends on how many nodes DoS is coming from though, as if it's coming from 1,000,000 nodes, it would only need to make let's say 5 requests per second and it would saturate network, and no firewall would effectively stop it. However, if DoS is going from 1000 nodes with 1000 requests per second trying to saturate your network - those types of DoS will easily be dropped by almost any properly configured firewall.

Essentially, I agree that if DoS was as large as what brought Sony to a crawl, then only good planning ahead could negate it. It involves a lot more than a firewall. But if you are running medium sized business, and it's some script kiddies that are coming together to try and bring your site down through DoS, it can easily be negated with a proper firewall.
Re:Load balancing and an experienced sysadmin by Bert64 · 2012-04-10 00:12 · Score: 2

This assumes they are just trying to flood the httpd with requests, because doing so requires less resources on their part, and generally only harms the target box and not the isp hosting it.
If you block an attack like this, you run the risk that the attacker will switch tactics and start simply flooding your line. If there were 31,000 attacking drones, and assuming a rather conservative 512kbit/sec upstream per box thats over 10gbit/sec of traffic hitting you...
Also when sending raw packets the source addresses can be spoofed because you dont care about receiving a reply (and conversely, a spoofed reply wastes the victims bandwidth without impacting the attackers, and may cause the victim machine to send unsolicited traffic to someone else), making it harder to trace the source of the attacks.
And ofcourse if you perform the filtering on the server itself, it has already traversed your line and is likely costing you... You really need to filter further upstream in order to be of much use.

--
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
Re:Load balancing and an experienced sysadmin by marcosdumay · 2012-04-10 06:35 · Score: 1

Instead of trying to "detect" a DDOS, you can simply replace 1) with rate limiting.
That's a great thing for doing in the kernel. Thus, I was out searching if it was alreay done, and yes, it is.

--
Rethinking email
Re:Load balancing and an experienced sysadmin by Snowhare · 2012-04-10 13:57 · Score: 1

Rate limiting IP addresses doesn't work when they are only hitting from any specific source IP address a few dozen times per hour. They bury you by having tens of thousands of different machines all hitting you independantly. You can be getting hundreds of requests per second and never trigger the rate limitting.
Re:Load balancing and an experienced sysadmin by Snowhare · 2012-04-10 14:21 · Score: 1

This assumes they are just trying to flood the httpd with requests, because doing so requires less resources on their part, and generally only harms the target box and not the isp hosting it.
If you block an attack like this, you run the risk that the attacker will switch tactics and start simply flooding your line.
True, they *could* have escalated it to a packet flood (and oddly enough naively dropping the TCP packets actually initially converted the HTTP Flood into a SYN flood - which didn't pose much of a problem for me at the rates they were running).
But it is much more resource intensive for the attacker and they are optimising return on investment. They can waste time dedicating their botnet to packet flooding a minor site with no financial payoff even if they succeed in bringing it down, or they can move on to easier targets where they can continue to 'time share' the botnet traffic among multiple targets.
It really is the 'why have locks on your doors and windows when the thief could kick them in' argument. Sure - he *could*. Or he could move down the street to the house that left their bathroom window open when they went to work.
Re:Load balancing and an experienced sysadmin by Intropy · 2012-04-10 17:23 · Score: 1

If you're only being hit a few dozen times per hour per machine then you're not talking about a very scary DDOS attack anyway. As you noted, that's only on the order or hundreds of requests per second even with tens of thousands of machines all attacking at once. It seems like a bad tradeoff to do intensive detection techniques like you were discussing on every single query to avoid serving traffic from a small DDOS attack should one arise.
Re:Load balancing and an experienced sysadmin by Snowhare · 2012-04-11 14:29 · Score: 1

A "small" DDOS attack is more than enough to down an unprotected machine. I experimented with less intensive approaches *first*. If I limitted the number of Apache connections they would run up the number of open connections until the server quit responding. If I let the number processes grow, they would keep adding connections until the machine ran out of memory to support additional connections. With a pool of more than 30K potentially attacking machines it takes an *incredible* amount of resources to just 'ride it out'.
You run into multiple limits: How many simultaneous TCP connections can your system handle? How much memory does it take per connection? How much CPU does it take to context switch between thousands of connections?
It was a simple yet very effective attack. If you didn't have a good sysadmin who *could* erect an intensive defense your choices would be
1) Let your site go down.
2) Pay a DDOS defense service to defend you.
Re:Load balancing and an experienced sysadmin by Intropy · 2012-04-11 18:19 · Score: 1

Fair enough. A small DDOS attack will take down a single web server. The OP was asking about mid size sites*, so I assumed you were as well. I think a small site probably is better off paying one of those services than spending its limited resources on the problem.
* Definitions of mid size will differ of course too. I had in mind something like netflix, photobucket, or yelp, who would definitely be running on a good number of machines and possibly at multiple physical locations (not that I know anything about how any of those randomly chosen examples actually operates)

Re:Change Apache to nginx by dreamchaser · 2012-04-09 12:13 · Score: 2, Insightful

If it helps against DDOS attacks, how is it stupid advice?

Because it doesn't really and you're just being a fanboi?

Lived Through This by ScentCone · 2012-04-09 12:15 · Score: 4, Interesting

It was a lot cheaper to pay a third party proxy a $400/month rate for 45 days (until the asshats attempting the extortion got bored and went away) than it wold have been to provision more server horsepower, pay for the bandwidth, and pay T&M for the DC's NOC to help with firewalling. A quick DNS change, use the credit card, hold your breath until it stops. Quick, cheap, and you can go on to other things.

--
Don't disappoint your bird dog. Go to the range.

Re:Lived Through This by Spikeles · 2012-04-09 12:28 · Score: 3, Interesting

Unless the third party proxy were the extortionists DDOSing you.

--
I don't need to test my programs.. I have an error correcting modem.
Re:Lived Through This by LordLucless · 2012-04-09 12:37 · Score: 1

What was the name of the service you used?

--
Just because you're paranoid doesn't mean there isn't an invisible demon about to eat your face
Re:Lived Through This by Taco+Cowboy · 2012-04-09 13:23 · Score: 1

It was a lot cheaper to pay a third party proxy ...

How do you determine if the third party proxy has sufficient bandwidth to handle the DDoS + regular traffic?

--
Muchas Gracias, Señor Edward Snowden !
Re:Lived Through This by million_monkeys · 2012-04-09 13:28 · Score: 2

Unless the third party proxy were the extortionists DDOSing you.
Good business model!
Re:Lived Through This by ScentCone · 2012-04-09 13:57 · Score: 3, Informative

How do you determine if the third party proxy has sufficient bandwidth to handle the DDoS + regular traffic?
They have a performance guarantee, and don't get paid if they can't keep up at the promised level. Any of the ones you'll want to use will have a dashboard that shows you a more-or-less-real-time view of the blocks/passes, and how much of the purchased throughput you're using.

--
Don't disappoint your bird dog. Go to the range.
Re:Lived Through This by ScentCone · 2012-04-09 14:00 · Score: 4, Informative

For that event, we used Zen Networks. They're at www.zenprotection.net, which describes their services pretty well. Not affiliated in any way, but they did solve the problem for us over the short stretch it was required. Honestly, we didn't shop around much ... the site in question was very much on fire. Not like a slashdotting, of course, but some fairly determined Eastern European punks looking for cash. They made my clients angry enough to have them asking, "Is there something we can do back to these guys?" We didn't, of course. Would have been a waste of time.

--
Don't disappoint your bird dog. Go to the range.
Re:Lived Through This by LordLucless · 2012-04-09 14:21 · Score: 1

Yeah, been there. Luckily enough, the guys that tried it with us were incompetent, and we could filter their attack because of it, but always on the lookout in case the next ones that try it are a bit smarter.

--
Just because you're paranoid doesn't mean there isn't an invisible demon about to eat your face
Re:Lived Through This by Intropy · 2012-04-09 19:54 · Score: 1

Definitely. For a mid size site, $400 a month is practically nothing. If you can make your DOS problems go away for that, do it.
Re:Lived Through This by Bert64 · 2012-04-10 00:48 · Score: 1

Not very competent asshats if they didnt notice the proxy setup, and simply continue hitting the original ip ranges...

--
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
Re:Lived Through This by ScentCone · 2012-04-10 03:16 · Score: 1

We changed the real server's IP address (on a different block) and pointed the proxy there, after pointing DNS at the proxy. The original IP address stopped listening for anything. They did keep trying to hit it until they realized nothing was there.

--
Don't disappoint your bird dog. Go to the range.

Don't post your link on Slashdot? by Anonymous Coward · 2012-04-09 12:16 · Score: 1

Yay. Someone finally got that right.

Gambling by Teppy · 2012-04-09 12:18 · Score: 1

I'm interested in this. I'm in the process of launching a real-money gambling MMORPG, and gambling sites (well, so I've been told) tend to get extorted. I spoke to Prolexic today and was shocked at how expensive they are: $3000/month minimum, plus setup fees.

Have any fellow Slashdot readers tried running a gambling site without such protection? Is it reasonable to assume we'll be enough under-the-radar at first to avoid the attacks?

Re:Gambling by Minupla · 2012-04-09 13:44 · Score: 3, Interesting

I used to run infosec for one of the mid-tier online gaming operations run out of the Caribbean. We got extorted by one of these gangs, and ended up paying Prolexic (they were Digidefense at the time) to solve this for us.
As for weather you can risk doing without it depends strongly on what your user tolerance for downtime is and how bursty your revenue stream is. The lower the tolerance and/or the more bursty the revenue stream the more vulnerable you are to these sort of attack methodology, as the opposition pays for the time they are actually attacking you, so if you can weather the attack they'll eventually give it up. If on the other hand they can cost you significant sums of cash by taking you out for 6 hrs (say sports betting, target the opening day games), that increases your susceptibility to these attacks.
Feel free to drop me a line if you have any more questions (my /. listed email will get to me).
Min

--
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before

Post-mortem: Admin investigates attack by microTodd · 2012-04-09 12:19 · Score: 5, Informative

Remember this really cool slashdot story about a sysadmin on the receiving end of a DDOS?

http://slashdot.org/story/01/05/31/1330202/post-mortem-of-a-dos-attack

The original writeup link is dead but I found it here (warning: PDF). This was a really cool story.

http://www.stanford.edu/class/msande91si/www-spr04/readings/week1/grcdos.pdf

--
"You cannot find out which view is the right one by science in the ordinary sense." - C.S. Lewis on Intelligent Design

Seen it by Anonymous Coward · 2012-04-09 12:20 · Score: 1

Manage your own IP ranges and disable any IPs that are not in use via your IP registry provider. You can run an IDS/IPS that can alert you to an attack in progress.

Have good upstream providers, preferrably ones that use the new DDOS prevention/detection appliances, and ones that can respond to block routing to attacked IPs temporarily if necessary.

What Are The Odds by sycodon · 2012-04-09 12:24 · Score: 4, Insightful

That all these "services" are part of a protection racket?

"Oh...having DDOS problems? Just sign up with our service and we can help you out."

While not as crude as burning down building, DDOS attacks are a perfect persuader to grow your business.

I figure this is half tin foil hat and half probably real, given the things organized crime has been into in the past. It's perfect actually, you don't have to hurt people, the attacks can't be traced and your "protection" can be fine tuned to avoid looking suspicious.

--
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.

Misunderstanding by lanner · 2012-04-09 12:26 · Score: 4, Informative

The mere question of how to mitigate a DDOS indicates a fundamental lack of understanding of how IP networking and DDOS works.

You (the ISP customer) have no ability to control what packets are sent to you over your uplink circuits. You can control what you send, but you have no ability to control what you receive.

Read the sentence above. Repeat as necessary.

Even if you knew with 100% certainty which packets were "bad" packets and which were "good" packets, if your uplink is saturated, dropping them on your edge router/firewall/whatever is 100% ineffective.

The best mitigating strategy is that you need to have an agreement with your ISP and plan in place prior to an attack. Identify the hostile addresses, give them to your ISP, and they will null-route those sources either within their core or even at the edges of their networks to prevent entry. Your ISP has the capacity to mitigate a DDOS attack, you as the little customer do not.

Re:Misunderstanding by Lando · 2012-04-09 13:07 · Score: 1

Depends on the type of DDOS you are receiving. If it is a distributed DDOS then you can't do much other than having multiple pipes into the system; however, if there are not too many systems launching the ddos then you can have your provider drop those packets before they are sent down the pipe, ie you can determine what gets sent to you.
For a distributed attack you can host a couple of gateways on a larger pipe that talk to a local backend. When I was handling mission critical stuff back in the day that is how I handled things. Put a proxy server at a couple of OC3 hosting companies and have those machines talk to my backend so that a ddos can't saturate my pipe. Static pages where appropriate, etc. Used to be that Microsoft's hosting was pretty bad at handling dropped connections so for clients that used them I'd throw up an apache proxy server in front to handle dropped connections, load balancing, etc.
I'm sure I'd have to evolve a bit from the ways I used to handle these problems since there a larger botnets and such nowdays, but it really depends on the attack. There are multiple ways to control what packets do reach your system though, ie not all ddos attacks are the same. Some are attacking the IP stack, some attempt to saturate the line and some try to exploit holes in the software you use.
For most sites, unless it's mission critical I wouldn't worry about it until/unless it becomes a problem.

--
/* TODO: Spawn child process, interest child in technology, have child write a new sig */
Re:Misunderstanding by Zaelath · 2012-04-09 13:10 · Score: 5, Interesting

From our experience packet flooding attacks are rare, most are application layer attacks because they're cheaper:
- If your landing page is dynamic chances are a small site can be choked at the database from a few hundred zombies, and it's much harder to detect the zombies from the genuine clients in a safe automated fashion
- If you don't have a lot of CPU at your firewall layer you can't create long enough rule tables to stop the bad traffic as you detect it
- Often you can simplify your rules but just starting by blocking China, Russia, Korea, then smaller countries that are hosting bots.
If they are genuine flood attacks:
The idea that your ISP will block a "list of addresses" is comical, it's not nearly responsive enough, and if you're lucky your ISP will agree to block countries and only if you have a business account which you're paying over the odds access fees for. Some will even null route YOUR IP instead of the attackers to save their own infrastructure: http://www.abc.net.au/4corners/content/2009/s2658405.htm

ANDREW FOWLER: The Russian cyber attack was so sustained it backed up through Telstra's network, knocking out the whole of Alice Springs, part of Adelaide, and Telstra central in Sydney.
DAN CRANE, FORMER TECHNOLOGY MANAGER, MULTIBET: And that's when they sort of started to panic a bit I think because all of a sudden it wasn't just a, you know run of the mill attack, this was a pretty hardcore attack because that's when it started, that's when it took out Alice Springs, that's when it degraded Adelaide and that's when it melted their routers in Sydney so that's when they said that's it, we don't want a bar of it.
ANDREW FOWLER: According to Dan Crane, Telstra stopped accepting any of Multibet's internet traffic from entering Australia.

Not to mention even creating this list is a continual task. Botnets rent out "so many connections", but the computers that are active at any time rotate in and out of the pool. We saw probably around 1000 computers at a time hitting the firewall, but from a pool of more like 100,000 addresses we discovered over the course of a week. We initially took a strategy of programmatically blocking individual IPs as they came in at a response rate of about 5 seconds with some scripting, but soon moved to blocking entire countries that we didn't do business with and doing some daily post processing of the IP list as well to consolidate IPs into /27s and sometimes as far as /24s
Our last client to have this issue used Black Lotus and they seemed to do a good job for the price and be quite responsive, though they were still learning their trade at that time... I don't think they were terribly cheap. And botnets are much much cheaper, so unless you're lucky and it's someone that loses interest and not a competitor attacking you it can end up making your web hosting very expensive.
Re:Misunderstanding by mysidia · 2012-04-09 13:18 · Score: 1

You (the ISP customer) have no ability to control what packets are sent to you over your uplink circuits. You can control what you send, but you have no ability to control what you receive.
Yes you do have lots of potential control. It's just expensive to exercise. Because you have to have multiple datacenters and multiple uplink circuits, to have any control (without intentionally breaking connectivity).
You buy transit and peer with major providers in different datacenters and anycast your server IP address space.
Apply appropriate BGP communities and prepending to manipulate which datacenter traffic from different places around the world winds up at.
In this manner you can 'diffuse' DDoS attacks by having a distributed infrastructure.
DDoS is highly effective against one target with a single presence. As soon as you have multiple points of presence though, you can divide and conquer
Each DoS attack node is only capable of sending traffic to one of your anycast nodes -- the selection depends on where the DoS attack node is located net-wise, e.g. which provider. Something you can actually manipulate, once the infrastructure is in place.
Also, if the DDoS attack nodes are concentrated on a particular network, it is likely they will all impact one target, leaving the vast majority of your presence unaffected.
Re:Misunderstanding by Anonymous Coward · 2012-04-09 13:23 · Score: 3, Funny

...a distributed DDOS...
Would that be a DDDOS? And what would you do if it were a distributed DDDOS? I bet you'd be totally hosed then.
Re:Misunderstanding by Liquid-Gecka · 2012-04-09 13:50 · Score: 5, Informative

This is a bit of a naive explanation.
Let me explain how a DDoS mitigation strategy works for many of the companies listed in the summary. They setup datacenters in 10, 15, or more places all hosting a proxy. Some of these solutions use DNS to route traffic around problems (GSLB) while others like CloudFlare use Anycast which is awesome and super hard to get right. Each of these services are typically setup with tons of bandwidth capacity, well over 10Gb, but often times into the 100Gb range. They also often have deals with upstream providers that can filter traffic at the edges meaning it never makes it onto the internet in the first place.
Since you servers are not exposed to the internet, and the ones are are have far, far more horsepower to deal with this than a DDoS will even manage from the client side they can easily just churn through the attack, discarding connections and never letting them hit your limited servers. This is how they can easily survive Anonymous style DDoS attacks.
The other thing is to ensure you have turned of every "feature" your load balancer is giving you. SSL termination at the LB, full session management, etc. All of these cost load balancer CPU which is easy to take advantage of, even if there is a DDoS mitigation system in front of your site. You can't just add a few more servers either. Adding capacity to a load balancer is nearly impossible to do mid-attack.
Even more interesting is that you can often times trick the crappy ddos software by doing things like excessively slow responses (tarpitting) making its loop take ages to try again. This is pretty much using the tactics of a DDoS directly against the attackers.
Another common tactic is to add attackers to a view in your bind config that resolves your hostname to 127.0.0.1 just for them. This works if you do not have long TTLs and they are using hostnames. If they are using direct IPs then you simply move your traffic to a second IP and drop the one they are attacking. Best case is if you can do this via BGP announcements so the traffic simply will fail to route and everybody wins.
And yes, I do this professionally but not for any commercial product.
Re:Misunderstanding by Lando · 2012-04-09 16:05 · Score: 2

Heh,
Good point. I guess my mind automatically equated DDOS with a DOS. I've called it DDOS so long that I didn't even think about what DDOS stood for. Thanks for the reference check.

--
/* TODO: Spawn child process, interest child in technology, have child write a new sig */
Re:Misunderstanding by hhw · 2012-04-10 03:57 · Score: 2

Null routes work on destination IPs only, not source IPs, so if one were to take your advice, they would effectively be doing the attackers' job for them. Null routes are for preventing a DoS attack from affecting other customers/services, not for keeping a server up and available during an attack. The only way to block source IPs on routers is with ACLs, but these are only effective against very simple DOS where a small number of sources are attacking you. For a large scale DDoS, with sources from several different ranges attacking protocol/ports on IP addresses where legitimate traffic needs to go (which any smart attacker would do), ACLs aren't going to do it.

--
http://astutehosting.com/
Re:Misunderstanding by hhw · 2012-04-10 04:00 · Score: 1

You can have a distributed DDoS protection service, where you BGP anycast the IP range being attacked to multiple cities, and deploy DDoS mitigation equipment to each of those locations. In the event that the attack is concentrated in one region, where your DDoS mitigation equipment is unable to keep up, you can then do a localized null route (that doesn't get advertised to the rest of your iBGP mesh), to drop the traffic for that particular region but still stay up for the rest of the world.

--
http://astutehosting.com/
Re:Misunderstanding by Lando · 2012-04-11 02:42 · Score: 1

Regional, Hmmm guess I don't look at things on a worldwide basis. I should look up more information, just for educational purposes. Do you have a link where I could look this stuff up more throughly?
Doesn't the first D stand for distributed so that distributed DDoS is redundant?

--
/* TODO: Spawn child process, interest child in technology, have child write a new sig */

Re:Change Apache to nginx by rev0lt · 2012-04-09 12:27 · Score: 3, Informative

So, are you saying nginx will work when you receive more TCP requests than your server can handle? Or the upstream router? Or when evey page render is a database hit? Nginx is a lot faster than apache when serving static content, and at the cost of some flexibility. Guess what? Most of the web isn't static content, even if it appears so. Do you think sessions and agent info are logged into ether? Get real.
And yes, I DO use nginx, and it rocks. It's just not the silver bullet you're talking about.

Re:Change Apache to nginx by bleedingsamurai · 2012-04-09 12:29 · Score: 1

You do realize that you can configure Apache to defend against DoS attacks, for example making the timeout threshold a shorter period of time.

If you are worried about bloat, why not go the whole nine yards and build your server from the ground up using Gentoo Linux or FreeBSD, compile everything from source and optimize binaries for the server's micro-architecture.

Re:Wow deja vu by rev0lt · 2012-04-09 12:35 · Score: 1

I'd suggest stop using a shared host, and looking for a real provider. Choose a dedicated server or colo solution (or AWS if you need the redundancy and HA), hire a good reputable company to set it up and mantain it for you. 3Gbps traffic is child's play even for small companies, and even if your dedicated server/colo is hit by this kind of stuff (3Gbps), I doubt it will be prejudicial to other clients.

Re:Change Apache to nginx by errandum · 2012-04-09 12:38 · Score: 1

Ignore this, since it wouldn't help all that much. I'd say that https://www.varnish-cache.org/ can help, but to be honest, if you want to be up, just stick them on Amazon Cloud services or something. They'll have a really hard time getting to that, and you'll leave all the defending and whatnot to be someone elses' job

Ah, but you CAN do something by Anonymous Coward · 2012-04-09 12:49 · Score: 5, Informative

Even if you knew with 100% certainty which packets were "bad" packets and which were "good" packets, if your uplink is saturated, dropping them on your edge router/firewall/whatever is 100% ineffective.

Your "best strategy" advice is very good, but it is not the "only strategy."

As others have said, you can also have multiple entry points all sharing the same back end. Each of these entry points can be on their own hosting provider. In principle, you can arrange for the front-end/back-end connection at your front-end provider to NOT share a physical wire with the "public" side of your front-end, so if it gets hit hard it crowd out traffic going to/from the back end.

Here's an example:

I run poormeddosvictim.com. I have servers at 3 sites around the country, 1.666.3.4 1.2.666.4, and 1.2.3.666.

For some reason, some mining company on Mars thinks I am evil so they keep DDOSing me.

Hosting provider A is widely connected. I advertise 1.666.3.4 so all but one of A's pipes see direct connections. I use A's remaining pipe to connect back to my back-end. I work with A so the traffic to the back-end never shares a wire or router with incoming traffic. Bang on A's incoming pipes all you want, I'll still be able to talk to my back end unless you crash me entirely.

I have similar arrangements with hosting providers B and C.

I put my back end at hosting provider D and, just for grins, have a backup back end on hosting provider E that syncs up regularly with the back-end on D.

Re:Change Apache to nginx by Shoten · 2012-04-09 12:59 · Score: 4, Informative

It doesn't help against DDoS attacks. Not even remotely, not even a little bit. To put the advice to a metaphor, a DDoS attack is where there are so many people loitering in the front lobby of a business that people can't even get into the front door of a building. Using a different web server is like having a receptionist who speaks faster; it doesn't address the nature of the attack in the slightest way possible. These attacks are either driven by saturation of network links or by leveraging vulnerabilities in underlying database-driven applications (hint: a little-known SQL command called WAITFOR is often to blame); using nginx won't help in the slightest bit.

Christ...these attacks are over a decade old; read up or be quiet.

--

For your security, this post has been encrypted with ROT-13, twice.

Re:Change Apache to nginx by postbigbang · 2012-04-09 13:05 · Score: 2

Apache can be configured to drop sessions more quickly to get past the trauma, but the hosts need more configuration. Shortening TCP session duration is key, but so also is going to something else than BIND, which is also less survivable than other DNS servers, IMHO. There are some reasonable TOE card, router, and layer 2/3 switch configs that can also help cut down the pain.

Load balancing helps, watching syslogs for weird behavior and using a syslog manager to alert you when various events occur, all these do as much good as the expense of fortress-ware.

--
---- Teach Peace. It's Cheaper Than War.

Re:DDoS != DoS by davidwr · 2012-04-09 13:08 · Score: 2

You are partly correct but you are oversimplifying things.

You assume that in a DDOS attack, your upstream capacity is 1) over-saturated and 2) the only thing that is over-saturated, or at least that nothing else would be saturated if your upstream capacity were bigger.

If your DDOS attack is not saturating your upstream, either a) you are successfully fending it off, or b) you are still suffering but increasing your upstream capacity is not the answer.

In the case of b), the suggestions you call irrelevant are worth looking at.

Remember, not all DDOS attacks are massively distributed. Sometimes it may be only 100 or 200 machines hitting you in a given moment. Sometimes it's 100 or 200 thousand machines or more trying to bang on your door. Sometimes it's in between.

--
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.

Re:good defesne by Robadob · 2012-04-09 13:09 · Score: 1

Recently a friends web-dev company was hit with a ddos over the course of a weekend to multiple sites across their hosting. Turns out that the ddos attack was actually testing for a point of php injection (or so they think), by the 3rd day the ddos had stopped. However all index.php, footer.php, header.php and some other common named files across many directories contained malicious code which rewrote this malicious code every time one of them was ran. Sure his above statement may not be totally correct, however ddos can equal a compromise (not that a 'secure connection' would stop php injection afaik).

Outsource it by Minupla · 2012-04-09 13:11 · Score: 2

I've lived through this (although in my case the twits doing it were holding us for ransom) Prolexic was the solution we went with and I endorse it. The economics of the situation strongly favor outsourcing to a third party. It's a service you'll likely need for a short period of time, provisioning it yourself would entail obtaining equipment and specialized expertise that you would have to commit to over a long period of time. A Prolexic can afford to obtain better equipment, and have specialized staff who can configure it to block the latest attack because they're dealing with it for clients constantly.

Min

--
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before

G-Wan ! by Taco+Cowboy · 2012-04-09 13:16 · Score: 2

Instead of nginx, I'd use g-wan

It's very efficient, even more than nginx

--
Muchas Gracias, Señor Edward Snowden !

Re:G-Wan ! by Ice+Station+Zebra · 2012-04-09 14:49 · Score: 1

Because you love writing web sites in C?
Re:G-Wan ! by Taco+Cowboy · 2012-04-09 16:00 · Score: 1

No, because it's fast
Besides, I figure there's no harm to utilize my knowledge in C to make my site super efficient

--
Muchas Gracias, Señor Edward Snowden !

Re:DDoS != DoS by mysidia · 2012-04-09 13:22 · Score: 1

Question: How to protect against DDoS attacks without regard to the availability of the target of the attack Answer: If your carrier supports it, initiate an RTBH on the targeted IP. If not, contact the helpdesk and have them null route it.

NO. That is not a DDos protection strategy. That is a "white flag" strategy. It is essentially surrender, because you are intentionally making the attack succeed against the target then.

First Hand Experience at Small Company by Anonymous Coward · 2012-04-09 13:28 · Score: 4, Interesting

Posting AC as I would prefer not to expose my employer in anyway.

I went through this exact situation the week before last Thanksgiving last year. I work for a gifting retailer that makes all of its money in Q4. Not a good situation. We're a small - mid sized business with about $20 million in sales from our Ecommerce site.

We went the cheap route first. The proxy service cost about $500/month and guaranteed 10 Mbps clean traffic to the site. Our DNS was changed swinging our domains to the proxy service and ACLs put in place on the "backend" to only allow connections from our new gateway in the proxy.

Things were fine for about 24 hours when the attack was stepped up. The service was seeing 450 Mbps inbound to our main domain. That is not a mistake - 450 Mbps is easily attained using a botnet or simply focusing the attention of some lurkers on pastebin links. We now had to change DNS AGAIN to "upgrade" to their better platform that could handle this attack. As we started this work, we also began talking to a couple of the higher end services...

After the $500/month service capped out and blew a gasket, we made the tough decision to go with the Cadillac. It was costly and they had us over a barrel (day before Thanksgiving, cheap service not working out, "sure would hate to see your site go down on Black Friday" mob pressure). But we knew even half a day of lost demand would pay for the yearly service (yes, it is yearly - no month to month option).

The difference was amazing. As soon as we had swung our DNS over to the new guys, the attack was mitigated within 5 minutes and abated within 20. This, of course, leads the paranoid to wonder whether it was the service doing the attacking to begin with, but we are a high profile target in the minds of the Occupy movement, so it made sense (I do not share my employers sense of community - it is only a job).

We have been attacked since then and every time the attack was mitigated within 5 minutes. If you require this type of uptime, build this service into your budget from the beginning.

Re:First Hand Experience at Small Company by shentino · 2012-04-09 14:20 · Score: 1

I guess it's expensive to be unpopular.
Re:First Hand Experience at Small Company by ScentCone · 2012-04-09 15:17 · Score: 2

I guess it's expensive to be unpopular.
No, it's expensive to be hated by a small group of people who happen to not mind being network abusing dicks so they can put on a show of hurting your business. There are plenty of people who don't like the Occupy nonsense, but most of them are too busy doing something constructive to sit around figuring out how to DDoS attack George Soro's hundreds of web properties.

--
Don't disappoint your bird dog. Go to the range.

Re:Change Apache to nginx by stanlyb · 2012-04-09 13:39 · Score: 1, Informative

Actually it does, and that's one of the reason for using nginx as a proxy and cache server before the appache server.

Unfortunate name by Livius · 2012-04-09 13:47 · Score: 1

No matter how many times I see it, my first thought on seeing "DOS attack" is that a virus downloaded MS-DOS onto a computer.

Which would almost be the worst thing to happen to a computer.

Re:Unfortunate name by dgrotto · 2012-04-09 15:17 · Score: 1

Every time I see Prolexic, I read prophylactic, which I am not sure was their intention...
Re:Unfortunate name by DamonHD · 2012-04-10 00:01 · Score: 1

Maybe it was: a prophylactic is a preventative measure. Like taking vitamins to avoid colds or locking your door to prevent theft. Prophylaxis need not involve genitalia, latex, and smirking. B^>
Rgds
Damon

--
http://m.earth.org.uk/

Akamai is a good alternative by K3ba · 2012-04-09 13:50 · Score: 2

I recommend Akamai's services as a CDN for static content (eliminates a lot of load from your own servers), a proxy for dynamic content (shield/reverse proxy effect services) as well as a protection against (D)DoS attacks. They have a number of great case studies ( http://www.akamai.com/html/customers/index.html ) which are well worth the time looking through, as they have successfully mitigated attacks against small, medium and large websites. Their (repackaged) Kona Security Services are surprisingly good.

--
Catapultam habeo. Nisi pecuniam omnem mihi dabis, ad caput tuum saxum immane mittam.

MaxCDN & CloudFlare by Jonah+Hex · 2012-04-09 13:52 · Score: 1

I have had a few large spikes and MaxCDN and CloudFlare have kept my site up with no issues, so the real answer is distributed caches and you get what you pay for. - HEX

--
Horror & SciFi Erotic Nudes

Re:Change Apache to nginx by CoderExpert · 2012-04-09 13:54 · Score: 1, Informative

You do understand that there are different kinds of DDOS attacks and flooding the available bandwidth is just one of them?

In fact most DDOS attacks rely on causing heavy load on the server. Bringing down server like that requires much less resources of your own than flooding it with pure traffic.

Geez, slashdot, this is one of the fundamentals of DDOS attacks.

Still waiting for Spoofarino ... by Taco+Cowboy · 2012-04-09 14:00 · Score: 1

I've lost count of how many years I've been waiting for the Spoofarino utility

And I'm not alone, other people also been talking about that utility

http://www.spywarepoint.com/gibsons-spoofarino-utility-t54570.html

--
Muchas Gracias, Señor Edward Snowden !

Re:Wow deja vu by shentino · 2012-04-09 14:14 · Score: 1

I agree. It seems like your hosting was using the attack as an excuse to try shoving an upsell down your throat.

Any transaction that allows someone else to take advantage of your own misfortune should be viewed skeptically.

Re:Change Apache to nginx by Bengie · 2012-04-09 14:27 · Score: 4, Informative

1) A properly configured FreeBSD router/firewall will handle 200k+ connections per second
2) Configure the firewall to proxy TCP hand-shakes, so your web servers don't get flooded with syn packets unless the hand-shake actually finishes
3) Mid-grade nginx web server will handle 70k+ requests/sec
4) Setup your DNS to round-robin to several web servers

Between your firewall and your webserver rules, you should be able to filter most obvious DDOS's. That which you can't filter, you'll just have to brute-force it and suck it up.

Your web servers can handle more requests than you have bandwidth, the next bottle-neck is your database.

There is not "silver bullet" like you said, but a properly designed system should be robust enough to leave your bandwidth your bottle-neck Most web apps I see aren't designed to properly make use of SQL. It's like someone trying to shoe-horn procedural logic into a database. Gotta get your DB architect to work with the programmers.

A properly architected web app with a properly architected DB should be able to handle more requests than your bandwidth can handle.

The only real DDOS to worry about is a flood and you can't really stop that unless it's a simple up-stream change. Enough machines DDOS'n ping floods at you will take you down. Filter all you want at your router, you won't have the bandwidth. Would be too simple to filter up-stream. A bunch of random forged TCP packets will suck up your bandwidth. If the attack is well distributed, ain't not'n you can do about it.

There is not "Silver Bullet" like you said, but a properly designed system should have bandwidth as its bottle-neck

Re:Change Apache to nginx by PiSkyHi · 2012-04-09 14:52 · Score: 4, Informative

Its quite normal in Slashdot for one person to rant, another rebuts everything cruelly and then another and another...

My take on this is that nginx is cool for static pages, we all should know that.... new optimisations in Apache 2.4 hope to address some of these and Apache is easier for me to configure for dynamic sites with controllers.

Regarding DDOS - neither of these will help... there are different types of DDOS attacks, sure. Any site that is dynamic in nature is screwed by any DDOS before it even saturates the entrance because an inability to disseminate requests in time causes the webserver to effectively stall. There are mitigations, one of the best is iptables rate limit for DOS attacks, of course defending DDOS attacks requires enough horsepower behind the scenes, so that when the entrance is saturated, requests can still be distributed usually by a load-balancer that places the bottleneck at the entrance alone - placing the site in the cloud with auto-scaling will solve this at a cost. Any type of DDOS attack that relies on an exploit though, requires a fix, removal or workaround before any horsepower mitigation can take place.

Re:Wow deja vu by sycodon · 2012-04-09 14:52 · Score: 1

Annoyingly we haven't had an attack since

What a coincidence eh?

--
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.

Re:Change Apache to nginx by Anonymous Coward · 2012-04-09 14:53 · Score: 1

Requests for static content -- the only area where anyone can make any kind of semi legitimized claim about Nginx's performance -- are never going to be about causing load on the server but about flooding its bandwidth. Dynamic content is where most DDoS attacks are successful especially for prolonged periods of time (perhaps even simply increasing cost without directly affecting service). Most people's shitty drupal and wordpress site are going to crumble easily regardless of what httpd.

Post a link on Slashdot! by Tyrannosaur · 2012-04-09 14:59 · Score: 1

I want to see this website!

Re:Change Apache to nginx by CoderExpert · 2012-04-09 15:09 · Score: 1, Informative

Actually, any website that is properly optimized is already serving most of it content as static. That is what caches are for. And yes, you can (and should) cache even parts of the page. However, even with dynamic content there is a very clear difference between serving with apache or nginx. Sure, someone who really knows Apache can maybe hack it to work as fast, but how many persons actually know? Let's be realistic here.

Most of the time just switching to nginx and properly caching your content can mitigate DDOS attacks. Sometimes you may need more, but the point is that you should fix these bottlenecks first anyway.

Don't ask on /. by Nethead · 2012-04-09 15:19 · Score: 4, Insightful

This is a discussion you need to take to the NANOG list. Don't ask the amateurs, ask the professionals. The answer will involve ACLs, BGP settings, and community strings. If you don't have your own ASN then you need to push the issue upstream and work with your provider. Period. If you do have your own ASN and are running BGP then you need to read the NANOG list (and learn to take shit from Randy Bush, et al. They know what they are talking about.) Asking on /. can only make things worse.

--
-- I have a private email server in my basement.

Re:Don't ask on /. by dgrotto · 2012-04-09 15:32 · Score: 1

+1 - From personal experience, if you do not have your own ASN and a STAFF of BGP engineers, outsource. Using the basic proxy service from the "large and mature" services mentioned above, plan to pay $5k - $6k per month for the privilege of an available website.

Re:Change Apache to nginx by PiSkyHi · 2012-04-09 15:27 · Score: 1

That's too generic. I manage a heavily dynamic site and so we use both, since page caching is a waste of time, the cache has to be a little further down and the impact of using any particular brand of webserver application will be held back by the processing behind the entrance point.

easy by bhcompy · 2012-04-09 15:33 · Score: 1

1) acquire a Gibson
2) change username/password of superuser account from 'god'
3) profit, since Gibsons easily survive ddos attacks/flooding as evidenced by the documentary released over a decade ago detailing attempts to hack a Gibson

Re:Change Apache to nginx by Gwala · 2012-04-09 15:55 · Score: 3, Interesting

Amazon AWS bills you bandwidth directly. A DDoS could get very expensive.

--
#!/bin/csh cat $0

a mirror by Max_W · 2012-04-09 16:02 · Score: 1

Use a mirror in another data-center for your e-commerce website. Post link on the main website to the mirror or mirrors.

Update mirrors with a script.

The ideas (prejudices) of monarchy, monotheism, etc. are set firmly in our heads, but why your e-commerce should have only one and only URL? Let it have 2 or 3 URLs.

We use a mirror for the 4th year, and it is the 4th year of sanity for me.

Re:Change Apache to nginx by TooMuchToDo · 2012-04-09 16:08 · Score: 1

You could server your site from multiple EC2 regions, and use anycasting if you don't need to manage state server-side. You're going to pay out the nose for traffic and instance time, but Amazon's AWS should have *more* that enough capacity to server as a sink for all but the largest of attackers.

Pay your protection money by Ryan+Amos · 2012-04-09 18:37 · Score: 2

To most of the commenters: WTF? You have obviously never been involved in a DDOS attack. Here is why:

1) A typical DDOS attack in 2012 will send traffic measured in hundreds of MBPS/GBPS down your pipe. Not only is this a massive volume of traffic, but almost all of it is in the form of SYN/ACK packets (which are exponentially more difficult for your frontend servers to handle; especially when they are never followed by a FIN.) This is many orders of magnitude more difficult to deal with than what most sites are scoped for. You cannot just "handle it," we're talking about something that is often 7-8 standard deviations away from your "normal" peak traffic levels. In other words, your infrastructure cannot handle it. Because if you overbuilt your infrastructure to those levels, you are an idiot. DDoS protection services cost a fraction of what it would cost you to build a network that could handle that.
2) Your normal DDOS doesn't come from one "large user." (hence, the first D in DDoS.) It comes from thousands (or hundreds of thousands) of IP addresses, all at once. Botnets? Yeah, they are real things, and they can be really destructive. And bad people control them, and you may have fired their mother at one point. Who knows why they have it in for you, but they probably will at some point.
3) Even if your infrastructure could handle an amount of legitimate traffic equal to the volume the DDoS will produce over the span of 6-12 hours, you would then have to pay for it. I promise you, you don't want to be in that position. Most hosting providers probably won't make you pay for all of it, but they will become real interested in what you're hosting that would make someone want to DDoS you in the first place. And your boss will probably make you find a proxy solution to solve the problem; so why not be proactive about it so you can say "Yeah, those /b/tards decided to DDoS us, but I took care of it 3 months ago."

TL;DR: DDoS proxy services like CloudFlare exist for a reason: it's simply not economically feasible to overbuild your infrastructure to the point where you could survive such an attack. Pay the man, keep your site up, and ignore the punks smashing cars in the street because you have insurance, so fuck em.

Re:Change Apache to nginx by 1s44c · 2012-04-09 21:41 · Score: 1

Nginx is not really going to help if you are getting DDOSed and your incoming pipes are full.

The problem isn't at the web server layer, it's at the network layer.

Wowsers... by way2trivial · 2012-04-09 21:47 · Score: 1

"They do wierd things like load web pages repeatedly while never loading images/running javascript/loading CSS stylesheets."

Just saying-- so do the blind....

--
every day http://en.wikipedia.org/wiki/Special:Random

Re:Wowsers... by DamonHD · 2012-04-09 23:36 · Score: 1

So *that's* where the blind societies have been getting all their gold and jet planes from! B^>
Rgds
Damon

--
http://m.earth.org.uk/
Re:Wowsers... by Snowhare · 2012-04-10 13:59 · Score: 1

Yes - but they don't change web browsers with every sequential request. ;)

Re:Change Apache to nginx by Dan+Dankleton · 2012-04-09 22:04 · Score: 1

Agreed. I've personally stopped DDoS attacks (the attack was coming from many sources - and therefore was distributed, and was stopping the customer's web server from responding - and therefore was a denial of service) by running Squid as a reverse proxy in front of the web server. In this case it was IIS which was being brought down by just 10Mb of malicious traffic. Yes, running a reverse proxy or changing Apache to nginx isn't going to solve 40Gb/s traffic hitting your 1Gb/s interface, but it may well fix a small attack which is depleting resources.

Re:Change Apache to nginx by rev0lt · 2012-04-09 22:07 · Score: 2

1) Yeah, in test environments with empty or very very small packets. The truth is, PF isn't threaded, so not only you're bound by ethernet I/O, but also by core processing power. Also, there are open port limits, internal buffer limits, and actual traffic limits (data). If you're pulling your data from a cluster (shared storage, database, whatever), it usually translates to more tcp connections. For real workloads and usual datacenter limits (100Mbps or 1Gbps uplink), you won't get nowhere near that value, unless you have multiple servers and/or your own upstream.
2) TCP connection limits (which FreeBSD handles somewhat gracefully), and if you have a fast link, you'll probably have PF performance issues
3) Requests/s is only a part of the equation, and it is meaningless in a DDOS context. Imagine your server receive a flood of requests from another continent - the RTT alone for the 3-way TCP handshake will probably exaust the available ports. So again, yes, I highly doubt you get those numbers in real-world usage (70k requests of a 1Kb file will be real close to saturating a Gigabit link, and that's assuming it's an instantaneous process).
4) That is a good solution, but it's not really nginx-awesomeness related, is it?

What you describe are the usual steps when you have the money to do it, the infrastructure, and the staff to mantain it. Installing nginx as a way of solving DDOS is plain stupid, specially when most optimizations you can apply to nginx are also valid in apache, and the result will be the same - a properly configured environment will usually have - as you said - bandwidth as its bottleneck, regardless of the web technology used.

Re:Change Apache to nginx by rev0lt · 2012-04-09 22:14 · Score: 2

Yeah, but if you have reverse proxy or caching servers, you might as well skip nginx and use varnish, or any other specialized solution. Or even Apache (with the plus of having mod_security). It's not like nginx automagically now enables you to do stuff that wasn't possible before. I actually never used nginx as a reverse proxy, but I've read some comments from people that had some issues with it, and switched to varnish instead.

Re:Change Apache to nginx by outsider007 · 2012-04-09 22:25 · Score: 1

Not to mention the bill for disk reads/writes. Staying up under a DDoS on ec2 could be the worst thing that ever happened to you. It could even be worse than bidding high on a spot instance and surviving a ridiculous rate spike.

--
If you mod me down the terrorists will have won

Re:Change Apache to nginx by tuomasb · 2012-04-09 22:37 · Score: 3, Informative

This was changed last year. AWS doesn't charge for inbound traffic. Amazon Web Services Pricing Changes Effective July 1, 2011

Re:Wow deja vu by rev0lt · 2012-04-09 23:05 · Score: 1

There are several available solutions, depending on your budget. You could opt for a cloud hosting solution (AWS, GoGrid, etc) so your cluster could scale dynamically according to demand. You could roll your own solution (and there are several options with different price ranges, from simple web cluster to redundant replicated clusters in different datacenters), preferably with a provider that won't panic over 3Gbps traffic. You could use a proxy service like those mentioned in the article.

To put it in perspective, many modern datacenters have 10G pipes to the rack. A modern server can handle 10G traffic with iptables (assuming somewhat simple rules) without issues. A midrange NAS probably can pull out more than 3Gbps of data.

The problem is, a DDOS targets resources. If your provider doesn't have the pipes to handle it, it won't even get close to your server, and all their infrastructure will probably drown. If the provider can handle it, then you can actually try to do something about it - front-end caching servers with round-robin DNS, firewalling and/or segment blackholing (some providers can easily configure AS blocking), and a scalable multi-tier architecture, so you don't get offline by TCP resource starvation, database or I/O overload. There are some nifty appliances for web filtering (like Citrix NetScaler or whatever is called), but I have no experience with them. They seem to work for big companies, so it may be an option.

Oh, and 100Mbps won't cut it (specially if it's those 10Mbs burstable to 100Mbps). I probably could DoS your server with my internet connection alone :D

Redirect everything to /. by hessian · 2012-04-09 23:30 · Score: 1

Obvious solution. You will have invented the reverse slashdot effect .

--
Futurist Traditionalism

Ignore gartner... by Bert64 · 2012-04-09 23:51 · Score: 1

Don't pay any heed to what the likes of gartner say, none of these things are "mandatory", they are all a factor of risk vs benefit vs cost...

If your running a webserver, it only has tcp/80 open and nothing else... Then you add a firewall which sits infront and only allows 80/tcp, what have you gained? You may have an extra point of monitoring, and mitigation against outbound traffic *if* the box gets compromised... But if someone exploits a vulnerability on port 80 it's not going to help you (and there was never anything else to exploit). On the other hand, your costs have now increased (Cost of firewall, + cost of management, + power/hosting costs), you now have an extra point of failure, an extra target for hackers and potentially reduced performance (increase risk of ddos) because the firewall device does far more processing per packet than your host does, and has a slower cpu.

Same with an IDS/IPS, cost of the device, cost of monitoring it (after all its largely pointless if not monitored), risk of false positives causing disruption with an IPS etc.

Similarly when it comes to risk of ddos, are you hosting anything thats likely to be a target? If someone does take your site down, what will it cost you for it to be down?
Also consider extra capacity to handle unusual loads (eg a link from slashdot), is the extra cost worth it to handle the 0.01% of the time you get that much traffic?

And most importantly, when you read advice from the likes of gartner take it all with a pinch of salt... Ask yourself this, who pays their wages? You will find that a lot of their funding comes from the very vendors that sell these products they are now claiming to be "mandatory"... Do you really think they are in any way impartial?

--
http://spamdecoy.net - free throwaway anonymous email - avoid spam!

Easy and Best Solution! by rullywowr · 2012-04-10 00:35 · Score: 1

Security by obscurity. The more obscure your website becomes, the less chance of a DDOS attack!

Re:Wow deja vu by Bert64 · 2012-04-10 00:58 · Score: 1

The question is..
Did the attacker throw 3gbps at it because that's all they had, or because that's all that was needed to do the job?

It's not uncommon for multiple colo customers to be on a shared switch with 1gb uplink especially when each individual customer only has a 100mbit port, a 3gbps attack will take all of those offline even if it doesn't harm the colo provider a a whole.

Also the isp has to pay for the bandwidth usage that hits them, even if most of it never actually reaches the end customer... A sustained 3gbps attack could become quite expensive if it lasts long enough to push their 95th percentile billing up. If their average customer only uses 1mbps, chances are they don't have an especially high commit rate.

--
http://spamdecoy.net - free throwaway anonymous email - avoid spam!

Re:Wow deja vu by rev0lt · 2012-04-10 01:51 · Score: 1

It's not uncommon for multiple colo customers to be on a shared switch with 1gb uplink especially when each individual customer only has a 100mbit port

Yes, you are right, but if you have DDNS worries, you'll keep away from those providers, and/or opt for a beefier connection. Many of them will happily provide you with detailed information regarding network structure, so you can be aware of what you are actually paying. Also, many of those "cheaper" providers have somewhat frequent DoS attacks inside their own network (some machines get compromised, and then try to scan/infect everybody else), so, again, handling/filtering a 3Gbps attack isn't usually a big deal.

Also the isp has to pay for the bandwidth usage that hits them, even if most of it never actually reaches the end customer...

That's the ISP's problem, and I seriously doubt that the pricing doesn't reflect these "hidden costs".

Re:Change Apache to nginx by rev0lt · 2012-04-10 02:35 · Score: 1

So, being aware that are issues when using nginx as a reverse proxy instead of varnish, is useless? And those comments were made here, on slashdot, so god forbid you use a search engine to confirm what issues are those...

Re:Change Apache to nginx by burne · 2012-04-10 03:57 · Score: 1

> In fact most DDOS attacks rely on causing heavy load on the server.

If that's the case you are far worse off using nginx. 'Untuned' apache will limit your poor webserver to 256 running threads. With defaults nginx will start 1000 processes, which makes your site four times as dead. And if you are clueless but know how to google you can easily get at 10.000.

The kind of attack that relies on high load preys on clueless admins who think that 'tuning a webserver' is maxing out all limits.

What we did by xcsalt01 · 2012-04-10 04:43 · Score: 1

When my company's software got DoS'd, we looked for areas that we could provide performance enhancements either in the database of application. A lot of times, you can handle tons more traffic by making sure that everything in the application is performing as well as possible. Also, traffic shapers couldn't hurt, since they give you a way to automatically monitor for DoS style attacks and block them. Just my 2 cents.

Re:Change Apache to nginx by errandum · 2012-04-10 04:54 · Score: 1

this is not true anymore, amazon does not charge for inbound traffic.

Had good results using blockdos.net by Wokan · 2012-04-10 06:58 · Score: 1

If your hosting service doesn't have an Anti-DDoS tier or option available, the people at blockdos.net were able to help in a pinch. If your host can change your IP address, you'll get the best results. You point your domain at the BlockDoS provided IP and then set your firewall (or server if firewall not an option) to block any inbound traffic not coming from a second BlockDoS provided IP. The downside is that you lose a lot of request header information and your server logs show all requests coming from a small set of IPs. (Using external services such as Google Analytics still works fine though.)

Re:Wow deja vu by rev0lt · 2012-04-10 11:02 · Score: 1

It seems Slashdot stopped notifying me of AC replies, weird :P
If you don't have the infrastructure and/or the staff, outsourcing DDOS protection seems a good choice. Trying to protect against the 3Gbps mark is a bit silly, since next time you may get hit by 10x that traffic. It is important to learn the nature of the attack (are we talking about hundreds of ip addresses, or a dozen? are they "residential connections" or compromised servers? is it a TCP handshake attack, or requests to specific pages? do those pages exist, or is the fancy 404 rewrite eating up resources by pulling the page from the database?), and probably some optimizations can be made to your ecommerce platform (such as reducing requests, pulling images from CDNs, etc - check tools.pingdom.com for basic hints) so that the traffic spent with legitimate customers can be lowered.

Ask yourself this: how much do you lose for each day your ecommerce platform is down? Since you seem to have a single-server solution, if you consider let's say, 5 days of downtime per year, how much would you lose? Do the losses are greater than 50% of cost of a fancy redundant infrastructure? If so, upgrade immediately.
Since you say you are from the UK (hi neighbour, I'm from Portugal), you may also want to check available other providers such as Rackspace. They probably have some cloud-based solution that may fit your budget.

There is an interesting presentation made by Mayhem a couple of years ago (mostly touting OpenBSD's awesomeness) regarding a DDOS case (also "small" traffic), and how they handled it. You can read the pdf at http://2009.confidence.org.pl/materialy/prezentacje/alessio_pennasilico_bakeca_ddos_confidence_2009.pdf

Re:Change Apache to nginx by Bengie · 2012-04-10 14:12 · Score: 1

You're a bit off. nginx by default has 1 thread per core. 4 core cpu is 4 threads. It never has more than those 4 threads. nginx uses async IO, apache uses threads.

Re:Change Apache to nginx by Bengie · 2012-04-10 14:43 · Score: 1

"PF isn't threaded"

I don't know for sure myself, but I recently saw a benchmark by Intel using FreeBSD and their main claim was making use of multiple cores. They had a FreeBSD box pushing 20Gb/sec of routing for with 1500byte packets and it could still handle over 10Gb/sec with 64byte packets.

One of the biggest changes for FreeBSD9 was the extra threading their network stack does, it is suppose to help dramatically with the stateful firewall.

"most optimizations you can apply to nginx are also valid in apache"

The biggest thing I like about nginx is not its great out of box speed, but how it scales past its peak. nginx is fairly async and when you get past peak performance, it only has mild negative scaling. Apache, on the other hand, has horrible negative scaling under load. Heck, properly programmed async IIS web apps are almost near perfect and just plateau once peak performance is reached.

Re:DDoS != DoS by AK+Marc · 2012-04-10 21:23 · Score: 1

Not true. A number of DDoS attacks are screens for penetration actions. If security is more important than uptime, pull the plug. If uptime is more important than security, then get any of the expensive anti-DDoS boxes, and work with your provider.

--
Learn to love Alaska

Re:Change Apache to nginx by rev0lt · 2012-04-10 23:34 · Score: 1

I guess you are confusing the FreeBSD TCP stack optimizations with stateful packet inspection - routing, by default, does not touch firewalling.
But hey, straight from the horse's mouth: http://www.openbsd.org/faq/pf/ru/perf.html . That's why there's a new (NetBSD) project called NPF, that aims the creation of a more sofisticated and scalable packet filter.And, afaik, FreeBSD 9.0 TCP optimizations relate to congestion algorithms, not necessarily firewalling.

I often see these misconception of trying to translate internal/testing/development benchmarks into real-case usage. Just because a piece of software measured 100k connections/s on a controlled environment and selected hardware, it doesn't mean it will perform similarly on your machine. Local network application tests are done with clean, well-formed, unrouted traffic. In internet-facing production servers, you'll get all kinds of garbage. The real amount of hits/s you'll get will depend of a plethora of factors, such as average content size, network latency, I/O latency, design of the application, quality of the hardware, quality of the drivers, etc. The choice of webserver is relevant, but not that much. TCP attacks can drown your machine (by network exaustion) long before you see any real advantage of using Nginx regarding DDOS scenarios. It is also very common to see people switching to nginx and being forced to learn what they are doing, so the idea they have of apache is "all modules loaded" and some random options on a control panel.

I use both webservers, for different applications. Nginx is faster in static content, and seems to handle stress very well. Apache is feature-complete with a plethora of third-party modules (such as mod_security) and behaves very well as a general-purpose server (I actually use a lot mpm_itk, so every vhost has its own uid/gid). Sometimes you can squeeze extra performance by trading Apache for Nginx, sometimes it will break your app (complex mod_rewrite rules, for example). But that's relevant for legitimate users - if your network pipes are clogged by requests, both webservers will appear unavailable.

Slashdot Mirror

Ask Slashdot: Experience Handling DDoS Attacks On a Mid-Tier Site?

130 of 197 comments (clear)