Slashdot Mirror
Ask Slashdot: How To Deal With a DDoS Attack?
First time accepted submitter TheUnFounded writes "A site that I administer was recently 'held hostage' for the vast sum of $800. We were contacted by a guy (who was, it turns out, in Lebanon), who told us that he had been asked to perform a DDoS on our site by a competitor, and that they were paying him $600. He then said for $800, he would basically go away. Not a vast sum, but we weren't going to pay just because he said he 'could' do something. Within 5 minutes, our site was down. The owner of the company negotiated with the guy, and he stopped his attack after receiving $400. A small price to pay to get the site online in our case. But obviously we want to come up with a solution that'll allow us to deal with these kinds of attacks in the future. While the site was down, I contacted our hosting company, Rackspace. They proceeded to tell me that they have 'DDoS mitigation services,' but they cost $6,000 if your site is under attack at the time you use the service. Once the attack was over, the price dropped to $1500. (Nice touch there Rackspace, so much for Fanatical support; price gouging at its worst). So, obviously, I'm looking for alternative solutions for DDoS mitigation. I'm considering CloudFlare as an option; does anyone have any other suggestions or thoughts on the matter?"
You just gave him $400 more than he had before, and he knows you're good for it.
What were you thinking?
Cloudflare are great, I use them on my sites and they can handle the traffic w/o issue.
Spend the 400$ on a computer-forensics investigator, find out who is doing this then contact law-enforcement.
I don't know the meaning of the word 'don't' - J
I don't know who you are. If you are looking for ransom, I can tell you I don't have money. But what I do have are a very particular set of skills; skills I have acquired over a very long career. Skills that make me a nightmare for people like you. If you let my computerr go now, that'll be the end of it. I will not look for you, I will not pursue you. But if you don't, I will look for you, I will find you, and I will kill you.
Undetectable Steganography? Yep, there's an app fo
Hi first time accepted submitter!
You may want to check this Ask Slashdot.
There was a gambling site in Australia that got on the wrong side of a gambling gang (stealing customers, nothing they did specifically to attract ire). The DDoS took down Australia. Keeping your servers up when your link is flooded isn't too hard. Keeping your site up when the DDoS takes down your ISP and their ISP is a little harder. The "best" solution is to log all IPs and sue all local IPs for hacking. Get some old lady fined $1,000,000 for hacking and maybe people will figure out that they should secure it or turn it off. If there were no botnets, there would be fewer, if any, DDoS attacks.
Learn to love Alaska
Try buying fire insurance when your house is on fire. It's a risk pool. Duh.
With due respect, in my view, this is like trying to buy homeowner's insurance while your house is on fire, and complaining that they won't sell it to you.
Why is it unreasonable for you to pay more for "OMG I NEED IT RIGHT NOW!" service?
It's easier to do some prevention than to try to and figure out and control the problem WHILE it's happening. Also, why is it unreasonable for them to give someone who sees the need for some complicated traffic monitoring and filtering a discount for letting them set it up, y'know, during normal business hours with forethought and preparation and not as part of a crazy firedrill?
(no, I don't work for Rackspace)
...would have been to ask him how much to get the name of the competitor. Would probably cost a bit, but documenting that exchange and turning it over to the FBI instead of just the DDoS info might have meant one fewer competitor...
Doing it at your own router won't work, because legitimate traffic has no room to get through
"When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
Judging from your post, you've never been the target of a DDoS as none of what you said would have any affect on a real attack.
If I wasn't even really trying, I'd just use your IDS against you and have you end up effectively firewalling yourself off the Internet.
Save my bandwidth for someone with skills while you try to figure out what's going on
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
Option a) Your best bet is go strait to law enforcement. The FBI is actually very interested in these sorts of things even if you are small fry. This might not be a such a hot idea though if the group extorting you actually has some capability. Usually they will set up a string, and track the money when you pay.
Option b) Just shut up and pay up. Never taken this approach myself. I assume it makes the problem go away for a while anyway. I imagine said problems come back for another fix later, and I'd wonder if the attacker ever really had the capability.
Option c) pay the back bone provider, ie ATT&T or whoever is your ISPs, ISP for their DDOS protection services. They actually DO have the resources to protect you from a DDOS. Everything else anyone else is selling is just snake oil because a large enough botnet can simply use all the bandwidth weather you attempt to ack tarpit, or not; They unanswered SYNs alone will consume your entire pipe. This option is terribly expensive, might be worthwhile if you are running a large and inadequately distributed eCommerce site or similar.
Option d) Distribute the hell out of your site. This leads to all sorts of complexity around replication and have the big CDN providers host all your static content and resources. This may help depending on the type of attack. You will want make sure your DNS resources are also well distributed you will basically use fast-flux DNS yourself to stay ahead of your attackers. Essentially you keep changing IPs every 300 seconds or so. You will have challenges preserving sessions and for lots of services its not viable, for WWW it can be made to work. Again this is serious money and time. It might be cheaper than Option c, if you want you are trying to be available for is a small amount if high dollar transactions, as opposed to a higher volume smaller dollar situation.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
was RESPONDING to the guy. Even to say "no." It's like responding "unsubscribe" to a spammer.
What you've done by replying is telling him a.) you GOT his e-mail (not by any means a sure bet with spam filters), b.) you ARE IN FACT the people who own the site in question, and c.) the REASON you're not paying is that you believe he can't carry out his threat.
Let's say I'm this guy. I'm probably a script kiddie with a small botnet under control. I troll for small ecommerce sites (ones that are probably not profitable enough to have good defenses, but would be seriously impacted by a DDoS attack). I try to find some contact information. Again, I'm running some kind of script to troll for these, which means my sample isn't amazing and my data quality is probably questionable.
Then I send out hundreds of e-mails. Like a spammer, I'm going for quantity. Most of these probably disappear into the ether. Whatever - I only need a few to hit a target to get paid. A few people will actually pay up from the e-mail (probably not many, but hey). Some will ignore me (and be impossible to tell from the "disappeared" group. Then there's the lunkheads like you who confirm I sent the threat to the right person and I do feel vulnerable, but I doubt your ability to follow through.
Perfect! I train my botnet on that guy. I'm pretty much guaranteed money. The "someone offered me $600" is a bluff, of course - no one offered him anything, and it's all profit to him. But it sets a nice mental scale for you, so that you'll foolishly think you "got off easy" giving him $400 (when you could have given him $0).
Again, this is a VOLUME play. He has enough bots to DDoS SOMEONE, but not to DDoS EVERYONE. You were attacked for one reason - because you responded.
Sure, there was network engineering involved, but make no mistake - you got SOCIAL engineered here, first and foremost. Fix THAT, not your network.
...but to be honest, Kuro5hin is paying us $1000 not to tell you. Perhaps if you would be willing to pony up $1500 we could do business.
There isn't much you can really do against a determined foe. There are just too many bot computers out there ready and willing to flood your servers with traffic. Huge companies with lots of staff, racks upon racks of servers, and really fat pipes have been hit with these attacks and failed to stop them.
Now there are a few things you can do to help... You'll note that these things are all extremely important for high-volume sites or major legit traffic spikes:
Have a switch in your website app that turns off all dynamic access, logins, session state, content generation, Ajax loading, etc and just serves static pages. This should also disable any kind of downloads unless you are already serving them from a CDN. If you are under attack (or just get featured on slashdot) throw the switch. Your website won't be terribly functional, but it will still be up. If you want to get fancy, have several levels of degradation where you can progressively turn features off to lighten database loads, etc. but without throwing up error pages or just having the site completely fall down. (ex if your sidebar typically shows recent comments via a database query, then just show a cached set of comments only updated once per day. Now every page access is using one less database query.) This is super critical because the first resource to be exhausted will be your database's ability to answer queries. The second will be your web server's ability to track session state and process requests. Especially if your site does anything even mildly complicated.
If your OS/Webserver/app support it, turn on kernel caching, install a cache plugin, etc. Especially make sure the parts of your pages, images, etc that can be cached are cached. If the under attack flag is set, vastly increase the cache timeouts. Make sure proxy caching is enabled too so any clients behind ISP proxies, etc don't hit your systems. Serve jQuery, fonts, etc from Google's CDN. That's just good practice anyway and free.
If possible, use a CDN for images and other content. CloudFlare is a good one. Companies like Dediserve offer cheap CDN. There are thousands of others. If the panic switch is set, you can even serve the static pages off the CDN if you structure things correctly. These help offset bandwidth saturation.
Take the time to setup a VM of at least your basic site and keep it on standby at Amazon/Azure. If you are under attack or heavy load, spin up a bunch of nodes using that VM image. If you leave your load balancing running on their systems 24/7 then it is trivial to add nodes to the pool. Running a bunch of extra servers for just a few minutes or hours shouldn't cost a ton and will encourage all but the most determined script kiddies to find an easier target once they see your site is still up.
The most common resources exhausted during an attack (in order):
1. Database servers
2. Web server CPU load or memory
3. Bandwidth
4. Load balancers
Again, like I said, none of this will stop a determined attacker with a million node DDoS botnet... But it will make you a less vulnerable target.
Natural != (nontoxic || beneficial)
You were blackmailed by someone claiming to be represent your competition and then by your service provider. Correct? There are two things you should consider, and do so quickly before you've completely hosed your server logs: Contact your local FBI field office and then contact US-CERT. Yes, I know - it's DHS, but they track this stuff and have access to tools/training they can provide.
Bark less. Wag more.
So you never bothered with DDoS prevention services for what is apparently a critical company web site, which would allow the provider to work pro-actively on protecting your assets. Then when your assets come under attack you expect your provider will just drop everything and tend to your immediate emergency without additional costs? Sounds like car insurance after the accident, or health insurance after you develop cancer.
It's 2012. DDoS are a real and credible threat today. 10 years ago, perhaps a passing thing, but today... do you not read the news?
Stipulating that your lack of preparedness is not your fault and over-sight, I want to address RackSpace's mitigation fees and perhaps defend your position at least a little. Being that it is 2012 and DDoS are a real and credible threat, depending on the costs of such protection, perhaps RackSpace (or another provider, free market thingie and all) could provide these mitigation services as standard for a bumped-up cost. Perhaps 400% mark-up is a little steep for immediate service when 200-300% might cover the costs of getting someone involved.
Nonetheless, my inclination is to side with RackSpace. When you work proactively, your provider can have technology in place and ready to go so that a DDoS doesn't affect you. But calling in when it's going on: first off, they have to deal with the increase in bandwidth, the abuse of the server, virtual service, or multi-hosted box you occupy and hence affects on other customers, getting someone or a team of someones involved to start the mitigation process and move your incoming traffic to the systems which perform this protection, amongst other issues.
No, you need to bite the bullet on this one and count it as a learning experience. And call your local and/or state authorities and start an investigation, since your costs will most likely be well over the threshold of damages necessary to start such an investigation.
Depending on the severity of the attack, CloudFlare may your cheapest option, but be aware that they are not interested in mitigating severe attacks.
A client of mine was DDOSed last year, and my ISP's (shall stay nameless) DDOS Mitigation service could not cope with the size of the attack.
I have briefly tried CloudFlare, but they turned us off within 20 minutes without any notice, and promptly refunded all the money.
Luckily, I had an old contact with DOS Arrest. It was a bit expensive to setup, but they quickly got us back online, so it was worth it in our case.
IDS will not help protect you from a DDOS. The closed it might come to offering any kind of DDOS protect is it may help your firewall thwart scanning and information gathering in preparation for a DDOS.
Some DDOS uses a smallish number of hosts and will attempt to exhaust a specific resource like like server session memory by speaking a the protocol for a little while, if there is something that makes you especially vulnerable to that. Big DDOS use large bot nets and will simply burn thru all your bandwidth with SYN (tcp session start) packets alone. You really can't do much. If you have some way to tell which traffic is bad, like you know traffic should only be sourced from a specific address you can drop these sessions at your firewall and maybe make things a little better for yourself but it won't do much because the traffic still comes to your firewall and its going to consume your entire outside downlink, choking out the legitimate traffic anyway.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
What makes you think they're going to keep their word? You're not signing a contract here, these are criminals! All you're doing is showing you're a soft touch. They'll be back, and they'll demand more money. They'll probably tell their friends, too. Not to mention the moral aspect that by giving in to these people you are directly funding crime.
No, you ignore them entirely. Don't even reply to the emails (but keep them safe). If they DDoS you, live with it. Remember that these guys rent their botnet from other criminals, so every second they're DDoSing you is costing them money. As soon as they realise that they're not going to get anything out of you they'll give up and move on to the next target. Yes, you'll probably be knocked offline for a while but (a) with a bit of marketing nous you can make this work for you, by issuing thundering press releases going on about not giving in the terrorist demands, issuing 'apologies' to your customers and giving them discounts to make up for it so driving sales, etc --- basically, free PR, make the most of it; and (b) your internet-facing servers should be coping anyway. Of course, given that they aren't, that last doesn't help right now. But beef them up because it'll help next time.
Rackspace's behaviour is contemptible, though. I'd suggest looking for a different provider.
That's one IP, only tens of thousands more to go!
*Distributed* Denial of Service, remember?
Null route the ip being attacked, not the ip attacking. Of course this assumes you have a network consisting of more than a single ip.
Anyway this is basically the best way to handle a DoS. Otherwise you basically need to have the bandwidth/resources to endure the attack. Many providers will allow either a remote-triggered black hole session to their BGP router or allow a burst rate above your committed bandwidth if the interface allows for it.
This is the simplest way to handle a DDoS, not the best. Well, might be best from the provider's point of view. The best solution is to scrub the attack and let legitimate traffic pass through, but they decided to pay $400 instead of $6000.
@OP: a simple Google search gives you quite a few options on solving this problem. Just input "ddos protection" and hit Enter. You'll find that there are a lot of companies providing the exact service that you need, for less or more money than Rackspace, with "instant" setup. I used quotes since it takes a while until the new DNS entries will propagate, but you DO have options. Since you got scammed once, there's a good chance they'll try it again, so I suggest you try to be prepared for the next time.
Born to raise hell.
If they actually contacted you, report that to the FBI. They're probably contacting other people, too. A pattern will emerge.
A useful technical solution that seems not to be used much is to make web site services "fair", rather than first-in, first out. If something has a queue, and you're handling an request from source X, take the next work item from a source other than X. The result is the volume of attacks coming from an individual IP address doesn't matter. Only the number of attacking IP addresses matters. Your real users will still get through, although there will be degradation in proportion to the number of hostile IP addresses.That really should be a feature in Apache.
We use this for a free API service we offer. If you make a request, it may either be satisfied immediately if we have the data available, or the request is queued for processing (this involves examining and rating a web site) and the caller gets a "try again later" status. The processing queue is "fair", so no single source can overwhelm it. (Once we rate a domain, we won't look at it again for 30 days, so our system can't be used to DDOS other web sites.)
We once had a user from an Italian university who was trying to request info on a huge number of web sites. He put over 100,000 requests into the queue, and it didn't hurt performance for other users. After a few days, though, we looked at the logs, and noticed that the requests that returned "try again later" were never being followed up with requests for the actual info. So it was all wasted work. I sent a note to the department chair of the university involved, indicating that we had no objection to their using our service, but that their client program was poorly written and wasn't doing anything useful. The traffic stopped.
Null route the ip being attacked
So to protect against someone taking your website down, you effectively take your website down? I think I've missed some detail in your suggestion.
IDS will not help protect you from a DDOS. The closed it might come to offering any kind of DDOS protect is it may help your firewall thwart scanning and information gathering in preparation for a DDOS.
I would have agreed with you until recently, but today you can get IPS boxes which will do TCP SYN proxy (with cookies) and similar at 10Gbps. Now you can obviously get hit by more than 10Gbps of traffic, but in most cases that means you need to ask your provider for help anyway, since your own Internet connection is full. Some providers offer that you can pass dynamic blacklists to them which they will then install at their end of the connection, and some IDS boxes know how to provide such blacklists.
Finally! A year of moderation! Ready for 2019?
There's no winning in my opinion. The ddos shields do work, but they are prices for companies who really lose a lot of money with downtime. Your best chance would be trying to figure out who ordered it and get evidence if it happens multiple times.
If anything, the more firewall and ids systems you have, the easier you are to dos...
A successful attack only needs to saturate one aspect of the target environment, and most firewall or ids systems are based on old server hardware from a few years ago so when faced with a flood of small packets they will often go down much quicker than the servers behind them. Not to mention all the extra ruleset parsing a typical firewall or ids has to do for each and every packet.
I have seen numerous occasions where a dos attack was successful because the firewall simply couldn't cope, while both the line and actual servers could easily have coped with that level of attack.
One thing to consider however, is that attackers will often only hit as hard as they need to... You may be face with what looks like a pitiful 20mbit/sec flood of tiny packets that cripples your firewall, and so long as your site stays offline its unlikely to get any worse than that. But as soon as you successfully filter the attack, whoever's attacking is likely to increase their attack. I have seen a few cases like this, a relatively small attack knocks out one user but leaves the ISP mostly unaffected, but once the ISP filtered it a much larger attack was launched which took the ISP down too.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
I have been happy with cloudflare but I am pretty unhappy with slashdot today. Other than cloudflare (which is free and pretty good but not the best) I have seen not one easily implementable solution. I am shocked that nobody here has much of a suggestion.
Did they use a botnet that was scattered all over the world, or just a specific set of systems? I would recommend going through your logs to see what you can find out about the attack, there may be some patterns there that you can learn from.
That said, a lot of people suggest you contact the authorities. I would suggest that those people have probably never tried that themselves. The authorities - local or federal - generally don't give a shit about cyber crime. They give it some (virtual) lip service on their websites but when presented with actual cyber crime they always find something more interesting to do with their time. After all, you said the criminal was in Lebanon, and the FBI has no jurisdiction there. Even if you found an FBI agent who cared, he wouldn't be able to get interpol working on it before the (electronic) check is cashed and the culprit has cleaned up his tracks.
In other words, you have to do the work yourself. Maybe you can learn something from the logs, or maybe you'll need to look at distributed hosting to better prepare yourself for a potential future attack.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Before I found that there was a lot more money and a lost less hours and stress doing consulting than being a cubicle drone, I worked for a large hosting company.
Handling a DDOS attack is a piece of cake. We handled a few a week and this was in the early 2000s. We would watch the router traffic graphs and see a spike that might be eating 5% or 10% of our capacity and just grin. All you need is money. Your ISP needs giant pipes, spare server capacity distributed around the world and sharp network guys, and for the right price, they'll simply make the problem go away for you.
However the cost of doing this means that if $1500 to Rackspace sounds like a lot of money, you're not in this league.
If you're at the "less than $200/month" level for hosting, your best course of action is to not piss people off, and if you're attacked just hope you can wait it out.
The "up side" of having a small site with cheap hosting is that it probably won't actually do much damage to your business if it's down for a few days.
>They proceeded to tell me that they have 'DDoS mitigation services,'
>but they cost $6,000 if your site is under attack at the time you use the
>service. Once the attack was over, the price dropped to $1500. (Nice
>touch there Rackspace, so much for Fanatical support; price gouging
>at its worst).
a) Ok. so now you could get it for $1500. The buy it. $1500 are roughly 18h of my time (as a consultant), so even the smalles action you coud do exceeds this. IFF you believe that this solves the problem then just do it and dont touch the rest. The advertisement on their web site sounds promising, bu did not test it.
b) Price gouging? No, it is reasonable, for several reasons. Doing the DDoS protection uses resources, which are allocated, but (according to your definition unsused). Why on earth should customers wise enough to see the necessity of a immediate reaction, which pay for this service provide the support, upkeep and unallocated ressources for the others? Such a service is like an insurance. In average you can offer it for a certain price, but if you know the risk hits, its not an insurance any more. Moreover: The service seems to be based on detecting deviations in the traffic patterns. If the attack is ongoing there is no way to detect the "ground truth" = the normal operation automatically. Which in turn will require *much* more human attention.